Methodology · beat-frontier
Eval before you swap the call
Land with measurement. A task model under the agent only ships when it beats your frontier baseline on held-out labeled outcomes — not demos, not vibes.
- RuleHeld-out firstSplit labels before training. Score frontier and base open-weight on the same holdout the task model will face.
- RuleJob metrics, not vibesUse recipe metrics: precision / recall / F1 / latency for decision-risk; add dollar-weighted recall when payment amounts matter.
- RuleShip only if it winsSwap the agent call only when the task model beats frontier on the metrics you care about. Otherwise keep frontier.
- RuleNo invented liftsWe do not publish fake customer percentages. Your eval numbers are the proof — Pilot exists to produce them.
Loop
- 1
Baseline
Score frontier (and raw open-weight) on holdout for the scoped job.
- 2
Train
Post-train via a recipe on Shared, BYOC, or confidential GPU.
- 3
Compare
Same metrics, same holdout. Record win/lose per metric that matters.
- 4
Cut over
Point the harness at the task-model endpoint only on a win.
FAQ
On your held-out labeled outcomes, the task model must outperform the frontier baseline on the metrics that matter for the job (e.g. F1, precision/recall, or dollar-weighted recall for fraud) before you swap the agent call.
Recipe-defined: decision/risk uses precision, recall, F1, and latency; fraud/payments adds dollar-weighted recall when amounts exist. You bring the labels; we do not invent customer lift percentages.
No. Pilot is framed as prove-then-swap. Keep the frontier call until held-out eval wins on the jobs you care about.
Run eval on Pilot
Beachhead teams with an agent + labels — $50 credits on Shared.