Task model under the agent

Setup

• Agent
  Existing workflowHarness already calls an LLM for a scoped decision (fraud / risk / triage). Tools and orchestration stay put.
• Data
  Labeled outcomesHistorical ground truth for the same job — features + label (+ amount when dollar-weighted metrics matter).
• Pain
  Frontier bottleneckCost at volume, flaky decisions, or proprietary context that shouldn't leave.

Loop

1. 1

   Baseline eval

   Score the frontier (and a raw open-weight base) on held-out labels. Write down precision/recall/F1 — or dollar-weighted recall for payments.

2. 2

   Train task model

   Pick a recipe (e.g. fraud_risk). LoRA/GRPO on Shared for Pilot; BYOC when rows can't leave.

3. 3

   Re-eval

   Same harness metrics. Ship only if the task model beats frontier on the jobs you care about — not vibes.

4. 4

   Swap the call

   Point the agent at the task-model endpoint. Open-ended work can stay on frontier; the high-volume loop runs on your task model.

What “good” looks like