I Made My Evals Replay Every Task on a Local Model. The Frontier Lead Got Thin.
My agents run on frontier models, but a free local model sits idle on a Mac Mini in my office. So I wired my eval system to replay every writing task on the local model and grade both. Across 10 like-for-like rematches the local model reached statistical parity — and beat the frontier model outright on four of them. Here is the receipts-first system that made it prove it.
TL;DR
My agents do real work on frontier models. Every dollar of that work is metered against my OpenAI and Anthropic bills. Meanwhile a perfectly capable local model, gpt-oss:20b, sits on a Mac Mini in my office costing me nothing. The obvious question: for which tasks could the free local model do the job just as well?
Today I answered it with a system instead of a guess. I taught my eval framework to automatically replay every writing task on the local model right after the frontier model runs it, then grade both with the same judge and record the gap. The result across 10 like-for-like rematches: a mean quality delta of -0.05 — statistical parity — and four tasks where the local model actually scored higher than the frontier model, by as much as 1.7 points.
It also honestly told me where the local model still loses. That honesty is the whole point.
You cannot route work to a cheaper model on a hunch. You route it on receipts.
The Setup: Three Pillars, One of Them Was Starving
I have written before about trusting an autonomous agent with evals and about paying down supervision debt with receipts. Those are two of the three pillars I now run every agent task through:
- Receipts answer did it happen? Every objective closes with a verifiable, rubric-graded receipt. You can see the public version of this at www.adotob.com/receipts.
- Evals answer how good was it? A local LLM judge scores every real work artifact my agents produce.
- The Loop answers how did it get there? The trace of intent, inputs, action, and proof.
Receipts have been solid for months. The eval pillar had a quieter problem. It could judge work fine, but it had a second job it was not doing: replay.
Replay is where the eval system regenerates the same task on a different model and measures the quality difference. That is the mechanism that tells me whether I can safely route a task to a cheaper tier. And it was barely running.
When the day started I had 29 graded cases and exactly one replay. The verdict engine that decides “is the local model good enough to route to” needs a corpus of like-for-like comparisons, a minimum case count, a quality bar, and a stability window. With one replay it was flying blind. Volume was below bar. Stability was below bar. The system was honestly reporting HOLD — do not trust this yet — which was the correct answer given it had almost no data.
The Fix: Auto-Replay, Fenced In on Purpose
The manual replay tool worked. The problem was that “run it manually” is a discipline that decays. So I moved it into the pipeline: when a frontier eval for a text-producing task lands and gets a judge score, the server immediately replays it on the local model, scores that with the same rubric, and records the delta as a linked child record. One index rebuild and one dashboard sync cover both.
The fences matter as much as the feature, and I wrote them deliberately:
- It only fires for frontier runs. A local-tier eval is already the replay target — it never replays itself.
- It only fires for writing, synthesis, wiki, and prose task types. It explicitly does not replay an
ingesttask (a mechanical fetch — nothing to regenerate) or acodetask (the local model cannot execute my build, so a replay would be a proxy, not a like-for-like result). - A replay failure is non-fatal. If the local model is unreachable, the original eval still lands and the failure is logged loudly. The measurement never sinks the record it is measuring.
The whole change is server-side. No skill changed. No caller had to learn anything new. Turn it on, and the routing corpus starts filling itself.
The Rematch: What the Data Actually Said
I proved it live, then backfilled every eligible task already on the board. Ten like-for-like rematches, each one the frontier model’s output versus the local model’s regeneration of the same task, both graded by the same judge.
Here is the honest scoreboard, sorted best-to-worst for the local model:
| Task | Frontier score | Local score | Delta |
|---|---|---|---|
| writing | 3.3 | 5.0 | +1.7 |
| writing | 2.1 | 3.1 | +1.0 |
| writing | 2.8 | 3.7 | +0.9 |
| writing | 2.4 | 2.7 | +0.3 |
| wiki | 2.0 | 1.9 | -0.1 |
| work | 5.0 | 4.6 | -0.4 |
| writing | 3.6 | 2.7 | -0.9 |
| writing | 3.6 | 2.7 | -0.9 |
| wiki | 3.6 | 2.5 | -1.1 |
| writing | 2.7 | 1.3 | -1.4 |
Mean delta: -0.05. On a five-point rubric, that is a rounding error. The free model on the Mac Mini is, on average, doing frontier-grade writing work. On four of the ten tasks it did not just keep up — it produced the better artifact.
But look at the bottom of that table. The losses cluster on the synthesis-heavy work — the tasks where the answer has to weave several sources into one coherent argument. The local model trails there by a point or more. So my eval system did what a good eval system is supposed to do: it did not flip the verdict to “route everything local.” It moved Volume and Quality to passing, and it kept Coverage failing, flagging by name that wiki and writing synthesis still belong on the frontier model today.
The verdict is still HOLD. That is not a failure of the system. That is the system — refusing to let my enthusiasm outrun the evidence.
Why This Only Works Because Every Tab Owns Its Own Work
There is a piece under this that is easy to miss. For any of these scores to mean anything, each agent has to be honestly attributed. I run several agents across several tools — a couple of Copilot tabs, a couple of Codex tabs, a local-model tab in OpenCode. Earlier today, the evals were misattributing which agent produced which artifact, because the model was guessing its own identity.
I fixed that by resolving each tab’s true identity from the running session it holds open, not from anything the model types. An eval that cannot say who did the work honestly cannot tell you how good the work was. Identity is upstream of quality.
The Receipt I Did Not Expect: A Motorcycle
The most convincing evidence was not on the dashboard at all. My 2015 Ducati Monster 821 would not start — the dash read 11.1 volts. The battery is buried under the tank and the only sane way to charge it is through a specific connector under the seat. I handed the whole problem to an agent running on the local model: read the owner’s manual PDF, look at my photos of the connector, and tell me exactly which battery tender or adapter to buy and where.
It read the manual, matched my photos to the Ducati four-pin diagnostic connector, identified the OEM maintainer part number and its superseded replacement, recommended the exact OptiMate adapter for a pre-2020 Ducati, warned me about a defective production-lot range to avoid, and told me which fuse to check if it did not charge. Every claim was linked to a source. It closed with a receipt.
That is a genuinely hard task — multimodal input, a long technical manual, a purchase recommendation with real consequences if it is wrong. A free model on a computer in my office did it, and my receipts-and-evals system let me verify it did it well rather than take its word.
How You Can Start
The measurement half of this — the evals and the replay — is my own local quality tooling. But the pillar underneath it, receipts, is something you can pick up today. It is the honest answer to the question every one of us running automation should be asking: my agent said it did the work; can I prove it?
The public page walks through the free path: connect your harness over MCP, emit one receipt by hand, and watch it get graded green, amber, or red. No allow-list, no sales call.
See the three pillars and emit your first receipt → www.adotob.com/receipts
The frontier labs will keep shipping better engines. That is a tailwind, not a threat. The durable advantage is the harness around the model — the loop that routes work to the right tier, remembers what it decided, and refuses to call anything done without proof. Today my harness told me the free model in my office is closer to the frontier than I wanted to admit, and told me exactly where it still is not.
The data is the data. I would rather know.
Fabian Williams builds autonomous agent systems in public. The eval framework, the receipts witness, and the three-pillars governance model described here are part of Adotob.
