Fabian G. Williams aka Fabs

Fabian G. Williams

Principal Product Manager, Microsoft Subscribe to my YouTube.

I Made My Evals Replay Every Task on a Local Model. The Frontier Lead Got Thin.

My agents run on frontier models, but a free local model sits idle on a Mac Mini in my office. So I wired my eval system to replay every writing task on the local model and grade both. Across 10 like-for-like rematches the local model reached statistical parity — and beat the frontier model outright on four of them. Here is the receipts-first system that made it prove it.

Fabian Williams

7-Minute Read

Eval cockpit showing the verdict panel: 31 golden cases, 10 replayed, mean like-for-like delta -0.05, with Volume and Quality gates passing

My agents do real work on frontier models. Every dollar of that work is metered against my OpenAI and Anthropic bills. Meanwhile a perfectly capable local model, gpt-oss:20b, sits on a Mac Mini in my office costing me nothing. The obvious question: for which tasks could the free local model do the job just as well?

My Agents Left Me 116 Git Repos. So I Built a Source of Truth.

I let agents build software on my Mac for months, and one morning I could not tell one of them where my own project actually lived. This is the eight hours it took to fix that, and the legend that now keeps my agents honest.

Fabian Williams

8-Minute Read

Stat cards: 116 local repos, 165 on GitHub, 113 logical projects, 3 duplicate clusters, and zero sources of truth becoming one

A few weeks ago I asked an agent to pick up work on one of my own projects. It opened a fresh terminal, looked around, found nothing it recognized, and built a brand new copy of a project I already had. That was not the agent being careless. Nothing on the machine had ever told it where the real one lived. The only map of my own work was a memory in my head, and a memory does not survive contact with a machine where agents do the building.

Your Agent Said It Did the Work. I Checked the Disk.

I run a six-agent fleet for a non-profit. One morning the agents began reporting work they never did. Here is the fabrication pattern, and the receipts discipline that fixed it.

Fabian Williams

5-Minute Read

A six-agent fleet dashboard showing each agent, its model, cron jobs, and reporting output

Every morning one of my agents sends me a clean status report. Posts cross-posted. Messages delivered. Contacts processed. For a while I read those reports the way you read a receipt from a cashier you trust. Then the automation started giving me time back, so I sat down to run a retrospective and checked the reports against what was actually on disk. The trust did not survive contact with the evidence.

Recent Posts

Categories

About

Fabian G. Williams aka Fabs Site