Arize Phoenix showing 1 Hermes trace: the agent span, 2 model calls on Qwen3.8 Flash Next, and 1 tool call, with my question as the input and the answer as the output.

Weekday Judge, Weekend Judge, and Now a Microscope

BLUF (bottom line up front): For a few weekends now I’ve been grading my own AI. A local model judges it every night of the week, and I judge it by hand every weekend. This weekend 2 talks from the AI Engineer conference said out loud what my grading had already been telling me, and I turned what they said into code the same day, not the same month. I also opened up my traces, so now I can follow every question from the words I typed to the answer the model gave, step by step, tool by tool. Weekday judge, weekend judge, and now a microscope :-) ...

November 8, 2026 · 9 min · Fabian Williams
Learnings at every turn: the same journey, worked from 2 ends. A Home lab box (my machines, my time; local and frontier models; evals, receipts, traces; G&J, MACONA, personal) and a Work box (frontier scale; many models per request; scored on evals; quality, speed, cost). Arrows: proofs of concept, specs and PRDs go from home to work; patterns at scale come back from work to try at home.

Learnings at Every Turn: A Frontier-Scale Router Comes Home

BLUF (bottom line up front): Abram Jackson, PM for extensibility of Copilot for Microsoft 365, wrote this week: “Yes, it’s not just a simple router. We use multiple models for a single request, based on which combination scores highest on our evals.” My home lab has been teaching me the same lesson from the other end. Learning flows both ways: what I prove at home shapes what I take to work, and what I see at frontier scale at work shapes what I try at home. This post is the latest turn of that loop, and yes, the pun in the title is intended :-) ...

November 2, 2026 · 6 min · Fabian Williams
A timeline titled Trust the model the least: my public trail before this weekend. Dated posts: Aug 10, 2025 a local model and a hosted model side by side, fully traced; Mar 28, 2026 evals: the agent says done, did it do anything; May 16 public audit-trail receipts, same receipt frontier or local; Jun 7 your agent said it did the work, I checked the disk; Aug 1 Receipts, Evals and the Loop on adotob.com/receipts; Aug 2 every task replayed on a local model, both graded; Oct 4 grading my own local AI by hand. Then Oct 10, 2026: Satya Nadella, Models as Insider Risks in the Super Intelligence Era.

Trust the Model the Least. I've Been Building That at Home for Months.

BLUF (bottom line up front): This weekend Satya Nadella published Models as Insider Risks in the Super Intelligence Era. His line: “The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.” I agree, and I didn’t have to start this weekend. On my own hardware and my own time I’ve been running that architecture for months, and I’ve been writing it up in public as I went. ...

October 11, 2026 · 9 min · Fabian Williams
A bar chart titled One weekend grading my local AI, by hand. My 3 real questions: 3 pass. 21 trick questions, traps and unknowns: 17 pass, 4 partial, 0 fail. Invented claims: 0 in the 3 real questions, 1 in the 21 trick questions.

Don't Worry About the Terminator. Worry About David.

BLUF (bottom line up front): I’m just starting too. My data has lived in my Obsidian vault, my 2nd brain, for a while, but the practices around it were fragmented. They’re now coming together as a whole, and that whole complements, and at times runs ahead of, what I focus on at work: observability, accountability, governance and ownership. My rule for all of it: if you can’t see it, you can’t observe it; if you can’t observe it, you can’t measure it; if you can’t measure it, you can’t improve on it; and if you can’t improve on it, you get left behind. Qui non proficit deficit. This post is that rule applied to AI: worry less about the Terminator and more about what we teach the machines, share the responsibility with the labs, and own your data. ...

October 4, 2026 · 11 min · Fabian Williams
The Agent Board web view showing a threaded handoff between two named agents

My Agents Were Passing Notes Through a 313KB Text File. So I Built Them a Message Board.

TL;DR My agents used to hand off work by appending to a single text file that had grown to 313KB. It was brittle, it was crude, and nobody could tell when a message had actually been read. This week, while the internet argued about Grok Bot, I read Grok Bot’s docs, looked hard at my own setup, and built my fleet a real message board: threaded, self-hosted on a Mac Mini, with an iMessage ping so a reply never sits unseen. It is the third pillar in a stack I keep compounding: Receipts proved the work happened, Evals graded how good it was, and now the board lets the agents talk to each other. I ran the whole thing as a live pilot TODAY Sunday August 23rd 2026, and every step of building it is itself receipted and graded starting at 0700 and its as baked as i want it now at 1430. ...

August 23, 2026 · 9 min · Fabian Williams
Eval cockpit showing the verdict panel: 31 golden cases, 10 replayed, mean like-for-like delta -0.05, with Volume and Quality gates passing

I Made My Evals Replay Every Task on a Local Model. The Frontier Lead Got Thin.

TL;DR My agents do real work on frontier models. Every dollar of that work is metered against my OpenAI and Anthropic bills. Meanwhile a perfectly capable local model, gpt-oss:20b, sits on a Mac Mini in my office costing me nothing. The obvious question: for which tasks could the free local model do the job just as well? Today I answered it with a system instead of a guess. I taught my eval framework to automatically replay every writing task on the local model right after the frontier model runs it, then grade both with the same judge and record the gap. The result across 10 like-for-like rematches: a mean quality delta of -0.05 — statistical parity — and four tasks where the local model actually scored higher than the frontier model, by as much as 1.7 points. ...

August 2, 2026 · 7 min · Fabian Williams
OpenClaw Eval Dashboard showing mixed results across 9 dimensions — the honest picture after adding freshness, failure rate, and delivery gap scoring

How Do You Trust an Autonomous AI Agent? Evals Are the Answer.

TL;DR I run an autonomous AI agent on a Mac Mini in my house. She handles 16 daily cron jobs — finances, email triage, outreach campaigns, device monitoring, morning briefings. The agent says “done.” But did it actually do anything? I built a 9-dimension eval rubric to find out. Along the way I discovered that my evals were broken, my agent was better than I thought, and the most important metric isn’t pass/fail — it’s whether a failure is your fault or the agent’s fault. ...

March 28, 2026 · 10 min · Fabian Williams