Weekday Judge, Weekend Judge, and Now a Microscope

BLUF (bottom line up front): For a few weekends now I’ve been grading my own AI. A local model judges it every night of the week, and I judge it by hand every weekend. This weekend 2 talks from the AI Engineer conference said out loud what my grading had already been telling me, and I turned what they said into code the same day, not the same month. I also opened up my traces, so now I can follow every question from the words I typed to the answer the model gave, step by step, tool by tool. Weekday judge, weekend judge, and now a microscope :-)

By day I work at Microsoft, where my focus is observability. Everything here is my own machines, my own time and public tools. I POC at home and bring what I learn to work, and I bring what I see at work back home to try. This post is the home half.

The setup, in 1 paragraph

2 boxes on my desk: a MacBook Pro M3 Max with 128 GB, and a Mac mini I call ADA. The MacBook Pro runs the local models and my Ontology: everything I know about my own work, notes and files, indexed so a question-answering layer I call the Genie can answer from it. The Mac mini runs the judges and the dashboards. Frontier models do the heavy lifting where they’re better; local models do what they can prove they can do. The whole point of the evals is to find out which is which, with numbers, not vibes.

Judge 1: a local model, every night

Image 1: my evals cockpit. 592 complete match sets since August 22, 2,711 legs, 6 contestants. Verdict: HOLD.

Every night, the same task goes to every contestant: 2 frontier models and 2 local ones, each in its own harness. A judge model running on the Mac mini scores them blind, and the cockpit asks 1 question per task type: can this move from frontier to local yet? The rule is strict. A local model only “flips” when it’s within 0.30 of frontier on average, its worst quarter is within 0.75, it doesn’t drop facts, and it actually opened its sources.

The honest answer today is HOLD. My best local model (Qwen3.8 27B, dense) now passes the quality, floor and fidelity gates on compiling my wiki, but it still fails the fact gate: it doesn’t reliably open its sources before it writes. That 1 red box is the whole story of where local models are for me right now.

Judge 2: me, every weekend

Every Friday at 6 PM the Genie answers a fixed set of questions from my Ontology: trap questions with known answers, plus 3 “would I actually act on this?” questions. Saturday at 6 AM a frontier model fact-checks every answer before I see it. Then I grade them in a workbook: pass, partial or fail, and did it invent anything.

4 weekends of my grades on the hardest questions (traps and unknowns):

(Nov 7 is out of 19, not 21: I retired 2 questions that no longer had a right answer.) 0 invented claims 2 weekends running, and 0 fails. The 3 “would I act on it?” questions have passed every week I’ve closed since October 4. I spend hours grading every weekend [today it was squeezed in between cutting the grass and picking up leaves] so my agents can use that learning the following week.

This weekend: other people’s talks confirmed what my grading found

Anirudh Veeraragavan from Fieldguide gave a talk at AI Engineer called “Your Financial Agent Needs to Show Its Work.” What I found revealing: “in this case we have 97% but out of nine of the misses four was because the eval itself was just wrong.” And: “if you skip evaluation integrity, you can arrive at a great flagship number of 97%. But you have no idea what [clears throat] the reality is.”

I’d been living that for weeks without naming it. Some of my “misses” weren’t the model’s fault at all: my answer key was out of date, or the question was bad. So on Saturday afternoon I built 4 things, live from this Friday’s run:

1. A cause on every grade that isn’t a pass: agent wrong, key wrong, key stale, or bad question. Only “agent wrong” counts against the model.
2. A “do not grade” guard if an answer is older than that week’s run (it happened: 1 question timed out and I almost graded last week’s answer).
3. Provenance in every answer: a fingerprint of every file it cited, and which model ran each step.
4. Grade only the exceptions: a question that passed 3 weekends in a row with a clean-enough fact check keeps its grade, with 1 random spot check a week so the shortcut is itself checked.

Dillon Rolnick from Nous Research (the people behind Hermes Agent) said at the same conference: “do not build on quicksand foundations. Build on open foundations.” That’s why the agent I talk to about my own data is Hermes, running on local models, on my machines, over my own Ontology. I upgraded it this week to the new version, and it now does far more than chat:

Image 2: Hermes on Qwen3.8 Flash Next, with its scheduled jobs: Genie health every morning, a weekly G&J summary, a pipeline check, a Saturday review.

Image 3: a 2nd bot, Ontology, wired straight into my Ontology’s search over my notes and files.

Image 4: both bots in 1 room. Scheduled posts land here, like the Genie health check every morning at 7:45.

And as of tonight I can reach my agents from my phone. I type /agent gandj <message> to Hermes on Telegram, a frontier agent picks it up, does the work, and replies in the same chat. The agents can message me too, and when I reply, it goes back to the agent that asked. Drafts only: nothing gets sent on my behalf until I say “Post”.

The microscope: every trace, end to end

Image 5: Arize Phoenix on the Mac mini. 1 project per harness. Last 7 days: Copilot CLI (440 traces), Hermes (142), OpenCode (1,080), and my local-models scripts.

Every harness sends OpenTelemetry traces to a collector on the Mac mini, which forwards them to Phoenix. Until Saturday I only kept the metadata: which model, how many tokens, how long, which tool. Then I asked myself why I was hiding my own prompts from my own dashboard on my own network, and switched on full content, with a filter that masks anything that looks like a key or a password.

Image 6: 1 Hermes question, end to end. My words in, the model asks for a tool, the tool runs, the model answers. 2 minutes 32 seconds on the local Flash model.

Image 7: 1 turn of my frontier coding agent: 6 model calls and 6 tool calls, all under 1 trace, in 1 session.

Now I can see how it all nests: a session holds many traces, 1 per question. Each trace has 1 agent span at the top, and under it 1 span per model call and 1 per tool call, in order. A simple 1-shot is 1 agent span and 1 model call. A tool question is model, tool, model. I want to be able to walk anyone through that, under every scenario, because if I can see it and explain it, I can trust it, and if I can’t, I can find out why. And I can line a trace up against the Genie’s grade for the same question.

The new model: Qwen3.8 Flash Next

I spent the weekend on a new local model, Qwen3.8 Flash Next (a 113 GB download, about 105 GB in memory). On the 4 trap questions I tried, it took about half the time of my dense model, and on my first read the answers were as good or better. 4 questions and my first read are a pilot, not a verdict, so it gets fact-checked and blind-graded this week.

The catch: it uses almost all of my 128 GB. I can hear the MacBook Pro fan spinning every time it thinks :-) My dense and fast models can’t run alongside it, so every comparison needs a scheduled swap overnight.

Where I’m not there yet

• Local still doesn’t clear my own bar on the nightly evals. HOLD means HOLD.
• My Mac hadn’t restarted in 104 days, and the macOS process that draws the screen had grown to 43 GB. A simple question sat for 8 minutes before the model wrote a word. A restart fixed it, and now there’s a checklist that brings everything back in the right order.
• 2 bots in a room talked to each other for 90 minutes, re-reading a 70,000-token room history every turn, until they hit the time limit. Rooms are now capped at 1 round per question.
• My agents’ summaries of where things stand went stale. Hermes answered from a morning snapshot and missed an afternoon update. Those summaries now refresh on a schedule every night, before the index rebuilds.
• The Genie once credited a podcast instead of the person speaking on it. The fact checker now looks for wrong attribution, not just wrong facts.
• This week’s 2 partial grades don’t have a cause marked yet. The new column only works if I fill it in.

Next week

• Tuesday night: Flash Next answers all 19 trap questions overnight, and Hermes replays my questions from Monday and Tuesday on it.
• Saturday: I grade Flash and the dense model side by side, blind: labelled A and B, and I don’t know which is which until after.
• If Flash is as accurate and at least 40% faster, it becomes my only local model for a week, checked every day. If it holds up, it replaces 2 models with 1.

None of this is novel or exotic. It’s evals, traces and a lot of grading, done consistently, on 2 Macs at home. What changed this weekend is speed: a talk I watched on Saturday was running code the same afternoon. I build in the open, so you get the numbers, the misses and the plan.

Q4U: who’s your judge, and when did you last look at a full trace of what your agent actually did?