BLUF (bottom line up front): Abram Jackson, PM for extensibility of Copilot for Microsoft 365, wrote this week: “Yes, it’s not just a simple router. We use multiple models for a single request, based on which combination scores highest on our evals.” My home lab has been teaching me the same lesson from the other end. Learning flows both ways: what I prove at home shapes what I take to work, and what I see at frontier scale at work shapes what I try at home. This post is the latest turn of that loop, and yes, the pun in the title is intended :-)
By day I work at Microsoft, where my focus is observability. Everything below about my lab is my own machines, my own time and public tools. The work side is what Abram and Charles Lamanna shared publicly on LinkedIn. This isn't me building a rival to anything; it's the same journey, worked from 2 ends.
What Abram and Charles said
Abram Jackson’s post, quoting Charles Lamanna. Screenshot from my phone.
Charles Lamanna, EVP of Copilot, Agents and Platform, opened his whiteboard post with: “There isn’t one model that’s the best at doing everything anymore.” He went on: “One model might be better at reasoning, another at building highly visual slides, and sometimes a smaller, faster model can do the job you need just as well.” And the line that matters most for me: “how one request can become multiple jobs, each handled by the model that’s right for the task.” He also shared that “more than 90% of Copilot queries use Auto today.”
Abram added the part that made me sit up: it’s not a router picking 1 model. It’s several models on 1 request, chosen by which combination scores highest on evals.
The replies are the part I can’t stop thinking about
The replies under Abram’s post.
Marlon Niklas Kaulich wrote: “It would be great if users could see which model was selected by the Auto Router.” Abram’s reply: “a response doesn’t use a single model, but I get what you mean about the main model.” Bryce Murray added: “I typically recommend people to choose a model. It’s helped a bit when they’re trying to debug why they didn’t get an output they were expecting. One less unknown. If ‘auto’ could show which model was being selected, that would be a game changer.”
That’s an observability ask, and observability is my home turf. When an answer is wrong, the first question is “which part of the system did that?” If you can’t see it, you can’t fix it.
Home to work: what my lab already does
For months I’ve run local and frontier models side by side at home, on a MacBook Pro M3 Max with 128 GB and a Mac mini, for my personal work and for 2 small organisations I volunteer with and help run. Every job ends with a receipt that needs outside proof, a judge that isn’t the model being judged, and traces of every call. I wrote that up in Trust the Model the Least.
My own numbers already say Charles is right that no single model wins everything:
- On my Maryland ballot guide test, a frontier model passed 2 independent judges on 12 of 13 pages. My local models passed 0 of 13. Open-ended writing about the world is not a job I give local models today.
- On my own files (the “Genie” that answers questions about my own notes and projects from a local index), my dense local model passed my 3 real test questions with 0 invented claims, and last week’s graded set came in at 18 pass, 3 partial, 0 fail. Grounded questions over my own data: local does the job.
- The fast local model, though, passed only 3 of 32 questions at baseline, against 17 of 32 for the dense one. Fast isn’t free.
That’s the kind of thing I bring to work: not slides, but a running proof of concept, and the specs and PRDs I write from it.
Work to home: 1 request, many models, scored on evals
Here’s the honest part. My Genie answers every question in 7 steps (check the question’s assumptions, search the wiki, pull the facts, compute the numbers with code, write, check any “couldn’t find” claims, revise) and all 7 run on 1 model. My run files record 1 model. If an answer is wrong, I can’t tell you which step’s model caused it.
Abram’s post is the frontier-scale version of the fix:
- Route per step, not per question. The fast model might be fine for searching, the dense one better at writing, and a frontier model is already my fact checker.
- Pick the mix by evals. I already score each step separately: a code check that every quote is really in the files, a code check on every number, and a weekly fact check of the written answer.
- Show which model did what. Marlon and Bryce asked for exactly this. My grading pages will record the model behind each step, so “which part did that?” has an answer.
Next weekend I’m testing a much bigger, sparse model (Qwen 3.8 Flash Next) on each step, not just end to end, and letting the evals pick the combination. Last night I cleared 163 GB of old models off the MacBook Pro to make room for a 111 GB download. I can already hear the fans :-)
Learnings at every turn
In a chat, every message is a turn, and at frontier scale each turn can become several jobs for several models. My journey works the same way. Every turn of it teaches me something, and the lesson doesn’t care which end it came from. Sometimes I prove something at home that I can write up for work. This week, someone at work said something in public that I can prove at home. Both count.
Absolutely none of this is about home versus work. It’s 1 continuous loop, and I’m lucky to get to run it from both ends.
Where I’m not there yet
- Per-step routing doesn’t exist in my pipeline yet. Next weekend is the 1st test.
- The model behind each step isn’t on my grading pages yet.
- Local models still can’t do open-world writing to my standard, so the frontier fact check stays no matter what the test says.
- A 111 GB model leaves little headroom on a 128 GB machine. We’ll see what it costs everything else running on it.
Which way does your learning flow? Do you bring the lab to work, or work to the lab?