A reader on Reddit asked me a sharp question about my last post. I had trialed a second local coding agent, Hermes, pointed at the same local Qwen 3.8 model my other agent already uses. His question was simple. Did I ever run both agents at the same time, two separate harnesses hammering one model on one Mac at once. And what about Hermes spawning its own sub-agents against that same endpoint. Is any of that predictable.

I gave him my honest first answer, which was that I would probably never do it, and that I assumed it would serialize into a single lane. Then I did the thing I keep telling everyone to do. I checked instead of assuming. I was wrong, and the real answer is more interesting than either of us guessed.

The Housekeeping Note

Last week’s post got called AI slop on Reddit. My full answer to that, and the standing standard I now hold my writing to, lives on its own page: On Slop, And Who Gets To Call It That. This post is a receipt of exactly the kind that page is about. Every number below was measured, not estimated, and the whole test rig is on disk and reproducible. Now to the experiment.

Why I Could Not Just Launch Two Agents

Here is the trap, and the reader helped me see it. If I open two terminals and start OpenCode in one and Hermes in the other, one of them starts first. That first request arrives first, claims the server, and sets the queue order. Whatever I measure after that is a measurement of my own launch timing, not of concurrency. A human cannot release two requests at the same instant. The experiment would be flawed before it began.

So the experiment has to be driven by code, not by hand. I wrote a small load driver that holds every request at a barrier and releases all of them at the same microsecond. That is the one thing a person at a keyboard cannot do, and it is the only way the numbers mean anything.

What The Server Actually Does

Before running anything, I read the source of the thing serving my model, which is mlx_lm.server. This matters, because the answer lives entirely in the serving layer, not in the agents. The agents are just HTTP clients. Two facts settled the question:

• The server does continuous batching. It accepts concurrent requests, folds them into one running batch, and decodes them together. The default batch width is 32.
• A request is only eligible for that batch when three things are true. There is no draft model loaded, the request does not pin a random seed, and it targets the model already in memory. Break any one of those and the request drops out of the batch and gets served on its own.

That already killed my single-lane assumption. Two agents on one model do not take turns by default. They ride the same batch. The real questions became how well that scales, and what quietly breaks it.

Finding One: Batching Is Real, But It Is Not Free

I fired 1, then 2, then 4, then 8, then 16 identical requests, all released at the same instant, and measured two things. Aggregate throughput, meaning all agents summed, and per-agent decode rate, meaning how fast any single agent actually gets its answer.

Read that carefully, because it is the whole point. Total throughput does climb. But the per-agent rate falls off a cliff, and the wait to even start a response climbs with it. The GPU has a roughly fixed throughput budget. Batching does not grow that budget. It divides it. More agents get you more total work only by giving each agent a thinner slice and a longer wait.

The honest practical read is that two agents share nicely. Four is the knee, where each one is already running at half speed. By eight and sixteen, each agent is crawling and waiting several seconds just to see its first token. So yes, you can run a small team of agents on one local model. No, you cannot run a crowd and expect any of them to feel fast.

Finding One, Continued: The Latency Tax

The same story shows up even more plainly if you only look at how long each agent waits to start.

At one agent, first token lands in about a quarter second. At sixteen, every agent waits nearly ten seconds before a single token appears. That is the tax of sharing one engine. It is predictable, it is smooth, and it is the number that will make a room full of agents feel broken even though nothing is broken.

Finding Two: The Seed Trap

This is the one almost nobody expects. Say you want reproducible runs, so you pin a random seed on your requests. Reasonable instinct. On this server it is a quiet catastrophe, because a pinned seed makes a request non-batchable. It drops out of the shared batch and gets served alone, and so does the next one, and they all line up behind each other.

I ran the same four agents three ways. Once with no seed, the normal batchable case. Once with the same fixed seed on all four. Once with a different seed on each.

Throughput collapses back to single-stream levels, and the wait to start explodes from under three seconds to fifteen seconds at the median and nearly thirty at the tail. There is a subtle twist worth naming. Each serialized request actually decodes faster on its own, because it briefly owns the whole GPU. But you pay for that with brutal queueing, because everyone is waiting in line. You traded a shared, steady system for a fast-but-jammed one. Chasing determinism silently destroyed both your concurrency and your latency.

Finding Three: Sub-Agents, And The Speculative Decoding Choice

The reader also asked about Hermes spawning sub-agents. There is nothing magic there. A parent that fans out to four sub-agents against one endpoint is just four more concurrent clients. It obeys the exact same physics as the chart above. Fan-out does not buy you free parallelism on one GPU. It redistributes the same fixed budget, and past the knee it makes every branch slower. On a single Mac, an agent that spawns a swarm of helpers against one local model can easily finish slower than if it had done the work in a tighter sequence.

I went further and tried to actually measure the speculative decoding tradeoff. My hypothesis was clean. The batchability rule in the source says a request is only eligible for the shared batch when no draft model is loaded, so turning on speculative decoding should flip the whole server into serialized mode and kill concurrency, exactly like the seed trap. I expected a fourth chart.

Then I tried to run it, and checking turned up something blunter than my hypothesis. On this stack, right now, I cannot turn speculative decoding on at all, for three independent reasons. The Qwen 3.8 family ships no small draft model, only full 27B variants, so there is nothing lightweight to speculate with. The multi-token-prediction variant that is supposed to do self-speculation, the one whose model type is qwen3_5_mtp, is not supported by my installed mlx_lm, it refuses to load with a plain not-supported error. And when I forced the issue by pointing the server at the 27B as its own draft, generation failed outright, because this architecture’s prompt cache is not trimmable and speculative decoding requires one that is.

So here is the honest correction to my own hypothesis. The tradeoff is real in the code, but it is currently moot on my machine, because speculative decoding does not run here at all. Batching is the only mode I actually have. I am leaving this section in, correction and all, because the entire point of this blog is that I check instead of assume, and this is precisely what checking is for. I set out to measure a tradeoff and instead measured that one side of it does not exist on my stack today.

The Bigger Point

The line I keep coming back to is rent the intelligence, own the harness. This week adds a footnote to it. The local model you own is not an infinite well of parallelism. It is a shared resource with a fixed throughput budget, and it behaves more like a database connection pool than like magic. Treat it that way. Know your knee, which on my machine is about four agents. Never pin a seed in a workload where you want concurrency. And confirm which modes your stack actually supports before you plan around them, because I assumed I could trade batching away for speculative decoding and found that speculative decoding does not run on my stack at all.

None of that is a reason to stop running agents locally. It is the opposite. It is the kind of thing you can only learn because the whole stack is yours to read, to probe, and to measure. You cannot open the source of a frontier endpoint and find out why your agents feel slow. Here I could, and now I know exactly where the walls are.

The Receipt For This Piece

This one is a receipt in the truest sense, because there was almost nothing to take on faith.

The full standard behind this, and why I now attach a receipt to work like this, is on the slop page.