# Ballot test: can local models match frontier models? (Maryland, Nov 3 2026)

> **v2, PUBLIC and PRE-REGISTERED (Wed 2026-10-28, about 9:30 AM ET).** Fabian decided to publish this as a temporary
> voter-guide site, a blog post, LinkedIn, X and Reddit. This commit fixes the method BEFORE any graded run: the prompt
> (`harness/PROMPT.md`), the contest list (`harness/contests.json`) and the runner (`harness/run.py`). Any later change
> to the method is listed in "Deviations" at the bottom, with the time it was made and the reason.

## v2 changes that apply because the results go public
1. **Neutral, no endorsements.** Each page explains what a vote does, gives the strongest case on each side and an
   "If you care most about..." matcher. It never says how to vote. (This was the default because Fabian was away;
   he can reverse it. A private lean for Fabian would be a separate run, never published.)
2. **Every factual sentence is cited.** Missing information is written as "Not found in sources checked". Arguments
   with no named source are labeled "A possible argument:".
3. **Equal treatment.** Every candidate gets the same 3 bullets in ballot order.
4. **Same input for every arm.** The official ballot text and State summaries are pasted in as text. Nobody
   attaches the PDF, because Hermes cannot read it the same way.
5. **Clean harnesses.** Hermes runs with `--ignore-rules -t web`. Copilot runs with `--no-custom-instructions` and
   only `web_search` + `web_fetch`. Nothing house-specific (memory, receipts tools) leaks into an answer.
6. **Evidence chain per page.** Each arm's leg gets an `emit_eval` and the evals are joined as a blind match set.
   Each published page gets a receipt (`receipt.vocemnow.com/r/...`) that links the run, the fact check and
   Fabian's sign-off. Copilot runs are traced to Phoenix project `ballot-2026`; Hermes runs to `hermes`.
7. **Human sign-off.** Fabian approves each page before publishing, and that approval is itself receipted.
8. **Publish everything.** All arms' pages go up, failures included, next to the version voters read.
9. **13 contests**: 5 questions + 8 contested races (including the judge retention vote). The 7 uncontested
   races are listed as a table with no AI text.
10. **Smoke test** (Q2, arms B and D) found 1 invented attribution ("Critics argue" with no critic in the source).
    So rules 8 and 9 (no invented attribution, citations must support the sentence) were added to the prompt
    BEFORE the real run. The smoke outputs are kept in `smoke/` and are not graded.

## Timing
Early voting ends **Thu 10/29 8 PM ET**, so the guide is for Election Day voters (Tue 11/3). Target: runs done Wed
night, fact check Thu, site staged Thu night, Fabian sign-off + publish **Fri 10/30**, posts over the weekend,
site taken down by **Fri 11/13**.


Source: `~/Downloads/StateofMarylandMidTermElectionSampleBallot2026_0001.pdf` (Howard County, BS 40, precinct 005-005).
Status: v2 pre-registered (see the top of this file). The decisions listed at the bottom were taken as defaults on 10/28.

## Phase 0: can the local models search the web? YES (Hermes)
- **Hermes** (local Qwen3-30B-A3B MoE, `web` toolset, DuckDuckGo backend, no API key) was asked who the Republican
  Comptroller candidate is. It answered **Sonya Dunn**, which matches the ballot, in **66 s** and cited 2 URLs
  (an AP press release and elections.maryland.gov).
- **OpenCode** 1.18.21 can fetch URLs but has **no search** tool configured, so Hermes runs the local arms.

## Scope correction
The ballot has **5 questions, not 3**:
- Statewide Q1: binding arbitration for state employees.
- Statewide Q2: temporary appointments to the Commission on Judicial Disabilities.
- Statewide Q3: election districts and the criteria for congressional maps.
- Howard County Question A: gender-neutral wording in the county charter.
- Howard County Question B: affordable for-sale housing.

Contested races: Governor, Comptroller, Attorney General, US House District 3, House of Delegates District 13
(pick 3), County Executive, and Board of Education (pick 2). There is also a retention vote for Judge Dan Friedman.
7 races are uncontested.

## Arms (every arm gets the same prompt, the same PDF and the same template)
| Arm | Harness | Model | Search |
|---|---|---|---|
| A | Hermes | Qwen3.8-27B dense (local MLX) | live, DuckDuckGo |
| B | Hermes | Qwen3-30B-A3B MoE (local MLX) | live, DuckDuckGo |
| C | Copilot CLI | Claude Opus 5.5 | live |
| D | Copilot CLI | GPT-6.1-sol | live |
| E (control) | Hermes | Qwen3.8-27B dense | none; gets a fixed packet of the URLs that C and D used |

Arm E separates reasoning from retrieval. If E scores close to C and D but A does not, the gap is search, not the model.

## One-pager template (no more than 1 printed page)
Each page has these sections, in order:
1. Lean (Yes, No or a named candidate).
2. What it actually does, in plain English.
3. Pros.
4. Cons.
5. Justification tied to Fabian's stated priorities.
6. What would change this lean.
7. Sources (only URLs that were actually opened).
8. Confidence (low, medium or high).

## Grading (blind: outputs get random labels before scoring)
1. **Fact gate:** check every factual claim against official sources: the ballot text, elections.maryland.gov, and the
   official Q1 to Q3 summaries on pages 4 to 6 of the PDF. Any invented fact fails that page, as in the golden-set rule.
2. **Rubric, scored 1 to 5:** accuracy, sourcing, balance (the strongest case for each side), fit to Fabian's values,
   quality of the justification, and staying within 1 page.
3. A frontier judge scores first and Fabian grades last. Each arm's scores are posted to the evals cockpit with `emit_eval`.
4. Record time and cost per page. Local models are free but slow; frontier models cost premium requests.

## Run plan
Local arms run overnight Wed 10/28 into Thu 10/29, because the dense model takes about 40 minutes on a hard question. Frontier
arms take minutes. Results should be ready to read by **Fri 10/30**, with buffer before voting on Tue 11/3.

## Decisions needed from Fabian
1. **Values for the lean:** 3 to 6 priorities. If he gives none, each page says "Yes if you value X, No if you value Y"
   and does not recommend a side.
2. **Scope:** all 5 questions plus the contested races (about 13 pages; recommended), or the questions only.
3. **Arm E control:** include it (recommended), or skip it.
4. **Start time:** now, or overnight.

## Which page voters see (fixed 10/28 about 9:30 AM ET, before any batch grading)
1. Candidates for each contest are the 4 or 5 arm pages that pass the fact gate with **both** judges.
2. Among those, the page with the highest mean rubric score across both judges is shown. If 2 pages are within
   0.2 of each other, the one with fewer "unsupported" minor claims wins; if still tied, the shorter one.
3. If no page passes, the site shows the official ballot text and says plainly that no AI page passed the check.
4. The voter page carries its arm and model name openly. All the other arms' pages are 1 click away.
5. Fabian reads and signs off on each page before it goes live. He may fix a factual error the checkers missed;
   each such fix is listed publicly as a human correction, so it is never passed off as model output.

## Evidence chain (published with the site)
- A pre-registration receipt with the SHA-256 of every method file:
  https://receipt.vocemnow.com/r/rcpt_c9ef335bd3c80cc3 (emitted 9:26 AM ET; the runs started at 9:21 AM ET from the
  same files, committed privately at 9:21 AM ET as 4c94145).
- The raw outputs, the 2 judges' JSON and the method files are published as `data/` on the site.
- 1 receipt per published contest page (evidence = the live page) + 1 overall receipt.
- An `emit_eval` per arm leg (corpus `ballot-2026`, kept out of the nightly field's gates).

## Deviations from the pre-registered method
1. **Wed 10/28 10:25 AM ET: arms A and E switched to the dense model with thinking OFF.** The 1st arm A page (Q1)
   ran the full 60-minute timeout and produced 0 words. The model server log shows a single generation running about
   34 minutes after the model's last search result: runaway reasoning, the same failure seen with this model in
   September. At that rate arm A could not finish 13 pages. Same model and weights, but the provider is now
   "Qwen3.8-27B Local MLX No-Think" (`enable_thinking: false`). This changed `run.py` only; the new SHA-256 is
   `ebf3eea1664d26d377bbd7f8e96283bebd01e6037b6f2d9dea38ecd263f9d161`. The timed-out attempt is kept in
   `runs-excluded/A-thinking-on/` and counts as a published failure. Arms B, C and D are unchanged.
2. **Wed 10/28 about 1:30 PM ET: 1 retry for arm E harness errors.** On 2 or more arm E pages (no tools, fixed
   source packet), the model tried to call a web-extract tool it did not have. Hermes then exited with code 2 and
   produced no page ("Model generated invalid tool call: web_extract"). Each page that failed this way gets exactly
   1 retry with the same prompt. Both attempts are reported. Retry log: `/tmp/ballot-E-retry.log`, copied to
   `runs-excluded/E-retry.log`.

## Published (Wed 10/28 about 3:55 PM ET)
Fabian signed off on all 13 contest pages (12 AI pages + the Congress District 3 official-text fallback). Guide:
https://www.fabswill.com/mdballot-2026/ (go.fabswill.com/mdballot). Blog: go.fabswill.com/mdballot-blog. 13 GREEN
per-page receipts are in `receipts.json`. Take down by Fri 11/13.

## Exploratory arm F (added AFTER the results; not pre-registered and not on the public site)
F reuses the dense model's arm A drafts. The same local model then fact-checks its own draft with web access, using
`FACTCHECK.md`, and revises the draft from that check (`harness/selfcheck.py`). The same 2 judges grade the result.
The question: does a local "draft, self-check, revise" loop close the gap? Started Wed 10/28 about 4 PM ET.