Closer than the spread is a tie

Here is the top of the leaderboard from the morning of the talk: three agents on four models, the same trained brain in every one, five runs each on the same ten frozen tickets.

Variant	Mean of 5 runs
OpenCode · Qwen 3.8 27B	0.910
Claude Code · Muse Glimmer 30B	0.908
Codex · Muse Glimmer 30B	0.900
Claude Code · Qwen 3.8 27B	0.894
Codex · Qwen 3.8 27B	0.887
OpenCode · Muse Glimmer 30B	0.866

Who won? The first five sit within 0.023 of each other. The median distance between one variant’s best and worst run was 0.052. So the honest answer is that nobody won: the top five are a tie, and the leaderboard says so in its own column. A variant has to beat another by more than the noise of running it again before the ranking means anything.

Every benchmark is an evaluation

The previous post built one evaluation: tickets, an agent, judges, a score. A benchmark is the same thing with three promises added:

Evaluation tickets + agent + judges → score

Benchmarkan evaluation, plus …

• + FrozenThe tickets and the recipe do not change.
• + SharedSomeone else can run it again and get the same task.
• + ComparableMany agents, models and brains, under the same conditions.

Take one promise away and you are back to an evaluation. Every benchmark is an evaluation. Not every evaluation is a benchmark.

What support-bench v1 freezes

“Frozen” is easy to say and easy to break, so here it is as a list:

Frozen	Value
Tickets	the ten validation tickets — always the same ten
Judges and their settings	the same six judges; the model judge takes the median of three samples
Judge model	gpt-oss-120b, pinned
Eval config	change any of the three above and it is a new benchmark version
Agent versions	OpenCode 1.18.30 · Claude Code 2.1.289 · Codex 0.154.0
Runs per variant	five for this grid: a mean and a range, not one lucky number

The tickets are the validation tickets for a reason. The adjuster never sees them during training, so no trained brain is ever ranked on its own training data. Evaluation and benchmarking use the same ten tickets here. They differ in purpose, not in data.

How big is the noise?

Brain v0, OpenCode on Qwen, three runs of the same evaluation: 0.376, 0.394, 0.365. The trained brain, same agent and model, five runs: 0.940, 0.909, 0.904, 0.900, 0.899. Same agent, same brain, same tickets, temperature zero — and four hundredths apart.

That is why a variant is never one run. One run cannot tell you which of two close variants is better, and the grid above is full of close variants. The console draws its ties from the real range of the runs, so a benchmark with one run per variant can mark no ties at all — and should not be trusted to rank anything.

The grid

Three agents (OpenCode, Claude Code, Codex) times four models (Qwen 3.8 27B and gpt-oss-120b on Cerebras, Gemma 4 31B and Muse Glimmer 30B on OpenRouter) is twelve variants. Five runs each is sixty evaluations, one Kubernetes Job each, up to twelve at a time. On the cluster: 59 evaluations ran, one was reused, none failed, 21 minutes and 21 seconds, 5.39 USD for the agents. The judge is not priced.

Below the top six, the Gemma variants landed between 0.72 and 0.85, and gpt-oss-120b came last with every one of the three agents, between 0.34 and 0.54. That gap is far wider than the noise, so that part of the ranking holds.

The mean is not the only way to read the grid. The same sixty evaluations give pass^5 per variant — the share of tickets that passed in all five runs, the number that says what a customer gets:

Variant	Mean	pass^5
OpenCode · Qwen	0.910	0.60
Claude Code · Glimmer	0.908	0.40
Codex · Glimmer	0.900	0.50
Claude Code · Qwen	0.894	0.70
Codex · Qwen	0.887	0.50
OpenCode · Glimmer	0.866	0.60

Second by the mean, Claude Code on Glimmer is last of these six by consistency. Fourth by the mean, Claude Code on Qwen is first. Picking the column is a decision about what you promise your users, and the benchmark cannot make it for you.

One brain, every agent

Every cell above ran the same brain: version 5 of a training run that used Claude Code on Qwen. It ran unchanged in OpenCode and Codex, on all four models, because its files are in open formats. Each agent’s runner puts them where that agent looks:

Agent	AGENTS.md	Skills
OpenCode	read as it is	.agents/skills/, as it is
Claude Code	a CLAUDE.md holding @AGENTS.md	copied to .claude/skills/
Codex	read as it is	.agents/skills/, as it is

With brain v0, OpenCode on Qwen scored 0.378 on average. With the brain trained in a different agent, it scored 0.910 — the top of the table. The instructions carried over between harnesses, which is the strongest argument for keeping them in formats no single agent owns.

The first design benchmarked two brains in one grid — v0 and the trained one, side by side — and even a “brain per cell”, a training run for every agent on every model. The second idea died on cost: a training run per cell is 1 to 15 USD and 10 to 45 minutes, and training belongs to training. A benchmark compares what training produced. The first idea died in the console, while I looked at the form that starts a benchmark:

Otherwise, the whole benchmark view would have to be like a 3D space because there will be agents, models, and brains. Let’s keep it simple for now.

One brain per benchmark. To compare two brains, start two benchmarks.

Never pay twice for one measurement

A benchmark reruns a lot of things that already ran. The training run that produced the brain had already scored it on the ten validation tickets with Claude Code on Qwen — that is one run of one cell, for free. So every evaluation is tagged with two hashes: the brain’s files, and a recipe — a hash of everything a score depends on except the agent, the model and the brain. The suite’s config, the set’s tickets, the policies, the runner scripts, the knowledge base, the agent settings and the judge model. A run with the same agent, model, brain hash and recipe is a run of that cell, whoever started it. Only the missing runs start. That is the “one reused” above.

The recipe is easy to get wrong in both directions. The first version hashed the runners’ price table and the whole agent config, model list included. Then a model in the grid changed — Muse Glimmer replaced a dearer Meta model the night before the talk — and every run already logged would have stopped counting. A price is not part of a measurement, and the run’s model is already in the key. The other direction is just as real: on the morning of the talk, a small fix to the Codex runner was ready, and building it would have changed the recipe and orphaned both of that morning’s benchmarks. The image went out without that line, and the benchmarks stayed reusable.

Two checks before anything runs

The model check. Before a model’s cells start, the benchmark sends it one call of sixteen tokens through the gateway. A model that does not answer — a missing key, a wrong route — skips its cells. Without the check, its evaluations would run, every answer would be empty, every score would be low, and the leaderboard would show a bad model instead of a broken one.

The quota. Qwen on Cerebras allows 150,000 tokens a minute, and a single scoring step at four tickets at a time already touched 165,000 in its busiest minute. So the benchmark caps evaluations per model — two at a time for Qwen — and runs the local engine one evaluation at a time, because it holds one model in memory and loading another unloads the first. During the talk, the live benchmark I started on stage used the OpenRouter models only: the training run started a few minutes later needed the Qwen quota.

Why our own runner, and not Harbor

Harbor is built for exactly this — agents on task sets, one container per task, many at once — and it runs the same eval.yaml. Before the talk I still chose a runner of our own, for four reasons. The runner is part of the frozen recipe, so on Harbor none of the training runs’ scores would count as benchmark runs. Harbor drove OpenCode with its own agent, so the brain, the quota retry and the traces needed a new and untested path. One pod per ticket per run for the original grid meant 720 pods, against one Job per cell. And the gateway works either way, so Harbor bought nothing there.

Harbor is the better choice for public task sets — Terminal-Bench, SWE-bench Verified, tau-bench — for task code you do not trust, and for many agents at once. Today it ran its first green tickets inside this project as a pluggable scoring backend, which turns “why not Harbor” from a wall into a switch. That is part of the last post.

A benchmark on OpenShift

A benchmark is a Job that creates evaluation Jobs. It plans every run that is missing, puts the brain into a ConfigMap, and starts one evaluation Job per run, named after its cell — bench-x7k2p-oc-qwen-b1-2, for example, is OpenCode, Qwen, brain 1, run 2. Every evaluation is an MLflow run tagged with its benchmark, and the benchmark’s own parent run holds the plan at the start and the result at the end. The evaluation Jobs are owned by the benchmark Job, so deleting the one deletes the sixty.

The console reads the grid back as a leaderboard with ranges and ties, an agent × model matrix, the judge each variant loses most on, and score against cost — where up and to the left is the corner you want.

The next post uses the same evaluation in a loop instead of a grid: score a brain, let a second agent adjust it, score it again, and keep the change only if the number says so.