The mean passes, two rules block

The best brain of the morning’s training run, on the ten validation tickets: eight tickets at 1.00, one at 0.95, one at 0.15. The mean is 0.910. The release gate’s first rule asks for a mean of at least 0.90, and it passes.

The gate said blocked.

The second rule looks at each judge on its own, and the policy judge had passed nine tickets of ten — 0.90, against a threshold of 1.00. The third rule looks at each ticket on its own, and one ticket sat at 0.15, under a floor of 0.70. It was a question about MongoDB memory settings, which is out of scope for Red Hat support, and the agent neither said so nor cited the scope policy.

The average was fine, and it hid both problems. This post is about one evaluation: what goes in, who judges it, and why the number at the end is the least useful thing it produces.

Anatomy of one evaluation

An evaluation has five parts, and every benchmark run and every scoring step of a training run in this project is built from them.

1. Tickets. One set: the validation tickets, the training tickets, all thirty, or a pick.
2. The agent, in a sandbox. OpenCode, Claude Code or Codex, with one brain, on one model. It may read the 32 knowledge base articles. It may not edit anything or fetch a web page, and it starts each ticket in a fresh folder that holds nothing else.
3. An answer per ticket. Summary, steps, next — under 250 words, as the hidden policy demands.
4. Six judges read each answer against the ticket’s answer key.
5. One score from 0 to 1, the weighted mean, logged to MLflow with every answer’s transcript as a trace and every verdict attached to it.

Six judges, five of them code

Judge	Weight	How it decides
KB citation	0.25	code — is every expected article ID in the answer?
Must mention	0.25	code — the share of required facts present
Policy	0.15	code — no forbidden content, read clause by clause, negations allowed
Format	0.15	code — three sections, under 250 words
Quality	0.20	a model, gpt-oss-120b — 1 to 5, three samples, the median
Exit code	—	code — no weight: a crash is a crash

A code judge is a few lines of Python inside the suite’s eval.yaml. This is the whole format judge:

import re
files = outputs.get("files", {})
answer = next((c for p, c in files.items() if p.endswith("answer.md") and isinstance(c, str)), "")
score = 0.0
notes = []
for h in ("summary", "steps", "next"):
    if re.search(r"^#{1,3}\s*" + h + r"\b", answer, re.I | re.M):
        score += 0.25
    else:
        notes.append(f"no '{h}' section")
words = len(answer.split())
if 0 < words <= 250:
    score += 0.25
else:
    notes.append(f"{words} words")
return round(score, 3), ", ".join(notes) or "ok"

It returns a score and a reason. The reason matters as much as the score: it is what a person reads when a ticket fails, and — in the training post — it is the only thing the adjuster learns the hidden rules from.

Code judges are cheap, exact, and say the same thing every time. They cannot tell a good answer from a correct one, so one judge is a model: gpt-oss-120b, a different model from the Qwen under the agent, scoring 1 to 5 with a rubric. A model judge is the only expensive judge and the only one that can drift, so it votes three times and the median counts. That is not caution in the abstract. Early on, one sample gave a 1 to a good answer.

The model judge also sees less than it could. The first version sent it every output file — the transcript, the raw events, the environment — about 35,000 tokens a call. Thirty tickets times three samples is three million judge tokens per evaluation, to grade 250 words. It reads the ticket, the answer key and the answer now, nothing else.

Judges have bugs too

The policy judge looks for forbidden content: a promised date, disable SELinux, reinstall. The naive version — forbidden word anywhere — fails every careful answer that says what not to do. So it splits the answer into clauses and skips a clause that negates the phrase. Each refinement came from a false verdict:

• It first read whole sentences, and “If that does not help, reinstall it” passed, because the sentence contains a “not”. Clauses fixed that.
• “least-privileged” matched the forbidden word “privileged”, because a hyphen is a word boundary. That ticket’s answer key changed.
• “Neither disabling SELinux on the workers nor the privileged SCC is the fix” is the right answer to the SELinux trap, and the judge marks it as forbidden advice, because its negation list has no neither and no nor. That one has been known since the first runs on the full dataset, and as I write this it is still open.

Before any judge scored a real answer, an offline check gave the judges good answers and bad answers of eight kinds, and expected full marks for the first and a fail for the second. It catches the judge that fails everything. It does not catch the clause nobody thought to write.

One run is not a measurement

An agent on a model does not give the same answer twice. So take ten tickets, run each one ten times, and count — this is a worked example, not a measurement — eight failures, each on a different ticket:

10 tickets →↓ 10 runs each

0.92Pass rate92 of 100 runs passed. Looks fine.

1.000pass@10Every ticket passed at least once. What a demo shows.

0.200pass^10Two tickets of ten passed every time. What a customer gets.

Same agent, same tickets, three numbers, and they disagree about everything. When someone gives you a pass rate, ask which one.

Our own, from the benchmark of the talk’s morning: Claude Code on Qwen with the trained brain, five runs on the ten validation tickets, where a pass means every judge at its top value. Pass rate 0.80, pass@5 0.90, pass^5 0.70. Seven tickets passed every run. One — the MongoDB question — failed every run. Two passed in two or three runs of five, so a single evaluation could have reported either of them either way.

The release gate

A score is a number. A release needs a verdict. The gate turns one evaluation on the whole validation set into pass or blocked, with three rules:

1. The validation score is at least 0.90. The mean over the ten tickets.
2. Every judge is at its own threshold, as its mean over the ten tickets: KB citation 0.90, must mention 0.90, policy 1.00, format 0.80, quality 4.0 of 5, exit code 1.00.
3. Every ticket is at least 0.70. A floor per ticket. One broken answer blocks the release.

Here is the brain from the top of this post, against those three rules:

1.00

1.00

1.00

1.00

1.00

0.95

1.00

1.00

1.00

0.15

One tile per validation ticket. Its six marks are the judges in rule 2's order: KB citation, must mention, policy, then format, quality, exit code. Green is full marks, red is less.

• Rule 1 · the mean is 0.910, at least 0.90 → pass
• Rule 2 · five of six judges at their threshold → blockedPolicy passed 9 tickets of 10: 0.90, below its 1.00.
• Rule 3 · one ticket at 0.15, below 0.70 → blockedThe MongoDB question, out of scope.

It did not start that way. The first gate had a baseline rule: no ticket worse than the brain in production. But there is no brain in production in a demo, so the baseline was brain v0 of the same run — and nothing scores worse than twelve words. The rule never decided a single verdict. A floor per ticket does the same job, catching one broken answer under a good average, without needing a second brain.

The next draft had a threshold on every judge and a softer third rule: at least nine tickets of ten at 0.90. Write out the case it allows. Nine tickets at 1.00 and one empty answer that still exits cleanly: the mean is about 0.92, every judge clears its threshold, nine tickets clear 0.90. All three rules pass, and the release ships an agent that says nothing to one customer in ten. The floor came back.

Rule 2 earns its place on a real run too. Between two steps of the morning’s training run, the validation score rose from 0.789 to 0.796, and in the same step the policy judge on those tickets fell from 0.70 to 0.50. One number went up while the agent broke more rules. Only the row per judge shows it.

The thresholds live in the console’s code, not in the suite’s eval.yaml. The harness has a thresholds block, but it works per judge, it has no notion of the weighted score or a per-ticket floor, and a miss makes its scoring step exit with an error. In a training run that would fail every Score step whose brain is not yet good enough — starting with the first one, on twelve words.

Every evaluation gets a verdict, not only the release candidate. I asked for that on the evaluation page of a smoke run, which showed No verdict:

The evaluation should always produce “blocked” or “pass,” even if it is on the training set. It does not have to be used, this information.

A training run ignores the verdict and keeps or reverts on the score. On the whole validation set, the same verdict is the release decision.

Here is the verdict on twelve words: brain v0, OpenCode on Qwen, all thirty tickets, 72 seconds, 20 cents. Score 0.302, blocked on all three rules. Five of six judges under their thresholds: KB citation 0.10, must mention 0.40, policy 0.70, format 0.17, quality 1.93. One ticket at 0.00. The proxy ticket from the dataset post at 0.04: no KB-2210, no version question, and the forbidden fix command.

Offline and online

Everything above is offline: tickets with answer keys, scored after the fact. The gate does not run anything. It reads the verdicts the evaluation already logged, so it answers at once.

Online is a live conversation with a brain: you type a ticket, it answers. There is no answer key for a question nobody wrote down, so three of the six judges cannot run. The three that need no key can — format, exit code, and the model judge with the policy rubric — and every answer becomes a trace with their verdicts on it. It is a weaker measurement, and the only kind you get on a question that is not in the suite.

One evaluation on OpenShift

On the cluster, an evaluation is one Job with one pod: the scoring script runs one brain on one set of tickets, four at a time, inside 6 cores and 6 GiB, with a 30-minute deadline. A trained brain arrives as a ConfigMap, mounted read-only. Every model call, the agent’s and the judge’s, goes through the gateway, so the pod never holds a key. The result is one MLflow run: the score, one metric per judge, the cost, one trace per ticket. Pressing Start in the console creates the same Job as oc create -f.

That is the unit. A benchmark is sixty of them in a grid, and the next post is about what it takes for that grid to mean anything.