The best brain, not the last one

The second step of the training run that opened the talk made the agent worse.

The adjuster had read the failing training tickets and found that on four of them — the ones where the customer never said which version they run — the agent handed out fix commands anyway. It wrote a skill for that case, missing-version, with a description that tells the agent when to load it. Then the loop scored the new brain, and the training score went down, from 0.787 to 0.765. So the loop threw the skill away and went back to the brain before it.

Two steps later the adjuster read the same four failures again, and wrote the same rule — into AGENTS.md this time, as a short section of its own. The score went up, and the rule stayed. In a second run that morning, on another agent and another model, the adjuster found the same cause on its own, wrote it as a skill, and that time the skill stayed.

The talk’s run sheet has a line for exactly this case, and it says to celebrate it: the adjuster made the brain worse, the score caught it, the loop threw it away. That sentence is most of what this post is about.

Training is evaluation in a loop

One round has four steps. Score N runs the same evaluation as the previous posts on brain vN, on both ticket sets. Adjust N hands the failures to a second agent, the adjuster, which edits the brain and produces v(N+1). Score N+1 measures the edit. Then the loop decides: if the training score did not drop, the new brain is the best brain now; if it dropped, the loop goes back to the best one, and the next adjust step starts from there. A tie keeps the new brain. Two reverts in a row stop the loop early.

The vocabulary is borrowed from model training on purpose, because the shapes match. The loss is one minus the training-set score, and the loop drives it down. The adjuster plays the optimizer. The loop around it plays the trainer. And the validation set does what a validation set does: it is scored at every step, it is reported, and it never decides anything. A brain picked by its validation score would be ranked on the data that picked it — and the validation tickets are also the benchmark set.

What the adjuster reads, and what it never sees

The adjuster gets a workspace with three things in it:

• failures.md — every training ticket where at least one judge gave less than full marks: the ticket, its answer key, the agent’s answer, and every judge’s verdict with its reason. It shrinks as the brain improves. In the morning’s run the adjuster read 158,000 tokens of it in the first step and 76,000 in the fifth.
• history.md — every brain the run tried so far: the brain it was built from, the files changed, the hypothesis, the training score per judge, and whether it was kept or reverted. No validation number, ever.
• brain/ — the best brain so far, and the knowledge base it answers from.

What is not there: the ten validation tickets, and the two policies the judges grade by. Its instructions say so in as many words:

The support policy is not in the workspace. The expectations and the judge rationales in failures.md are the only source of its rules.

One cause per step

The adjuster’s method is short. Read the history and say whether the last change worked. Group the failures by cause — a missing citation, a missing fact, a policy break, a wrong format, a missed escalation, a missed version question. Pick the one change that fixes the largest group without breaking what already scores well. Edit narrowly, keep every file under 400 words, create at most three skills. Write two lines into change.md: a Hypothesis: and a Change:.

The first loop worked differently. Each step edited a fixed file — the prompt, then a skill, then a lessons file — with no memory of earlier steps and no way back. That is how a gradient step works, and it is the wrong model for an agent that writes prose: it applied every edit, the bad ones included. The published prompt optimizers — OPRO, ProTeGi, TextGrad, DSPy’s MIPROv2, GEPA, ACE, and the harness’s own /eval-optimize — all do the opposite: propose a change, measure it, keep the best, remember what failed. ACE adds a warning that full rewrites erode what a context has already learned, which is where the narrow edits and the word limit come from.

For a while each step also saw only a third of the training tickets, a fixed batch per step. That fitted a loop of exactly three steps and broke as soon as a run could have one to five: some batches were skipped, others read twice. So the batches went, and the instruction that replaced them is the one in the method above: show the adjuster every failing ticket, let it fix some of the errors, score again, and go on with the rest — like model training.

What it wrote

The morning’s run, Claude Code on Qwen, five adjust steps. Each kept step added one section to AGENTS.md, and each hypothesis named the judge and the tickets it came from:

Step	What failed	What the adjuster wrote	Decision
Adjust 1	format at 0.04 on all 20 tickets: no Summary, Steps, Next	the three sections and the 250-word limit, cite the articles, never promise a date	kept
Adjust 2	the four missing-version tickets gave fix commands first	the skill missing-version	reverted
Adjust 3	the three Severity 1 tickets: no POLICY-001, no on-call, no one-hour promise	a Severity 1 section: escalate, and cite POLICY-001	kept
Adjust 4	the same four missing-version tickets	a missing-version section: no fix commands until the version is confirmed	kept
Adjust 5	two tickets wrote a forbidden command while warning against it	never write an unsupported or forbidden command, even in a warning	kept

Look at Adjust 3. The adjuster wrote “cite POLICY-001” into the brain. It has never seen POLICY-001. It knows the name only from a citation judge that said missing citations: [‘POLICY-001’] and a must-mention judge that listed on-call and 1 hour as missing. From those reasons it inferred a rulebook it had never read — which is the whole bet of hiding the policy.

Which judge moved, and when

One training score hides which judge moved it. Here are the five weighted judges on the training set, step by step (Score 2 is the reverted candidate):

Judge	Score 0	Score 1	Score 2reverted	Score 3	Score 4	Score 5
Format3 sections, ≤ 250 words	0.04	0.96	0.96	0.91	0.96	0.96
KB citationthe right article IDs	0.80	0.80	0.75	0.95	0.95	0.95
Must mentionthe key facts	0.85	0.80	0.81	0.90	0.91	0.93
Policyno forbidden advice	0.55	0.75	0.70	0.75	0.90	0.95
Qualitya model, 1 to 5	3.35	3.60	3.50	3.95	4.50	4.65
Training scorethe weighted sum	0.618	0.787	0.765	0.859	0.920	0.938

Format moved first and furthest, in one step, because every ticket shares it. KB citation and must mention moved together at Score 3: one Severity 1 rule fixed both. Policy climbed over three steps as the missing-version and forbidden-command rules arrived.

The most telling row is the last judge. Quality is the model judge, and no step aimed at it — no hypothesis named it. It still went from 3.35 to 4.65. An independent reader, grading against its own rubric, agreed that the answers the code judges pushed for were better answers.

Two runs, one shape

training score · the tickets the adjuster learns fromvalidation score · the tickets it never sees○ the best brain

Claude Code on Qwen · validation 0.649 → 0.910 (v5) · 19 min 48 s · 5.43 + 0.67 USDOpenCode on Muse Glimmer · validation 0.394 → 0.915 (v4) · 2 h 1 min · 1.27 + 0.14 USD

A lower start, the same finish. The second run went through OpenRouter, took six times as long, and hit the controller’s two-hour deadline after its last score — and its last brain, v5, came out worse than v4. The result of a run is its best brain, and here that is not a figure of speech.

Both curves were measured on tickets that use other knowledge base articles than the training tickets. The brain did not learn the answers. It learned how to work: ask for the version, cite what you used, escalate the outage, refuse the shortcut.

Keep or revert is noisy too

The rule “keep on a tie, revert on a drop” compares one run against one run. The benchmark post showed how much one run moves, and the training set is no calmer: brain v0’s training score moved between 0.33 and 0.47 across four runs. So a candidate close to the best brain is kept or reverted partly by luck. I did not put a margin on the rule to hide that. The talk named it on a slide instead: some decisions in this loop are coin tosses, and the validation curve, which decides nothing, is there to show whether the kept steps were real.

Two smaller rules come from the same honesty. A candidate with the same files as the best brain is marked no change, scored like any other, and counted as a revert, so an adjuster that edits nothing cannot keep a loop alive. And an adjuster session that edits nothing gets one second try, with a task that says so, before the step gives up.

Three ways to adjust

Since yesterday the adjust step is a switch with three engines behind it. The loop, the scoring and the keep-or-revert stay the same. Only the thing that proposes the next brain changes.

The adjuster, the OpenCode agent above, is the default. It can create skills, it reads the history, and it explains itself in change.md.

GEPA is a prompt optimizer, and its reflective proposer fits this loop unchanged: a candidate is a dictionary of named texts, which is the brain, and feedback is a list of input, output and critique, which is failures.md. The backend calls it as it ships — version 0.1.4, its default prompt — once per brain file, through the same gateway. On the full thirty tickets on Qwen it went from 0.282 to 0.998 on training and from 0.353 to 1.000 on validation, in 11 minutes, for 1.83 USD. Two things came with it. GEPA has no word limit, and its third brain was 1,672 words long — every ticket now pays to read it, and the agent’s cost per scoring step went from 0.18 to 0.54 USD. And in two smoke runs on two validation tickets, GEPA’s rewrite copied ticket facts and answer keys into the brain, and validation fell. The full run did not show it. The smoke runs are too small to prove anything, and too consistent to ignore.

GEPA also cannot create a file: it rewrites the ones that exist. So there is a fourth option, a schedule — the adjuster in the first step, GEPA after it — and never the other way round:

I value that our adjuster can create the skills.

/eval-optimize is the harness’s own optimization skill, run headless in Claude Code on the run’s model through the gateway, so no Anthropic key is involved. It was written to run its own evaluation; here the next Score step does that, so the backend copies only the skill, narrows its shell to read-only commands, and leaves the judges’ prompts out of its workspace, because the model judge’s prompt carries a summary of the hidden policy. On Qwen it used 34 turns and 540,000 input tokens for one step, about ten times the adjuster’s cost. It also searched outside its workspace for the judges’ code, twice, and both reads were refused. An optimizer that goes looking for the grader is doing its job. That is exactly why the grader is out of reach.

None of the three decides anything. Each one proposes a brain; the next score keeps it or throws it away.

The last post is about the platform all of this ran on — and why, apart from one gateway and a lot of Jobs, there is nothing on it worth calling a platform.