Nothing here is special
Every step is a Job, every brain is a ConfigMap, every score is an MLflow run, and one gateway holds every key. The platform under the agent flywheel, the libraries it is built from — the evaluation harness, Harbor, EvalHub, GEPA — what each one needed wrapped around it, and the sandbox that is built against the one that is only designed.
At minute nineteen, the talk’s run sheet leaves the slides and the console for
the plain OpenShift web console. Workloads, Jobs: the controller of the
training run started a minute earlier, and its first Score Job. ConfigMaps: brain-v0, with
twelve words in it. The Score pod: not root, every capability dropped, 6 cores
and 6 GiB, and no Secret mounted. Then the ConfigMaps of the morning’s run,
flywheel-8l25r-brain-v0 to -v5 — six versions of an agent’s instructions,
as text files you can diff, review and roll back.
The cluster is the lab notebook. This last post is about that notebook: what a run puts on the cluster, the libraries it is assembled from, and what each one needed wrapped around it before its numbers could be trusted.
What one training run creates
One oc create -f starts a controller Job, and the controller makes the rest.
For every step of the loop it creates a Job from a template: <id>-score-N
runs the evaluation, <id>-adjust-N runs the adjuster. Every version of the
brain becomes a ConfigMap, <id>-brain-vN, mounted read-only at /brain. The
controller owns all of them, so when its Job expires a day later, everything
it made goes with it. Names carry the run’s id, so a second run never collides
with the first.
The division of labour is strict, and it is the part I would copy into any other project:
- The steps talk to the gateway and to MLflow. They never call the Kubernetes API.
- The controller talks to the Kubernetes API and to MLflow. It never sees a model key.
- The handoff between them is MLflow. An adjust step logs the next brain as an artifact, the controller downloads it and turns it into the next ConfigMap. MLflow is the record. The ConfigMaps are the copy the cluster runs.
Because the handoff is MLflow, the same loop runs three ways: the steps as processes on the laptop, as Podman containers from the same image, or as Jobs on the cluster. A free run on a local model — Gemma 4 26B on Unsloth Studio, with thinking off — exercises exactly the handoff the cluster uses. One image, built on the cluster by a BuildConfig, holds everything the Jobs need: the three agents, the tickets, the judges, brain v0 and the console.
One gateway, all the keys
Every model call — the agent’s, the judge’s, the adjuster’s — goes through one LiteLLM gateway in the namespace. It is the only pod with a model key. A Score pod has no key to leak, because it never had one.
The gateway also turned out to be a translator, which I did not plan for.
Cerebras speaks one API, chat completions, and validates it strictly: a
reasoning_effort object, a thinking block on an assistant message, an unknown
top-level field, or a system message after the first user message — each one is
a 400. Claude Code speaks the Anthropic Messages API and puts its environment in
a system message after the user’s. Codex speaks the Responses API and sends a
reasoning object. A small pre-call hook in the gateway reshapes every request
into what Cerebras accepts. The first gateway I tried, Envoy AI Gateway, passed
the Responses API through unchanged, so Codex could not reach Cerebras at all,
and round-tripped Claude Code’s thinking blocks, so every second turn failed.
Two more lessons cost a run each:
- Pin the gateway by digest. On the day before the talk, the moving image tag pulled a newer LiteLLM onto the cluster that refused to start without a master key. CrashLoopBackOff, on the last day of preparation.
- A gateway is not transparent until you have measured it. OpenCode treats a custom provider’s models as having every capability off unless the config declares it — and with temperature off, it sends no temperature at all. The first loop through the gateway ran Qwen at the provider’s default temperature, and the agent skipped the knowledge base on four tickets of six. The scores dropped. Declaring the capabilities made the request bodies match the direct route again, field for field, apart from two the gateway itself adds.
Three agents, three ideas of a sandbox
The agent under test is OpenCode, Claude Code or Codex, picked per evaluation. Each one gets a runner script that puts the brain where that agent looks, a fresh folder per ticket, a fresh config folder so the developer’s own settings never reach it, and the gateway. Each one also needed something different:
| Agent | Surprise | What the runner does about it |
|---|---|---|
| OpenCode 1.18.30 | reads ~/.claude/CLAUDE.md and ~/.claude/skills/ by default |
sets OPENCODE_DISABLE_CLAUDE_CODE=1 — before that, every laptop run’s support agent could see ten of my own skills |
| Claude Code 2.1.289 | no Glob, Grep or LS tool, and Read cannot list a folder | grants Bash with an allow-list of read-only commands; its don’t-ask mode refuses any read outside the folder |
| Codex 0.154.0 | its read-only sandbox blocks writes, not reads, and cannot start inside another sandbox | runs with its sandbox bypassed inside the pod, so the fresh folder and the pod are the guard |
Neither Claude Code nor Codex knows the price of a Cerebras model, so the runners price the tokens themselves and every agent reports cost the same way. Claude Code originally had a 15-turn cap; on Severity 1 tickets it hit the cap and ended with no answer at all, which failed the whole evaluation. The cap is gone — the per-ticket timeout is the only limit.
And agents leave things behind. After one benchmark the node’s disk was 82% full: a finished Codex pod kept 90 MB of unpacked plugins per ticket, 61 finished pods held 22 GB until their Jobs expired, and they pinned old images against the kubelet’s cleanup. Every runner now deletes its per-ticket folders when it exits.
The libraries, and what each needed around it
| Library | Its role here |
|---|---|
| agent-eval-harness | the engine in every Score Job: eval.yaml, the judges, the runners |
| Harbor | a Score backend: one container or pod per ticket |
| EvalHub | a Score backend: job records, status, cancel, tenancy |
| MLflow | the record: runs, metrics, one trace per ticket, every brain |
| LiteLLM | the gateway, and the only holder of keys |
| GEPA | an Adjust backend: its reflective proposer |
| OpenCode, Claude Code, Codex | the agents under test |
The harness runs headless: five of its scripts — workspace, execute, collect, score, report — called by the Score step, which is the mode its own CI guide documents. It is pinned, unmodified, and none of its Claude Code skills run in the scoring path.
Since Friday the Score step is a switch, SCORE_BACKEND, and the other two
libraries are behind it. Both were planned one way and built another.
Harbor was meant to run the harness’s own Harbor driver, with Harbor’s verifier running the judges inside each container. A spike showed why that cannot work for this suite: the task bundle’s config carries no dataset path, so the judges in the container get no answer keys, and the judges read a file only our runner writes. So the backend inverts it. Each ticket becomes a Harbor task whose agent is a small class of ours that runs our runner inside the container; the files come back out, and the judges run outside with the real answer keys. Harbor adds the container and costs about ten seconds a ticket. On OpenShift each trial is a pod with no ServiceAccount token, scratch volumes for the paths Harbor writes, and an owner reference to the Score Job’s pod — so a stopped or crashed Score Job takes its trial pods with it. In the stop test they were gone 36 seconds after the Stop. Only Score Jobs that use Harbor run as a ServiceAccount that may create pods and exec into them; a Job made by hand without it failed every ticket in ten seconds.
EvalHub was meant to run through the harness’s EvalHub adapter. That adapter
returns the judges’ means — and drops the per-ticket reasons the adjust step
reads, and our runners with them. So the backend is our own EvalHub provider,
written with the SDK’s adapter for bringing your own framework: EvalHub’s
server runs it as one job per ticket set, it runs the harness step with our
runners, and the full run folder comes back through MLflow. EvalHub keeps what
it is good at — the job record, the status, the cancel — for about two seconds a
set. No operator release ships EvalHub yet, so on the cluster it runs from the
TrustyAI operator’s main, pinned to one commit and three image digests. Three
gaps showed up only live: the server did not read the provider folder the
operator mounts, the operator gave the server no right to create the
NetworkPolicy it makes for every job, and our provider, as process 1 of its pod,
ignored SIGTERM — a cancelled job finished its set anyway.
The backend is part of the benchmark recipe, because it can change a score. Brain v0 on the validation set yesterday, one run each: 0.562 on the harness, 0.531 on Harbor, 0.470 on EvalHub. Whether that gap is the backend or the noise needs more than one run each to say. Until then, a benchmark reuses a run only on the same backend and the same environment.
Two updates from yesterday
GEPA learned the answers. The training post reported GEPA taking validation from 0.353 to 1.000, and two smoke runs where it copied ticket facts into the brain instead. Yesterday a full run of the schedule — the adjuster first, then GEPA — settled it the second way: training 0.389 to 0.988, validation 0.433 to 0.558. The same day, the adjuster alone went from 0.361 to 1.000 on training and from 0.410 to 0.958 on validation, every candidate kept, in 13 and a half minutes for 2.46 USD. Only the validation column tells those two runs apart, and it can do that because it decides nothing.
The judge got fixed. The evaluation post ended its list of judge bugs with
one still open: a neither … nor sentence read as forbidden advice. Yesterday
it blocked the adjuster’s best brain at the gate — policy 0.90, on one sentence
alone: “Neither disabling SELinux nor the privileged SCC is the correct
fix.” neither and nor
joined the negation words. Replayed offline on 300 logged answers, the fix
changed three verdicts, all of them that sentence. After a rebuild, the same
brain scored 1.000 and passed all three rules. And because the suite’s config is
part of the recipe, no benchmark after the fix will reuse a run scored before
it.
Stopping is a feature
The console can stop a run. On the cluster that means suspending the controller Job, then every step Job that has not finished. The first stop on the cluster did not stop anything: the controller ran on for its 30-second grace period and created the next step’s Job. A script running as process 1 of its pod gets no default handler for SIGTERM, so it simply ignores it. Every step script now exits on SIGTERM itself. Five days later the EvalHub provider had the same bug, for the same reason.
The sandbox, built and designed
An agent has files, tools and a network. It must not read what it should not know — in an evaluation, the answer key, and then the score lies — and it must not touch what it should not. Here are the layers, outside in. Three run today, and the two dashed ones are designed and not built:
The keys sit in Secrets that only the gateway reads.
The pod runs in a VM with its own kernel.
Not root, every capability dropped, no privilege escalation, the default seccomp profile, CPU and memory limits, the brain mounted read-only. OpenShift applies that SCC to every pod by default.
Egress only to allowed hosts, a placeholder instead of the real key, and a record of every request.
For OpenCode: edit, bash, webfetch, task and external_directory denied. The closest equivalent for the other two. A fresh folder per ticket that holds nothing else.
The two dashed layers wait for a cluster that can run them. Kata Containers need a runtime class and nested virtualization this cluster does not have. OpenShell records every request the agent makes, allowed or denied, and nothing here runs it yet. One gap is known in what is built: the step pods still mount the default ServiceAccount token. No agent tool can read it, but it is there.
For an evaluation, OpenShell is the more interesting of the two, and not for the blocking. The judges see only the answer. They never see what the agent reached for on the way. The design document puts it in two sentences:
A sandbox that only blocks is a wall. A sandbox that records is a judge’s witness.
Build the evaluation first
The talk ended with three sentences, and a week later they still hold.
An evaluation measures an agent and its brain. A benchmark compares agents. Training improves the brain. All three run the same evaluation, again and again — so build the evaluation first.
One number lies. A mean hides the broken ticket, and one run hides the noise. Read every judge and every ticket, and measure more than once.
An agent is code you cannot trust. Measure it, and run it on a platform built for that. On OpenShift that platform is mostly the defaults — which is why nothing here is special.