Thirty tickets nobody filed

The first full run on the new dataset looked like a success, and it was a failure.

It was the evening of 29 September, on the laptop. The plain agent — twelve words of instructions, nothing else — scored 0.837 on the twenty training tickets and 0.765 on the ten it would never learn from. One adjust step later, the numbers were 0.983 and 1.000. A support agent went from twelve words to perfect in one step.

A measurement that reaches the ceiling after one step cannot tell you anything about step two. And the reason it got there was worse than the number. The event logs showed it: the plain agent had opened a policy file on 17 of the 20 training tickets. The rulebook the judges graded by was sitting in its folder, and the agent found it with a recursive glob. Then the adjuster read the whole policy too and copied it into the prompt. Score 0.98, nothing learned.

This post is about the dataset under every number in this series, and most of it is about what the agent must not be able to see.

Why a support desk

I needed a job where an answer is right or wrong, so that code can score it. A sales email is an opinion. A support answer is not: it cites the right article or it does not, it asks for the version or it guesses, it escalates the outage or it tries to fix it alone. Each of those is a check a few lines of Python can make, and the one judgement that needs a reader — is this a good answer? — is left to a single model judge.

So the agent answers tickets for Red Hat products from a local folder of knowledge base articles. The product names are real. The articles, versions, fixes, customers and rules are invented.

A knowledge base built to confuse

There are 32 articles. Each one has the same shape: the product, the affected versions, the version that fixes it, symptoms, cause, resolution, and a section called Notes for support engineers that holds exactly one rule the agent is supposed to apply. KB-2210 ends like this:

## Notes for support engineers

- Always ask for the RHEL version (`cat /etc/redhat-release`) before you apply this article.
  On RHEL 8 the cause is different, and this fix does nothing.

The dangerous part is the neighbours. Several articles come in look-alike pairs: the same symptom, a different version or platform, a different fix.

Pair	Same symptom	What separates them
KB-1042 / KB-1043	the image registry operator is Removed, oc start-build fails	OpenShift 4.20 single-node, fixed in 4.20.8 — against 4.19 on any platform without a storage API, not a bug
KB-2210 / KB-2211	subscription-manager register fails behind a corporate proxy	hangs for minutes on RHEL 9.4 to 9.6 — against fails at once with certificate verify failed behind a TLS-inspecting proxy

An agent that matches on the symptom picks the wrong one half the time. The plan counted on the look-alikes for headroom. They were not where it was: the plain agent cited a wrong look-alike on 3 of the 20 training tickets. What the pairs really do is make the version question matter. Without the version, there is no way to pick between them, and a confident answer is a coin toss.

Thirty tickets, eight kinds

Every ticket is written the way customers write: some precise, some panicked, some far too relaxed for what they describe.

hey, not sure how urgent this is, but after the power cut in our datacenter last night none of the nodes of our prod cluster came back. oc get nodes says NotReady for all of them and our web shop is offline since then. It’s OpenShift 4.19.14. can someone have a look when you get a minute? thx

That is a Severity 1 — production down, no workaround — in the voice of a Severity 4. The policy says to judge severity from the facts, not the tone, so the right answer escalates it to the on-call engineer and asks for a must-gather. The other kinds plant other traps:

Kind	What a good answer does	Train	Validation
plain fix	cites the right article and the fix for the stated version	6	3
missing version	asks for the version, says how to find it, names the candidate articles — and gives no fix steps yet	4	2
Severity 1	escalates to on-call with the product’s diagnostic data, whatever the tone	3	1
trap: ETA	gives no date for a fix, because engineering owns the timeline	1	1
trap: SELinux	refuses to turn off a security control, even when asked which is faster	1	1
trap: reinstall	does not reinstall as a first step, even on “only a test box”	2	0
two articles	handles two problems with two articles	2	1
out of scope	says so, points to the vendor, guesses nothing	1	1

The SELinux trap hands the agent the wrong question on purpose: “Our admin wants to disable SELinux on the worker nodes or give the pods the privileged SCC. Which one is faster?” The right answer is neither, and an article that fixes the actual bug.

The answer key

Every ticket has a second file next to it, which the agent never sees. It is what the judges read. Take this ticket, Severity 3:

subscription-manager register hangs for about five minutes and then says it cannot reach subscription.rhsm.redhat.com. We are behind a corporate proxy and curl works fine with https_proxy set. What should we do?

No version. Its key:

kind: missing-version
expected_kb_ids: ["KB-2210"]
must_mention: ["which version|what version|exact version|confirm the version|your version|version you|version are you|version do you", "redhat-release"]
must_not: ["subscription-manager config"]
needs_version_question: true
needs_escalation: false

expected_kb_ids are the articles the answer must cite. Each entry in must_mention is a required fact, and a|b means any of them counts, because a customer can be asked for their version in a dozen ways. must_not lists forbidden content, as whole words, and a clause that negates it does not count — do not reinstall is fine, reinstall it is not.

Look at what is forbidden here: subscription-manager config. That is the fix. KB-2210’s resolution starts with exactly that command. But the ticket gives no version, KB-2210 only covers RHEL 9.4 to 9.6, and on RHEL 8 “this fix does nothing”. So on this ticket the right fix is the wrong answer until the customer confirms the version. A judge that only checks for the correct command would reward the guess.

The rulebook nobody may read

After the run that hit the ceiling, I moved both policies — POLICY-001 for severity, escalation and answer rules, POLICY-002 for scope — out of the agent’s folder and out of the adjuster’s. They live next to the judges now. The judges grade by them and the slides quote them. Neither agent can open them.

That changes what the agent can learn, and from where. Of the 49 required facts on the training tickets, 13 exist only in the hidden policy: the version question, the Severity 1 procedure, the out-of-scope refusal. So do the answer format and the citation rule. The only way to learn them is the judges’ feedback: a format judge that says “no ‘next’ section, 312 words”, a citation judge that says “missing citations: [‘POLICY-001’]”. The agent learns the name of the rulebook from the reason it was marked down — like a new hire who never gets the handbook and learns the rules from code review.

The same evening, on Cerebras, the loop ran again on the hidden-policy dataset:

Step	Train	Validation	Failing train tickets
twelve words	0.434	0.447	20 of 20
after adjust 1	0.838	0.909	8
after adjust 2	0.856	0.943	5
after adjust 3	0.995	0.975	1

A start near the middle, three steps that each found something, and a finish just under the ceiling. The format judge alone went from 0.00 to 0.99 in the first step, because every ticket shares the format. That is the shape a dataset has to allow before any number on it means something.

Train and validation share no articles

A ticket counts as failing when any judge is below its top. The adjuster reads every failing training ticket with its answer key — which is the point of training, and also the risk. It could copy facts. So the ten validation tickets use other knowledge base articles than the twenty training tickets. Only the two policies are shared.

That makes the validation score honest in a specific way. The adjuster cannot copy a validation answer into the brain, because it never saw one and the facts differ. If the validation score rises, the agent got better at the process — ask, cite, escalate, refuse — not at the answers. In that run it rose with the training score at every step, and sat above it after two of the three.

Freezing ten of them

The ten validation tickets have a second life: they are the benchmark set, support-bench v1. The rest of the suite is free to grow — tickets from real incidents, fixes to the judges — but the benchmark set changes only as a new version. A benchmark has to give the same result to anyone who runs it again, so it cannot share the lifetime of a suite that changes with every incident. That is a story for the benchmark post.

There is also a third set, for plumbing: six smoke tickets, four from training and two from validation, a mix of kinds. The rule for it is one line in the plan: never for a number that goes on a slide.

What thirty tickets cannot do

Thirty invented tickets cannot tell you how good an agent is at real support. They are too few, the facts are made up, and every score in this series comes with a spread I measured. What they can do is make an agent’s behaviour countable — the version question, the escalation, the refusal — and that part carries over to your own tickets unchanged.

The next post is about the judges that do the counting, and about the release gate that refuses to trust their average.