Your test suite bills by the token

Dex runs a project as a loop of stages, each one a fresh Claude Code agent, and it commits after every stage so the whole run is navigable afterwards as a branching timeline.

Which raises a question I did not think about until it was expensive: how do you test the timeline?

Not the agents. The agents are somebody else’s correctness problem. I mean the part I wrote — does the fourth stage of the second cycle land on the right branch, does clicking a mid-history node fork correctly, does a mid-cycle pause resume at the stage it claims. That code is completely deterministic. It is also, in the original design, only reachable by spending twenty minutes and several dollars of inference to produce the git history it operates on.

Version 1: run it for real

There was no version 1. That is the point. Verifying a change to the timeline looked like this:

1. Reset the example project.
2. Start a real loop.
3. Wait through prerequisites, clarification, constitution, manifest extraction, gap analysis, specify, plan, tasks — none of which you are testing — for fifteen to twenty minutes.
4. Get to implement, watch the thing you actually changed.
5. Notice it is wrong.
6. Go to 1.

And a single cycle is not even sufficient. Half the interesting timeline behaviour is cross-cycle: does cycle 3’s gap analysis node attach to the right parent, does a jump back into cycle 1 correctly detach, does the branch prune when you leave it empty. So the real loop is that loop, three times over, for every iteration on a UI change.

Where it broke

I stopped testing. Obviously. Not as a decision — nobody decides to stop testing — but as a slow drift into “I’ll just eyeball it”, which works until you ship a timeline that renders cycle 3 in the wrong column and only notice a week later because by then nobody wants to spend twenty minutes finding out.

The diagnosis is that three properties had collided in one place:

Property	True of
Nondeterministic	the agent
Slow	the agent
Expensive	the agent
Deterministic, fast, free	everything else in the system

Every one of those problems lives on one side of one boundary. The orchestrator’s own logic — git operations, state writes, event emission, timeline layout, resume arithmetic — has none of those properties. It only looks like it does, because it happens to be downstream of something that has all three.

So don’t fix the agent. Find the seam and put something behind it.

The seam

The seam turned out to be small. An agent backend has to do exactly three things:

export interface AgentRunner {
  runStep(ctx: StepContext): Promise<StepResult>;
  runTaskPhase(ctx: TaskPhaseContext): Promise<TaskPhaseResult>;
  /** 014 — generic ad-hoc invocation. v1 caller is the conflict resolver. */
  runOneShot(ctx: OneShotContext): Promise<OneShotResult>;
}

Three methods. runStep is one stage of the loop, runTaskPhase is one phase of a tasks.md build, runOneShot is any small ad-hoc gesture. Everything the orchestrator knows about “calling an AI” is behind those signatures, and selection is one switch:

export const KNOWN_AGENTS = ["claude", "mock"] as const;

export function createAgentRunner(agent: string, config: RunConfig, projectDir: string): AgentRunner {
  switch (agent) {
    case "claude": return new ClaudeAgentRunner(config, projectDir);
    case "mock":   return new MockAgentRunner(config, projectDir);
    default:       throw new UnknownAgentError(agent);
  }
}

There was briefly a dynamic registry here, with registration functions and a lookup map, because that is what you build when you have been told plugins are good. It got deleted in favour of a switch. At two implementations a switch is easier to read, easier to grep, and adding a case is one line.

Set agent: "mock" in the project’s .dex/dex-config.json and the entire loop runs against a script. A three-cycle run finishes in under a minute at zero API cost, and produces the same stage commits, the same branches, the same .dex/ artifacts, and the same event stream the real thing would.

The mock that refuses to improvise

The important design decision in MockAgentRunner is not what it does. It is what it refuses to do.

The script is a JSON file — .dex/mock-config.json — keyed by phase, cycle and stage. Each entry says how long to pretend to take, what files to write, and what structured output to return:

{
  "dex_loop": {
    "cycles": [
      {
        "feature": { "id": "f-001", "title": "Product catalogue" },
        "stages": {
          "specify": {
            "delay": 400,
            "writes": [
              { "path": "{specDir}/spec.md", "content": "# Feature Specification\n..." }
            ]
          },
          "plan": { "delay": 400, "writes": [{ "path": "{specDir}/plan.md", "content": "..." }] }
        }
      }
    ]
  }
}

Now: what should the mock do when the orchestrator asks for a stage the script does not cover?

The tempting answer is something reasonable. Return an empty success, write nothing, keep the run going. It is one line and it makes the mock feel robust.

It is also the single worst thing you could do, because the mock’s entire job is to be a truthful stand-in, and a stand-in that quietly invents behaviour is generating false confidence at machine speed. So it throws, loudly, with the coordinates in the message:

MockConfigMissingEntryError: no script entry for phase=dex_loop, step=verify,
cycle=2, feature=f-001. Update .dex/mock-config.json.

Same for a path template containing a token it does not recognise — the allowed set is {specDir}, {cycle}, {feature}, {runId}, {shortRunId}, and anything else is an error naming the bad token and listing the legal ones, not a literal {featureName} directory appearing on disk.

The second lever: replay, not just simulation

The mock buys speed by replacing the agent. Sometimes you need the opposite — real artifacts from a real run, because what you are testing is how the orchestrator handles the messy output an actual model produced, not the tidy output you wrote into a fixture by hand.

That is what checkpoint replay is for. Since every completed stage commits, any point in any past run is a ref, and a named tag can be pinned to it:

scripts/promote-checkpoint.sh ../dex-ecommerce after-tasks
scripts/reset-example-to.sh list          # every save point
scripts/reset-example-to.sh after-tasks   # restore the tree to it

Reset to after-tasks and the next launch boots straight into implement, against specs and a plan that a real agent really wrote, with a real state.json describing a real half-finished run. Fifteen minutes of upstream inference, paid for once, replayed for free forever.

The two levers answer different questions, and it took me a while to say that crisply:

	Mock backend	Checkpoint replay
Replaces	the agent	the time spent getting there
Artifacts are	authored by me	authored by a real model
Good for	orchestration, timeline, resume logic	anything downstream of real output
Cost	free	free after the first real run

The aside that paid for itself

While making runs inspectable I deleted the database.

Dex used to keep its audit trail in SQLite at ~/.dex/db/data.db. Answering “what did that run cost” meant opening a SQL shell, in a home directory, against a schema you had to remember. Nobody does that. So nobody looked at the audit trail, so the audit trail may as well not have existed.

It is now one JSON file per run, inside the project it describes:

jq '.totalCostUsd, (.phases | length)' .dex/runs/<runId>.json

The ergonomic win is the point, but the build win was the bigger surprise: better-sqlite3 is a native module, and native modules in Electron mean rebuilds against the right ABI, on every machine, on every version bump. Deleting it removed an entire category of “works on my machine”. The engine’s only dependencies for state are now node:fs, node:path and node:crypto.

What the mock does not buy you

Now the honest part, because a post about testing that only lists wins is marketing.

The mock is easier to satisfy than the real agent. It never writes a spec directory with a slightly different name than it announced. It never formats tasks.md with three spaces where the parser wants two. It never returns a paragraph of prose where a schema was expected. Every one of those has actually happened, and none of them can happen against a script I wrote, because I wrote the script with the parser in mind.

So a green mock run proves the orchestrator is internally consistent. It does not prove the system works. That gap is covered by three other things — the replay path above, which uses genuinely messy real artifacts; the unit tests over the parsers and the timeline layout; and, in the end, real runs, which I still do before anything ships.

The general lesson

The mistake I made was treating “this system is slow, nondeterministic and expensive” as a property of the system. It was a property of one dependency, and everything else had been dragged into its cost profile because there was no boundary between them.

That generalises well past agents. Whenever some category of test feels impossibly expensive, the useful question is not “how do I make this cheaper” but which single component actually carries the expensive property, and what would it take to put an interface in front of it? In this case: three method signatures and a switch statement, in exchange for a test cycle that went from twenty minutes to under one.

The other lesson is smaller and I feel more strongly about it. A test double’s job is to be honest, not to be helpful. Make yours throw.

Next: the bug that made me stop trusting my own state file — three sources of truth, one Stop click, three different answers.