"It looked like it worked": grading a non-deterministic agent

The agent finishes. It writes a summary. The summary says the task was completed successfully. The summary is wrong.

Not maliciously — it genuinely cannot tell. It clicked what looked like the right thing, the screen changed in a way consistent with success, and its own self-assessment is generated by the same model that made the decisions being assessed. Asking it to grade itself is asking a student to mark their own exam using the notes they revised from.

So we built something that grades from the outside.

Three layers, in increasing order of trust

Layer	Cost	Trust	What it is good for
Agent self-report	free	low	detecting obvious failure, driving the loop
Deterministic evaluator	cheap	high	the actual pass/fail number
Human review of the video	expensive	highest	anything surprising

The self-report still exists — the agent needs an internal notion of “done” to terminate. It just does not decide the grade. And the video recording exists because when a benchmark says 3 of 5 runs failed, nothing except watching them ever explains why.

The interesting layer is the middle one.

Why the evaluator lives inside the guest

server_evaluator runs on port 5053 inside the Windows VM, alongside the other control servers. That placement is the single most important design decision in it.

The orchestrator, outside, can see pixels. That is all. If it wants to know whether a file was created it has to infer it from a screenshot of a file manager, which is exactly the fragile inference we are trying to eliminate.

The evaluator, inside, can read C:\. It can inspect the filesystem, query process state, and read the app’s own telemetry off disk. It gets facts where the orchestrator gets appearances.

def check_file_exists(path, file_name):
    full_path = os.path.join(path, file_name)
    return {"exists": os.path.exists(full_path)}

Four lines, and infinitely more reliable than any amount of vision-model reasoning about whether a file icon appeared.

The interface is a plain HTTP POST, called from the scenario’s teardown hook:

{
  "evaluation": [{
    "evaluator": "teams_scenarios",
    "scenarios": ["simple_collab_switch"],
    "telemetry_file": "/data/logs/teams-telemetry/teams-telemetry-0.log"
  }]
}

Grading from the application’s own telemetry

The best evaluator we built does not look at the filesystem at all. It reads Microsoft Teams’ own telemetry stream — captured via a man-in-the-middle proxy — and checks whether the application itself reported the events that a successful run would produce.

The check is ordered marker matching:

marker_index = 0
for line_num, line in enumerate(f):
    record = json.loads(line)
    if marker_index >= len(markers):
        break

    marker = markers[marker_index]
    if match_marker(record, marker):
        found.append(marker)
        match_indices.append(line_num)
        marker_index += 1

matched_in_order = len(found) == len(markers)
return { "found": found, "not_found": not_found,
         "matched_in_order": matched_in_order, "success": matched_in_order }

Note marker_index only advances on a match, and never resets. This is subsequence matching, not set membership: the markers must appear in order, though other events may occur between them.

That ordering constraint is doing real work. “Switch between 5 chats” is not satisfied by five chat-open events in any arrangement — a bug that opens the same chat five times produces five events. It is satisfied by a specific sequence. Order is most of the semantics of a workflow, and a naive “did all these events happen” check throws it away.

Markers are matched with dot-path lookups into nested telemetry:

def deep_get(dictionary, keys: str):
    """Retrieve nested value using dot notation (e.g., data.Action.Gesture)"""
    for key in keys.split('.'):
        if not isinstance(dictionary, dict):
            return None
        dictionary = dictionary.get(key)
    return dictionary

So a marker can be as specific as data.Action.Gesture == "click" on an event called chat_switch, and as loose as just the event name.

The result carries found and not_found rather than a bare boolean. When a run fails you immediately see which marker it got to — which is the difference between “it failed” and “it completed four of five steps and stalled on the summarise.”

The four-word commit

let's do 5 times

That is the entire commit message, and it is the most consequential one in the project.

Up to that point, a scenario ran once and the result was pass or fail. Then we ran the same scenario five times and got three passes and two failures — with identical configuration, an identical starting state, and no code change.

That is not a bug. That is what a stochastic policy in a non-deterministic environment does. Temperature, timing, whether a background process happened to steal 200ms — all of it moves the trajectory.

The consequences cascaded:

• A single run is not a result. It is one sample from a distribution.
• “The agent can do X” is not a claim you can make. “The agent completes X in 4 of 5 runs” is.
• Comparing two prompts, two models, two anything requires many runs each. A change that takes you from 3/5 to 4/5 might be an improvement or might be noise, and you cannot tell from one run.
• Therefore you need twenty desktops, not one.

That last inference is the whole reason the second repository exists. Running one Windows VM on a laptop answers “can the agent do this?” Running a hundred answers “how often, at what cost, and did last week’s change help?” — and those are the only questions that matter once the demo works.

Six days after that commit, Terraform and Helm charts landed in the repo. A week after that, OpenOperator-Infra got its first commit.

Cost as a first-class result

Alongside pass/fail, every run records money and time, per step:

from test.helpers import ..., sum_costs

“It succeeded” and “it succeeded for $4.20 in eleven minutes” are different results, and only the second one tells you whether the approach is viable. We had scenarios that passed reliably and were economically absurd — the agent would find its way there eventually via twelve replans. Without cost accounting those look identical to an efficient success on the dashboard.

There is also a pathology cost tracking catches that nothing else does: the run that passes because it burned its whole budget wandering into the right answer. Same grade, ten times the price, and a strong signal that the planner is bad at that task even though the number is green.

What I would build first, next time

In order:

1. A deterministic oracle with privileged access to the environment. Before the agent is good. You cannot improve what you cannot measure, and building the measurement afterwards means every result before it is uninterpretable.
2. Multiple runs by default. Make “run this once” the special case. If your harness makes single runs easy and repeats hard, you will draw conclusions from single runs. I did.
3. Cost and duration per step, recorded from day one.
4. Video, always on. It is cheap and it is the only artifact that ever explains a weird failure.

The thing I would not do again is let the agent’s self-report anywhere near the grade. It took two weeks of confusing results to fully separate them, and every one of those results had to be thrown away.

Next: MCP for mouse and keyboard — turning a desktop into a tool server, and the one design decision in it I still think is underrated.