The API is the screen

Every integration project starts with the same optimistic question: does it have an API?

For the work that actually costs companies money, the answer is a qualified no. There is an API for “send a message.” There is no API for “find the thread where someone mentioned the invoice, check whether the finance person ever replied, and summarise where it stands.” That task lives in the gap between five features that each have an endpoint and a workflow that has none. It is performed, tens of thousands of times a day, by a human looking at a screen.

So we stopped asking for an API and pointed an agent at the screen instead.

This is the first post in a series about OpenOperator — five months of work across two repositories, from one Windows VM on a laptop to a Kubernetes cluster that provisions disposable desktops on demand and benchmarks agents against them. I want to write it up honestly, in the order it happened, including the parts that took a week and shouldn’t have.

What the thing does

The loop is simple to describe and unreasonable to build.

Take a screenshot. Work out what is on it. Decide the next action. Perform it with a real mouse and a real keyboard. Take another screenshot. Repeat until the goal is met or you have burned your budget.

Every arrow in that diagram is a subsystem, and only one of them is the model.

The uncomfortable ratio

Here is the thing nobody puts in the demo video. Across five months, the split looked roughly like this:

Work	Share
Agent logic — planning, execution, replanning	~20%
Making a real OS reproducible, observable, disposable, cheap	~80%

The agent was the fun part and the small part. The rest was:

• Getting a real Windows 11 to exist as a disposable unit — which turns out to mean QEMU inside a container, /dev/kvm, an unattended-install answer file, and a 6 GB ISO you cannot legally bake into an image.
• Getting software installed into that guest with no SSH, no cloud-init, and no config management — just a .bat file, a SMB share the hypervisor provides, and Windows Task Scheduler flags whose documentation does not mention the one behaviour you need.
• Getting eyes on it, because you cannot debug a computer-use agent from logs. Failures are visual. The click landed four pixels off. A modal stole focus. The app was still painting.
• Getting fifty of them at once, on Kubernetes, without waiting forty minutes per boot — which is where snapshots, node pools and a control plane come in.
• Getting ground truth, because the single least trustworthy signal in the entire system is the agent’s own report that it succeeded.

The two repositories

The work splits cleanly, and so does the series.

OpenOperator — started 22 February 2025. The agent, the perception stack, the control servers that live inside the guest, and the local Docker Compose setup that runs the whole thing on one machine. This is the “can it work at all” repo.

OpenOperator-Infra — started 2 April 2025, six weeks later, on the day it became obvious that one VM on a laptop does not answer any question worth asking. Terraform for AKS, Helm charts for everything, a FastAPI/React control plane, and a benchmarking system that runs N tests × M repeats and reports a pass rate instead of a pass.

The six-week gap between those two first commits is the story of the series in miniature. You build the agent, you get it working once, you feel great, and then you try to run it twice and discover you have built a science experiment rather than a system.

Why a pass rate, not a pass

The commit that changed how I think about agent testing is four words long:

let's do 5 times

Agents are stochastic. The same instruction against the same starting state produces a different trajectory every run — sometimes a different outcome. A single green run tells you almost nothing, and a single red run tells you only slightly more. What you need is a rate, and the moment you need a rate you need to run the same scenario five, ten, twenty times.

Which means you need twenty desktops. Which means you cannot spend forty minutes installing Windows. Which means golden images, which means snapshots, which means a storage layer, which means a control plane to orchestrate it and a database to record what happened.

One four-word commit, one entire second repository.

What the series covers

Roughly in the order the work happened:

1. The computer — QEMU in Docker, unattended Windows installs, and the bootstrap chain that gets seven Python servers running inside a guest you cannot SSH into.
2. The perception — set-of-mark parsing as a service, and a fairly brutal bake-off of every vision model we could get our hands on.
3. The agent — four generations in four weeks, from an AutoGen prototype to a hand-rolled plan/execute/replan graph, and why most of a computer-use agent is not acting but checking whether the last action worked.
4. The measurement — an evaluator that runs inside the guest, scenarios as config rather than code, and cost-per-step accounting.
5. The platform — nested virtualisation on AKS, VolumeSnapshots as golden images, shipping a 6 GB ISO to every pod, remote desktop for N ephemeral VMs, and a control plane that streams helm upgrade output to a browser.
6. The weird one — decoding Microsoft Bond binary telemetry through mitmproxy to get ground truth out of an application that has no test hooks.

Then a retrospective, because five months in, the most valuable output was not the agent. It was the list of things I now know are hard.

Next up: the container that is not really a container — how you run a genuine Windows 11 VM as a Docker service, and the three lines of compose.yml that make the difference between an eight-minute install and a forty-minute one.