Five months, two repos, one lesson: the model was the easy part
A retrospective on OpenOperator. What I would keep, what I would redo, and why a commit called "remove hardcoded API keys" landing a year late is the most honest thing in the history.
The last substantive commit in the OpenOperator repositories is called remove hardcoded API keys from env files and compose.
The first commit was in February 2025. That cleanup landed thirteen months later.
I want to open the retrospective there, because it is the most honest single fact about the project. Not the architecture diagrams, not the benchmark tables — a security cleanup that took over a year to get to, on a codebase that was doing real work the whole time.
The ratio, revisited
The first post claimed roughly 20% agent logic, 80% everything else. Having written the series, I would revise it: the 80% was not “everything else.” It was four specific problems, and every one of them took longer than the agent.
Making a real OS disposable. QEMU in a container, then in a Kubernetes pod, then cloned from a snapshot. Roughly two months of elapsed time, and the highest-leverage single change in the project was in there — forty minutes to ninety seconds.
Making a guest provisionable without SSH. Three hundred lines of batch, four rewrites, and one Task Scheduler flag that decided whether screenshots were black.
Making success measurable. An evaluator with privileged access, and a binary protocol decoder to get ground truth out of an application with no test hooks.
Making it run more than once. Node pools, queues, cost
accounting — the entire second
repository, which exists because of a four-word commit that said let's do 5 times.
The agent itself — four generations — was about four weeks, and by the end it was a few hundred lines of ordinary Python with no framework in it.
What I would keep, unchanged
Set-of-mark parsing as a service. Giving the model a numbered list of elements instead of a coordinate space was worth more than any model upgrade, and it kept working as models changed. The parser being a separate server is what made it cheap to run agents anywhere.
Deterministic evaluators with privileged access to the environment. Put the oracle where the truth is. Pixels are appearances; the filesystem and the app’s own telemetry are facts.
Snapshots as golden images. Not just for speed. For identity — every clone is byte-identical, which removes a source of variance you cannot afford when the thing you are measuring is already stochastic.
Scenarios as config. JSON with lifecycle hooks, a function registry for anything that needs logic, and the discipline that a config format which resists becoming a language is a feature.
Pass rate, not pass. The single most important epistemic change. One green run is a sample, not a result.
Hard bounds on every loop, in config, per scenario.
Price annotations next to the Terraform variable. Ten characters, and it turns an architecture conversation into an architecture-and-cost conversation.
What I would redo
Security first, not last. The credential arc — public container, then a connection string in an env var, then workload identity, then network access closed, then finally the key cleanup thirteen months later — cost more in total than doing it properly on day one. Not because the final design is complicated (a managed identity and a role assignment is a small amount of Terraform) but because every intermediate stage left artifacts. Connection strings in git history. Default gateway credentials in integration code. A CORS rule allowing a Vite dev server, in production, for months.
The mistake was not the sequencing decision, which was reasonable. It was believing that “we will tidy it up later” describes a scheduled activity rather than a hope.
Versioned artifacts, everywhere. Server code reached production by a human
running an upload script onto a file share. The API git-cloned its own chart repo
at deploy time and used whatever was on main. Both worked; neither can answer
“which version did this result come from,” which is the question a benchmarking
platform exists to answer. Container images with tags, charts in an OCI registry
pulled by digest, versions recorded on the run.
A controller, not a shell-out. The API shelling out to
helm with a PAT in its environment was fast
to build and structurally wrong. A Computer CRD and a reconciler would have
given us idempotency, retries, status and drift correction for free — all of
which we implemented badly by hand.
Four agent generations was two too many. In fairness, generations one and two falsified something specific and cheaply. Three and four were mostly churn, and I could have got there by writing down what the state object needed to hold and building for that directly.
Design for the cardinality you will have. One proxy pod per VM was correct at one machine and a liability at twenty, with no warning in between. The question I did not ask early enough: what does this look like at ten times the instances? If the answer is “ten times the objects,” redesign now.
What is still unsolved
Guest state drift. A golden image is a photograph. The software in it ages, certificates expire, the vendor ships a UI redesign. You need a rebuild cadence and you need every result to record which image it ran on — otherwise “the agent got worse in June” is unattributable between the agent, the model, and a frozen application.
Long-horizon cost. Short tasks are affordable. Tasks needing forty steps with a screenshot at each are not, at least not at the frequency real testing wants. Image compression, cheap models for the many small judgements, and aggressive bounds all help. None of them change the shape of the curve.
The last mile of automation. Interactive sign-in with MFA is not automatable by design. Snapshots amortise it to once per image, which is the right answer, and it means a human is still in the loop and the image contains credentials.
“Did it do the right thing” is still partly human. Telemetry markers verify that a sequence of events occurred. They do not verify that the summary the agent wrote was any good. For anything with a qualitative output, the honest state of the art is still a person reading it — or an LLM judge, which is a different set of problems in a similar-shaped box.
If you are starting this today
Much has improved since early 2025. Frontier models ground clicks better. Computer-use APIs exist. MCP is mature rather than a curiosity. If you are building on this now, most of the agent post is a historical document.
Almost none of the rest is.
The hypervisor still needs /dev/kvm. Windows still has no cloud-init. Task
Scheduler still needs /IT or your screenshots are black. The autoscaler still
wants to evict your VM. Snapshots are still the difference between a test suite
and a weekend. And your agent will still tell you, with total sincerity, that it
completed a task it did not complete.
That is the durable part. The model layer moves under you every few months, mostly in your favour. The desktop underneath does not move at all.
Build the boring parts well. They are the parts you keep.
This is the last post in a series about OpenOperator. It starts here.