Four agents in four weeks: from AutoGen to a hand-rolled plan-and-solve loop

There is a directory in the repository called old_agents/. It contains six abandoned agents:

old_agents/
├── agent-waa/       # the starting point
├── agent-waa-oo/    # ours, forked from it
├── agent_oogen1/    # AutoGen
├── agent_oogen2/    # AutoGen, more structure
├── agent_oogen3/    # AutoGen, merged with 4
├── agent_oogen4/    # the one that became agent_oo4

Four weeks, six architectures. I am not proud of the throughput, but I would do it again — each one was cheap and each one falsified something specific.

Generation 1: AutoGen

The first real agent was built on AutoGen. It got to a demo in about two days, which is exactly what a multi-agent framework is for.

Then it started to fight us. The commit that best captures the moment is real, and I have left it in the history:

not the cleanest, but we are going to need to add some dummy states
to the function args for now

Here is what happened. In a conversational agent framework, tools are functions with typed arguments and the framework marshals a conversation into calls. That is a fine model when your tools are get_weather(city). It is a poor model when every tool needs access to a large mutable object — the current screenshot, the parsed element list, the plan, the iteration counter, the tracker — that is not conceptually an argument at all.

We ended up threading state through function signatures as dummy parameters that the LLM was never supposed to fill in, then stripping them out again. That is the smell of a framework whose abstraction does not fit your domain. Followed by:

adding state to the other agents so they can run

At which point you are maintaining a state-passing convention and a framework that does not know about it.

Generations 2–4: converging on a graph

The oogen2/oogen3/oogen4 sequence was progressive de-frameworking. By the end there was no framework — just classes, an explicit state object, and a loop. The shape it converged on is Plan-and-Solve (arXiv.04091) with dynamic replanning:

                 ┌──────────────┐
      goal ─────▶│   Planner    │◀────────────┐
                 └──────┬───────┘             │
                        │ plan v1..vN         │
                        ▼                     │
                 ┌──────────────┐             │
                 │  Executor    │             │
                 │  (agent_me)  │             │
                 └──────┬───────┘             │
                        │ step failed         │
                        ▼                     │
                 ┌──────────────┐             │
                 │  Replanner   │─────────────┘
                 └──────────────┘

The planner turns a goal into ordered steps. The executor tries to complete one step. If it cannot, the replanner rewrites the remaining plan given what was learned, and control returns to execution. Plans are versioned, so you can look at a failed run and see the agent’s model of the task evolving.

The surprise: most of the agent is not acting

Here is the executor’s node list, which is the most useful artifact in the whole architecture:

agents/agent_oo4/workflow/agent_me/
├── node_get_step.py                  # what am I doing?
├── node_is_step_done.py              # is it already done?
├── agent_computer/                   # ← the only thing that ACTS
├── node_summarize_actions.py         # what did I just do?
├── node_validate_plan_step.py        # did it achieve the step?
├── node_is_step_done_by_validation.py# ...really?
└── node_summarize_plan_step.py       # record it for the plan

One of seven nodes performs an action. The other six are about knowing where you are and whether what you did worked.

That ratio was not designed; it accreted, one painful failure at a time. Every node past the first two exists because a run went wrong in a way that node now catches. And it is the deepest thing I learned on this project: in a non-deterministic environment you do not control, state estimation dominates action selection. Picking the click is easy. Knowing what happened after the click is the hard part.

Before and after, every iteration

The core loop captures the screen twice per iteration — t1 before the action, t2 after — and both go into validation:

while planStepValidation == False and planStepIteration < self.config.workflow.params.max_plan_step_iterations:
    planStepIteration += 1

    screenshot_t1 = self.computer.get_screenshot()
    screenshot_t1_resized = resize_and_compress_image(screenshot_t1)
    ...
    all_messages = await self.agentComputer.run()   # ← act

    screenshot_t2 = self.computer.get_screenshot()
    ...

A single frame cannot answer “did my click do anything?” — a menu that is opening and a menu that is closing look identical in a still. A pair can. This is the cheap approximation of the video experiment that we ended up preferring: two frames and a targeted question, rather than a clip and a general one.

The screenshots are resized and compressed before they go to the model. A 1080p PNG is a lot of tokens, and you send several per iteration, and iterations run for minutes. Image compression is a direct line item on your bill.

Prompting the checker to be a pessimist

node_is_step_done is one of the shortest files and one of the most important. Its system prompt:

- Determine if the user has already **completed** the step.
- Do not assume the step is done just because the UI element is visible.
- Only mark the step as done if there is clear **evidence** that the action
  (e.g. click, selection, navigation) was performed.
- Be conservative. If there is uncertainty, assume the step is NOT completed.

Output format:
RESULT: <TRUE or FALSE>
REASON: <a short explanation of why you made this decision, based only on visible evidence>

Three deliberate choices in there:

“Do not assume the step is done just because the UI element is visible.” The single most common false positive. The step is “click the Chat button.” The Chat button is visible. A helpful model concludes the step is complete. It is not — it is possible.

“Be conservative.” The asymmetry again: a false “done” skips work and corrupts everything downstream. A false “not done” costs one wasted iteration, bounded by max_plan_step_iterations. Bias toward the recoverable error.

A structured RESULT: / REASON: format. RESULT drives control flow; REASON goes into logs and into the replanner’s context. Forcing an explanation also measurably improves the decision, and it gives you something to read when you are trying to work out why a run went sideways.

Note the model choice, too:

self.llm = LLMClient("azure", model="gpt-4o-mini", deployment="gpt-4o-mini-deployment")

A cheap model. This node runs several times per step, many times per plan. Using your most expensive model for “does this screenshot show a completed step” is how a benchmark run costs more than the engineer running it. Reserve the good model for planning; use the cheap one for the many small judgements.

Circuit breakers, or: how not to get a surprise invoice

Every scenario carries three limits, and they are the difference between an agent and an unbounded loop with a credit card:

"workflow": {
  "params": {
    "max_plan_versions": 20,
    "max_plan_step_iterations": 3,
    "max_plan_step_actions": 5
  }
}

• max_plan_versions — how many times the replanner may rewrite the plan. Caps the outer loop, which is where a genuinely stuck agent will spin forever, producing subtly different plans for a task that is impossible.
• max_plan_step_iterations — retries per step. Bounds “click, check, click again” against an element that will never respond.
• max_plan_step_actions — actions within one iteration. Bounds a model that decides the correct move is forty keystrokes.

Three nested loops, three independent bounds. All in config, per scenario, because the right budget for “open Notepad and type a word” is not the right budget for “summarise a chat history you have to scroll through.”

Human-in-the-loop, as a stepping stone

Before there was any automated evaluation, agent-oogen grew a human-in-the-loop mode: the agent pauses, shows you the plan or the proposed action, and waits for approval.

It is not a production feature. It is a development feature, and an excellent one. Standing at the approval prompt for an afternoon is how you build intuition about where your agent’s judgement is bad — and it turned out to be bad in specific, patterned ways that then became nodes in the graph.

We turned it off once the evaluator could answer the same question without a human. But it was the fastest way to learn.

What I would keep

If I started again tomorrow:

1. Skip the framework for this problem shape. Plan/execute/replan is a few hundred lines of ordinary Python and you will want to modify all of them.
2. Make state an explicit object from the first commit. Not conversation history — a real State with the plan, the current step, the screenshots, the iteration counters, persisted so you can inspect a run afterwards.
3. Budget for validation. Assume most of your nodes will be about knowing where you are, and design the state object so it is cheap to ask.
4. Bound every loop and put the bounds in config.
5. Tier your models. Expensive for planning, cheap for the hundred small yes/no calls.

Next: the scenario format — because a task that only exists as a Python function is a task nobody but you can write.