9 minOpenOperatorwindowsdockerinfraautomation

The bootstrap from hell: provisioning a Windows guest you cannot SSH into

No SSH, no cloud-init, no config management. Just a .bat file, an SMB share the hypervisor provides, and one Task Scheduler flag that decides whether your screenshots are black.

Last time we got a real Windows 11 VM booting as a Docker service. Stock Windows. No Python, no automation, and no way for anything outside the machine to make it click a button.

Closing that gap took two months and produced the most miserable file in the repository — 300 lines of Windows batch script that I rewrote four times. It is worth writing up in detail, because almost every problem in it is a problem you will hit too, and almost none of them are documented anywhere useful.

The constraint

On Linux you would use cloud-init, or Ansible over SSH, or just bake a better image. On Windows-in-QEMU you have:

  • No SSH. Not by default, and enabling it requires… running a command on the machine, which is the problem you are trying to solve.
  • No cloud-init. There is cloudbase-init, which is real, and which we did not use because our entry point was already the unattended answer file.
  • No package manager you can rely on. winget exists. It is not on PATH in a fresh automated install. More on that below.
  • One hook. The FirstLogonCommands block in autounattend.xml, which runs a command the first time a user logs in.

So: one command, one shot, and everything has to descend from it.

The chain

autounattend.xml
  └─ FirstLogonCommands
      └─ C:\OEM\install.bat            ← 300 lines of batch, the misery
          ├─ robocopy \\host.lan\Data → C:\Data
          ├─ install Python 3.10 (silent, system-wide)
          ├─ python C:\Data\init\main.py   ← the sane part, in Python
          │    ├─ install software from software.json
          │    ├─ install winget packages
          │    ├─ verify node + npm, install Appium
          │    ├─ enable Developer Mode
          │    ├─ install Playwright Chromium
          │    └─ configure Teams
          ├─ pip install -r for 7 control servers
          ├─ 8 × netsh firewall rules
          ├─ generate 7 × start_*.bat launchers
          └─ 8 × schtasks /Create /SC ONLOGON

The design goal was to make install.bat as short as possible and get into Python as fast as possible — Windows batch is a language where string handling and error handling are both approximately theoretical. It is still 300 lines, because a surprising amount has to happen before you have an interpreter.

Problem 1: getting files in

The guest needs our code. It is not in the image (it changes daily) and there is no network path to the host by default.

QEMU’s wrapper solves this by running Samba and exposing the container’s /data directory to the guest as \\host.lan\Data. So:

set "SOURCE=\\host.lan\Data"
set "DEST=C:\Data"

robocopy "%SOURCE%" "%DEST%" /MIR /Z /NP /NFL /NDL /NJH /NJS /R:3 /W:5 >> "%LOGFILE%" 2>&1

if %ERRORLEVEL% GEQ 8 (
    echo File copy failed with error code %ERRORLEVEL%. >> "%LOGFILE%"
    exit /b %ERRORLEVEL%
)

Two things here that cost me time.

robocopy exit codes are not exit codes. Success is not 0. Robocopy returns a bitmask: 0 means nothing copied, 1 means files copied, 2 means extra files present, 3 means both, and so on. Anything ≥ 8 is an actual error. If you write the obvious if %ERRORLEVEL% neq 0 you will treat every successful copy as a failure. This is documented, and it is documented in a place nobody reads until after they have been bitten.

/MIR mirrors, which means it deletes. It makes the destination identical to the source, including removing anything in the destination that is not in the source. That is exactly what we want for idempotency — re-running gives the same state — and it is a loaded gun pointed at any directory you also write into. We learned to keep logs at C:\Logs, deliberately outside C:\Data:

REM Setup log directory (outside C:\Data to prevent robocopy overwrite)
set "LOGDIR=C:\Logs"

That comment is a scar.

Problem 2: /IT, or why your screenshots are black

This is the one. If you take nothing else from this post, take this.

The control servers must run automatically when the machine boots. The natural tool is Task Scheduler:

schtasks /Create /TN "StartServer-ComputerControl" /SC ONLOGON ^
    /TR "\"%STARTUP_SERVER_COMPUTER_CONTROL_BAT%\"" ^
    /RU "%USERNAME%" /RL HIGHEST /IT /F

Four flags, all load-bearing:

Flag Meaning What breaks without it
/SC ONLOGON run at user logon starts before a desktop exists
/RU %USERNAME% run as the desktop user wrong session, no user profile
/RL HIGHEST elevated cannot bind ports, cannot touch the registry
/IT interactive screenshots come back black

/IT — “interactive token” — is the killer. Without it the task runs in a non-interactive session. The process starts. It logs happily. It responds to health checks. And every screenshot it captures is a black rectangle, because it is not attached to a session with a rendered desktop, so there is no framebuffer to read.

The failure is silent, the service looks healthy, and the symptom appears three layers away in the agent — which concludes that the screen is blank and starts planning around it. I lost a day and a half to this. The same flag governs video recording, which was the original symptom in a commit charmingly titled “fix ffmpeg issues, fix scheduled tasks permissions.” Two bugs, one root cause.

Problem 3: the console window that got clicked

We generate a small .bat launcher per server:

start /b "ComputerControlServer" "C:\Program Files\Python310\pythonw.exe" "C:\Data\server_computer_control\server.py"

pythonw.exe, not python.exe. python.exe is a console application: launch it and Windows opens a console window. Seven servers, seven console windows, stacked on the desktop.

Which the agent then sees. It screenshots the desktop, the parser dutifully detects seven black rectangles as UI elements, and the planner starts reasoning about them. On one memorable run it clicked one, which put a console into mark mode, which paused the process, which broke that server.

pythonw.exe is the windowless variant. The start /b also matters — it starts without creating a new window and without blocking the parent script.

Problem 4: winget is not on PATH

winget is genuinely good and it is not where you expect it in an automated install. It lives under the user profile:

setx PATH "%PATH%;%LOCALAPPDATA%\Microsoft\WinGet\Links" /M >nul

%LOCALAPPDATA%\Microsoft\WinGet\Links holds the shims. In an interactive session Windows sets this up for you. In a FirstLogonCommands context it may not have happened yet, so winget resolves to nothing and every install silently does not happen.

Note /M — machine-wide. And note that setx writes the persisted environment; it does not change the environment of the currently running process. If you need it in the same script, set it twice.

Software itself is declarative, at least, which is one of the few pleasant things in this stack:

{
  "Microsoft Teams": {
    "mirrors": ["https://aka.ms/teams64bitmsi"],
    "alias": "teams"
  }
}

Multiple mirrors per package, because vendor download URLs rot and a provisioning run that dies on a 404 at minute six is a bad afternoon.

Problem 5: Windows Firewall silently drops your ports

The control servers listen on 5050–5056, 4723, 9222. Windows Firewall blocks inbound connections to them by default, with no log anywhere the caller can see. From outside, the port simply does not answer.

netsh advfirewall firewall add rule name="SERVER_COMPUTER_CONTROL"     dir=in action=allow protocol=TCP localport=5050
netsh advfirewall firewall add rule name="MCP_SERVER_COMPUTER_CONTROL" dir=in action=allow protocol=TCP localport=5055
netsh advfirewall firewall add rule name="SERVER_BROWSER_CONTROL"      dir=in action=allow protocol=TCP localport=5051
netsh advfirewall firewall add rule name="SERVER_NETWORK_PROXY"        dir=in action=allow protocol=TCP localport=5052
netsh advfirewall firewall add rule name="SERVER_EVALUATOR"            dir=in action=allow protocol=TCP localport=5053
netsh advfirewall firewall add rule name="SERVER_TEAMS_CONTROL"        dir=in action=allow protocol=TCP localport=5056
netsh advfirewall firewall add rule name="Chrome Remote Debugging Port" dir=in action=allow protocol=TCP localport=9222
netsh advfirewall firewall add rule name="Appium"                       dir=in action=allow protocol=TCP localport=4723

Tedious, unavoidable, and the first thing to check when a server is “running” but unreachable.

Problem 6: the certificate store that has two of everything

To intercept the guest’s own HTTPS traffic — which we do, for reasons that get their own post — the mitmproxy CA has to be trusted. Windows has per-user and per-machine certificate stores, and installing into the wrong one produces a certificate that is trusted by nothing that matters:

certutil -addstore "Root" "$env:USERPROFILE/.mitmproxy/mitmproxy-ca-cert.cer"

Root is the machine-wide Trusted Root Certification Authorities store, and it needs elevation. The user store looks identical in the UI and is ignored by services and by anything running as another user.

Observability, because you cannot attach a debugger

Everything logs to C:\Logs, and each server writes a startup record. There is also a small PowerShell script that runs on logon and writes the timing data you need to debug ordering problems:

$bootTime = (Get-CimInstance -ClassName Win32_OperatingSystem).LastBootUpTime

$logonEvent = Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4624} -MaxEvents 50 |
    Where-Object {
        $_.Properties[5].Value -eq $currentUser -and
        ($_.Properties[8].Value -eq "2" -or $_.Properties[8].Value -eq "10")
    } | Sort-Object TimeCreated -Descending | Select-Object -First 1

Boot time, logon time, and when each task fired. Security event 4624 is “an account was successfully logged on”; logon type 2 is interactive and 10 is remote interactive. With those three timestamps you can finally answer “did my server start before the network share was mounted?” — which was the cause of an entire class of intermittent failures.

We also enable Task Scheduler’s operational log, which is off by default:

wevtutil set-log "Microsoft-Windows-TaskScheduler/Operational" /enabled:true

Without it, a task that fails to start leaves no trace at all.

The part that is still manual

The README has a post-installation checklist, and I want to be straight about it:

  • Show hidden files in Explorer.
  • Verify the mitmproxy cert actually landed in the machine Root store.
  • Verify Teams picked up its configuration.json.
  • Log in to Teams, and dismiss the “Last Step” dialog.
  • Set Notepad to always open in a new window.

Most of these were automatable and some were automated later. One was not: signing in to a Microsoft account. Interactive OAuth with MFA is, by design, resistant to automation. That is the point of it.

So the honest architecture is: automate to the last possible step, then take a snapshot of the disk after a human has signed in once, and clone every subsequent machine from that snapshot. The human step happens once per golden image rather than once per test run.

Which is a neat segue, because that snapshot turned out to be worth 40 minutes per machine — but first, the agent that was going to drive all this.