One proxy pod per VM does not scale

Remote desktop for one virtual machine is a solved problem. You bolt websockify in front of VNC, point a browser at it, and you are done in an afternoon.

Remote desktop for N machines that appear and disappear on demand is a different problem wearing the same clothes. It took us two months to notice.

Version 1: a proxy per computer

Each computer got its own translation layer, as Helm templates inside the computer’s own chart:

k8s-Helm/OOUser/computers/<os>/templates/
├── dockur/stateful-set.yaml       # the VM
├── dockur/service.yaml
├── vnc-ws-proxy/deployment.yaml   # ← per-computer proxy
├── vnc-ws-proxy/service.yaml
└── traefik-ingressroute.yaml      # ← per-computer route

Plus an rdp-tls-proxy — HAProxy with a self-signed certificate — for RDP, which needs TLS termination that noVNC’s path does not provide.

The proxy deployment is unremarkable, and that is rather the point:

        - name: '{{ include "proxy.name" . }}-container'
          image: {{ .Values.vnc_ws_proxy.image }}
          env:
            - name: VNC_HOST
              value: {{ .Values.computerName }}    # the computer's Service
            - name: VNC_PORT
              value: "{{ .Values.vnc_ws_proxy.vnc_port }}"
          readinessProbe:
            httpGet: { path: /, port: 8888 }
          livenessProbe:
            httpGet: { path: /, port: 8888 }

Correct Kubernetes. Probes, a Service, an ingress route, deployed with the computer and cleaned up with it. At one computer it is elegant.

Where it broke

Object count. Per computer: a StatefulSet, a headless Service, a PVC, a VNC proxy Deployment, its Service, an RDP proxy Deployment, its Service, and one or two IngressRoutes. Call it nine objects, of which four exist purely to translate a protocol. At twenty computers that is eighty objects of pure overhead and a routing table nobody can hold in their head.

Resource waste. Each proxy is a pod. It needs a request, so the scheduler reserves capacity, so nodes fill up with processes whose entire job is copying bytes between two sockets. Twenty computers meant forty extra pods on a node pool sized for virtual machines.

Hostname sprawl. Every computer needed a routable name. In staging that meant <computer>.<ingress-ip>.nip.io — nip.io resolves anything.1.2.3.4.nip.io to 1.2.3.4, which is a genuinely good trick for wildcard-ish DNS without owning a zone. It also meant the ingress configuration grew linearly with the fleet.

WebSocket routing. The wss upgrade through Traefik has to be right for every route, and there is a commit called fix ingressroutes for wss that represents several hours of learning that a misconfigured upgrade header produces a connection which opens and then produces nothing.

Lifecycle coupling. The proxy has to come up after the computer’s Service exists, be torn down with it, and not leave orphans when something fails midway — which is exactly the class of bug that shows up at 2am with forty Terminating pods.

Version 2: one gateway, dynamic registration

Apache Guacamole is a clientless remote desktop gateway. Two components: guacd, which speaks VNC, RDP and SSH natively and translates to the Guacamole protocol, and a web application that serves the JavaScript client and holds connection configuration in Postgres.

One deployment, for the whole cluster. Computers are registered with it when they are provisioned, over its REST API:

def register_computer_with_guacamole(computer_name: str):
    session = requests.Session()

    auth_res = session.post(f"{guac_url}/api/tokens",
                            data={"username": guac_username, "password": guac_password})
    token = auth_res.json()["authToken"]
    data_source = auth_res.json()["dataSource"]   # usually "postgresql"
    headers = {"Guacamole-Token": token}

    vnc_payload = {
        "name": f"{computer_name}-vnc",
        "protocol": "vnc",
        "parameters": {
            "hostname": f"{computer_name}.computer.svc.cluster.local",
            "port": "5900",
            "password": comp_password,
        },
        "attributes": {"max-connections": "5", "max-connections-per-user": "2"},
    }
    session.post(f"{guac_url}/api/session/data/{data_source}/connections",
                 json=vnc_payload, headers=headers)
    # ...and the same for RDP on 3389

The key line is the hostname:

{computer_name}.computer.svc.cluster.local

Guacamole reaches the computer over cluster-internal DNS. No ingress route, no external hostname, no proxy pod. The computer’s Service already exists; that is sufficient. And guacd is inside the cluster too, so the VNC and RDP traffic never leaves it.

Provisioning gained one line — register_computer_with_guacamole(name) — and teardown gained its mirror. Both are ordinary HTTP calls from the control plane, not Kubernetes objects with their own lifecycle.

The commit

remove VNC and RDP proxies

118 lines deleted, the charts moved to backup/, and per-computer IngressRoutes removed from the computer templates.

I have a lot of affection for that commit. Deleting working infrastructure is harder than it should be — it works, someone built it, and the replacement is a new dependency. But two months of operating the first design produced enough evidence to make the argument, and the fleet got quieter overnight.

Moving the old charts to backup/ rather than deleting them outright is a habit I would defend. Git remembers, but a directory named backup/vnc-proxy sitting next to the current design is a much more discoverable answer to “did you consider per-computer proxies?” than a commit from three months ago. It documents a road not taken, at the cost of a directory.

What the migration cost

Not free. Three things came with it.

CORS. The dashboard is served from one origin, Guacamole from another, and the browser cares. A Traefik middleware handles it:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: guac-cors
spec:
  headers:
    accessControlAllowOriginList:
      - "http://localhost:5173"   # dev UI origin
    accessControlAllowMethods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"]
    accessControlAllowHeaders: ["*"]
    accessControlAllowCredentials: true

Note that origin — the Vite dev server. Convenient during development, and exactly the sort of allowance that must not survive into production. It did survive longer than it should have.

A database. Guacamole needs Postgres for connection state, which is a new stateful component to run, back up and upgrade, in exchange for deleting forty pods. Worth it, but it is a trade, not a pure win.

A more centralised failure mode. Twenty independent proxies fail independently. One gateway fails for everybody. In practice this was fine — it is a mature, boring piece of software — but it is a real change in the shape of your outages and worth naming rather than discovering.

The general lesson

The first design was not stupid. It was a correct solution to the problem as it existed when it was written: one computer, get a picture of it in a browser.

What changed was not the requirement but the cardinality, and cardinality changes are the ones you do not notice. There is no moment where per-computer proxies stop working. They work at two. They work at five. At twenty they are merely expensive and annoying, and by then they are load-bearing.

The question worth asking early, and I did not ask it: what does this design look like at ten times the instance count? If the answer is “ten times as many objects,” you have found the thing to redesign — and it is much cheaper to redesign it before the answer is measured in pods.

Next: running the benchmarks that made all of this necessary — node pools, job queues, and cost per step.