mave@doof:~$

~/notes / console-outage

#postmortem · #dns #reverse-proxy #proxmox · 2026-09-09

Postmortem: when every console said "Connecting…"

Every VM and container console in my Proxmox web UI stopped working at once. The cause wasn't Proxmox at all. It was one DNS record I had added that day.

$ cat summary

I added a local DNS record in Pi-hole for the Proxmox web UI's domain. It pointed the name straight at the Proxmox host instead of at the reverse proxy in front of it. The web console runs over a WebSocket, and that connection never made it back to the server, so every console timed out. Removing the record fixed it.

$ cat impact

  • No web console for any VM, any container or the node shell. Each one sat on "Connecting…" forever.
  • Everything else in the web UI worked: dashboards, settings, starting and stopping guests.
  • The guests themselves were fine, and pct console over SSH still worked. Only access through the browser was broken.

$ cat investigation

The task log had the first real clue. Every console attempt ended the same way after about 10 seconds:

TASK ERROR: command '/usr/bin/termproxy ...' failed: exit code 1

It failed the same way for every guest, so I started with the host. I checked and ruled out:

  • Disk space: under half full.
  • The Proxmox services (pveproxy, pvedaemon, pvestatd): all running, nothing odd in their logs.
  • Clock and NTP: in sync, so not a certificate or ticket timing problem.
  • The termproxy binary itself, and AppArmor: no errors or denials.
  • The local auth API termproxy calls: reachable.
  • A port clash in the VNC range: nothing else was listening there.

With the host clean, the 10 seconds started to matter. termproxy doesn't fail on start. It opens a port and waits for the browser to connect back, then gives up. So the server side was fine and the browser's connection was never arriving. The question became what had changed between my browser and the server, and the answer was the DNS record I'd added earlier that day.

$ cat root-cause

Correct and broken paths for the console WebSocket Normally the domain resolves to the reverse proxy, which forwards to the Proxmox web port, so the console connects. With the bad record, the domain resolved straight to the Proxmox host on the standard HTTPS port, where the WebSocket had nowhere to go, so termproxy timed out. before · working browser opens console DNS → proxy reverse proxy TLS termination web UI port Proxmox termproxy connects after the new record · broken browser opens console DNS → Proxmox host directly, HTTPS port skips the proxy · WebSocket never arrives Proxmox waits 10 s · exit 1
Same domain, two different paths. Only one of them can carry the console.

I reach the Proxmox web UI through a domain that resolves to my reverse proxy. The proxy handles TLS and forwards to the web UI's port. The new Pi-hole record pointed that same domain at the Proxmox host itself. That skipped the proxy, and nothing on the host's standard HTTPS port could take the console's WebSocket.

$ cat fix

I deleted the record, and the consoles worked again right away, both by IP and by domain.

If I want local DNS for it later, the record has to point at the reverse proxy, not the host, and the proxy host needs WebSocket support turned on. Then local clients take the same path everyone else does.

$ cat lessons

  • Start with what changed. I spent most of the time checking a healthy host. The DNS change from that same day should have been my first suspect.
  • Read the timeouts. A clean exit after a fixed 10 seconds meant something was waiting and gave up, not that something crashed. That pointed at the network, not the server.
  • Split DNS has to match the proxy. A local record that skips the reverse proxy changes more than the address. It changes which ports, certificates and protocols are on the path.
lab up — · 9 notes ·