doof-homelab/

$ cat overview

doof started as an old desktop, a small TP-Link router and a switch. The first note in my lab docs says it will be "tiny at first," and it was. Before Proxmox it was a Windows and Hyper-V dual boot. Before that, the data disk belonged to a TrueNAS install I'd abandoned.

I wanted to run my own services instead of renting them, and to secure my own network. Mostly, though, I wanted to know where my data was going. Almost everything below came out of that.

$ cat architecture

Service	Runs as	Why it's there
Pi-hole v6	LXC	DNS for every VLAN, ~760k blocked domains, and a log of what's talking to what
Nginx Proxy Manager	LXC (Docker)	One wildcard certificate; routes each *.doof.quest name to its backend
cloudflared	LXC	Outbound tunnel for SSH when Tailscale is blocked
Tailscale	LXC	Subnet router so my devices reach the LAN from anywhere
CasaOS + Immich	LXC (Docker)	My own Google Photos, with the library on ZFS
doof-watcher	LXC	Prometheus, exporters, Grafana and Discord alerts

$ cat network

The network runs on an Omada stack (router, managed switch, wall AP) split into Admin, Server, Guest, and IoT VLANs. The default VLAN only holds the router.

The ACLs needed a full rebuild. I'd written them assuming the router tracked connections between LAN segments, and it doesn't: LAN-to-LAN rules on the router are stateless, so a one-way permit silently broke return traffic. The rebuilt set has explicit permits in both directions where needed, lets guest and IoT devices reach Pi-hole on port 53 and nothing else, and ends with a blanket guest-to-LAN deny. Pi-hole is targeted with a /32 IP group, not a whole subnet.

I learned one DNS lesson from my own laptop. The router was handing out Pi-hole plus 1.1.1.1 as a fallback, and macOS quietly preferred the fallback, so its queries never touched Pi-hole. DHCP now hands out Pi-hole only.

$ cat internal-https

Every service gets a real address like https://<service>.doof.quest with a valid padlock, on the LAN and away from home, and nothing listens on the internet. Three pieces make that work.

• One Let's Encrypt wildcard certificate for *.doof.quest, validated through a Cloudflare DNS-01 challenge. The API token can edit DNS on that one zone and nothing else.
• Pi-hole answers every *.doof.quest lookup with the proxy's address (split-horizon DNS), so public DNS never holds internal records.
• Tailscale advertises the LAN and uses Pi-hole as the tailnet's resolver, which is why the same addresses work on cellular.

$ cat remote-ssh

Tailscale is blocked on my school's Wi-Fi, and CGNAT rules out port forwarding. So SSH goes through a Cloudflare Tunnel: a small connector container makes an outbound HTTPS connection, and SSH rides back through it on its own hostname.

• A Cloudflare Access app sits in front. It sends a one-time PIN, and the policy allows exactly one email address. I created the Access app before the route existed, so the hostname was never open even for a minute.
• The Proxmox web UI isn't published through the tunnel. Only SSH is.
• The host accepts keys only. When I checked with sshd -T, password login was still on: the line in sshd_config was commented out, which means OpenSSH's default of "yes" applied. A drop-in file fixed it, and I verified the effective config, not just the file.
• At home, Pi-hole's wildcard sent the SSH hostname to the proxy, which can't carry SSH. A more specific rule now sends that one name upstream.

$ cat storage

• The NVMe holds Proxmox, every service container, and the databases that need speed.
• A ZFS pool on a 1 TB HDD has LZ4 compression and one dataset per service. Immich's photo library lives there, while its Postgres database stays on the SSD. I take a snapshot before every upgrade.
• A 2 TB SMR drive is split into an ext4 directory and an LVM-thin pool for VMs I rarely run. SMR drives handle heavy random writes badly, so nothing busy goes there.

Two things confused me for a while. Unprivileged containers map their root user to UID 100000 on the host, so a dataset created by host root looks read-only from inside until you chown it into that range. And ZFS shows about 899 GiB on a 930 GiB disk because it keeps space for metadata and a safety reserve. That's also why my dataset quotas are 445G instead of 450G.

$ cat monitoring

A single container runs Prometheus, which scrapes every 15 seconds: node_exporter for the host's hardware and OS, and pve-exporter for per-guest stats through a read-only Proxmox API user. Grafana sits behind the proxy with HTTPS. Everything else listens on localhost, and the host's exporter port is firewalled to the monitoring container alone. ss -tlnp confirms it.

Alerts go to a private Discord channel:

Alert	Fires when
Target down	any scrape target is down for 2 min
Storage filling	any Proxmox storage is over 85%
Root disk filling	root filesystem is over 85%
CPU too hot	CPU is over 85°C for 5 min
Memory pressure	RAM is over 90% for 10 min
Guest down	a VM or container set to autostart isn't running

I gave the messages a wizard theme ("A disturbance stirs in the realm of doof") because plain alerts are boring, and each one's description doubles as a short runbook. I tested every rule by stopping an exporter and watching it go pending, firing, then resolved. The dashboards already caught real things: one storage pool sitting near 80%, HDD I/O saturation spikes, and guests promised more RAM than the host has.

$ cat incidents

• Three reboots in one day. I suspected an intrusion first. SSH logs, cron, scheduled jobs and sessions were all clean, and so were RAM, SMART and kernel errors. The logs just stopped mid-write with no shutdown, which points at hardware. The cause was a faulty UPS that caused brownouts instead of covering them. The fixes also turned up containers without autostart and a firewall rule that existed but was never enabled.
• Every web console stuck on "Connecting…". A DNS record I'd added skipped the proxy and broke the console's WebSocket. Full postmortem →
• The server wouldn't boot. An HDD mount in fstab had no nofail. I recovered through an init=/bin/bash shell and added a device timeout.

$ cat lessons

• Check the effective state, not the config file: sshd -T, ss -tlnp, promtool check config.
• A firewall rule isn't done until it's enabled and tested.
• Least privilege everywhere: one-zone API tokens, a read-only monitoring user, a one-address access policy.
• systemctl status tells you that something failed. journalctl tells you why.
• I document everything. Each part of doof has its own doc with numbered troubleshooting tickets (symptom, cause, fix).

$ cat todo

• Replace the UPS.
• Ship SSH logs to Loki and alert on them.
• Finish the IoT VLAN rollout, and stand up a shared Proxmox box for friends on its own isolated VLAN.