feat(deploy): halkyon.li's forward through ramiel and its host at home (ADR 0039)

ramiel's HAProxy passes halkyon.li's TLS connections, unopened, through a
WireGuard tunnel to the instance at home, with the visitor's address in a
PROXY header, and everything else to ramiel's own Caddy, moved to the
loopback. Each end's nftables table lets the tunnel carry HAProxy's 443 and
the SSH jump's 22 alone, and nothing from home to ramiel.

The host at home: sshd with keys alone, the root disk unlocked over SSH at
boot (dropbear-initramfs, D20), Docker's and containerd's data bind-mounted
from the HDD, and home's end of the tunnel before Docker. An instance behind
the forward (compose.forwarded.yaml) publishes 443 on the tunnel's address
alone, trusts the PROXY header from ramiel alone, and gets its certificates
through TLS-ALPN, without HTTP/3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

deploy/forward/Caddyfile.halkyon-li

@@ -0,0 +1,33 @@
+# What ramiel's Caddy (/etc/caddy/Caddyfile) gains for HAProxy in front of it (ADR 0039,
+# D3, D10, D11). Not a Caddyfile of its own: the global options below go at the very
+# top of ramiel's, before `code.halkyon.app {`, and the site at its end. Setting it up:
+# forward/README.md.
+
+# --- global options: the top of /etc/caddy/Caddyfile ---
+{
+	# HAProxy holds 443 and hands Caddy what isn't halkyon.li's on 8443, which ufw
+	# keeps closed from outside. Port 80 stays Caddy's.
+	https_port 8443
+	servers :8443 {
+		listener_wrappers {
+			# The visitor's address from HAProxy's PROXY header, trusted from the
+			# loopback alone, so the forge's log keeps visitors rather than
+			# 127.0.0.1.
+			proxy_protocol {
+				timeout 5s
+				allow 127.0.0.1/32 ::1/128
+			}
+			tls
+		}
+		# HTTP/3 runs over UDP, which HAProxy doesn't route: none, so no client is
+		# sent to look for it.
+		protocols h1 h2
+	}
+}
+
+# --- a site: the end of /etc/caddy/Caddyfile ---
+# halkyon.li over plain HTTP, sent on to HTTPS, which HAProxy passes home; nothing of
+# halkyon.li is served or certified here.
+http://halkyon.li, http://*.halkyon.li {
+	redir https://{host}{uri} 308
+}

deploy/forward/README.md

@@ -0,0 +1,120 @@
+# ramiel's forward for halkyon.li
+
+ramiel, the halkyon.app server, publishes halkyon.li, which runs at home (ADR 0039).
+HAProxy takes ramiel's 443, reads the name each TLS connection asks for, and passes
+halkyon.li's, unopened, through a WireGuard tunnel to the instance at home. Every other
+name goes to ramiel's own Caddy, which serves `code.halkyon.app`, `halkyon.app` and
+`learn.halkyon.app` as before, now on `127.0.0.1:8443`. Port 80 stays Caddy's.
+
+| File | On ramiel | What for |
+|---|---|---|
+| `haproxy.cfg` | `/etc/haproxy/haproxy.cfg` | the front on 443 (D3, D10) |
+| `Caddyfile.halkyon-li` | merged into `/etc/caddy/Caddyfile` | Caddy on 8443 with the PROXY header, halkyon.li's HTTP to HTTPS (D11) |
+| `wg-halkyon.conf` | `/etc/wireguard/wg-halkyon.conf` | ramiel's end of the tunnel (D8) |
+| `wg-halkyon.nft` | `/etc/wireguard/wg-halkyon.nft` | what the tunnel may carry (D9, D17) |
+
+`test/deploy/halkyon_li_forward_test.dart` holds these files to ADR 0039's rules.
+
+## Steps
+
+All as root on ramiel (`sudo`), but step 1 on melchior. The home end is set up
+alongside (`../host/README.md`): each side needs the other's public key.
+
+1. **The files**, from a checkout on melchior, into the admin's home on ramiel:
+
+   ```sh
+   scp deploy/forward/haproxy.cfg deploy/forward/Caddyfile.halkyon-li \
+     deploy/forward/wg-halkyon.conf deploy/forward/wg-halkyon.nft ramiel:
+   ```
+
+2. **The packages**:
+
+   ```sh
+   sudo apt install wireguard-tools haproxy nftables
+   ```
+
+   Ubuntu's HAProxy starts with a configuration that listens nowhere, so nothing changes
+   yet.
+
+3. **ramiel's key** for the tunnel, made on ramiel and never leaving it but into Proton
+   Pass (vault *Infra*, an item *ramiel wg-halkyon*: a new server needs it again):
+
+   ```sh
+   sudo install -d -m 700 /etc/wireguard
+   sudo sh -c 'umask 077; wg genkey > /etc/wireguard/wg-halkyon.key'
+   sudo cat /etc/wireguard/wg-halkyon.key       # into Proton Pass
+   sudo wg pubkey < /etc/wireguard/wg-halkyon.key   # ramiel's public key: home's config
+   ```
+
+4. **The tunnel's files**, with home's public key (from `../host/README.md`, step 4) in
+   place of `<home's public key>`:
+
+   ```sh
+   sudo install -m 600 -o root -g root ~/wg-halkyon.conf ~/wg-halkyon.nft /etc/wireguard/
+   sudoedit /etc/wireguard/wg-halkyon.conf
+   sudo nft -c -f /etc/wireguard/wg-halkyon.nft   # checks the rules, loads nothing
+   ```
+
+5. **WireGuard's port**, opened in ufw:
+
+   ```sh
+   sudo ufw allow 51820/udp comment 'WireGuard: halkyon.li at home'
+   ```
+
+6. **The tunnel**, up now and at every boot:
+
+   ```sh
+   sudo systemctl enable --now wg-quick@wg-halkyon
+   sudo wg show wg-halkyon    # a handshake once home is up too
+   sudo nft list table inet halkyon_tunnel
+   ```
+
+7. **Caddy moved behind HAProxy.** Keep the Caddyfile as it is, then put
+   `Caddyfile.halkyon-li`'s global options at the very top of `/etc/caddy/Caddyfile`
+   (before `code.halkyon.app {`) and its site at the end, and check it:
+
+   ```sh
+   sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.before-haproxy
+   sudoedit /etc/caddy/Caddyfile
+   sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
+   sudo install -m 644 ~/haproxy.cfg /etc/haproxy/haproxy.cfg
+   sudo haproxy -c -f /etc/haproxy/haproxy.cfg
+   ```
+
+8. **The switch**, the forge away for the seconds between the two commands: Caddy lets go
+   of 443, HAProxy takes it.
+
+   ```sh
+   sudo systemctl reload caddy && sudo systemctl restart haproxy
+   sudo ss -tlnp | grep -E ':(443|8443) '   # haproxy on 443, caddy on 8443
+   curl -sI https://code.halkyon.app | head -1   # from melchior: HTTP/2 200
+   ```
+
+   If anything fails, step 8 is undone by `sudo systemctl stop haproxy`, the old
+   Caddyfile copied back and `sudo systemctl reload caddy`.
+
+9. **Checks** once home's Caddy runs (`../host/README.md`):
+
+   ```sh
+   curl -sI https://learn.halkyon.li | head -1   # from anywhere: an answer from home
+   curl -sI http://learn.halkyon.li | head -3    # 308 to https://, from ramiel
+   ```
+
+## Taking it away
+
+```sh
+sudo systemctl disable --now wg-quick@wg-halkyon haproxy
+sudo ufw delete allow 51820/udp
+sudo cp /etc/caddy/Caddyfile.before-haproxy /etc/caddy/Caddyfile
+sudo systemctl reload caddy
+```
+
+Caddy then holds 443 and serves HTTP/3 again, as before.
+
+## What changes for the forge
+
+- Caddy serves no HTTP/3 any more (`protocols h1 h2`). Browsers and git use HTTP/2,
+  which they fall back to on their own.
+- Its log keeps visitors' addresses, from HAProxy's PROXY header, which Caddy trusts from
+  the loopback alone.
+- HAProxy logs nothing.

deploy/forward/haproxy.cfg

@@ -0,0 +1,45 @@
+# ramiel's front on 443 (ADR 0039, D3, D10): HAProxy reads the name a TLS connection
+# asks for (SNI) and passes halkyon.li's, unopened, through the tunnel to its instance
+# at home; every other name goes to ramiel's own Caddy on the loopback, which serves
+# code.halkyon.app and halkyon.app as before. Both get the visitor's address in a PROXY
+# header (v2). As /etc/haproxy/haproxy.cfg on ramiel; check it with
+# `haproxy -c -f /etc/haproxy/haproxy.cfg`. Setting it up: forward/README.md.
+#
+# Nothing is logged: no `log` line, so neither the visitors of halkyon.li nor those of
+# the forge are kept here (Caddy behind each side logs what it logs, as before).
+
+global
+	user haproxy
+	group haproxy
+	daemon
+	maxconn 2000
+
+defaults
+	mode tcp
+	timeout connect 5s
+	# Jellyfin's streams, Serverpod's WebSockets and a forge's long clones stay open:
+	# an hour without a byte either way, as a tunnel's idle time.
+	timeout client 1h
+	timeout server 1h
+	timeout tunnel 1h
+
+frontend tls
+	bind :::443 v4v6
+	# Wait for the ClientHello, which names the host; a connection that sends none
+	# within 5 s goes to the local Caddy, which answers it as before.
+	tcp-request inspect-delay 5s
+	tcp-request content accept if { req.ssl_hello_type 1 }
+	# halkyon.li's names exactly, as its DNS zone has them (D7): no wildcard, so no
+	# other name reaches home.
+	acl halkyon_li req.ssl_sni -i halkyon.li learn.halkyon.li todo.halkyon.li train.halkyon.li meditate.halkyon.li jellyfin.halkyon.li
+	use_backend home if halkyon_li
+	default_backend local
+
+# halkyon.li's Caddy at home, at its address in the tunnel (D8), which it publishes
+# 443 on alone.
+backend home
+	server balthasar 10.61.97.2:443 send-proxy-v2
+
+# ramiel's own Caddy, moved off 443 (Caddyfile.halkyon-li).
+backend local
+	server caddy 127.0.0.1:8443 send-proxy-v2

deploy/forward/wg-halkyon.conf

@@ -0,0 +1,17 @@
+# ramiel's end of the tunnel to halkyon.li at home (ADR 0039, D8), as
+# /etc/wireguard/wg-halkyon.conf on ramiel (root, 0600), brought up by
+# wg-quick@wg-halkyon. Its private key is a file of its own beside it, never here.
+# Setting it up: forward/README.md.
+
+[Interface]
+Address = 10.61.97.1/30
+ListenPort = 51820
+PostUp = wg set %i private-key /etc/wireguard/wg-halkyon.key
+# HAProxy's 443 and the jump's 22 into the tunnel, nothing from it (D9, D17).
+PostUp = nft -f /etc/wireguard/wg-halkyon.nft
+PreDown = nft delete table inet halkyon_tunnel
+
+# Home, which dials in: no endpoint, so ramiel never names its address.
+[Peer]
+PublicKey = <home's public key>
+AllowedIPs = 10.61.97.2/32

deploy/forward/wg-halkyon.nft

@@ -0,0 +1,39 @@
+# The rules around ramiel's end of the tunnel to halkyon.li (ADR 0039, D9, D17), loaded
+# by wg-quick as the tunnel comes up and deleted as it goes (wg-halkyon.conf), so they
+# live exactly as long as it. As /etc/wireguard/wg-halkyon.nft on ramiel (root, 0600).
+#
+# A table of their own, apart from ufw's: a packet the table drops is dropped whatever
+# ufw says, and what it accepts still meets ufw's rules after it.
+#
+# - Into the tunnel: new connections to home's Caddy (443, from HAProxy) and sshd (22,
+#   the jump from away), then their replies; nothing else.
+# - From the tunnel: replies alone, so a taken-over home reaches neither ramiel's sshd
+#   nor Forgejo nor anything else listening here.
+# - Through the tunnel: nothing; ramiel routes nothing between it and anywhere else.
+#
+# IPv4 alone: the tunnel carries no IPv6 (D8).
+
+# Made and deleted first, so loading this again replaces it whole.
+table inet halkyon_tunnel {}
+delete table inet halkyon_tunnel
+
+table inet halkyon_tunnel {
+	chain input {
+		type filter hook input priority filter - 1; policy accept;
+		iifname "wg-halkyon" ct state established,related accept
+		iifname "wg-halkyon" drop
+	}
+
+	chain output {
+		type filter hook output priority filter - 1; policy accept;
+		oifname "wg-halkyon" ct state established,related accept
+		oifname "wg-halkyon" ip daddr 10.61.97.2 tcp dport { 22, 443 } accept
+		oifname "wg-halkyon" drop
+	}
+
+	chain forward {
+		type filter hook forward priority filter - 1; policy accept;
+		iifname "wg-halkyon" drop
+		oifname "wg-halkyon" drop
+	}
+}