halkyon/deploy/forward/README.md

ramiel's forward for halkyon.li

ramiel, the halkyon.app server, publishes halkyon.li, which runs at home (ADR 0039). HAProxy takes ramiel's 443, reads the name each TLS connection asks for, and passes halkyon.li's, unopened, through a WireGuard tunnel to the instance at home. Every other name goes to ramiel's own Caddy, which serves code.halkyon.app, halkyon.app and learn.halkyon.app as before, now on 127.0.0.1:8443. Port 80 stays Caddy's.

File	On ramiel	What for
haproxy.cfg	/etc/haproxy/haproxy.cfg	the front on 443 (D3, D10)
Caddyfile.halkyon-li	merged into /etc/caddy/Caddyfile	Caddy on 8443 with the PROXY header, halkyon.li's HTTP to HTTPS (D11)
wg-halkyon.conf	/etc/wireguard/wg-halkyon.conf	ramiel's end of the tunnel (D8)
wg-halkyon.nft	/etc/wireguard/wg-halkyon.nft	what the tunnel may carry (D9, D17)

test/deploy/halkyon_li_forward_test.dart holds these files to ADR 0039's rules.

Steps

All as root on ramiel (sudo), but step 1 on melchior. The home end is set up alongside (../host/README.md): each side needs the other's public key.

1. The files, from a checkout on melchior, into the admin's home on ramiel:

   scp deploy/forward/haproxy.cfg deploy/forward/Caddyfile.halkyon-li \
     deploy/forward/wg-halkyon.conf deploy/forward/wg-halkyon.nft ramiel:
   

2. The packages:

   sudo apt install wireguard-tools haproxy nftables
   

   Ubuntu's HAProxy starts with a configuration that listens nowhere, so nothing changes yet.

3. ramiel's key for the tunnel, made on ramiel and never leaving it but into Proton Pass (vault Infra, an item ramiel wg-halkyon: a new server needs it again):

   sudo install -d -m 700 /etc/wireguard
   sudo sh -c 'umask 077; wg genkey > /etc/wireguard/wg-halkyon.key'
   sudo cat /etc/wireguard/wg-halkyon.key       # into Proton Pass
   sudo sh -c 'wg pubkey < /etc/wireguard/wg-halkyon.key'   # ramiel's public key: home's config
   

4. The tunnel's files, with home's public key (from ../host/README.md, step 5) in place of <home's public key>:

   sudo install -m 600 -o root -g root ~/wg-halkyon.conf ~/wg-halkyon.nft /etc/wireguard/
   sudoedit /etc/wireguard/wg-halkyon.conf
   sudo nft -c -f /etc/wireguard/wg-halkyon.nft   # checks the rules, loads nothing
   

5. WireGuard's port, opened in ufw:

   sudo ufw allow 51820/udp comment 'WireGuard: halkyon.li at home'
   

6. The tunnel, up now and at every boot:

   sudo systemctl enable --now wg-quick@wg-halkyon
   sudo wg show wg-halkyon    # a handshake once home is up too
   sudo nft list table inet halkyon_tunnel
   

7. Caddy moved behind HAProxy. Keep the Caddyfile as it is, then put Caddyfile.halkyon-li's global options at the very top of /etc/caddy/Caddyfile (before code.halkyon.app {) and its sites at the end, and check it; 8443 must stay closed from outside (ufw's default for incoming: deny):

   sudo ufw status verbose | grep -E 'Default|8443'   # deny (incoming), no 8443 rule
   sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.before-haproxy
   sudoedit /etc/caddy/Caddyfile
   sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
   sudo install -m 644 ~/haproxy.cfg /etc/haproxy/haproxy.cfg
   sudo haproxy -c -f /etc/haproxy/haproxy.cfg
   

8. The switch, the forge away for the seconds between the two commands: Caddy lets go of 443, HAProxy takes it.

   sudo systemctl reload caddy && sudo systemctl restart haproxy
   sudo ss -tlnp | grep -E ':(443|8443) '   # haproxy on 443, caddy on 8443
   curl -sI https://code.halkyon.app | head -1   # from melchior: HTTP/2 200
   curl -sI http://code.halkyon.app | grep -i '^location'   # https://code.halkyon.app/, no :8443
   

   If anything fails, step 8 is undone by sudo systemctl stop haproxy, the old Caddyfile copied back and sudo systemctl reload caddy.

9. Checks once home's Caddy runs (../host/README.md):

   curl -sI https://learn.halkyon.li | head -1   # from anywhere: an answer from home
   curl -sI http://learn.halkyon.li | head -3    # 308 to https://, from ramiel
   

Taking it away

sudo systemctl disable --now wg-quick@wg-halkyon haproxy
sudo ufw delete allow 51820/udp
sudo cp /etc/caddy/Caddyfile.before-haproxy /etc/caddy/Caddyfile
sudo systemctl reload caddy

Caddy then holds 443 and serves HTTP/3 again, as before.

What changes for the forge

• Caddy serves no HTTP/3 any more (protocols h1 h2). Browsers and git use HTTP/2, which they fall back to on their own.
• Its log keeps visitors' addresses, from HAProxy's PROXY header, which Caddy trusts from the loopback alone.
• HAProxy logs nothing.