halkyon/deploy/forward/README.md

# ramiel's forward for halkyon.li

ramiel, the halkyon.app server, publishes halkyon.li, which runs at home (ADR 0039).
HAProxy takes ramiel's 443, reads the name each TLS connection asks for, and passes
halkyon.li's, unopened, through a WireGuard tunnel to the instance at home. Every other
name goes to ramiel's own Caddy, which serves `code.halkyon.app`, `halkyon.app` and
`learn.halkyon.app` as before, now on `127.0.0.1:8443`. Port 80 stays Caddy's.

| File | On ramiel | What for |
|---|---|---|
| `haproxy.cfg` | `/etc/haproxy/haproxy.cfg` | the front on 443 (D3, D10) |
| `Caddyfile.halkyon-li` | merged into `/etc/caddy/Caddyfile` | Caddy on 8443 with the PROXY header, halkyon.li's HTTP to HTTPS (D11) |
| `wg-halkyon.conf` | `/etc/wireguard/wg-halkyon.conf` | ramiel's end of the tunnel (D8) |
| `wg-halkyon.nft` | `/etc/wireguard/wg-halkyon.nft` | what the tunnel may carry (D9, D17) |

`test/deploy/halkyon_li_forward_test.dart` holds these files to ADR 0039's rules.

## Steps

All as root on ramiel (`sudo`), but step 1 on melchior. The home end is set up
alongside (`../host/README.md`): each side needs the other's public key.

1. **The files**, from a checkout on melchior, into the admin's home on ramiel:

   ```sh
   scp deploy/forward/haproxy.cfg deploy/forward/Caddyfile.halkyon-li \
     deploy/forward/wg-halkyon.conf deploy/forward/wg-halkyon.nft ramiel:
   ```

2. **The packages**:

   ```sh
   sudo apt install wireguard-tools haproxy nftables
   ```

   Ubuntu's HAProxy starts with a configuration that listens nowhere, so nothing changes
   yet.

3. **ramiel's key** for the tunnel, made on ramiel and never leaving it but into Proton
   Pass (vault *Infra*, an item *ramiel wg-halkyon*: a new server needs it again):

   ```sh
   sudo install -d -m 700 /etc/wireguard
   sudo sh -c 'umask 077; wg genkey > /etc/wireguard/wg-halkyon.key'
   sudo cat /etc/wireguard/wg-halkyon.key       # into Proton Pass
   sudo sh -c 'wg pubkey < /etc/wireguard/wg-halkyon.key'   # ramiel's public key: home's config
   ```

4. **The tunnel's files**, with home's public key (from `../host/README.md`, step 5) in
   place of `<home's public key>`:

   ```sh
   sudo install -m 600 -o root -g root ~/wg-halkyon.conf ~/wg-halkyon.nft /etc/wireguard/
   sudoedit /etc/wireguard/wg-halkyon.conf
   sudo nft -c -f /etc/wireguard/wg-halkyon.nft   # checks the rules, loads nothing
   ```

5. **WireGuard's port**, opened in ufw:

   ```sh
   sudo ufw allow 51820/udp comment 'WireGuard: halkyon.li at home'
   ```

6. **The tunnel**, up now and at every boot:

   ```sh
   sudo systemctl enable --now wg-quick@wg-halkyon
   sudo wg show wg-halkyon    # a handshake once home is up too
   sudo nft list table inet halkyon_tunnel
   ```

7. **Caddy moved behind HAProxy.** Keep the Caddyfile as it is, then put
   `Caddyfile.halkyon-li`'s global options at the very top of `/etc/caddy/Caddyfile`
   (before `code.halkyon.app {`) and its sites at the end, and check it; 8443 must stay
   closed from outside (ufw's default for incoming: deny):

   ```sh
   sudo ufw status verbose | grep -E 'Default|8443'   # deny (incoming), no 8443 rule
   sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.before-haproxy
   sudoedit /etc/caddy/Caddyfile
   sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
   sudo install -m 644 ~/haproxy.cfg /etc/haproxy/haproxy.cfg
   sudo haproxy -c -f /etc/haproxy/haproxy.cfg
   ```

8. **The switch**, the forge away for the seconds between the two commands: Caddy lets go
   of 443, HAProxy takes it.

   ```sh
   sudo systemctl reload caddy && sudo systemctl restart haproxy
   sudo ss -tlnp | grep -E ':(443|8443) '   # haproxy on 443, caddy on 8443
   curl -sI https://code.halkyon.app | head -1   # from melchior: HTTP/2 200
   curl -sI http://code.halkyon.app | grep -i '^location'   # https://code.halkyon.app/, no :8443
   ```

   If anything fails, step 8 is undone by `sudo systemctl stop haproxy`, the old
   Caddyfile copied back and `sudo systemctl reload caddy`.

9. **Checks** once home's Caddy runs (`../host/README.md`):

   ```sh
   curl -sI https://learn.halkyon.li | head -1   # from anywhere: an answer from home
   curl -sI http://learn.halkyon.li | head -3    # 308 to https://, from ramiel
   ```

## Taking it away

```sh
sudo systemctl disable --now wg-quick@wg-halkyon haproxy
sudo ufw delete allow 51820/udp
sudo cp /etc/caddy/Caddyfile.before-haproxy /etc/caddy/Caddyfile
sudo systemctl reload caddy
```

Caddy then holds 443 and serves HTTP/3 again, as before.

## What changes for the forge

- Caddy serves no HTTP/3 any more (`protocols h1 h2`). Browsers and git use HTTP/2,
  which they fall back to on their own.
- Its log keeps visitors' addresses, from HAProxy's PROXY header, which Caddy trusts from
  the loopback alone.
- HAProxy logs nothing.