halkyon/deploy/host/README.md

# The instance's host at home

halkyon.li runs at home on balthasar (ADR 0039) and is published through ramiel
(`../forward/README.md`). This folder holds what the host needs beyond Docker: SSH for
melchior, the root disk unlocked over SSH at boot, Docker's data on the HDD, and home's
end of the tunnel.

| File | On the host | What for |
|---|---|---|
| `sshd-halkyon.conf` | `/etc/ssh/sshd_config.d/10-halkyon.conf` | keys alone, two users (D17) |
| `dropbear.conf` | `/etc/dropbear/initramfs/dropbear.conf` | the passphrase over SSH at boot (D20) |
| `initramfs-modules` | appended to `/etc/initramfs-tools/modules` | the USB network adapter at boot (D20) |
| `fstab-docker` | appended to `/etc/fstab` | Docker's and containerd's data on the HDD (D12) |
| `docker-after-data.conf` | `docker.service.d/` and `containerd.service.d/` | both wait for it (D12) |
| `wg-halkyon.conf` | `/etc/wireguard/wg-halkyon.conf` | home's end of the tunnel (D8) |
| `wg-halkyon.nft` | `/etc/wireguard/wg-halkyon.nft` | what the tunnel may carry (D9, D17) |
| `wg-quick-before-docker.conf` | `wg-quick@wg-halkyon.service.d/` | the tunnel before Docker (D12) |

`test/deploy/halkyon_li_forward_test.dart` holds these files to ADR 0039's rules.

## Steps

As the developer at the iMac (`sudo`), from this checkout (`~/project-halkyon/halkyon`),
in this order: each step leaves the machine working, and the early ones keep a way in
for the later.

1. **sshd for melchior** (D17). On balthasar, then from melchior, then on balthasar:

   ```sh
   sudo apt install openssh-server
   ```

   ```sh
   ssh-copy-id raisondetredev@192.168.178.53   # on melchior, with balthasar's password
   ```

   ```sh
   sudo install -m 644 deploy/host/sshd-halkyon.conf /etc/ssh/sshd_config.d/10-halkyon.conf
   sudo sshd -t && sudo systemctl reload ssh
   ```

   Then, before closing anything, from melchior: `ssh raisondetredev@192.168.178.53`
   must still get in, and `ssh -o PubkeyAuthentication=no raisondetredev@192.168.178.53`
   must be refused. In the FRITZ!Box, give balthasar always the same address
   (*Heimnetz › Netzwerk › balthasar › Diesem Netzwerkgerät immer die gleiche
   IPv4-Adresse zuweisen*), so that 192.168.178.53 stays its address.

2. **The passphrase over SSH at boot** (D20):

   ```sh
   sudo apt install dropbear-initramfs
   sudo install -m 644 deploy/host/dropbear.conf /etc/dropbear/initramfs/dropbear.conf
   # melchior's public key alone (from ~/.ssh/authorized_keys, step 1), restricted:
   printf 'no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="/usr/bin/cryptroot-unlock" %s\n' \
     "$(cat ~/.ssh/authorized_keys)" | sudo tee /etc/dropbear/initramfs/authorized_keys
   grep -v '^#' deploy/host/initramfs-modules | sudo tee -a /etc/initramfs-tools/modules
   echo 'IP=::::balthasar:enx0050b6bffcff:dhcp' | sudo tee -a /etc/initramfs-tools/initramfs.conf
   sudo update-initramfs -u
   ```

   At the next boot, at home: `ssh -p 2222 root@192.168.178.53` from melchior asks for the
   root disk's passphrase, and the boot goes on. The keyboard at the iMac still works as
   before, so a mistake here costs no more than the walk to it.

3. **No desktop** (D12): it stops starting at boot, and Docker Desktop, which runs a VM of
   its own beside Docker, goes. Multipass only if it isn't used:

   ```sh
   sudo systemctl set-default multi-user.target   # back: graphical.target
   sudo apt remove docker-desktop
   sudo snap disable multipass
   ```

4. **Docker's data on the HDD** (D12). Everything Docker holds now (the home Jellyfin's
   images and containers) moves with it. ADR 0032's home Jellyfin stops here for good;
   its configuration and library stay where they are and move into the instance later
   (`../sidecars/README.md`, *Jellyfin on halkyon.li*):

   ```sh
   sudo docker compose -p home-jellyfin down
   sudo systemctl stop docker docker.socket containerd
   sudo mkdir -p /data/docker /data/containerd
   sudo rsync -aHAX --numeric-ids /var/lib/docker/ /data/docker/
   sudo rsync -aHAX --numeric-ids /var/lib/containerd/ /data/containerd/
   sudo mv /var/lib/docker /var/lib/docker.old
   sudo mv /var/lib/containerd /var/lib/containerd.old
   sudo mkdir /var/lib/docker /var/lib/containerd
   grep -v '^#' deploy/host/fstab-docker | sudo tee -a /etc/fstab
   for unit in docker containerd; do
     sudo install -D -m 644 deploy/host/docker-after-data.conf \
       /etc/systemd/system/$unit.service.d/after-data.conf
   done
   sudo systemctl daemon-reload
   sudo mount /var/lib/docker && sudo mount /var/lib/containerd
   sudo systemctl start containerd docker
   sudo docker images   # as before
   ```

   Once Docker runs as before: `sudo rm -rf /var/lib/docker.old /var/lib/containerd.old`.

5. **Home's end of the tunnel** (D8, D9, D12). Home's key, made here and never leaving
   but into Proton Pass (vault *Infra*, an item *balthasar wg-halkyon*):

   ```sh
   sudo apt install wireguard-tools
   sudo install -d -m 700 /etc/wireguard
   sudo sh -c 'umask 077; wg genkey > /etc/wireguard/wg-halkyon.key'
   sudo cat /etc/wireguard/wg-halkyon.key           # into Proton Pass
   sudo sh -c 'wg pubkey < /etc/wireguard/wg-halkyon.key'   # home's public key: ramiel's config
   ```

   Then the files, with ramiel's public key (`../forward/README.md`, step 3) in place of
   `<ramiel's public key>`:

   ```sh
   sudo install -m 600 deploy/host/wg-halkyon.conf deploy/host/wg-halkyon.nft /etc/wireguard/
   sudoedit /etc/wireguard/wg-halkyon.conf
   sudo nft -c -f /etc/wireguard/wg-halkyon.nft   # checks the rules, loads nothing
   sudo install -D -m 644 deploy/host/wg-quick-before-docker.conf \
     /etc/systemd/system/wg-quick@wg-halkyon.service.d/before-docker.conf
   sudo systemctl daemon-reload
   sudo systemctl enable --now wg-quick@wg-halkyon
   sudo wg show wg-halkyon   # a handshake once ramiel is up too
   ping -c 1 10.61.97.1      # refused by home's own rules: nothing new goes to ramiel
   ```

   From melchior, away from home: `ssh -J ramiel raisondetredev@10.61.97.2`.

6. **Reboot**, at home, to see each step hold: the passphrase over SSH (step 2), no
   desktop (step 3), Docker on the HDD (step 4), the tunnel up before Docker (step 5).

The instance itself follows (its users, checkout, `.env` and first deploy). Then:

7. **The CI runner in its nightly window** (D6, D16), once `deploy` and its checkout
   exist. Its user, its folder on the HDD, Forgejo's runner (v13.1.0, the version
   `../runner/config.yml` is written for) and the units:

   ```sh
   sudo useradd --system --uid 61000 --user-group --no-create-home \
     --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
   sudo install -d -o runner -g runner -m 700 /data/forgejo-runner /var/lib/forgejo-runner
   sudo install -m 755 forgejo-runner /usr/local/bin/forgejo-runner   # v13.1.0, its checksum checked
   sudo install -d -m 700 /etc/forgejo-runner
   sudo install -m 644 deploy/runner/config.yml /etc/forgejo-runner/config.yml
   sudo install -m 644 deploy/runner/forgejo-runner.service deploy/runner/forgejo-runner.timer \
     deploy/runner/halkyon-night.service deploy/runner/halkyon-night.timer \
     deploy/runner/halkyon-morning.service deploy/runner/halkyon-morning.timer \
     /etc/systemd/system/
   ```

   In Forgejo (`code.halkyon.app`, the repository's *Settings › Actions › Runners*): a new
   runner, `balthasar`, its UUID into `/etc/forgejo-runner/config.yml` (`RUNNER_UUID`) and
   its token into `/etc/forgejo-runner/token` (root, 0600); sahaquiel's runner removed
   there once it is gone. Then:

   ```sh
   sudo systemctl daemon-reload
   sudo systemctl enable --now forgejo-runner.timer halkyon-night.timer halkyon-morning.timer
   systemctl list-timers 'forgejo-runner*' 'halkyon-*'   # 01:55, 02:00, 05:00
   ```

   To try it by day: `sudo systemctl start forgejo-runner` (it stops itself three hours
   later) and push a branch. The job's first step, *Sandbox*, fails if the runner
   reaches the instance, the home network, the tunnel or any IPv6 address, or sees `/data`.