halkyon/deploy/host/README.md

# The instance's host at home

halkyon.li runs at home on balthasar (ADR 0039) and is published through ramiel
(`../forward/README.md`). This folder holds what the host needs beyond Docker: SSH for
melchior, the root disk unlocked over SSH at boot, Docker's data on the HDD, and home's
end of the tunnel.

| File | On the host | What for |
|---|---|---|
| `sshd-halkyon.conf` | `/etc/ssh/sshd_config.d/10-halkyon.conf` | keys alone, two users (D17) |
| `dropbear.conf` | `/etc/dropbear/initramfs/dropbear.conf` | the passphrase over SSH at boot (D20) |
| `initramfs-modules` | appended to `/etc/initramfs-tools/modules` | the USB network adapter at boot (D20) |
| `initramfs-ip` | `/etc/initramfs-tools/conf.d/halkyon-ip` | its address by DHCP at boot (D20) |
| `grub-halkyon.cfg` | `/etc/default/grub.d/halkyon.cfg` | GRUB's menu for 5 s, the boot in text (D20), the screen off after a minute idle |
| `grub-dracut-fallback` | `/etc/grub.d/41_dracut_fallback` | the way back to dracut's image (D20) |
| `fstab-docker` | appended to `/etc/fstab` | Docker's and containerd's data on the HDD (D12) |
| `docker-after-data.conf` | `docker.service.d/` and `containerd.service.d/` | both wait for it (D12) |
| `wg-halkyon.conf` | `/etc/wireguard/wg-halkyon.conf` | home's end of the tunnel (D8) |
| `wg-halkyon.nft` | `/etc/wireguard/wg-halkyon.nft` | what the tunnel may carry (D9, D17) |
| `wg-quick-halkyon.conf` | `wg-quick@wg-halkyon.service.d/halkyon.conf` | the tunnel before Docker, its rules around it (D9, D12) |

`test/deploy/halkyon_li_forward_test.dart` holds these files to ADR 0039's rules.

## Steps

As the developer at the iMac (`sudo`), from this checkout (`~/project-halkyon/halkyon`),
in this order: each step leaves the machine working, and the early ones keep a way in
for the later.

1. **sshd for melchior** (D17). On balthasar, then from melchior, then on balthasar:

   ```sh
   sudo apt install openssh-server
   ```

   ```sh
   ssh-copy-id raisondetredev@192.168.178.53   # on melchior, with balthasar's password
   ```

   ```sh
   sudo install -m 644 deploy/host/sshd-halkyon.conf /etc/ssh/sshd_config.d/10-halkyon.conf
   sudo sshd -t && sudo systemctl reload ssh
   ```

   Then, before closing anything, from melchior: `ssh raisondetredev@192.168.178.53`
   must still get in, and `ssh -o PubkeyAuthentication=no raisondetredev@192.168.178.53`
   must be refused. In the FRITZ!Box, give balthasar always the same address
   (*Heimnetz › Netzwerk › balthasar › Diesem Netzwerkgerät immer die gleiche
   IPv4-Adresse zuweisen*), so that 192.168.178.53 stays its address.

2. **The passphrase over SSH at boot** (D20). Ubuntu 26.04 builds its boot images with
   dracut; `dropbear-initramfs` needs initramfs-tools, which replaces it. So first the
   way back: today's image, dracut's, kept for the kernel running now, as an entry in
   GRUB's menu, which shows for 5 seconds from now on:

   ```sh
   sudo cp /boot/initrd.img-$(uname -r) /boot/initrd.img-$(uname -r).dracut
   sudo install -m 644 deploy/host/grub-halkyon.cfg /etc/default/grub.d/halkyon.cfg
   sudo install -m 755 deploy/host/grub-dracut-fallback /etc/grub.d/41_dracut_fallback
   sudo update-grub                     # "Found the way back: … with dracut's image"
   sudo grep -c 'the way back' /boot/grub/grub.cfg   # 1 (root alone reads it)
   ```

   Then initramfs-tools, with what opens the root disk (`cryptsetup-initramfs`) and
   dropbear. apt lists `dracut` and `dracut-core` to be removed, and nothing else:

   ```sh
   sudo apt install initramfs-tools cryptsetup-initramfs dropbear-initramfs
   sudo install -m 644 deploy/host/dropbear.conf /etc/dropbear/initramfs/dropbear.conf
   # melchior's public key alone (from ~/.ssh/authorized_keys, step 1), restricted:
   printf 'no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="/usr/bin/cryptroot-unlock" %s\n' \
     "$(cat ~/.ssh/authorized_keys)" | sudo tee /etc/dropbear/initramfs/authorized_keys
   grep -v '^#' deploy/host/initramfs-modules | sudo tee -a /etc/initramfs-tools/modules
   sudo install -m 644 deploy/host/initramfs-ip /etc/initramfs-tools/conf.d/halkyon-ip
   sudo update-initramfs -u -k all
   ```

   Before any reboot, each new image must hold what opens the disk, dropbear, its key
   and the adapter's driver:

   ```sh
   for image in /boot/initrd.img-*-generic; do
     echo "== $image"
     sudo lsinitramfs "$image" | grep -E 'cryptroot/crypttab$|sbin/cryptsetup$|sbin/dropbear$|/authorized_keys$|cdc_ncm\.ko|sbin/lvm$'
   done   # for each: crypttab, cryptsetup, dropbear, authorized_keys, cdc_ncm, lvm
   sudo sh -c 'unmkinitramfs /boot/initrd.img-$(uname -r) /tmp/initrd &&
     find /tmp/initrd -path "*cryptroot/crypttab" -exec cat {} +; rm -rf /tmp/initrd'
   # dm_crypt-0 UUID=… none luks…
   sudo dropbearkey -y -f /etc/dropbear/initramfs/dropbear_ed25519_host_key | grep -i fingerprint
   ```

   The last line is dropbear's own host key, which melchior sees on port 2222 at the
   next boot. melchior's key isn't at one of ssh's default paths, so it is named for the
   unlock in melchior's `~/.ssh/config`, its host key apart from sshd's:

   ```
   Host balthasar-unlock
       HostName 192.168.178.53
       Port 2222
       User root
       IdentityFile ~/.ssh/raisondetredev@ramiel
       IdentitiesOnly yes
       HostKeyAlias balthasar-unlock
   ```

   At the next boot, at home: `ssh balthasar-unlock` from melchior asks for the root
   disk's passphrase, and the boot goes on. Both the boot image and the system ask the
   FRITZ!Box for their address, differently (the system with a client ID), so the
   FRITZ!Box keeps 192.168.178.53 for balthasar's MAC address, `00:50:B6:BF:FC:FF`
   (step 1); without it, the system got another address than the boot image. The keyboard at the iMac still
   works as before. Should the new image not open the disk, choose *the way back* in
   GRUB's menu at the iMac: it boots as before this step. Once the new images have
   booted: `sudo rm /etc/grub.d/41_dracut_fallback /boot/initrd.img-*.dracut && sudo update-grub`.

3. **No desktop** (D12): it stops starting at boot, and Docker Desktop, which runs a VM of
   its own beside Docker, goes. Multipass is switched off while no VM is needed: its
   daemon alone held 505 MB with none (8 Nov 2026). `snap disable` keeps it installed,
   and `sudo snap enable multipass` brings it back; a VM then gets a fixed size
   (`--memory`), as it shares the instance's memory:

   ```sh
   sudo systemctl set-default multi-user.target   # back: graphical.target
   sudo apt remove docker-desktop
   sudo snap disable multipass
   ```

4. **Docker's data on the HDD** (D12). Everything Docker holds now (the home Jellyfin's
   images and containers) moves with it. ADR 0032's home Jellyfin stops here for good;
   its configuration and library stay where they are and move into the instance later
   (`../sidecars/README.md`, *Jellyfin on halkyon.li*):

   ```sh
   sudo docker compose -p halkyon-home-jellyfin down
   sudo systemctl stop docker docker.socket containerd
   sudo mkdir -p /data/docker /data/containerd
   sudo rsync -aHAX --numeric-ids /var/lib/docker/ /data/docker/
   sudo rsync -aHAX --numeric-ids /var/lib/containerd/ /data/containerd/
   sudo mv /var/lib/docker /var/lib/docker.old
   sudo mv /var/lib/containerd /var/lib/containerd.old
   sudo mkdir /var/lib/docker /var/lib/containerd
   grep -v '^#' deploy/host/fstab-docker | sudo tee -a /etc/fstab
   for unit in docker containerd; do
     sudo install -D -m 644 deploy/host/docker-after-data.conf \
       /etc/systemd/system/$unit.service.d/after-data.conf
   done
   sudo systemctl daemon-reload
   sudo mount /var/lib/docker && sudo mount /var/lib/containerd
   sudo systemctl start containerd docker
   sudo docker images   # as before
   ```

   Once Docker runs as before: `sudo rm -rf /var/lib/docker.old /var/lib/containerd.old`.

5. **Home's end of the tunnel** (D8, D9, D12). Home's key, made here and never leaving
   but into Proton Pass (vault *Infra*, an item *balthasar wg-halkyon*):

   ```sh
   sudo apt install wireguard-tools
   sudo install -d -m 700 /etc/wireguard
   sudo sh -c 'umask 077; wg genkey > /etc/wireguard/wg-halkyon.key'
   sudo cat /etc/wireguard/wg-halkyon.key           # into Proton Pass
   sudo sh -c 'wg pubkey < /etc/wireguard/wg-halkyon.key'   # home's public key: ramiel's config
   ```

   Then the files, with ramiel's public key (`../forward/README.md`, step 3) in place of
   `<ramiel's public key>`:

   ```sh
   sudo install -m 600 deploy/host/wg-halkyon.conf deploy/host/wg-halkyon.nft /etc/wireguard/
   sudoedit /etc/wireguard/wg-halkyon.conf
   sudo nft -c -f /etc/wireguard/wg-halkyon.nft   # checks the rules, loads nothing
   sudo install -D -m 644 deploy/host/wg-quick-halkyon.conf \
     /etc/systemd/system/wg-quick@wg-halkyon.service.d/halkyon.conf
   sudo systemctl daemon-reload
   sudo systemctl enable --now wg-quick@wg-halkyon
   sudo wg show wg-halkyon   # a handshake once ramiel is up too
   ping -c 1 10.61.97.1      # refused by home's own rules: nothing new goes to ramiel
   ```

   From melchior, away from home: `ssh -J ramiel raisondetredev@10.61.97.2`.

6. **Reboot**, at home, to see each step hold: the passphrase over SSH (step 2), no
   desktop (step 3), Docker on the HDD (step 4), the tunnel up before Docker (step 5).

The instance itself follows, once: `first-start.md` (its users, checkout, `.env`, the
databases, the first start, the first operator, the backups). Then:

7. **The CI runner in its nightly window** (D6, D16), once `deploy` and its checkout
   exist. Its user, its folder on the HDD, Forgejo's runner (v13.1.0, the version
   `../runner/config.yml` is written for) and the units:

   ```sh
   sudo useradd --system --uid 61000 --user-group --no-create-home \
     --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
   sudo install -d -o runner -g runner -m 700 /data/forgejo-runner /var/lib/forgejo-runner
   sudo install -m 755 forgejo-runner /usr/local/bin/forgejo-runner   # v13.1.0, its checksum checked
   sudo install -d -m 755 /etc/forgejo-runner   # runner reads config.yml in it; the token stays 0600
   sudo install -m 644 deploy/runner/config.yml /etc/forgejo-runner/config.yml
   sudo install -m 644 deploy/runner/forgejo-runner.service deploy/runner/forgejo-runner.timer \
     deploy/runner/halkyon-night.service deploy/runner/halkyon-night.timer \
     deploy/runner/halkyon-morning.service deploy/runner/halkyon-morning.timer \
     /etc/systemd/system/
   sudo install -m 755 deploy/runner/ci-now.sh /usr/local/sbin/halkyon-ci-now
   ```

   In Forgejo (`code.halkyon.app`, the repository's *Settings › Actions › Runners*): a new
   runner, `balthasar`, its UUID into `/etc/forgejo-runner/config.yml` (`RUNNER_UUID`) and
   its token into `/etc/forgejo-runner/token` (root, 0600); sahaquiel's runner removed
   there once it is gone. Then:

   ```sh
   sudo systemctl daemon-reload
   sudo systemctl enable --now forgejo-runner.timer halkyon-night.timer halkyon-morning.timer
   systemctl list-timers 'forgejo-runner*' 'halkyon-*'   # 01:55, 02:00, 05:00
   ```

   To run the queue by day: `sudo halkyon-ci-now` (`deploy/runner/ci-now.sh`, which says
   how from melchior too) rests the heavy sidecars, runs every queued job and starts them
   again; `--keep-sidecars` keeps them, the job then slower, in swap. The job's first step, *Sandbox*, fails if the runner
   reaches the instance, the home network, the tunnel or any IPv6 address, or sees `/data`.