halkyon/deploy/host

The instance's host at home

halkyon.li runs at home on balthasar (ADR 0039) and is published through ramiel (../forward/README.md). This folder holds what the host needs beyond Docker: SSH for melchior, the root disk unlocked over SSH at boot, Docker's data on the HDD, and home's end of the tunnel.

File	On the host	What for
sshd-halkyon.conf	/etc/ssh/sshd_config.d/10-halkyon.conf	keys alone, two users (D17)
dropbear.conf	/etc/dropbear/initramfs/dropbear.conf	the passphrase over SSH at boot (D20)
initramfs-modules	appended to /etc/initramfs-tools/modules	the USB network adapter at boot (D20)
initramfs-ip	/etc/initramfs-tools/conf.d/halkyon-ip	its address by DHCP at boot (D20)
grub-halkyon.cfg	/etc/default/grub.d/halkyon.cfg	GRUB's menu for 5 s, the boot in text (D20), the screen off after a minute idle
grub-dracut-fallback	/etc/grub.d/41_dracut_fallback	the way back to dracut's image (D20)
fstab-docker	appended to /etc/fstab	Docker's and containerd's data on the HDD (D12)
docker-after-data.conf	docker.service.d/ and containerd.service.d/	both wait for it (D12)
wg-halkyon.conf	/etc/wireguard/wg-halkyon.conf	home's end of the tunnel (D8)
wg-halkyon.nft	/etc/wireguard/wg-halkyon.nft	what the tunnel may carry (D9, D17)
wg-quick-halkyon.conf	wg-quick@wg-halkyon.service.d/halkyon.conf	the tunnel before Docker, its rules around it (D9, D12)

test/deploy/halkyon_li_forward_test.dart holds these files to ADR 0039's rules.

Steps

As the developer at the iMac (sudo), from this checkout (~/project-halkyon/halkyon), in this order: each step leaves the machine working, and the early ones keep a way in for the later.

1. sshd for melchior (D17). On balthasar, then from melchior, then on balthasar:

   sudo apt install openssh-server
   

   ssh-copy-id raisondetredev@192.168.178.53   # on melchior, with balthasar's password
   

   sudo install -m 644 deploy/host/sshd-halkyon.conf /etc/ssh/sshd_config.d/10-halkyon.conf
   sudo sshd -t && sudo systemctl reload ssh
   

   Then, before closing anything, from melchior: ssh raisondetredev@192.168.178.53 must still get in, and ssh -o PubkeyAuthentication=no raisondetredev@192.168.178.53 must be refused. In the FRITZ!Box, give balthasar always the same address (Heimnetz › Netzwerk › balthasar › Diesem Netzwerkgerät immer die gleiche IPv4-Adresse zuweisen), so that 192.168.178.53 stays its address.

2. The passphrase over SSH at boot (D20). Ubuntu 26.04 builds its boot images with dracut; dropbear-initramfs needs initramfs-tools, which replaces it. So first the way back: today's image, dracut's, kept for the kernel running now, as an entry in GRUB's menu, which shows for 5 seconds from now on:

   sudo cp /boot/initrd.img-$(uname -r) /boot/initrd.img-$(uname -r).dracut
   sudo install -m 644 deploy/host/grub-halkyon.cfg /etc/default/grub.d/halkyon.cfg
   sudo install -m 755 deploy/host/grub-dracut-fallback /etc/grub.d/41_dracut_fallback
   sudo update-grub                     # "Found the way back: … with dracut's image"
   sudo grep -c 'the way back' /boot/grub/grub.cfg   # 1 (root alone reads it)
   

   Then initramfs-tools, with what opens the root disk (cryptsetup-initramfs) and dropbear. apt lists dracut and dracut-core to be removed, and nothing else:

   sudo apt install initramfs-tools cryptsetup-initramfs dropbear-initramfs
   sudo install -m 644 deploy/host/dropbear.conf /etc/dropbear/initramfs/dropbear.conf
   # melchior's public key alone (from ~/.ssh/authorized_keys, step 1), restricted:
   printf 'no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="/usr/bin/cryptroot-unlock" %s\n' \
     "$(cat ~/.ssh/authorized_keys)" | sudo tee /etc/dropbear/initramfs/authorized_keys
   grep -v '^#' deploy/host/initramfs-modules | sudo tee -a /etc/initramfs-tools/modules
   sudo install -m 644 deploy/host/initramfs-ip /etc/initramfs-tools/conf.d/halkyon-ip
   sudo update-initramfs -u -k all
   

   Before any reboot, each new image must hold what opens the disk, dropbear, its key and the adapter's driver:

   for image in /boot/initrd.img-*-generic; do
     echo "== $image"
     sudo lsinitramfs "$image" | grep -E 'cryptroot/crypttab$|sbin/cryptsetup$|sbin/dropbear$|/authorized_keys$|cdc_ncm\.ko|sbin/lvm$'
   done   # for each: crypttab, cryptsetup, dropbear, authorized_keys, cdc_ncm, lvm
   sudo sh -c 'unmkinitramfs /boot/initrd.img-$(uname -r) /tmp/initrd &&
     find /tmp/initrd -path "*cryptroot/crypttab" -exec cat {} +; rm -rf /tmp/initrd'
   # dm_crypt-0 UUID=… none luks…
   sudo dropbearkey -y -f /etc/dropbear/initramfs/dropbear_ed25519_host_key | grep -i fingerprint
   

   The last line is dropbear's own host key, which melchior sees on port 2222 at the next boot. melchior's key isn't at one of ssh's default paths, so it is named for the unlock in melchior's ~/.ssh/config, its host key apart from sshd's:

   Host balthasar-unlock
       HostName 192.168.178.53
       Port 2222
       User root
       IdentityFile ~/.ssh/raisondetredev@ramiel
       IdentitiesOnly yes
       HostKeyAlias balthasar-unlock
   

   At the next boot, at home: ssh balthasar-unlock from melchior asks for the root disk's passphrase, and the boot goes on. Both the boot image and the system ask the FRITZ!Box for their address, differently (the system with a client ID), so the FRITZ!Box keeps 192.168.178.53 for balthasar's MAC address, 00:50:B6:BF:FC:FF (step 1); without it, the system got another address than the boot image. The keyboard at the iMac still works as before. Should the new image not open the disk, choose the way back in GRUB's menu at the iMac: it boots as before this step. Once the new images have booted: sudo rm /etc/grub.d/41_dracut_fallback /boot/initrd.img-*.dracut && sudo update-grub.

3. No desktop (D12): it stops starting at boot, and Docker Desktop, which runs a VM of its own beside Docker, goes. Multipass is switched off while no VM is needed: its daemon alone held 505 MB with none (8 Nov 2026). snap disable keeps it installed, and sudo snap enable multipass brings it back; a VM then gets a fixed size (--memory), as it shares the instance's memory:

   sudo systemctl set-default multi-user.target   # back: graphical.target
   sudo apt remove docker-desktop
   sudo snap disable multipass
   

4. Docker's data on the HDD (D12). Everything Docker holds now (the home Jellyfin's images and containers) moves with it. ADR 0032's home Jellyfin stops here for good; its configuration and library stay where they are and move into the instance later (../sidecars/README.md, Jellyfin on halkyon.li):

   sudo docker compose -p halkyon-home-jellyfin down
   sudo systemctl stop docker docker.socket containerd
   sudo mkdir -p /data/docker /data/containerd
   sudo rsync -aHAX --numeric-ids /var/lib/docker/ /data/docker/
   sudo rsync -aHAX --numeric-ids /var/lib/containerd/ /data/containerd/
   sudo mv /var/lib/docker /var/lib/docker.old
   sudo mv /var/lib/containerd /var/lib/containerd.old
   sudo mkdir /var/lib/docker /var/lib/containerd
   grep -v '^#' deploy/host/fstab-docker | sudo tee -a /etc/fstab
   for unit in docker containerd; do
     sudo install -D -m 644 deploy/host/docker-after-data.conf \
       /etc/systemd/system/$unit.service.d/after-data.conf
   done
   sudo systemctl daemon-reload
   sudo mount /var/lib/docker && sudo mount /var/lib/containerd
   sudo systemctl start containerd docker
   sudo docker images   # as before
   

   Once Docker runs as before: sudo rm -rf /var/lib/docker.old /var/lib/containerd.old.

5. Home's end of the tunnel (D8, D9, D12). Home's key, made here and never leaving but into Proton Pass (vault Infra, an item balthasar wg-halkyon):

   sudo apt install wireguard-tools
   sudo install -d -m 700 /etc/wireguard
   sudo sh -c 'umask 077; wg genkey > /etc/wireguard/wg-halkyon.key'
   sudo cat /etc/wireguard/wg-halkyon.key           # into Proton Pass
   sudo sh -c 'wg pubkey < /etc/wireguard/wg-halkyon.key'   # home's public key: ramiel's config
   

   Then the files, with ramiel's public key (../forward/README.md, step 3) in place of <ramiel's public key>:

   sudo install -m 600 deploy/host/wg-halkyon.conf deploy/host/wg-halkyon.nft /etc/wireguard/
   sudoedit /etc/wireguard/wg-halkyon.conf
   sudo nft -c -f /etc/wireguard/wg-halkyon.nft   # checks the rules, loads nothing
   sudo install -D -m 644 deploy/host/wg-quick-halkyon.conf \
     /etc/systemd/system/wg-quick@wg-halkyon.service.d/halkyon.conf
   sudo systemctl daemon-reload
   sudo systemctl enable --now wg-quick@wg-halkyon
   sudo wg show wg-halkyon   # a handshake once ramiel is up too
   ping -c 1 10.61.97.1      # refused by home's own rules: nothing new goes to ramiel
   

   From melchior, away from home: ssh -J ramiel raisondetredev@10.61.97.2.

6. Reboot, at home, to see each step hold: the passphrase over SSH (step 2), no desktop (step 3), Docker on the HDD (step 4), the tunnel up before Docker (step 5).

The instance itself follows, once: first-start.md (its users, checkout, .env, the databases, the first start, the first operator, the backups). Then:

7. The CI runner in its nightly window (D6, D16), once deploy and its checkout exist. Its user, its folder on the HDD, Forgejo's runner (v13.1.0, the version ../runner/config.yml is written for) and the units:

   sudo useradd --system --uid 61000 --user-group --no-create-home \
     --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
   sudo install -d -o runner -g runner -m 700 /data/forgejo-runner /var/lib/forgejo-runner
   sudo install -m 755 forgejo-runner /usr/local/bin/forgejo-runner   # v13.1.0, its checksum checked
   sudo install -d -m 755 /etc/forgejo-runner   # runner reads config.yml in it; the token stays 0600
   sudo install -m 644 deploy/runner/config.yml /etc/forgejo-runner/config.yml
   sudo install -m 644 deploy/runner/forgejo-runner.service deploy/runner/forgejo-runner.timer \
     deploy/runner/halkyon-night.service deploy/runner/halkyon-night.timer \
     deploy/runner/halkyon-morning.service deploy/runner/halkyon-morning.timer \
     /etc/systemd/system/
   sudo install -m 755 deploy/runner/ci-now.sh /usr/local/sbin/halkyon-ci-now
   

   In Forgejo (code.halkyon.app, the repository's Settings › Actions › Runners): a new runner, balthasar, its UUID into /etc/forgejo-runner/config.yml (RUNNER_UUID) and its token into /etc/forgejo-runner/token (root, 0600); sahaquiel's runner removed there once it is gone. Then:

   sudo systemctl daemon-reload
   sudo systemctl enable --now forgejo-runner.timer halkyon-night.timer halkyon-morning.timer
   systemctl list-timers 'forgejo-runner*' 'halkyon-*'   # 01:55, 02:00, 05:00
   

   To run the queue by day: sudo halkyon-ci-now (deploy/runner/ci-now.sh, which says how from melchior too) rests the heavy sidecars, runs every queued job and starts them again; --keep-sidecars keeps them, the job then slower, in swap. The job's first step, Sandbox, fails if the runner reaches the instance, the home network, the tunnel or any IPv6 address, or sees /data.