halkyon/deploy/host/README.md

The instance's host at home

halkyon.li runs at home on balthasar (ADR 0039) and is published through ramiel (../forward/README.md). This folder holds what the host needs beyond Docker: SSH for melchior, the root disk unlocked over SSH at boot, Docker's data on the HDD, and home's end of the tunnel.

File	On the host	What for
sshd-halkyon.conf	/etc/ssh/sshd_config.d/10-halkyon.conf	keys alone, two users (D17)
dropbear.conf	/etc/dropbear/initramfs/dropbear.conf	the passphrase over SSH at boot (D20)
initramfs-modules	appended to /etc/initramfs-tools/modules	the USB network adapter at boot (D20)
fstab-docker	appended to /etc/fstab	Docker's and containerd's data on the HDD (D12)
docker-after-data.conf	docker.service.d/ and containerd.service.d/	both wait for it (D12)
wg-halkyon.conf	/etc/wireguard/wg-halkyon.conf	home's end of the tunnel (D8)
wg-halkyon.nft	/etc/wireguard/wg-halkyon.nft	what the tunnel may carry (D9, D17)
wg-quick-before-docker.conf	wg-quick@wg-halkyon.service.d/	the tunnel before Docker (D12)

test/deploy/halkyon_li_forward_test.dart holds these files to ADR 0039's rules.

Steps

As the developer at the iMac (sudo), from this checkout (~/project-halkyon/halkyon), in this order: each step leaves the machine working, and the early ones keep a way in for the later.

1. sshd for melchior (D17). On balthasar, then from melchior, then on balthasar:

   sudo apt install openssh-server
   

   ssh-copy-id raisondetredev@192.168.178.53   # on melchior, with balthasar's password
   

   sudo install -m 644 deploy/host/sshd-halkyon.conf /etc/ssh/sshd_config.d/10-halkyon.conf
   sudo sshd -t && sudo systemctl reload ssh
   

   Then, before closing anything, from melchior: ssh raisondetredev@192.168.178.53 must still get in, and ssh -o PubkeyAuthentication=no raisondetredev@192.168.178.53 must be refused. In the FRITZ!Box, give balthasar always the same address (Heimnetz › Netzwerk › balthasar › Diesem Netzwerkgerät immer die gleiche IPv4-Adresse zuweisen), so that 192.168.178.53 stays its address.

2. The passphrase over SSH at boot (D20):

   sudo apt install dropbear-initramfs
   sudo install -m 644 deploy/host/dropbear.conf /etc/dropbear/initramfs/dropbear.conf
   # melchior's public key alone (from ~/.ssh/authorized_keys, step 1), restricted:
   printf 'no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="/usr/bin/cryptroot-unlock" %s\n' \
     "$(cat ~/.ssh/authorized_keys)" | sudo tee /etc/dropbear/initramfs/authorized_keys
   grep -v '^#' deploy/host/initramfs-modules | sudo tee -a /etc/initramfs-tools/modules
   echo 'IP=::::balthasar:enx0050b6bffcff:dhcp' | sudo tee -a /etc/initramfs-tools/initramfs.conf
   sudo update-initramfs -u
   

   At the next boot, at home: ssh -p 2222 root@192.168.178.53 from melchior asks for the root disk's passphrase, and the boot goes on. The keyboard at the iMac still works as before, so a mistake here costs no more than the walk to it.

3. No desktop (D12): it stops starting at boot, and Docker Desktop, which runs a VM of its own beside Docker, goes. Multipass only if it isn't used:

   sudo systemctl set-default multi-user.target   # back: graphical.target
   sudo apt remove docker-desktop
   sudo snap disable multipass
   

4. Docker's data on the HDD (D12). Everything Docker holds now (the home Jellyfin's images and containers) moves with it. ADR 0032's home Jellyfin stops here for good; its configuration and library stay where they are and move into the instance later (../sidecars/README.md, Jellyfin on halkyon.li):

   sudo docker compose -p home-jellyfin down
   sudo systemctl stop docker docker.socket containerd
   sudo mkdir -p /data/docker /data/containerd
   sudo rsync -aHAX --numeric-ids /var/lib/docker/ /data/docker/
   sudo rsync -aHAX --numeric-ids /var/lib/containerd/ /data/containerd/
   sudo mv /var/lib/docker /var/lib/docker.old
   sudo mv /var/lib/containerd /var/lib/containerd.old
   sudo mkdir /var/lib/docker /var/lib/containerd
   grep -v '^#' deploy/host/fstab-docker | sudo tee -a /etc/fstab
   for unit in docker containerd; do
     sudo install -D -m 644 deploy/host/docker-after-data.conf \
       /etc/systemd/system/$unit.service.d/after-data.conf
   done
   sudo systemctl daemon-reload
   sudo mount /var/lib/docker && sudo mount /var/lib/containerd
   sudo systemctl start containerd docker
   sudo docker images   # as before
   

   Once Docker runs as before: sudo rm -rf /var/lib/docker.old /var/lib/containerd.old.

5. Home's end of the tunnel (D8, D9, D12). Home's key, made here and never leaving but into Proton Pass (vault Infra, an item balthasar wg-halkyon):

   sudo apt install wireguard-tools
   sudo install -d -m 700 /etc/wireguard
   sudo sh -c 'umask 077; wg genkey > /etc/wireguard/wg-halkyon.key'
   sudo cat /etc/wireguard/wg-halkyon.key           # into Proton Pass
   sudo wg pubkey < /etc/wireguard/wg-halkyon.key   # home's public key: ramiel's config
   

   Then the files, with ramiel's public key (../forward/README.md, step 3) in place of <ramiel's public key>:

   sudo install -m 600 deploy/host/wg-halkyon.conf deploy/host/wg-halkyon.nft /etc/wireguard/
   sudoedit /etc/wireguard/wg-halkyon.conf
   sudo nft -c -f /etc/wireguard/wg-halkyon.nft   # checks the rules, loads nothing
   sudo install -D -m 644 deploy/host/wg-quick-before-docker.conf \
     /etc/systemd/system/wg-quick@wg-halkyon.service.d/before-docker.conf
   sudo systemctl daemon-reload
   sudo systemctl enable --now wg-quick@wg-halkyon
   sudo wg show wg-halkyon   # a handshake once ramiel is up too
   ping -c 1 10.61.97.1      # refused by home's own rules: nothing new goes to ramiel
   

   From melchior, away from home: ssh -J ramiel raisondetredev@10.61.97.2.

6. Reboot, at home, to see each step hold: the passphrase over SSH (step 2), no desktop (step 3), Docker on the HDD (step 4), the tunnel up before Docker (step 5).

The instance itself follows (its users, checkout, .env and first deploy). Then:

7. The CI runner in its nightly window (D6, D16), once deploy and its checkout exist. Its user, its folder on the HDD, Forgejo's runner (v13.1.0, the version ../runner/config.yml is written for) and the units:

   sudo useradd --system --uid 61000 --user-group --no-create-home \
     --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
   sudo install -d -o runner -g runner -m 700 /data/forgejo-runner /var/lib/forgejo-runner
   sudo install -m 755 forgejo-runner /usr/local/bin/forgejo-runner   # v13.1.0, its checksum checked
   sudo install -d -m 700 /etc/forgejo-runner
   sudo install -m 644 deploy/runner/config.yml /etc/forgejo-runner/config.yml
   sudo install -m 644 deploy/runner/forgejo-runner.service deploy/runner/forgejo-runner.timer \
     deploy/runner/halkyon-night.service deploy/runner/halkyon-night.timer \
     deploy/runner/halkyon-morning.service deploy/runner/halkyon-morning.timer \
     /etc/systemd/system/
   

   In Forgejo (code.halkyon.app, the repository's Settings › Actions › Runners): a new runner, balthasar, its UUID into /etc/forgejo-runner/config.yml (RUNNER_UUID) and its token into /etc/forgejo-runner/token (root, 0600); sahaquiel's runner removed there once it is gone. Then:

   sudo systemctl daemon-reload
   sudo systemctl enable --now forgejo-runner.timer halkyon-night.timer halkyon-morning.timer
   systemctl list-timers 'forgejo-runner*' 'halkyon-*'   # 01:55, 02:00, 05:00
   

   To try it by day: sudo systemctl start forgejo-runner (it stops itself three hours later) and push a branch. The job's first step, Sandbox, fails if the runner reaches the instance, the home network, the tunnel or any IPv6 address, or sees /data.