build(ci,deploy)!: the runner at home in a nightly window, deploys by hand (ADR 0039, D6, D13, D16)

The CI runner moves to halkyon.li's host at home and runs from 02:00 to
05:00 alone: a timer starts it, its unit's RuntimeMaxSec ends it, and
night.sh rests the instance's heavy sidecars around it as deploy, while
Caddy, PostgreSQL and the apps' servers keep serving. Jobs pushed during the
day wait in Forgejo's queue.

Its sandbox fits the new host: its folder on the HDD, the rest of /data
hidden, containerd's store out of reach, no private IPv4 network and no
IPv6 at all (the home network's IPv6 addresses are public ones under a
changing prefix, on the internet's own interface), 5 GB of memory with swap
beyond. The Sandbox step checks the tunnel, the home router and IPv6.

BREAKING CHANGE: CI deploys nothing any more: the deploy job goes with
learn.halkyon.eu's server, and halkyon.li deploys by hand with redeploy.sh
once a commit's night's run passed. Jobs run on the runner labelled
balthasar; sahaquiel's runner picks up none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

deploy/host/README.md

@@ -129,5 +129,36 @@ for the later.
 6. **Reboot**, at home, to see each step hold: the passphrase over SSH (step 2), no
    desktop (step 3), Docker on the HDD (step 4), the tunnel up before Docker (step 5).
 
-The instance itself follows (its users, checkout, `.env` and first deploy), once its
-compose files for Meditate and Jellyfin are built.
+The instance itself follows (its users, checkout, `.env` and first deploy). Then:
+
+7. **The CI runner in its nightly window** (D6, D16), once `deploy` and its checkout
+   exist. Its user, its folder on the HDD, Forgejo's runner (v13.1.0, the version
+   `../runner/config.yml` is written for) and the units:
+
+   ```sh
+   sudo useradd --system --uid 61000 --user-group --no-create-home \
+     --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
+   sudo install -d -o runner -g runner -m 700 /data/forgejo-runner /var/lib/forgejo-runner
+   sudo install -m 755 forgejo-runner /usr/local/bin/forgejo-runner   # v13.1.0, its checksum checked
+   sudo install -d -m 700 /etc/forgejo-runner
+   sudo install -m 644 deploy/runner/config.yml /etc/forgejo-runner/config.yml
+   sudo install -m 644 deploy/runner/forgejo-runner.service deploy/runner/forgejo-runner.timer \
+     deploy/runner/halkyon-night.service deploy/runner/halkyon-night.timer \
+     deploy/runner/halkyon-morning.service deploy/runner/halkyon-morning.timer \
+     /etc/systemd/system/
+   ```
+
+   In Forgejo (`code.halkyon.app`, the repository's *Settings › Actions › Runners*): a new
+   runner, `balthasar`, its UUID into `/etc/forgejo-runner/config.yml` (`RUNNER_UUID`) and
+   its token into `/etc/forgejo-runner/token` (root, 0600); sahaquiel's runner removed
+   there once it is gone. Then:
+
+   ```sh
+   sudo systemctl daemon-reload
+   sudo systemctl enable --now forgejo-runner.timer halkyon-night.timer halkyon-morning.timer
+   systemctl list-timers 'forgejo-runner*' 'halkyon-*'   # 01:55, 02:00, 05:00
+   ```
+
+   To try it by day: `sudo systemctl start forgejo-runner` (it stops itself three hours
+   later) and push a branch. The job's first step, *Sandbox*, fails if the runner
+   reaches the instance, the home network, the tunnel or any IPv6 address, or sees `/data`.

deploy/night.sh

@@ -0,0 +1,37 @@
+#!/bin/sh
+# Rests the instance's heavy sidecars during the CI runner's nightly window and starts them
+# again after it (ADR 0039, D6): `night.sh stop` at 01:55, `night.sh start` at 05:00, as
+# deploy, from deploy/runner/'s halkyon-night and halkyon-morning units. Caddy, PostgreSQL
+# and the apps' servers keep serving through the night, so people sign in and sync;
+# Jellyfin, LanguageTool, the dictionaries, routes, map tiles and place search, with their
+# refreshers, wait until the morning. Only those .env switches on; a deploy in between
+# starts them with the rest, which is harmless.
+set -eu
+
+# Every service that takes memory or builds data, by its compose name.
+heavy="jellyfin languagetool dictionaries brouter brouter-segments map-tiles map-tiles-refresh photon photon-data"
+
+cd "$(dirname "$0")"
+action=${1:-}
+case $action in
+  stop | start) ;;
+  *)
+    echo "usage: $0 stop|start" >&2
+    exit 2
+    ;;
+esac
+
+on=$(docker compose config --services)
+services=
+for service in $heavy; do
+  if printf '%s\n' "$on" | grep -qx "$service"; then
+    services="$services $service"
+  fi
+done
+if [ -z "$services" ]; then
+  echo "No heavy sidecar is switched on."
+  exit 0
+fi
+# shellcheck disable=SC2086 # one service a word
+docker compose "$action" $services
+echo "night.sh $action:$services"

deploy/redeploy.sh

@@ -1,12 +1,13 @@
 #!/bin/sh
-# Deploys learn.halkyon.eu from main. The only command deploy's keys may run: deploy's
-# ~/.ssh/authorized_keys on the server pins both to this script, CI's from the server itself:
+# Deploys the instance (halkyon.li, ADR 0039, D13; learn.halkyon.eu until November 2026)
+# from main, by hand, once the commit's night's CI run passed. The only command deploy's
+# keys may run: deploy's ~/.ssh/authorized_keys on the instance's host pins the
+# developer's key to this script:
 #
-#   restrict,from="127.0.0.1,::1",command="/home/deploy/halkyon/deploy/redeploy.sh" ssh-ed25519 … (CI deploy trigger)
-#   restrict,command="/home/deploy/halkyon/deploy/redeploy.sh" ssh-ed25519 … (manual deploys)
+#   restrict,command="/home/deploy/halkyon/deploy/redeploy.sh" ssh-ed25519 … (deploys by hand)
 #
-# CI passes the commit it has checked. This deploys it only
-#   - while it is still the tip of main (otherwise the CI run of the newer tip deploys),
+# The developer passes the commit. This deploys it only
+#   - while it is still the tip of main (otherwise the newer tip is deployed instead),
 #   - as a fast-forward of the deployed commit,
 #   - if every new commit is signed by a key in deploy/allowed-signers, as the deployed
 #     checkout lists them (so no commit can add its own key),
@@ -14,10 +15,10 @@
 # too where .env switches each on, compose.train.yaml, compose.todo.yaml,
 # compose.meditate.yaml), applies the migrations
 # before switching to the new server, starts it once more after a migration (see below),
-# and fails unless that server answers afterwards. A failure leaves the checkout at the deployed commit, so
-# running the CI job again retries. By hand, with your own key, which passes the commit as CI does:
+# and fails unless that server answers afterwards. A failure leaves the checkout at the
+# deployed commit, so running it again retries. From melchior, at home or through ramiel:
 #
-#   ssh sahaquiel-deploy <commit>
+#   ssh balthasar-deploy <commit>
 set -eu
 
 main() {

deploy/runner/config.yml

@@ -1,10 +1,11 @@
-# The Forgejo Actions runner of this repository's CI, on learn.halkyon.eu's server. Installed as
+# The Forgejo Actions runner of this repository's CI, on halkyon.li's host at home (ADR 0039,
+# D16), in the nightly window alone (forgejo-runner.timer). Installed as
 # /etc/forgejo-runner/config.yml, read by forgejo-runner.service (next to this file).
 # Written for forgejo-runner v13.1.0; its full reference: `forgejo-runner generate-config`.
 #
 # Jobs run directly on the host as the unprivileged user `runner`, never in containers:
-# running containers would take Docker access, which is root on the production server.
-# The systemd unit keeps the jobs away from production.
+# running containers would take Docker access, which is root on the instance's host.
+# The systemd unit keeps the jobs away from the instance and the home network.
 
 log:
   level: info
@@ -21,7 +22,7 @@ runner:
   fetch_interval: 10s
   report_interval: 1s
   # `runs-on: sahaquiel` in a workflow means: here, on the instance's server, on the host.
-  labels: ["sahaquiel:host"]
+  labels: ["balthasar:host"]
 
 cache:
   # No cache server (it would listen on a port of its own); the SDKs and package

deploy/runner/forgejo-runner.service

@@ -1,6 +1,10 @@
-# The Forgejo Actions runner of this repository's CI, on learn.halkyon.eu's server. Installed as
-# /etc/systemd/system/forgejo-runner.service; its configuration is config.yml, next to
-# this file.
+# The Forgejo Actions runner of this repository's CI, on halkyon.li's host at home (ADR
+# 0039, D6, D16; on learn.halkyon.eu's server until it retired in November 2026).
+# Installed as /etc/systemd/system/forgejo-runner.service; its configuration is
+# config.yml, next to this file. It runs in the nightly window alone: started at 02:00 by
+# forgejo-runner.timer, stopped three hours later (RuntimeMaxSec), while night.sh rests
+# the instance's heavy sidecars (halkyon-night.timer, halkyon-morning.timer). Jobs pushed
+# during the day wait in Forgejo's queue.
 #
 # Jobs run directly on the host as the unprivileged user `runner` ("without any
 # isolation" from that user, as Forgejo's security guide puts it), and they run our
@@ -12,14 +16,19 @@
 # container's processes (PostgreSQL and the server run as 999; Docker doesn't remap
 # them): processes of one user ID can read each other's environment and, through
 # /proc, their files. `useradd --system` picks 999 when it is free, so create it with
-#   useradd --system --uid 61000 --user-group --create-home \
+#   useradd --system --uid 61000 --user-group --no-create-home \
 #     --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
+# Its home is /data/forgejo-runner on the HDD (the root disk has no room for Flutter and
+# the caches), seen by the runner as /var/lib/forgejo-runner (BindPaths, below):
+#   install -d -o runner -g runner -m 700 /data/forgejo-runner /var/lib/forgejo-runner
 # CI's first step checks this sandbox and fails if it leaks.
 [Unit]
 Description=Forgejo Actions runner (CI of halkyon_learn; jobs on the host)
 Documentation=https://forgejo.org/docs/latest/admin/actions/security/
 After=network-online.target
 Wants=network-online.target
+# Its folder is on the HDD, which its LUKS key file unlocks once the root disk is open.
+RequiresMountsFor=/data
 
 [Service]
 User=runner
@@ -32,18 +41,29 @@ Restart=on-failure
 RestartSec=30
 # The runner waits up to 10 minutes for a running job (shutdown_timeout).
 TimeoutStopSec=11min
+# The nightly window's end: three hours after forgejo-runner.timer started it at 02:00. A
+# job still running is stopped, and the next one's Leftovers step clears what it left.
+RuntimeMaxSec=3h
 
 # Away from production: /home (deploy's .env, the checkout, the database dumps) is
-# invisible, the rest of the system read-only except the runner's own directory, other
-# users' processes hidden, and Docker's container networks, where the database listens,
-# unreachable, and so is the tunnel to Jellyfin at home (10.61.96.0/30, ADR 0032, D6).
+# invisible, and so is /data, where /home really lies, with Docker's data and the media,
+# but for the runner's own folder; the rest of the system is read-only, other users'
+# processes are hidden, and Docker and containerd out of reach.
 ProtectHome=yes
 ProtectSystem=strict
+TemporaryFileSystem=/data:ro
+BindPaths=/data/forgejo-runner:/var/lib/forgejo-runner
 ReadWritePaths=/var/lib/forgejo-runner
 PrivateTmp=yes
 ProtectProc=invisible
-InaccessiblePaths=-/run/docker.sock -/run/containerd -/var/lib/docker
-IPAddressDeny=172.16.0.0/12 192.168.0.0/16 fd00::/8 10.61.96.0/30
+InaccessiblePaths=-/run/docker.sock -/run/containerd -/var/lib/docker -/var/lib/containerd
+# The network: the internet over IPv4 alone. No private IPv4 network (the home network
+# and its router, Docker's networks, where the database listens, the tunnel to ramiel,
+# Multipass' bridges, CGNAT and link-local addresses), and no IPv6 at all: the home
+# network's IPv6 addresses are public ones, under a prefix the provider changes, on the
+# same interface as the internet, so no address or interface could tell them apart. All
+# CI fetches (pub.dev, GitHub, code.halkyon.app) answers over IPv4.
+IPAddressDeny=10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 169.254.0.0/16 ::/0
 NoNewPrivileges=yes
 CapabilityBoundingSet=
 AmbientCapabilities=
@@ -66,20 +86,19 @@ SystemCallFilter=~@clock @cpu-emulation @debug @module @mount @obsolete @raw-io
 SystemCallErrorNumber=EPERM
 UMask=0077
 
-# Never at the instance's expense: two of the four cores, at most 6 GB of the 8 (the
-# kernel reclaims from CI first above 5.5 GB), the lowest priority for CPU and disk, and
-# CI's processes the first the kernel kills should the whole server run out of memory.
-# A cold `dart analyze` of the workspace with its lint plugin needs about 5.4 GB since
-# Todo and the shared packages joined it (3.7 GB warm); below that the kernel slowed it
-# to a crawl and then killed it, and OOMPolicy's default then stopped the whole unit.
-CPUQuota=200%
-MemoryHigh=5.5G
-MemoryMax=6G
+# Never at the instance's expense, which keeps serving through the night: three of the
+# four cores (its servers idle then), at most 5 GB of the 7.2 (the kernel reclaims from CI
+# first above 4.5 GB), and swap beyond, the lowest priority for CPU and disk, and CI's
+# processes the first the kernel kills should the whole machine run out of memory. A
+# cold `dart analyze` of the workspace with its lint plugin needs about 5.4 GB (3.7 GB
+# warm), so it swaps a little rather than being killed: slower, within the window.
+CPUQuota=300%
+MemoryHigh=4.5G
+MemoryMax=5G
 OOMScoreAdjust=1000
 TasksMax=2048
 Nice=10
 IOSchedulingClass=best-effort
 IOSchedulingPriority=7
 
-[Install]
-WantedBy=multi-user.target
+# No [Install]: never started at boot, only by forgejo-runner.timer.

deploy/runner/forgejo-runner.timer

@@ -0,0 +1,15 @@
+# The CI runner's nightly window (ADR 0039, D6): started at 02:00, after night.sh has
+# rested the heavy sidecars (halkyon-night.timer, 01:55), and stopped by its own unit three
+# hours later (RuntimeMaxSec). Never started to catch up after a night the machine was
+# off: the queue waits for the next night. As /etc/systemd/system/forgejo-runner.timer,
+# enabled with `systemctl enable --now forgejo-runner.timer`.
+[Unit]
+Description=The CI runner's nightly window
+
+[Timer]
+OnCalendar=*-*-* 02:00:00
+Persistent=false
+Unit=forgejo-runner.service
+
+[Install]
+WantedBy=timers.target

deploy/runner/halkyon-morning.service

@@ -0,0 +1,12 @@
+# night.sh start as deploy: starts the heavy sidecars again after the CI runner's window (ADR 0039, D6). As
+# /etc/systemd/system/halkyon-morning.service, started by halkyon-morning.timer.
+[Unit]
+Description=halkyon.li: night.sh start
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+User=deploy
+WorkingDirectory=/home/deploy/halkyon/deploy
+ExecStart=/home/deploy/halkyon/deploy/night.sh start

deploy/runner/halkyon-morning.timer

@@ -0,0 +1,12 @@
+# halkyon-morning.service at 05:00:00 daily (ADR 0039, D6). As
+# /etc/systemd/system/halkyon-morning.timer, enabled with
+# `systemctl enable --now halkyon-morning.timer`.
+[Unit]
+Description=halkyon.li: night.sh start at 05:00:00
+
+[Timer]
+OnCalendar=*-*-* 05:00:00
+Persistent=false
+
+[Install]
+WantedBy=timers.target

deploy/runner/halkyon-night.service

@@ -0,0 +1,12 @@
+# night.sh stop as deploy: rests the heavy sidecars before the CI runner's window (ADR 0039, D6). As
+# /etc/systemd/system/halkyon-night.service, started by halkyon-night.timer.
+[Unit]
+Description=halkyon.li: night.sh stop
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+User=deploy
+WorkingDirectory=/home/deploy/halkyon/deploy
+ExecStart=/home/deploy/halkyon/deploy/night.sh stop

deploy/runner/halkyon-night.timer

@@ -0,0 +1,12 @@
+# halkyon-night.service at 01:55:00 daily (ADR 0039, D6). As
+# /etc/systemd/system/halkyon-night.timer, enabled with
+# `systemctl enable --now halkyon-night.timer`.
+[Unit]
+Description=halkyon.li: night.sh stop at 01:55:00
+
+[Timer]
+OnCalendar=*-*-* 01:55:00
+Persistent=false
+
+[Install]
+WantedBy=timers.target