build(ci,deploy)!: the runner at home in a nightly window, deploys by hand (ADR 0039, D6, D13, D16)

The CI runner moves to halkyon.li's host at home and runs from 02:00 to
05:00 alone: a timer starts it, its unit's RuntimeMaxSec ends it, and
night.sh rests the instance's heavy sidecars around it as deploy, while
Caddy, PostgreSQL and the apps' servers keep serving. Jobs pushed during the
day wait in Forgejo's queue.

Its sandbox fits the new host: its folder on the HDD, the rest of /data
hidden, containerd's store out of reach, no private IPv4 network and no
IPv6 at all (the home network's IPv6 addresses are public ones under a
changing prefix, on the internet's own interface), 5 GB of memory with swap
beyond. The Sandbox step checks the tunnel, the home router and IPv6.

BREAKING CHANGE: CI deploys nothing any more: the deploy job goes with
learn.halkyon.eu's server, and halkyon.li deploys by hand with redeploy.sh
once a commit's night's run passed. Jobs run on the runner labelled
balthasar; sahaquiel's runner picks up none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

deploy/runner/config.yml

@@ -1,10 +1,11 @@
-# The Forgejo Actions runner of this repository's CI, on learn.halkyon.eu's server. Installed as
+# The Forgejo Actions runner of this repository's CI, on halkyon.li's host at home (ADR 0039,
+# D16), in the nightly window alone (forgejo-runner.timer). Installed as
 # /etc/forgejo-runner/config.yml, read by forgejo-runner.service (next to this file).
 # Written for forgejo-runner v13.1.0; its full reference: `forgejo-runner generate-config`.
 #
 # Jobs run directly on the host as the unprivileged user `runner`, never in containers:
-# running containers would take Docker access, which is root on the production server.
-# The systemd unit keeps the jobs away from production.
+# running containers would take Docker access, which is root on the instance's host.
+# The systemd unit keeps the jobs away from the instance and the home network.
 
 log:
   level: info
@@ -21,7 +22,7 @@ runner:
   fetch_interval: 10s
   report_interval: 1s
   # `runs-on: sahaquiel` in a workflow means: here, on the instance's server, on the host.
-  labels: ["sahaquiel:host"]
+  labels: ["balthasar:host"]
 
 cache:
   # No cache server (it would listen on a port of its own); the SDKs and package

deploy/runner/forgejo-runner.service

@@ -1,6 +1,10 @@
-# The Forgejo Actions runner of this repository's CI, on learn.halkyon.eu's server. Installed as
-# /etc/systemd/system/forgejo-runner.service; its configuration is config.yml, next to
-# this file.
+# The Forgejo Actions runner of this repository's CI, on halkyon.li's host at home (ADR
+# 0039, D6, D16; on learn.halkyon.eu's server until it retired in November 2026).
+# Installed as /etc/systemd/system/forgejo-runner.service; its configuration is
+# config.yml, next to this file. It runs in the nightly window alone: started at 02:00 by
+# forgejo-runner.timer, stopped three hours later (RuntimeMaxSec), while night.sh rests
+# the instance's heavy sidecars (halkyon-night.timer, halkyon-morning.timer). Jobs pushed
+# during the day wait in Forgejo's queue.
 #
 # Jobs run directly on the host as the unprivileged user `runner` ("without any
 # isolation" from that user, as Forgejo's security guide puts it), and they run our
@@ -12,14 +16,19 @@
 # container's processes (PostgreSQL and the server run as 999; Docker doesn't remap
 # them): processes of one user ID can read each other's environment and, through
 # /proc, their files. `useradd --system` picks 999 when it is free, so create it with
-#   useradd --system --uid 61000 --user-group --create-home \
+#   useradd --system --uid 61000 --user-group --no-create-home \
 #     --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
+# Its home is /data/forgejo-runner on the HDD (the root disk has no room for Flutter and
+# the caches), seen by the runner as /var/lib/forgejo-runner (BindPaths, below):
+#   install -d -o runner -g runner -m 700 /data/forgejo-runner /var/lib/forgejo-runner
 # CI's first step checks this sandbox and fails if it leaks.
 [Unit]
 Description=Forgejo Actions runner (CI of halkyon_learn; jobs on the host)
 Documentation=https://forgejo.org/docs/latest/admin/actions/security/
 After=network-online.target
 Wants=network-online.target
+# Its folder is on the HDD, which its LUKS key file unlocks once the root disk is open.
+RequiresMountsFor=/data
 
 [Service]
 User=runner
@@ -32,18 +41,29 @@ Restart=on-failure
 RestartSec=30
 # The runner waits up to 10 minutes for a running job (shutdown_timeout).
 TimeoutStopSec=11min
+# The nightly window's end: three hours after forgejo-runner.timer started it at 02:00. A
+# job still running is stopped, and the next one's Leftovers step clears what it left.
+RuntimeMaxSec=3h
 
 # Away from production: /home (deploy's .env, the checkout, the database dumps) is
-# invisible, the rest of the system read-only except the runner's own directory, other
-# users' processes hidden, and Docker's container networks, where the database listens,
-# unreachable, and so is the tunnel to Jellyfin at home (10.61.96.0/30, ADR 0032, D6).
+# invisible, and so is /data, where /home really lies, with Docker's data and the media,
+# but for the runner's own folder; the rest of the system is read-only, other users'
+# processes are hidden, and Docker and containerd out of reach.
 ProtectHome=yes
 ProtectSystem=strict
+TemporaryFileSystem=/data:ro
+BindPaths=/data/forgejo-runner:/var/lib/forgejo-runner
 ReadWritePaths=/var/lib/forgejo-runner
 PrivateTmp=yes
 ProtectProc=invisible
-InaccessiblePaths=-/run/docker.sock -/run/containerd -/var/lib/docker
-IPAddressDeny=172.16.0.0/12 192.168.0.0/16 fd00::/8 10.61.96.0/30
+InaccessiblePaths=-/run/docker.sock -/run/containerd -/var/lib/docker -/var/lib/containerd
+# The network: the internet over IPv4 alone. No private IPv4 network (the home network
+# and its router, Docker's networks, where the database listens, the tunnel to ramiel,
+# Multipass' bridges, CGNAT and link-local addresses), and no IPv6 at all: the home
+# network's IPv6 addresses are public ones, under a prefix the provider changes, on the
+# same interface as the internet, so no address or interface could tell them apart. All
+# CI fetches (pub.dev, GitHub, code.halkyon.app) answers over IPv4.
+IPAddressDeny=10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 169.254.0.0/16 ::/0
 NoNewPrivileges=yes
 CapabilityBoundingSet=
 AmbientCapabilities=
@@ -66,20 +86,19 @@ SystemCallFilter=~@clock @cpu-emulation @debug @module @mount @obsolete @raw-io
 SystemCallErrorNumber=EPERM
 UMask=0077
 
-# Never at the instance's expense: two of the four cores, at most 6 GB of the 8 (the
-# kernel reclaims from CI first above 5.5 GB), the lowest priority for CPU and disk, and
-# CI's processes the first the kernel kills should the whole server run out of memory.
-# A cold `dart analyze` of the workspace with its lint plugin needs about 5.4 GB since
-# Todo and the shared packages joined it (3.7 GB warm); below that the kernel slowed it
-# to a crawl and then killed it, and OOMPolicy's default then stopped the whole unit.
-CPUQuota=200%
-MemoryHigh=5.5G
-MemoryMax=6G
+# Never at the instance's expense, which keeps serving through the night: three of the
+# four cores (its servers idle then), at most 5 GB of the 7.2 (the kernel reclaims from CI
+# first above 4.5 GB), and swap beyond, the lowest priority for CPU and disk, and CI's
+# processes the first the kernel kills should the whole machine run out of memory. A
+# cold `dart analyze` of the workspace with its lint plugin needs about 5.4 GB (3.7 GB
+# warm), so it swaps a little rather than being killed: slower, within the window.
+CPUQuota=300%
+MemoryHigh=4.5G
+MemoryMax=5G
 OOMScoreAdjust=1000
 TasksMax=2048
 Nice=10
 IOSchedulingClass=best-effort
 IOSchedulingPriority=7
 
-[Install]
-WantedBy=multi-user.target
+# No [Install]: never started at boot, only by forgejo-runner.timer.

deploy/runner/forgejo-runner.timer

@@ -0,0 +1,15 @@
+# The CI runner's nightly window (ADR 0039, D6): started at 02:00, after night.sh has
+# rested the heavy sidecars (halkyon-night.timer, 01:55), and stopped by its own unit three
+# hours later (RuntimeMaxSec). Never started to catch up after a night the machine was
+# off: the queue waits for the next night. As /etc/systemd/system/forgejo-runner.timer,
+# enabled with `systemctl enable --now forgejo-runner.timer`.
+[Unit]
+Description=The CI runner's nightly window
+
+[Timer]
+OnCalendar=*-*-* 02:00:00
+Persistent=false
+Unit=forgejo-runner.service
+
+[Install]
+WantedBy=timers.target

deploy/runner/halkyon-morning.service

@@ -0,0 +1,12 @@
+# night.sh start as deploy: starts the heavy sidecars again after the CI runner's window (ADR 0039, D6). As
+# /etc/systemd/system/halkyon-morning.service, started by halkyon-morning.timer.
+[Unit]
+Description=halkyon.li: night.sh start
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+User=deploy
+WorkingDirectory=/home/deploy/halkyon/deploy
+ExecStart=/home/deploy/halkyon/deploy/night.sh start

deploy/runner/halkyon-morning.timer

@@ -0,0 +1,12 @@
+# halkyon-morning.service at 05:00:00 daily (ADR 0039, D6). As
+# /etc/systemd/system/halkyon-morning.timer, enabled with
+# `systemctl enable --now halkyon-morning.timer`.
+[Unit]
+Description=halkyon.li: night.sh start at 05:00:00
+
+[Timer]
+OnCalendar=*-*-* 05:00:00
+Persistent=false
+
+[Install]
+WantedBy=timers.target

deploy/runner/halkyon-night.service

@@ -0,0 +1,12 @@
+# night.sh stop as deploy: rests the heavy sidecars before the CI runner's window (ADR 0039, D6). As
+# /etc/systemd/system/halkyon-night.service, started by halkyon-night.timer.
+[Unit]
+Description=halkyon.li: night.sh stop
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+User=deploy
+WorkingDirectory=/home/deploy/halkyon/deploy
+ExecStart=/home/deploy/halkyon/deploy/night.sh stop

deploy/runner/halkyon-night.timer

@@ -0,0 +1,12 @@
+# halkyon-night.service at 01:55:00 daily (ADR 0039, D6). As
+# /etc/systemd/system/halkyon-night.timer, enabled with
+# `systemctl enable --now halkyon-night.timer`.
+[Unit]
+Description=halkyon.li: night.sh stop at 01:55:00
+
+[Timer]
+OnCalendar=*-*-* 01:55:00
+Persistent=false
+
+[Install]
+WantedBy=timers.target