The CI runner moves to halkyon.li's host at home and runs from 02:00 to
05:00 alone: a timer starts it, its unit's RuntimeMaxSec ends it, and
night.sh rests the instance's heavy sidecars around it as deploy, while
Caddy, PostgreSQL and the apps' servers keep serving. Jobs pushed during the
day wait in Forgejo's queue.
Its sandbox fits the new host: its folder on the HDD, the rest of /data
hidden, containerd's store out of reach, no private IPv4 network and no
IPv6 at all (the home network's IPv6 addresses are public ones under a
changing prefix, on the internet's own interface), 5 GB of memory with swap
beyond. The Sandbox step checks the tunnel, the home router and IPv6.
BREAKING CHANGE: CI deploys nothing any more: the deploy job goes with
learn.halkyon.eu's server, and halkyon.li deploys by hand with redeploy.sh
once a commit's night's run passed. Jobs run on the runner labelled
balthasar; sahaquiel's runner picks up none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The instance serves jellyfin.halkyon.eu and its Caddy proxies through a
WireGuard tunnel to a Jellyfin on the operator's own machine, which dials
out, so nothing names the home network's address and its router opens no
port. compose.jellyfin-home.yaml takes the sidecar's place, with its
site's rules unchanged (jellyfin.caddy's upstream is now a setting), and
leaves JELLYFIN_URL unset, so the instance suggests it only while it
answers.
On the instance's server the tunnel reaches Jellyfin from Caddy alone and
nothing of the host (wg-jellyfin-rules.sh, set and removed with the
tunnel); the CI runner's unit denies it, and CI's Sandbox step fails if a
job reaches it. The home end keeps Jellyfin in WireGuard's network
namespace, its port on the loopback alone. The steps are in
deploy/sidecars/README.md, the rules held by test/deploy/jellyfin_home_test.dart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Caddy's network had IPv4 alone, so Docker's userland proxy forwarded
every IPv6 visitor to Caddy from the network's gateway, and the server
counted all of them as one network (the second reading's M5, reasoned).
The edge network gets a unique local /64 (fd4a:6c6b:7e80::/64), Caddy a
fixed address in it beside its IPv4 one, both trusted as the proxy; the
CI runner's unit denies fd00::/8 and its sandbox check tries Caddy's IPv6
address too. A commit of its own: it needs Docker 27 or later with
ip6tables on the server and the network made anew (compose down, never
-v, then up), checked by hand first (NEXT.md).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
With Todo and the shared packages in the workspace, a cold `dart analyze`
peaks at about 5.4 GB (3.7 GB warm). Above the 4 GB soft limit the
kernel slowed it to a crawl, and at 5 GB it killed the job. Analysing
parts from the root drops the lint plugin's rules, so the limit rises
instead, and the kernel kills CI first should the server run out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The servers' names are no secret, so the settings that carry them stay
as they were, and nothing on a server or a laptop needs changing: CI
runs on `sahaquiel` again, the release uploads to `ramiel-fdroid`, and
deploys by hand go through `sahaquiel-deploy`. Prose still names a
server by the domain it serves, which says what a reader needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Files called the servers by their own host names, which are the
operator's and say nothing a reader needs. They now say which domain a
server serves: learn.halkyon.eu's server (the instance and its CI
runner), the halkyon.app server (Forgejo and the F-Droid repository).
Two names are settings rather than words. CI's runner label becomes
`instance` (`runs-on: instance`, `labels: ["instance:host"]`): the
runner reads its installed copy of deploy/runner/config.yml, so jobs
wait in the queue until that copy is updated and the runner restarted.
The SSH aliases the scripts and docs use become `learn-deploy` and
`fdroid-upload`. AGENTS.md says to keep it so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Decided 2026-10-01. CI run #32 died in its analysis: a cold `dart analyze`
of the workspace with the lint plugin now peaks at about 3.7 GB (2.1 GB
warm), measured with an empty analysis cache, while the runner's unit
allowed 3 GB. The kernel killed it, systemd's default OOMPolicy (stop) then
stopped the whole unit, and the job's remaining processes got SIGTERM: the
"signal: terminated" in the log. The packages this branch added tipped it
over.
The runner may now use 5 GB of sahaquiel's 8, and above 4 GB the kernel
reclaims from CI first (MemoryHigh), so the instance keeps at least 3 GB;
it uses under 1 GB today. Installed by hand on sahaquiel, like the rest of
the unit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Installing the runner on sahaquiel, `useradd --system` gave `runner` user ID 999: the ID
under which PostgreSQL and the server run inside their containers, which Docker doesn't
remap. To the kernel, those processes were the runner's own, so a job could have read the
server's environment (database password, session pepper, GitHub secret) and, through
/proc, PostgreSQL's files and its local socket, which the official image trusts. The
sandbox's other walls don't help there: ProtectProc hides only other users' processes,
IPAddressDeny doesn't cover Unix sockets. Only our own check ran as 999; `runner` now has
user and group ID 61000.
The check job now starts with a Sandbox step that fails unless only the runner's own
processes are visible, all in one process namespace, /home/deploy is invisible, the
Docker socket is unusable and Docker's bridge network unreachable. It would have caught
this; outside a sandbox it fails as it should. The unit and AGENTS.md say how to create
the user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Pushes to main were deployed by hand. Now Forgejo Actions runs tool/check.sh on every
push, and a green main deploys learn.halkyon.eu by itself.
The runner (forgejo-runner v13.1.0) runs on sahaquiel as the unprivileged user `runner`,
with jobs directly on the host: containers would take Docker access, which is root on the
production server, and Forgejo warns that host jobs have no isolation from the runner's
user. So its systemd unit (deploy/runner/forgejo-runner.service) provides it: /home and
the Docker socket invisible, Docker's container networks unreachable, no capabilities,
two cores and 3 GB at most, lowest priority. It is registered to this repository only,
with the UUID and token of Forgejo's "Create new runner" (the old `register` is
deprecated); the token reaches it as a systemd credential. The configuration
(deploy/runner/config.yml) was dry-run against v13.1.0, the unit rates 1.8 in
`systemd-analyze security`. CI runs on pushes only: a pull request from a fork would run
its code on the production server.
The deploy job calls deploy/redeploy.sh as `deploy`, through a key that authorized_keys
pins to that script and to connections from sahaquiel itself. The script deploys the
commit CI checked while it is still main's tip, as a fast-forward, only if every new
commit is signed by a key in deploy/allowed-signers (as the deployed checkout lists
them), after dumping the database (the last ten are kept), and fails unless the server
answers afterwards; a failure puts the checkout back. Tested against the local stack:
refusals of a malformed argument, a stale commit, a forged unsigned commit and a
non-fast-forward, a real deploy with its dump, a repeat, and a failed build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
