2026-10-13 23:11:40 +02:00
|
|
|
# The Forgejo Actions runner of this repository's CI, on learn.halkyon.eu's server. Installed as
|
ci: check every push on sahaquiel and deploy a green main
Pushes to main were deployed by hand. Now Forgejo Actions runs tool/check.sh on every
push, and a green main deploys learn.halkyon.eu by itself.
The runner (forgejo-runner v13.1.0) runs on sahaquiel as the unprivileged user `runner`,
with jobs directly on the host: containers would take Docker access, which is root on the
production server, and Forgejo warns that host jobs have no isolation from the runner's
user. So its systemd unit (deploy/runner/forgejo-runner.service) provides it: /home and
the Docker socket invisible, Docker's container networks unreachable, no capabilities,
two cores and 3 GB at most, lowest priority. It is registered to this repository only,
with the UUID and token of Forgejo's "Create new runner" (the old `register` is
deprecated); the token reaches it as a systemd credential. The configuration
(deploy/runner/config.yml) was dry-run against v13.1.0, the unit rates 1.8 in
`systemd-analyze security`. CI runs on pushes only: a pull request from a fork would run
its code on the production server.
The deploy job calls deploy/redeploy.sh as `deploy`, through a key that authorized_keys
pins to that script and to connections from sahaquiel itself. The script deploys the
commit CI checked while it is still main's tip, as a fast-forward, only if every new
commit is signed by a key in deploy/allowed-signers (as the deployed checkout lists
them), after dumping the database (the last ten are kept), and fails unless the server
answers afterwards; a failure puts the checkout back. Tested against the local stack:
refusals of a malformed argument, a stale commit, a forged unsigned commit and a
non-fast-forward, a real deploy with its dump, a repeat, and a failed build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 21:49:04 +02:00
|
|
|
# /etc/systemd/system/forgejo-runner.service; its configuration is config.yml, next to
|
|
|
|
|
# this file.
|
|
|
|
|
#
|
|
|
|
|
# Jobs run directly on the host as the unprivileged user `runner` ("without any
|
|
|
|
|
# isolation" from that user, as Forgejo's security guide puts it), and they run our
|
|
|
|
|
# dependencies' build code. On the production server, so everything below keeps them
|
|
|
|
|
# away from production and makes sure CI can't starve the instance. Docker is out of
|
|
|
|
|
# reach entirely: `runner` isn't in the docker group, and the socket is hidden too.
|
ci: stop a job whose sandbox leaks, first thing
Installing the runner on sahaquiel, `useradd --system` gave `runner` user ID 999: the ID
under which PostgreSQL and the server run inside their containers, which Docker doesn't
remap. To the kernel, those processes were the runner's own, so a job could have read the
server's environment (database password, session pepper, GitHub secret) and, through
/proc, PostgreSQL's files and its local socket, which the official image trusts. The
sandbox's other walls don't help there: ProtectProc hides only other users' processes,
IPAddressDeny doesn't cover Unix sockets. Only our own check ran as 999; `runner` now has
user and group ID 61000.
The check job now starts with a Sandbox step that fails unless only the runner's own
processes are visible, all in one process namespace, /home/deploy is invisible, the
Docker socket is unusable and Docker's bridge network unreachable. It would have caught
this; outside a sandbox it fails as it should. The unit and AGENTS.md say how to create
the user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 22:12:31 +02:00
|
|
|
#
|
|
|
|
|
# `runner` has user and group ID 61000, and must never share a user ID with a
|
|
|
|
|
# container's processes (PostgreSQL and the server run as 999; Docker doesn't remap
|
|
|
|
|
# them): processes of one user ID can read each other's environment and, through
|
|
|
|
|
# /proc, their files. `useradd --system` picks 999 when it is free, so create it with
|
|
|
|
|
# useradd --system --uid 61000 --user-group --create-home \
|
|
|
|
|
# --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
|
|
|
|
|
# CI's first step checks this sandbox and fails if it leaks.
|
ci: check every push on sahaquiel and deploy a green main
Pushes to main were deployed by hand. Now Forgejo Actions runs tool/check.sh on every
push, and a green main deploys learn.halkyon.eu by itself.
The runner (forgejo-runner v13.1.0) runs on sahaquiel as the unprivileged user `runner`,
with jobs directly on the host: containers would take Docker access, which is root on the
production server, and Forgejo warns that host jobs have no isolation from the runner's
user. So its systemd unit (deploy/runner/forgejo-runner.service) provides it: /home and
the Docker socket invisible, Docker's container networks unreachable, no capabilities,
two cores and 3 GB at most, lowest priority. It is registered to this repository only,
with the UUID and token of Forgejo's "Create new runner" (the old `register` is
deprecated); the token reaches it as a systemd credential. The configuration
(deploy/runner/config.yml) was dry-run against v13.1.0, the unit rates 1.8 in
`systemd-analyze security`. CI runs on pushes only: a pull request from a fork would run
its code on the production server.
The deploy job calls deploy/redeploy.sh as `deploy`, through a key that authorized_keys
pins to that script and to connections from sahaquiel itself. The script deploys the
commit CI checked while it is still main's tip, as a fast-forward, only if every new
commit is signed by a key in deploy/allowed-signers (as the deployed checkout lists
them), after dumping the database (the last ten are kept), and fails unless the server
answers afterwards; a failure puts the checkout back. Tested against the local stack:
refusals of a malformed argument, a stale commit, a forged unsigned commit and a
non-fast-forward, a real deploy with its dump, a repeat, and a failed build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 21:49:04 +02:00
|
|
|
[Unit]
|
|
|
|
|
Description=Forgejo Actions runner (CI of halkyon_learn; jobs on the host)
|
|
|
|
|
Documentation=https://forgejo.org/docs/latest/admin/actions/security/
|
|
|
|
|
After=network-online.target
|
|
|
|
|
Wants=network-online.target
|
|
|
|
|
|
|
|
|
|
[Service]
|
|
|
|
|
User=runner
|
|
|
|
|
Group=runner
|
|
|
|
|
WorkingDirectory=/var/lib/forgejo-runner
|
|
|
|
|
ExecStart=/usr/local/bin/forgejo-runner daemon --config /etc/forgejo-runner/config.yml
|
|
|
|
|
# The runner's token, from a root-only file.
|
|
|
|
|
LoadCredential=token:/etc/forgejo-runner/token
|
|
|
|
|
Restart=on-failure
|
|
|
|
|
RestartSec=30
|
|
|
|
|
# The runner waits up to 10 minutes for a running job (shutdown_timeout).
|
|
|
|
|
TimeoutStopSec=11min
|
|
|
|
|
|
|
|
|
|
# Away from production: /home (deploy's .env, the checkout, the database dumps) is
|
|
|
|
|
# invisible, the rest of the system read-only except the runner's own directory, other
|
|
|
|
|
# users' processes hidden, and Docker's container networks, where the database listens,
|
feat(deploy): Jellyfin at home, published through WireGuard (ADR 0032)
The instance serves jellyfin.halkyon.eu and its Caddy proxies through a
WireGuard tunnel to a Jellyfin on the operator's own machine, which dials
out, so nothing names the home network's address and its router opens no
port. compose.jellyfin-home.yaml takes the sidecar's place, with its
site's rules unchanged (jellyfin.caddy's upstream is now a setting), and
leaves JELLYFIN_URL unset, so the instance suggests it only while it
answers.
On the instance's server the tunnel reaches Jellyfin from Caddy alone and
nothing of the host (wg-jellyfin-rules.sh, set and removed with the
tunnel); the CI runner's unit denies it, and CI's Sandbox step fails if a
job reaches it. The home end keeps Jellyfin in WireGuard's network
namespace, its port on the loopback alone. The steps are in
deploy/sidecars/README.md, the rules held by test/deploy/jellyfin_home_test.dart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-30 22:49:47 +01:00
|
|
|
# unreachable, and so is the tunnel to Jellyfin at home (10.61.96.0/30, ADR 0032, D6).
|
ci: check every push on sahaquiel and deploy a green main
Pushes to main were deployed by hand. Now Forgejo Actions runs tool/check.sh on every
push, and a green main deploys learn.halkyon.eu by itself.
The runner (forgejo-runner v13.1.0) runs on sahaquiel as the unprivileged user `runner`,
with jobs directly on the host: containers would take Docker access, which is root on the
production server, and Forgejo warns that host jobs have no isolation from the runner's
user. So its systemd unit (deploy/runner/forgejo-runner.service) provides it: /home and
the Docker socket invisible, Docker's container networks unreachable, no capabilities,
two cores and 3 GB at most, lowest priority. It is registered to this repository only,
with the UUID and token of Forgejo's "Create new runner" (the old `register` is
deprecated); the token reaches it as a systemd credential. The configuration
(deploy/runner/config.yml) was dry-run against v13.1.0, the unit rates 1.8 in
`systemd-analyze security`. CI runs on pushes only: a pull request from a fork would run
its code on the production server.
The deploy job calls deploy/redeploy.sh as `deploy`, through a key that authorized_keys
pins to that script and to connections from sahaquiel itself. The script deploys the
commit CI checked while it is still main's tip, as a fast-forward, only if every new
commit is signed by a key in deploy/allowed-signers (as the deployed checkout lists
them), after dumping the database (the last ten are kept), and fails unless the server
answers afterwards; a failure puts the checkout back. Tested against the local stack:
refusals of a malformed argument, a stale commit, a forged unsigned commit and a
non-fast-forward, a real deploy with its dump, a repeat, and a failed build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 21:49:04 +02:00
|
|
|
ProtectHome=yes
|
|
|
|
|
ProtectSystem=strict
|
|
|
|
|
ReadWritePaths=/var/lib/forgejo-runner
|
|
|
|
|
PrivateTmp=yes
|
|
|
|
|
ProtectProc=invisible
|
|
|
|
|
InaccessiblePaths=-/run/docker.sock -/run/containerd -/var/lib/docker
|
feat(deploy): Jellyfin at home, published through WireGuard (ADR 0032)
The instance serves jellyfin.halkyon.eu and its Caddy proxies through a
WireGuard tunnel to a Jellyfin on the operator's own machine, which dials
out, so nothing names the home network's address and its router opens no
port. compose.jellyfin-home.yaml takes the sidecar's place, with its
site's rules unchanged (jellyfin.caddy's upstream is now a setting), and
leaves JELLYFIN_URL unset, so the instance suggests it only while it
answers.
On the instance's server the tunnel reaches Jellyfin from Caddy alone and
nothing of the host (wg-jellyfin-rules.sh, set and removed with the
tunnel); the CI runner's unit denies it, and CI's Sandbox step fails if a
job reaches it. The home end keeps Jellyfin in WireGuard's network
namespace, its port on the loopback alone. The steps are in
deploy/sidecars/README.md, the rules held by test/deploy/jellyfin_home_test.dart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-30 22:49:47 +01:00
|
|
|
IPAddressDeny=172.16.0.0/12 192.168.0.0/16 fd00::/8 10.61.96.0/30
|
ci: check every push on sahaquiel and deploy a green main
Pushes to main were deployed by hand. Now Forgejo Actions runs tool/check.sh on every
push, and a green main deploys learn.halkyon.eu by itself.
The runner (forgejo-runner v13.1.0) runs on sahaquiel as the unprivileged user `runner`,
with jobs directly on the host: containers would take Docker access, which is root on the
production server, and Forgejo warns that host jobs have no isolation from the runner's
user. So its systemd unit (deploy/runner/forgejo-runner.service) provides it: /home and
the Docker socket invisible, Docker's container networks unreachable, no capabilities,
two cores and 3 GB at most, lowest priority. It is registered to this repository only,
with the UUID and token of Forgejo's "Create new runner" (the old `register` is
deprecated); the token reaches it as a systemd credential. The configuration
(deploy/runner/config.yml) was dry-run against v13.1.0, the unit rates 1.8 in
`systemd-analyze security`. CI runs on pushes only: a pull request from a fork would run
its code on the production server.
The deploy job calls deploy/redeploy.sh as `deploy`, through a key that authorized_keys
pins to that script and to connections from sahaquiel itself. The script deploys the
commit CI checked while it is still main's tip, as a fast-forward, only if every new
commit is signed by a key in deploy/allowed-signers (as the deployed checkout lists
them), after dumping the database (the last ten are kept), and fails unless the server
answers afterwards; a failure puts the checkout back. Tested against the local stack:
refusals of a malformed argument, a stale commit, a forged unsigned commit and a
non-fast-forward, a real deploy with its dump, a repeat, and a failed build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 21:49:04 +02:00
|
|
|
NoNewPrivileges=yes
|
|
|
|
|
CapabilityBoundingSet=
|
|
|
|
|
AmbientCapabilities=
|
|
|
|
|
SystemCallArchitectures=native
|
|
|
|
|
RestrictSUIDSGID=yes
|
|
|
|
|
ProtectKernelTunables=yes
|
|
|
|
|
ProtectKernelModules=yes
|
|
|
|
|
ProtectKernelLogs=yes
|
|
|
|
|
ProtectControlGroups=yes
|
|
|
|
|
ProtectClock=yes
|
|
|
|
|
ProtectHostname=yes
|
|
|
|
|
LockPersonality=yes
|
|
|
|
|
RestrictRealtime=yes
|
|
|
|
|
RestrictNamespaces=yes
|
|
|
|
|
RemoveIPC=yes
|
|
|
|
|
# Netlink stays: Dart lists network interfaces through it.
|
|
|
|
|
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
|
|
|
|
|
# System calls no CI job needs fail with an error rather than killing the job.
|
|
|
|
|
SystemCallFilter=~@clock @cpu-emulation @debug @module @mount @obsolete @raw-io @reboot @swap
|
|
|
|
|
SystemCallErrorNumber=EPERM
|
|
|
|
|
UMask=0077
|
|
|
|
|
|
2026-10-22 19:04:57 +02:00
|
|
|
# Never at the instance's expense: two of the four cores, at most 6 GB of the 8 (the
|
|
|
|
|
# kernel reclaims from CI first above 5.5 GB), the lowest priority for CPU and disk, and
|
|
|
|
|
# CI's processes the first the kernel kills should the whole server run out of memory.
|
|
|
|
|
# A cold `dart analyze` of the workspace with its lint plugin needs about 5.4 GB since
|
|
|
|
|
# Todo and the shared packages joined it (3.7 GB warm); below that the kernel slowed it
|
|
|
|
|
# to a crawl and then killed it, and OOMPolicy's default then stopped the whole unit.
|
ci: check every push on sahaquiel and deploy a green main
Pushes to main were deployed by hand. Now Forgejo Actions runs tool/check.sh on every
push, and a green main deploys learn.halkyon.eu by itself.
The runner (forgejo-runner v13.1.0) runs on sahaquiel as the unprivileged user `runner`,
with jobs directly on the host: containers would take Docker access, which is root on the
production server, and Forgejo warns that host jobs have no isolation from the runner's
user. So its systemd unit (deploy/runner/forgejo-runner.service) provides it: /home and
the Docker socket invisible, Docker's container networks unreachable, no capabilities,
two cores and 3 GB at most, lowest priority. It is registered to this repository only,
with the UUID and token of Forgejo's "Create new runner" (the old `register` is
deprecated); the token reaches it as a systemd credential. The configuration
(deploy/runner/config.yml) was dry-run against v13.1.0, the unit rates 1.8 in
`systemd-analyze security`. CI runs on pushes only: a pull request from a fork would run
its code on the production server.
The deploy job calls deploy/redeploy.sh as `deploy`, through a key that authorized_keys
pins to that script and to connections from sahaquiel itself. The script deploys the
commit CI checked while it is still main's tip, as a fast-forward, only if every new
commit is signed by a key in deploy/allowed-signers (as the deployed checkout lists
them), after dumping the database (the last ten are kept), and fails unless the server
answers afterwards; a failure puts the checkout back. Tested against the local stack:
refusals of a malformed argument, a stale commit, a forged unsigned commit and a
non-fast-forward, a real deploy with its dump, a repeat, and a failed build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 21:49:04 +02:00
|
|
|
CPUQuota=200%
|
2026-10-22 19:04:57 +02:00
|
|
|
MemoryHigh=5.5G
|
|
|
|
|
MemoryMax=6G
|
|
|
|
|
OOMScoreAdjust=1000
|
ci: check every push on sahaquiel and deploy a green main
Pushes to main were deployed by hand. Now Forgejo Actions runs tool/check.sh on every
push, and a green main deploys learn.halkyon.eu by itself.
The runner (forgejo-runner v13.1.0) runs on sahaquiel as the unprivileged user `runner`,
with jobs directly on the host: containers would take Docker access, which is root on the
production server, and Forgejo warns that host jobs have no isolation from the runner's
user. So its systemd unit (deploy/runner/forgejo-runner.service) provides it: /home and
the Docker socket invisible, Docker's container networks unreachable, no capabilities,
two cores and 3 GB at most, lowest priority. It is registered to this repository only,
with the UUID and token of Forgejo's "Create new runner" (the old `register` is
deprecated); the token reaches it as a systemd credential. The configuration
(deploy/runner/config.yml) was dry-run against v13.1.0, the unit rates 1.8 in
`systemd-analyze security`. CI runs on pushes only: a pull request from a fork would run
its code on the production server.
The deploy job calls deploy/redeploy.sh as `deploy`, through a key that authorized_keys
pins to that script and to connections from sahaquiel itself. The script deploys the
commit CI checked while it is still main's tip, as a fast-forward, only if every new
commit is signed by a key in deploy/allowed-signers (as the deployed checkout lists
them), after dumping the database (the last ten are kept), and fails unless the server
answers afterwards; a failure puts the checkout back. Tested against the local stack:
refusals of a malformed argument, a stale commit, a forged unsigned commit and a
non-fast-forward, a real deploy with its dump, a repeat, and a failed build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-09-30 21:49:04 +02:00
|
|
|
TasksMax=2048
|
|
|
|
|
Nice=10
|
|
|
|
|
IOSchedulingClass=best-effort
|
|
|
|
|
IOSchedulingPriority=7
|
|
|
|
|
|
|
|
|
|
[Install]
|
|
|
|
|
WantedBy=multi-user.target
|