ci: stop a job whose sandbox leaks, first thing

Installing the runner on sahaquiel, `useradd --system` gave `runner` user ID 999: the ID
under which PostgreSQL and the server run inside their containers, which Docker doesn't
remap. To the kernel, those processes were the runner's own, so a job could have read the
server's environment (database password, session pepper, GitHub secret) and, through
/proc, PostgreSQL's files and its local socket, which the official image trusts. The
sandbox's other walls don't help there: ProtectProc hides only other users' processes,
IPAddressDeny doesn't cover Unix sockets. Only our own check ran as 999; `runner` now has
user and group ID 61000.

The check job now starts with a Sandbox step that fails unless only the runner's own
processes are visible, all in one process namespace, /home/deploy is invisible, the
Docker socket is unusable and Docker's bridge network unreachable. It would have caught
this; outside a sandbox it fails as it should. The unit and AGENTS.md say how to create
the user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

.forgejo/workflows/ci.yaml

@@ -19,6 +19,28 @@ jobs:
       # The Flutter the app is developed with; move it together with the local one.
       FLUTTER_VERSION: 3.47.5
     steps:
+      # Before anything else runs, the job proves it can neither see nor reach production
+      # (the runner's sandbox, deploy/runner/). If it can, nothing else should run.
+      - name: Sandbox
+        run: |
+          fail() { echo "The runner's sandbox leaks: $*" >&2; exit 1; }
+          # Only the runner's own processes are visible. A container's process shows up
+          # (in a process namespace of its own) if it runs under the runner's user ID,
+          # and then its environment and files are readable too.
+          [ "$(ps -e -o uid= | sort -u | wc -l)" -eq 1 ] || fail "other users' processes are visible"
+          namespaces=$(for p in /proc/[0-9]*; do readlink "$p/ns/pid"; done 2>/dev/null | sort -u | wc -l)
+          [ "$namespaces" -le 1 ] || fail "processes of $namespaces process namespaces are visible"
+          # deploy's home (.env, the checkout, the dumps) and the Docker socket are out of
+          # reach.
+          [ ! -e /home/deploy ] || fail "/home/deploy is visible"
+          { [ ! -r /run/docker.sock ] && [ ! -w /run/docker.sock ]; } || fail "the Docker socket is usable"
+          # Docker's networks, where the database listens, are unreachable. Caddy listens on
+          # every address of the host, the Docker bridge's included.
+          if timeout 5 bash -c 'exec 3<>/dev/tcp/172.17.0.1/443' 2>/dev/null; then
+            fail "Docker's network is reachable"
+          fi
+          echo "This job can neither see nor reach production."
+
       - name: Check out
         env:
           TOKEN: ${{ github.token }}

AGENTS.md

@@ -106,7 +106,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
 
 ## CI and deploys
 
-- **CI** (`.forgejo/workflows/ci.yaml`) runs `tool/check.sh` on every push, on the Forgejo runner on sahaquiel (`deploy/runner/`). Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. CI runs on pushes only: a pull request from a fork would run its code on the production server.
+- **CI** (`.forgejo/workflows/ci.yaml`) runs `tool/check.sh` on every push, on the Forgejo runner on sahaquiel (`deploy/runner/`). Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and the server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. CI runs on pushes only: a pull request from a fork would run its code on the production server.
 - **A green `main` deploys learn.halkyon.eu** through `deploy/redeploy.sh`, the only command the runner's key may run as `deploy` (pinned in `authorized_keys`, from sahaquiel itself only). It deploys the commit CI checked while it is still the tip, as a fast-forward, only if every new commit is signed by a key in `deploy/allowed-signers` (the deployed copy decides), after dumping the database (the last 10 dumps stay in `~deploy/backups`), and fails unless the server answers. A failed deploy leaves the checkout at the deployed commit; run the workflow again to retry. By hand: `ssh sahaquiel-deploy halkyon-learn/deploy/redeploy.sh <commit>`. Never `docker compose down -v` there.
 
 ## Rules the code relies on

deploy/runner/forgejo-runner.service

@@ -7,6 +7,14 @@
 # dependencies' build code. On the production server, so everything below keeps them
 # away from production and makes sure CI can't starve the instance. Docker is out of
 # reach entirely: `runner` isn't in the docker group, and the socket is hidden too.
+#
+# `runner` has user and group ID 61000, and must never share a user ID with a
+# container's processes (PostgreSQL and the server run as 999; Docker doesn't remap
+# them): processes of one user ID can read each other's environment and, through
+# /proc, their files. `useradd --system` picks 999 when it is free, so create it with
+#   useradd --system --uid 61000 --user-group --create-home \
+#     --home-dir /var/lib/forgejo-runner --shell /usr/sbin/nologin runner
+# CI's first step checks this sandbox and fails if it leaks.
 [Unit]
 Description=Forgejo Actions runner (CI of halkyon_learn; jobs on the host)
 Documentation=https://forgejo.org/docs/latest/admin/actions/security/