ci: a job first stops what earlier jobs left running

On 22 Oct a cancelled job left its analysis server running in the
runner, 4 GB, until someone noticed. forgejo-runner v13.1.0 starts every
host step as a process group and sends the group SIGKILL on a cancel or
at timeout-minutes (act/common/cmdgroup.go), and after that no later step
runs, `if: always()` ones neither (they get the cancelled context; only
actions' post steps get a fresh one), so a cleanup at the job's end
can't be relied on. The check job's second step, after the sandbox's
proof, stops every process of a job's working folder (`$HOME/work/`,
also one removed since) that is neither the runner nor the step itself,
TERM then KILL after 5 s, and names each in the log. The runner takes
one job at a time (capacity: 1), so nothing of a running job is touched.
No change on the server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

.forgejo/workflows/ci.yaml

@@ -55,6 +55,44 @@ jobs:
           fi
           echo "This job can neither see nor reach production."
 
+      # A job cancelled, or stopped at timeout-minutes, is killed with its step's process
+      # group (forgejo-runner v13.1.0 starts every host step as a group and sends it
+      # SIGKILL), and no later step runs, `if: always()` ones neither. Yet on 22 Oct a
+      # cancelled job left its analysis server running, 4 GB, until someone noticed. The
+      # runner takes one job at a time (capacity: 1), so a process here in a job's working
+      # folder that is neither the runner nor this step is such a leftover: it is stopped
+      # before anything else starts, and the log names it.
+      - name: Leftovers
+        run: |
+          own_group=$(ps -o pgid= -p $$ | tr -d ' ')
+          # This step and the runner above it.
+          keep=' '
+          pid=$$
+          while [ -n "$pid" ] && [ "$pid" -gt 1 ]; do
+            keep="$keep$pid "
+            pid=$(ps -o ppid= -p "$pid" | tr -d ' ')
+          done
+          left=$(ps -e -o pid=,pgid= | while read -r pid pgid; do
+            case "$keep" in *" $pid "*) continue ;; esac
+            [ "$pgid" != "$own_group" ] || continue
+            # Only what a job started: its working folder, or one since removed.
+            case "$(readlink "/proc/$pid/cwd" 2>/dev/null)" in
+              "$HOME/work/"*) echo "$pid" ;;
+            esac
+          done)
+          if [ -z "$left" ]; then
+            echo 'Nothing left behind by earlier jobs.'
+            exit 0
+          fi
+          echo 'Left behind by earlier jobs, stopped now:'
+          # shellcheck disable=SC2086 # one process ID a word
+          ps -o pid=,etime=,rss=,args= -p "$(echo $left | tr ' ' ,)" | cut -c1-200
+          # shellcheck disable=SC2086
+          kill -TERM $left 2>/dev/null || true
+          sleep 5
+          # shellcheck disable=SC2086
+          kill -KILL $left 2>/dev/null || true
+
       # A commit that passed this job before isn't checked again: a fast-forward merge
       # brings the commit a branch's push checked to main unchanged, and only its deploy
       # is new. The marks live in the runner's home; only pushes run this workflow, so

AGENTS.md

@@ -84,5 +84,5 @@ Checklist after doing changes, in this order:
 
 ## CI and deploys
 
-- **CI** (`.forgejo/workflows/ci.yaml`) runs `dart run tool/check.dart` on every push of a branch (not of a tag: a release pushes its tag right after `main`, on the same commit), on the Forgejo runner on learn.halkyon.eu's server (`deploy/runner/`), except for a commit that passed before: the job marks passed commits in the runner's home (`checked/`, kept 90 days), so a branch's commit fast-forwarded into `main` goes straight to its deploy. It checks every package, whichever a push changed. Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and Learn's server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. Every server package's tests get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since no `passwords.yaml` is ever committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI installs the Serverpod CLI at `SERVERPOD_VERSION` (which the root's tests hold to the lock's Serverpod), so the checks' generated-code step fails there too if the committed generated code of any server differs from what the generator makes, as it does on the laptop since 23 Oct 2026 (before, only CI compared, and a push went red for an endpoint's changed comment). So never run `dart format` over a server's `lib/` or `test/` as a whole: it reformats the generated code (`lib/src/generated/`, `serverpod_test_tools.dart`); format the files you wrote. CI runs on pushes only: a pull request from a fork would run its code on the production server.
+- **CI** (`.forgejo/workflows/ci.yaml`) runs `dart run tool/check.dart` on every push of a branch (not of a tag: a release pushes its tag right after `main`, on the same commit), on the Forgejo runner on learn.halkyon.eu's server (`deploy/runner/`), except for a commit that passed before: the job marks passed commits in the runner's home (`checked/`, kept 90 days), so a branch's commit fast-forwarded into `main` goes straight to its deploy. It checks every package, whichever a push changed. Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and Learn's server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Its second (*Leftovers*) stops whatever an earlier job left running in a job's working folder: the runner (forgejo-runner v13.1.0) kills a cancelled or timed-out step's process group and runs no later step, not even an `if: always()` one, and on 22 Oct 2026 a cancelled job still left its 4 GB analysis server behind; with one job at a time, any such process is a leftover. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. Every server package's tests get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since no `passwords.yaml` is ever committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI installs the Serverpod CLI at `SERVERPOD_VERSION` (which the root's tests hold to the lock's Serverpod), so the checks' generated-code step fails there too if the committed generated code of any server differs from what the generator makes, as it does on the laptop since 23 Oct 2026 (before, only CI compared, and a push went red for an endpoint's changed comment). So never run `dart format` over a server's `lib/` or `test/` as a whole: it reformats the generated code (`lib/src/generated/`, `serverpod_test_tools.dart`); format the files you wrote. CI runs on pushes only: a pull request from a fork would run its code on the production server.
 - **A green `main` deploys learn.halkyon.eu**: this repository's `main`, whichever app or package a push changed, through `deploy/redeploy.sh` (Learn's `AGENTS.md`, *CI and deploys*, with the rules a deploy checks, the sidecars, the backups and the releases).