ci: a job first stops what earlier jobs left running

On 22 Oct a cancelled job left its analysis server running in the
runner, 4 GB, until someone noticed. forgejo-runner v13.1.0 starts every
host step as a process group and sends the group SIGKILL on a cancel or
at timeout-minutes (act/common/cmdgroup.go), and after that no later step
runs, `if: always()` ones neither (they get the cancelled context; only
actions' post steps get a fresh one), so a cleanup at the job's end
can't be relied on. The check job's second step, after the sandbox's
proof, stops every process of a job's working folder (`$HOME/work/`,
also one removed since) that is neither the runner nor the step itself,
TERM then KILL after 5 s, and names each in the log. The runner takes
one job at a time (capacity: 1), so nothing of a running job is touched.
No change on the server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-23 15:22:17 +02:00
commit c4ad437fc1
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64

View file

@ -84,5 +84,5 @@ Checklist after doing changes, in this order:
## CI and deploys
- **CI** (`.forgejo/workflows/ci.yaml`) runs `dart run tool/check.dart` on every push of a branch (not of a tag: a release pushes its tag right after `main`, on the same commit), on the Forgejo runner on learn.halkyon.eu's server (`deploy/runner/`), except for a commit that passed before: the job marks passed commits in the runner's home (`checked/`, kept 90 days), so a branch's commit fast-forwarded into `main` goes straight to its deploy. It checks every package, whichever a push changed. Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and Learn's server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. Every server package's tests get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since no `passwords.yaml` is ever committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI installs the Serverpod CLI at `SERVERPOD_VERSION` (which the root's tests hold to the lock's Serverpod), so the checks' generated-code step fails there too if the committed generated code of any server differs from what the generator makes, as it does on the laptop since 23 Oct 2026 (before, only CI compared, and a push went red for an endpoint's changed comment). So never run `dart format` over a server's `lib/` or `test/` as a whole: it reformats the generated code (`lib/src/generated/`, `serverpod_test_tools.dart`); format the files you wrote. CI runs on pushes only: a pull request from a fork would run its code on the production server.
- **CI** (`.forgejo/workflows/ci.yaml`) runs `dart run tool/check.dart` on every push of a branch (not of a tag: a release pushes its tag right after `main`, on the same commit), on the Forgejo runner on learn.halkyon.eu's server (`deploy/runner/`), except for a commit that passed before: the job marks passed commits in the runner's home (`checked/`, kept 90 days), so a branch's commit fast-forwarded into `main` goes straight to its deploy. It checks every package, whichever a push changed. Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and Learn's server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Its second (*Leftovers*) stops whatever an earlier job left running in a job's working folder: the runner (forgejo-runner v13.1.0) kills a cancelled or timed-out step's process group and runs no later step, not even an `if: always()` one, and on 22 Oct 2026 a cancelled job still left its 4 GB analysis server behind; with one job at a time, any such process is a leftover. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. Every server package's tests get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since no `passwords.yaml` is ever committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI installs the Serverpod CLI at `SERVERPOD_VERSION` (which the root's tests hold to the lock's Serverpod), so the checks' generated-code step fails there too if the committed generated code of any server differs from what the generator makes, as it does on the laptop since 23 Oct 2026 (before, only CI compared, and a push went red for an endpoint's changed comment). So never run `dart format` over a server's `lib/` or `test/` as a whole: it reformats the generated code (`lib/src/generated/`, `serverpod_test_tools.dart`); format the files you wrote. CI runs on pushes only: a pull request from a fork would run its code on the production server.
- **A green `main` deploys learn.halkyon.eu**: this repository's `main`, whichever app or package a push changed, through `deploy/redeploy.sh` (Learn's `AGENTS.md`, *CI and deploys*, with the rules a deploy checks, the sidecars, the backups and the releases).