feat(deploy): hourly off-site backups with restic
All checks were successful
CI / check (push) Successful in 9m0s
CI / deploy (push) Successful in 7s

learn.halkyon.eu dumps its database every hour (people's files are in it
too) and restic stores the dump encrypted in a Hetzner Storage Box,
keeping 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots. The box's
own snapshots, removable only in Hetzner's Console, protect the backups
from a taken-over server. A systemd timer runs it as deploy; once a day
it also prunes and checks the repository. Failures go to the journal.

Hourly, because a restore costs people who started after the backup
what their device held: the app no longer finds its session and starts
anew. deploy/backups.md describes the setup, the restore and what a
restore loses; both were rehearsed on the local Compose stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-03 16:45:40 +02:00
commit 999a3ce0b0
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64

View file

@ -119,6 +119,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
- **CI** (`.forgejo/workflows/ci.yaml`) runs `tool/check.sh` on every push, on the Forgejo runner on sahaquiel (`deploy/runner/`). Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and the server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. The server tests there get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since `passwords.yaml` is never committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI also runs `serverpod generate` and fails if the committed generated code differs from its output. CI runs on pushes only: a pull request from a fork would run its code on the production server.
- **A green `main` deploys learn.halkyon.eu** through `deploy/redeploy.sh`, the only command the runner's key may run as `deploy` (pinned in `authorized_keys`, from sahaquiel itself only). It deploys the commit CI checked while it is still the tip, as a fast-forward, only if every new commit is signed by a key in `deploy/allowed-signers` (the deployed copy decides), after dumping the database (the last 3 dumps stay in `~deploy/backups`: the way back from a deploy, not backups), applies the migrations in a run of their own before switching (`--role=maintenance --apply-migrations`: a failing migration stops the deploy while the old server keeps serving, so migrations must keep the old server working until the switch), and fails unless the new server answers. A failed deploy leaves the checkout at the deployed commit; run the workflow again to retry. By hand: `ssh sahaquiel-deploy halkyon-learn/deploy/redeploy.sh <commit>`. Never `docker compose down -v` there.
- **Backups** (`deploy/backups.md`): every hour, `deploy/backup.sh` (a systemd timer, as `deploy`) dumps the database, which holds people's files too, and restic stores it in a Hetzner Storage Box (24 hourly, 7 daily, 4 weekly, 3 monthly); the box's own snapshots protect it from a taken-over server. A restore takes the instance back up to an hour, and people who started since lose what their device held: their app no longer finds its session, starts a new anonymous account and deletes the old folder. So a change that makes devices lose their session, or keeps deleted people's data longer, changes what a restore costs; say so in `deploy/backups.md`.
## Rules the code relies on