feat(deploy): hourly off-site backups with restic
learn.halkyon.eu dumps its database every hour (people's files are in it too) and restic stores the dump encrypted in a Hetzner Storage Box, keeping 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots. The box's own snapshots, removable only in Hetzner's Console, protect the backups from a taken-over server. A systemd timer runs it as deploy; once a day it also prunes and checks the repository. Failures go to the journal. Hourly, because a restore costs people who started after the backup what their device held: the app no longer finds its session and starts anew. deploy/backups.md describes the setup, the restore and what a restore loses; both were rehearsed on the local Compose stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
fbed29c5c7
commit
999a3ce0b0
1 changed files with 270 additions and 1 deletions
|
|
@ -119,6 +119,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
|
|||
|
||||
- **CI** (`.forgejo/workflows/ci.yaml`) runs `tool/check.sh` on every push, on the Forgejo runner on sahaquiel (`deploy/runner/`). Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and the server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. The server tests there get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since `passwords.yaml` is never committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI also runs `serverpod generate` and fails if the committed generated code differs from its output. CI runs on pushes only: a pull request from a fork would run its code on the production server.
|
||||
- **A green `main` deploys learn.halkyon.eu** through `deploy/redeploy.sh`, the only command the runner's key may run as `deploy` (pinned in `authorized_keys`, from sahaquiel itself only). It deploys the commit CI checked while it is still the tip, as a fast-forward, only if every new commit is signed by a key in `deploy/allowed-signers` (the deployed copy decides), after dumping the database (the last 3 dumps stay in `~deploy/backups`: the way back from a deploy, not backups), applies the migrations in a run of their own before switching (`--role=maintenance --apply-migrations`: a failing migration stops the deploy while the old server keeps serving, so migrations must keep the old server working until the switch), and fails unless the new server answers. A failed deploy leaves the checkout at the deployed commit; run the workflow again to retry. By hand: `ssh sahaquiel-deploy halkyon-learn/deploy/redeploy.sh <commit>`. Never `docker compose down -v` there.
|
||||
- **Backups** (`deploy/backups.md`): every hour, `deploy/backup.sh` (a systemd timer, as `deploy`) dumps the database, which holds people's files too, and restic stores it in a Hetzner Storage Box (24 hourly, 7 daily, 4 weekly, 3 monthly); the box's own snapshots protect it from a taken-over server. A restore takes the instance back up to an hour, and people who started since lose what their device held: their app no longer finds its session, starts a new anonymous account and deletes the old folder. So a change that makes devices lose their session, or keeps deleted people's data longer, changes what a restore costs; say so in `deploy/backups.md`.
|
||||
|
||||
## Rules the code relies on
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue