feat(deploy): hourly off-site backups with restic

learn.halkyon.eu dumps its database every hour (people's files are in it
too) and restic stores the dump encrypted in a Hetzner Storage Box,
keeping 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots. The box's
own snapshots, removable only in Hetzner's Console, protect the backups
from a taken-over server. A systemd timer runs it as deploy; once a day
it also prunes and checks the repository. Failures go to the journal.

Hourly, because a restore costs people who started after the backup
what their device held: the app no longer finds its session and starts
anew. deploy/backups.md describes the setup, the restore and what a
restore loses; both were rehearsed on the local Compose stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

AGENTS.md

@@ -119,6 +119,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
 
 - **CI** (`.forgejo/workflows/ci.yaml`) runs `tool/check.sh` on every push, on the Forgejo runner on sahaquiel (`deploy/runner/`). Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and the server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. The server tests there get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since `passwords.yaml` is never committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI also runs `serverpod generate` and fails if the committed generated code differs from its output. CI runs on pushes only: a pull request from a fork would run its code on the production server.
 - **A green `main` deploys learn.halkyon.eu** through `deploy/redeploy.sh`, the only command the runner's key may run as `deploy` (pinned in `authorized_keys`, from sahaquiel itself only). It deploys the commit CI checked while it is still the tip, as a fast-forward, only if every new commit is signed by a key in `deploy/allowed-signers` (the deployed copy decides), after dumping the database (the last 3 dumps stay in `~deploy/backups`: the way back from a deploy, not backups), applies the migrations in a run of their own before switching (`--role=maintenance --apply-migrations`: a failing migration stops the deploy while the old server keeps serving, so migrations must keep the old server working until the switch), and fails unless the new server answers. A failed deploy leaves the checkout at the deployed commit; run the workflow again to retry. By hand: `ssh sahaquiel-deploy halkyon-learn/deploy/redeploy.sh <commit>`. Never `docker compose down -v` there.
+- **Backups** (`deploy/backups.md`): every hour, `deploy/backup.sh` (a systemd timer, as `deploy`) dumps the database, which holds people's files too, and restic stores it in a Hetzner Storage Box (24 hourly, 7 daily, 4 weekly, 3 monthly); the box's own snapshots protect it from a taken-over server. A restore takes the instance back up to an hour, and people who started since lose what their device held: their app no longer finds its session, starts a new anonymous account and deletes the old folder. So a change that makes devices lose their session, or keeps deleted people's data longer, changes what a restore costs; say so in `deploy/backups.md`.
 
 ## Rules the code relies on
 

README.md

@@ -73,7 +73,8 @@ licences require offering the source of what the instance runs to the people who
 
 learn.halkyon.eu itself deploys from `main` once CI is green: `deploy/redeploy.sh`, run by
 the CI runner described in `deploy/runner/`, deploys only commits signed by a key in
-`deploy/allowed-signers`.
+`deploy/allowed-signers`. It backs up its database every hour, off-site and encrypted;
+[`deploy/backups.md`](deploy/backups.md) says how to set that up and how to restore.
 
 ## More
 

deploy/backup.sh

@@ -0,0 +1,61 @@
+#!/bin/sh
+# Backs up learn.halkyon.eu off-site. Runs every hour from
+# deploy/backup/halkyon-backup.timer, as deploy on sahaquiel; deploy/backups.md says
+# how it was set up and how to restore.
+#
+# Everything people keep is in the database, their files too (Serverpod's database
+# cloud storage), so one dump covers the instance; the secrets in deploy/.env are kept
+# in Proton Pass instead. restic stores the dump encrypted in a Hetzner Storage Box
+# and keeps 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots (decided 2026-10-03).
+# Hourly, because a restore loses what came after the backup, and for someone who
+# started since, that is everything: their app no longer finds its session and starts
+# anew (deploy/backups.md). After 3 months, a deleted account is gone from every
+# backup (the privacy page says so). The box's own automatic snapshots, which only
+# Hetzner's Console can remove, keep the backups safe from a taken-over sahaquiel,
+# which can delete what restic wrote.
+#
+# Failures go to the journal only (decided 2026-10-03):
+#
+#   systemctl status halkyon-backup; journalctl -u halkyon-backup
+#
+# restic finds the repository and its password in deploy's files
+# ~/.config/restic/repository and ~/.config/restic/password (both 0600); a run by hand
+# can point elsewhere with RESTIC_REPOSITORY_FILE and RESTIC_PASSWORD_FILE.
+set -eu
+
+export RESTIC_REPOSITORY_FILE="${RESTIC_REPOSITORY_FILE:-$HOME/.config/restic/repository}"
+export RESTIC_PASSWORD_FILE="${RESTIC_PASSWORD_FILE:-$HOME/.config/restic/password}"
+
+cd "$(dirname "$0")"
+
+# The dump goes to a file first: Ubuntu 24.04's restic (v0.16) would store a dump
+# piped into it even when pg_dump fails halfway, and the file is the newest copy at
+# hand for a quick restore. Uncompressed, so restic finds what hasn't changed since
+# the last hour and stores it only once; restic compresses what it stores.
+mkdir -p -m 700 "$HOME/backups"
+dump="$HOME/backups/latest.dump"
+(
+  umask 077
+  # Never during a deploy (redeploy.sh holds this lock): its migrations would wait
+  # for the dump, or the dump for them.
+  exec 9>"$HOME/.redeploy.lock"
+  flock 9
+  docker compose exec -T postgres pg_dump -U halkyon_learn -Fc -Z 0 halkyon_learn >"$dump.partial"
+)
+mv "$dump.partial" "$dump"
+
+restic backup --quiet "$dump"
+restic forget --quiet --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 3
+
+# Once a day, in the run at 03:00: free the space of forgotten snapshots, and read the
+# repository's structure; on Sundays also a tenth of the stored data, chosen at random,
+# so damage to what was stored long ago shows within weeks.
+if [ "$(date +%H)" = 03 ]; then
+  restic prune --quiet
+  if [ "$(date +%u)" = 7 ]; then
+    restic check --read-data-subset=10%
+  else
+    restic check
+  fi
+fi
+restic snapshots --compact --latest 1

deploy/backup/halkyon-backup.service

@@ -0,0 +1,24 @@
+# The hourly off-site backup of learn.halkyon.eu, on sahaquiel: deploy/backup.sh, as
+# deploy, started by halkyon-backup.timer. Installed with the timer as
+# /etc/systemd/system/halkyon-backup.service; deploy/backups.md has the steps.
+#
+# deploy is in the docker group, which is as good as root, so sandboxing this unit
+# would protect little: it keeps to the basics.
+[Unit]
+Description=Off-site backup of learn.halkyon.eu's database
+Documentation=file:///home/deploy/halkyon-learn/deploy/backups.md
+After=network-online.target docker.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=deploy
+Group=deploy
+ExecStart=/home/deploy/halkyon-learn/deploy/backup.sh
+PrivateTmp=yes
+NoNewPrivileges=yes
+ProtectSystem=full
+# Behind the instance for CPU and disk.
+Nice=10
+IOSchedulingClass=best-effort
+IOSchedulingPriority=7

deploy/backup/halkyon-backup.timer

@@ -0,0 +1,13 @@
+# Starts halkyon-backup.service at the start of every hour; the run at 03:00,
+# sahaquiel's time (Europe/Berlin), also prunes and checks, done before the security
+# updates may reboot at 04:00. A run missed while sahaquiel was off happens as soon as
+# it is back. Installed as /etc/systemd/system/halkyon-backup.timer.
+[Unit]
+Description=Hourly off-site backup of learn.halkyon.eu
+
+[Timer]
+OnCalendar=hourly
+Persistent=true
+
+[Install]
+WantedBy=timers.target

deploy/backups.md

@@ -0,0 +1,169 @@
+# Backups
+
+learn.halkyon.eu backs up its database every hour, encrypted, to a Hetzner Storage Box:
+`backup.sh`, started by the systemd timer in `backup/`. The database is all there is to
+back up: people's files are in it too (Serverpod's database cloud storage). The other
+state of the instance is `deploy/.env`, whose secrets are kept in a password manager,
+and Caddy's certificates, which it fetches again.
+
+| | |
+|---|---|
+| When | every hour, on the hour; a run missed while the server was off happens when it is back |
+| What | `pg_dump` of the whole database, stored by [restic](https://restic.net) (encrypted, deduplicated, compressed) |
+| Kept | 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots; a deleted account is gone from every backup within about 3 months |
+| Where | a sub-account of the Storage Box (SFTP), used for nothing else |
+| Protected by | the box's automatic snapshots, daily, which only Hetzner's Console can remove: a taken-over server can delete what restic wrote, but not those |
+| Checked | every night at 03:00: forgotten snapshots pruned, the repository's structure read; on Sundays also a random tenth of the data |
+| Failures | in the journal only: `systemctl status halkyon-backup`, `journalctl -u halkyon-backup` |
+
+The newest dump also stays on the server, as `~deploy/backups/latest.dump`.
+
+**Without the restic password, the backups can't be read by anyone, you included.** Keep
+it, with the box's login, somewhere other than the server.
+
+## What a restore loses
+
+A restore takes the instance back to the time of the backup, up to an hour ago. Devices
+then sync with a server that has forgotten their last hour:
+
+- **Someone who used the instance before the backup** loses nothing: their devices still
+  have what the server lost, and send it again.
+- **Someone who started after the backup** is unknown to the server. At the next start,
+  their app finds its session gone, starts a new anonymous account and deletes the old
+  one's data from the device, the only copy left. The same happens to a device that
+  signed in after the backup, though a GitHub account can be signed into again.
+- **An account deleted after the backup** comes back. Delete it again if you know which
+  one it was.
+
+## Setting it up
+
+Once, in this order. The commands name who runs them, and where.
+
+### 1. The Storage Box (you, in Hetzner Console)
+
+1. Order a Storage Box (BX11, 1 TB, is plenty) in any location; its main login and
+   password go into your password manager.
+2. Turn on **SSH support**.
+3. Set up **automatic snapshots**: daily, at a fixed time, with as many kept as the box
+   allows (10 on a BX11).
+4. Add a **sub-account** for this server: its own directory (e.g. `sahaquiel`), SSH
+   support on, every other protocol off, not read-only (restic deletes what it prunes).
+   Note its user name (`uXXXXX-subN`) and password; the password is needed once, below.
+
+### 2. restic (an admin with sudo, on the server)
+
+```sh
+sudo apt install restic
+```
+
+### 3. The repository (deploy, on the server)
+
+As `deploy` (`ssh sahaquiel-deploy` on learn.halkyon.eu). A key without a passphrase,
+since nobody is there to type it at 03:00:
+
+```sh
+ssh-keygen -t ed25519 -N '' -C 'deploy@sahaquiel: backups' -f ~/.ssh/storagebox
+cat >>~/.ssh/config <<'EOF'
+Host storagebox
+    HostName uXXXXX-subN.your-storagebox.de
+    User uXXXXX-subN
+    Port 23
+    IdentityFile ~/.ssh/storagebox
+    IdentitiesOnly yes
+EOF
+```
+
+Install the key, with the sub-account's password. SSH first shows the box's host key:
+accept it only if its ED25519 fingerprint is
+`SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM`
+([Hetzner's list](https://docs.hetzner.com/storage/storage-box/general/)).
+
+```sh
+ssh storagebox install-ssh-key <~/.ssh/storagebox.pub
+ssh storagebox ls   # must not ask for the password any more
+```
+
+Then restic's two files and the repository. The password is generated here; copy it into
+your password manager before going on.
+
+```sh
+mkdir -p -m 700 ~/.config/restic
+echo 'sftp:storagebox:restic' >~/.config/restic/repository
+(umask 077 && openssl rand -base64 32 >~/.config/restic/password)
+cat ~/.config/restic/password   # into the password manager
+export RESTIC_REPOSITORY_FILE=~/.config/restic/repository RESTIC_PASSWORD_FILE=~/.config/restic/password
+restic init
+~/halkyon-learn/deploy/backup.sh   # the first backup, by hand
+```
+
+### 4. The timer (an admin with sudo, on the server)
+
+```sh
+sudo cp /home/deploy/halkyon-learn/deploy/backup/halkyon-backup.service \
+  /home/deploy/halkyon-learn/deploy/backup/halkyon-backup.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now halkyon-backup.timer
+systemctl list-timers halkyon-backup.timer
+```
+
+The units name `/home/deploy/halkyon-learn`; another checkout needs them changed.
+
+## Looking at the backups
+
+As `deploy`, with the two variables from step 3 exported:
+
+```sh
+restic snapshots                     # every snapshot, the newest last
+restic check --read-data-subset=5%   # reads a twentieth of the data back
+```
+
+## Restoring
+
+As `deploy`, in `~/halkyon-learn/deploy`, with the two variables from step 3 exported:
+
+```sh
+export RESTIC_REPOSITORY_FILE=~/.config/restic/repository RESTIC_PASSWORD_FILE=~/.config/restic/password
+```
+
+A restore replaces the whole database, so dump the current one first, if there is one:
+the way back from a restore that went wrong.
+
+1. **The dump to restore.** The newest is `~/backups/latest.dump`, if the server still
+   has it. Otherwise from the repository (on a new server, set up steps 1–3 first, with
+   the restic password from your password manager instead of a new one, and without
+   `restic init`):
+
+   ```sh
+   restic snapshots
+   restic dump <snapshot> /home/deploy/backups/latest.dump >~/backups/restore.dump
+   ```
+
+   `latest` in place of a snapshot's ID takes the newest.
+2. **Stop the server**, so nobody writes meanwhile:
+
+   ```sh
+   docker compose stop server
+   ```
+
+3. **Replace the database** with the dump (it drops the database and creates it anew):
+
+   ```sh
+   docker compose exec -T postgres \
+     pg_restore -U halkyon_learn -d postgres --clean --create <~/backups/restore.dump
+   ```
+
+4. **Start the server.** It applies the migrations that came after the dump:
+
+   ```sh
+   docker compose up -d
+   ```
+
+To check a backup without touching the instance, restore it into a database of its own
+next to the real one, look, and drop it:
+
+```sh
+docker compose exec -T postgres createdb -U halkyon_learn restore_check
+docker compose exec -T postgres pg_restore -U halkyon_learn -d restore_check <~/backups/restore.dump
+docker compose exec -T postgres psql -U halkyon_learn -d restore_check -c 'SELECT count(*) FROM studyset'
+docker compose exec -T postgres dropdb -U halkyon_learn restore_check
+```