feat(deploy): hourly off-site backups with restic

learn.halkyon.eu dumps its database every hour (people's files are in it
too) and restic stores the dump encrypted in a Hetzner Storage Box,
keeping 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots. The box's
own snapshots, removable only in Hetzner's Console, protect the backups
from a taken-over server. A systemd timer runs it as deploy; once a day
it also prunes and checks the repository. Failures go to the journal.

Hourly, because a restore costs people who started after the backup
what their device held: the app no longer finds its session and starts
anew. deploy/backups.md describes the setup, the restore and what a
restore loses; both were rehearsed on the local Compose stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

deploy/backup.sh

@@ -0,0 +1,61 @@
+#!/bin/sh
+# Backs up learn.halkyon.eu off-site. Runs every hour from
+# deploy/backup/halkyon-backup.timer, as deploy on sahaquiel; deploy/backups.md says
+# how it was set up and how to restore.
+#
+# Everything people keep is in the database, their files too (Serverpod's database
+# cloud storage), so one dump covers the instance; the secrets in deploy/.env are kept
+# in Proton Pass instead. restic stores the dump encrypted in a Hetzner Storage Box
+# and keeps 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots (decided 2026-10-03).
+# Hourly, because a restore loses what came after the backup, and for someone who
+# started since, that is everything: their app no longer finds its session and starts
+# anew (deploy/backups.md). After 3 months, a deleted account is gone from every
+# backup (the privacy page says so). The box's own automatic snapshots, which only
+# Hetzner's Console can remove, keep the backups safe from a taken-over sahaquiel,
+# which can delete what restic wrote.
+#
+# Failures go to the journal only (decided 2026-10-03):
+#
+#   systemctl status halkyon-backup; journalctl -u halkyon-backup
+#
+# restic finds the repository and its password in deploy's files
+# ~/.config/restic/repository and ~/.config/restic/password (both 0600); a run by hand
+# can point elsewhere with RESTIC_REPOSITORY_FILE and RESTIC_PASSWORD_FILE.
+set -eu
+
+export RESTIC_REPOSITORY_FILE="${RESTIC_REPOSITORY_FILE:-$HOME/.config/restic/repository}"
+export RESTIC_PASSWORD_FILE="${RESTIC_PASSWORD_FILE:-$HOME/.config/restic/password}"
+
+cd "$(dirname "$0")"
+
+# The dump goes to a file first: Ubuntu 24.04's restic (v0.16) would store a dump
+# piped into it even when pg_dump fails halfway, and the file is the newest copy at
+# hand for a quick restore. Uncompressed, so restic finds what hasn't changed since
+# the last hour and stores it only once; restic compresses what it stores.
+mkdir -p -m 700 "$HOME/backups"
+dump="$HOME/backups/latest.dump"
+(
+  umask 077
+  # Never during a deploy (redeploy.sh holds this lock): its migrations would wait
+  # for the dump, or the dump for them.
+  exec 9>"$HOME/.redeploy.lock"
+  flock 9
+  docker compose exec -T postgres pg_dump -U halkyon_learn -Fc -Z 0 halkyon_learn >"$dump.partial"
+)
+mv "$dump.partial" "$dump"
+
+restic backup --quiet "$dump"
+restic forget --quiet --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 3
+
+# Once a day, in the run at 03:00: free the space of forgotten snapshots, and read the
+# repository's structure; on Sundays also a tenth of the stored data, chosen at random,
+# so damage to what was stored long ago shows within weeks.
+if [ "$(date +%H)" = 03 ]; then
+  restic prune --quiet
+  if [ "$(date +%u)" = 7 ]; then
+    restic check --read-data-subset=10%
+  else
+    restic check
+  fi
+fi
+restic snapshots --compact --latest 1