halkyon/deploy/backups.md

Backups

learn.halkyon.eu backs up its database every hour, encrypted, to a Hetzner Storage Box: backup.sh, started by the systemd timer in backup/. The database holds nearly all there is to back up, people's files too (Serverpod's database cloud storage); the releases of published studysets are git repositories in the server's repositories volume (ADR 0005, rule 3), backed up beside it as a git bundle each. The other state of the instance is deploy/.env, whose secrets are kept in a password manager, Caddy's certificates, which it fetches again, and the dictionary files, which it downloads again.

When	every hour, on the hour; a run missed while the server was off happens when it is back
What	pg_dump of the whole database, then a git bundle of each published studyset's repository in one tar file, stored by restic (encrypted, deduplicated, compressed)
Kept	24 hourly, 7 daily, 4 weekly and 3 monthly snapshots; a deleted account is gone from every backup within about 3 months
Where	a sub-account of the Storage Box (SFTP), used for nothing else
Protected by	the box's automatic snapshots, daily, which only Hetzner's Console can remove: a taken-over server can delete what restic wrote, but not those
Checked	every night at 03:00: forgotten snapshots pruned, the repository's structure read; on Sundays also a random tenth of the data
Failures	in the journal only: systemctl status halkyon-backup, journalctl -u halkyon-backup

The newest dump also stays on the server, as ~deploy/backups/latest.dump, and the newest bundles as ~deploy/backups/repositories.tar.

Without the restic password, the backups can't be read by anyone, you included. Keep it, with the box's login, somewhere other than the server.

What a restore loses

A restore takes the instance back to the time of the backup, up to an hour ago. Devices then sync with a server that has forgotten their last hour:

• Someone who used the instance before the backup loses nothing: their devices still have what the server lost, and send it again.
• Someone who started after the backup is unknown to the server. At the next start, their app finds its session gone and asks how to start; starting anew deletes the old account's data from the device, the only copy left. The same happens to a device that signed in after the backup, though a GitHub account can be signed into again.
• An account deleted after the backup comes back. Delete it again if you know which one it was.
• The instance's other services (ADR 0016; none on learn.halkyon.eu yet), whose accounts this server's register keeps: an account made since the backup is unknown to the register, and each member's daily pruning would delete that person's data there. Stop the members (or their pruning) until those people are dealt with. An account deleted since comes back here, but stays deleted at the members.
• A release published after the backup is gone from the instance; learners who took it keep their copies. Its version number is given again by the next release only if the bundles lack it too: the server numbers releases past both the database's and the repository's highest.

Setting it up

Once, in this order. The commands name who runs them, and where.

1. The Storage Box (you, in Hetzner Console)

1. Create a Storage Box (BX11, 1 TB, is plenty) in any location (Germany or Finland, both in the EU; the app shows where, from BACKUP_LOCATION in deploy/.env, so set it to the box's country: DE or FI, shown as "Germany (EU)"), with a password, which goes into your password manager. Turn on SSH Support (port 23, which restic uses) and External Reachability: without it, the box only answers inside Hetzner's network. Samba and WebDAV stay off; FTP and SFTP on port 22 can't be turned off.
2. Set up automatic snapshots: daily, at a fixed time, with as many kept as the box allows (10 on a BX11).
3. Add a sub-account for this server: its own directory (e.g. learn-halkyon-eu), SSH support and External reachability on, Samba and WebDAV off, not read-only (restic deletes what it prunes). Note its user name (uXXXXX-subN) and password; the password is needed once, below.

2. restic (an admin with sudo, on the server)

sudo apt install restic

3. The repository (deploy, on the server)

As deploy (ssh sahaquiel-deploy on learn.halkyon.eu). A key without a passphrase, since nobody is there to type it at 03:00:

ssh-keygen -t ed25519 -N '' -C 'deploy: learn.halkyon.eu backups' -f ~/.ssh/storagebox
cat >>~/.ssh/config <<'EOF'
Host storagebox
    HostName uXXXXX-subN.your-storagebox.de
    User uXXXXX-subN
    Port 23
    IdentityFile ~/.ssh/storagebox
    IdentitiesOnly yes
EOF

Install the key, with the sub-account's password. SSH first shows the box's host key: accept it only if its ED25519 fingerprint is SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM, or, for another key type, the one Hetzner lists for it.

ssh storagebox install-ssh-key <~/.ssh/storagebox.pub
ssh storagebox ls   # must not ask for the password any more

Then restic's two files and the repository. The password is generated here; copy it into your password manager before going on.

mkdir -p -m 700 ~/.config/restic
echo 'sftp:storagebox:restic' >~/.config/restic/repository
(umask 077 && openssl rand -base64 32 >~/.config/restic/password)
cat ~/.config/restic/password   # into the password manager
export RESTIC_REPOSITORY_FILE=~/.config/restic/repository RESTIC_PASSWORD_FILE=~/.config/restic/password
restic init
~/halkyon-learn/deploy/backup.sh   # the first backup, by hand

4. The timer (an admin with sudo, on the server)

sudo cp /home/deploy/halkyon-learn/deploy/backup/halkyon-backup.service \
  /home/deploy/halkyon-learn/deploy/backup/halkyon-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now halkyon-backup.timer
systemctl list-timers halkyon-backup.timer

The units name /home/deploy/halkyon-learn; another checkout needs them changed.

Looking at the backups

As deploy, with the two variables from step 3 exported:

restic snapshots                     # every snapshot, the newest last
restic check --read-data-subset=5%   # reads a twentieth of the data back

Restoring

As deploy, in ~/halkyon-learn/deploy, with the two variables from step 3 exported:

export RESTIC_REPOSITORY_FILE=~/.config/restic/repository RESTIC_PASSWORD_FILE=~/.config/restic/password

A restore replaces the whole database, so dump the current one first, if there is one: the way back from a restore that went wrong.

1. The dump to restore. The newest is ~/backups/latest.dump, if the server still has it. Otherwise from the repository (on a new server, set up steps 1–3 first, with the restic password from your password manager instead of a new one, and without restic init):

   restic snapshots
   restic dump <snapshot> /home/deploy/backups/latest.dump >~/backups/restore.dump
   

   latest in place of a snapshot's ID takes the newest.

2. Stop the server, so nobody writes meanwhile:

   docker compose stop server
   

3. Replace the database with the dump (it drops the database and creates it anew):

   docker compose exec -T postgres \
     pg_restore -U halkyon_learn -d postgres --clean --create <~/backups/restore.dump
   

4. Replace the repositories of published studysets with the bundles of the same snapshot (or ~/backups/repositories.tar, beside the newest dump), in a container of the server's image that does only this:

   restic dump <snapshot> /home/deploy/backups/repositories.tar >~/backups/restore-repositories.tar
   docker compose run --rm --no-deps -T --entrypoint sh server -ec '
     mkdir /tmp/bundles && tar -C /tmp/bundles -xf -
     rm -rf /app/repositories/*.git
     for bundle in /tmp/bundles/*.bundle; do
       [ -f "$bundle" ] || continue
       repository=/app/repositories/$(basename "$bundle" .bundle).git
       git init --bare --quiet --initial-branch=main "$repository"
       git -C "$repository" config gc.auto 0
       git -C "$repository" fetch --quiet "$bundle" "refs/*:refs/*"
     done
   ' <~/backups/restore-repositories.tar
   

5. Start the server. It applies the migrations that came after the dump:

   docker compose up -d
   

To check a backup without touching the instance, restore it into a database of its own next to the real one, look, and drop it:

docker compose exec -T postgres createdb -U halkyon_learn restore_check
docker compose exec -T postgres pg_restore -U halkyon_learn -d restore_check <~/backups/restore.dump
docker compose exec -T postgres psql -U halkyon_learn -d restore_check -c 'SELECT count(*) FROM studyset'
docker compose exec -T postgres dropdb -U halkyon_learn restore_check