halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 37 Packages Wiki Activity Actions

feat(publish): releases in git (public tier, stage 2)

Browse source 
A person with an identity and a public profile publishes a studyset with
everything inside it as its next release (ADR 0005, rule 3): the server
reads their synced rows and writes them through the git command as a
commit on main and the annotated tag vN, in a bare repository of the
studyset's own, named by a random public id made at the first publish.

A release is a directory per studyset with a file per card, note, path,
step and PDF (its id, position, tag names and payload as synced, keys
sorted and indented) and the files the content names, without what is
only the author's (a card's report, a PDF's deadline). Publishing is
refused without a licence allowing passing it on in every studyset ("no
derivatives" allowed), with revisions to settle, with files not uploaded
yet, when nothing changed, or for a name that won't do; the name is
fixed from the first release, as is the person's handle.

In the app, Share in a studyset's action row opens the Publish sheet;
publishing syncs first, and settles by itself the revisions that merge
cleanly, listing only real conflicts. The server's image gets git, the
repositories a volume of their own, and the hourly backup a git bundle of
each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-09 22:01:00 +02:00
commit f635b0816a
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
3 changed files with 20287 additions and 77 deletions
Download patch file Download diff file Expand all files Collapse all files

29
deploy/backup.sh
View file

@ -4,8 +4,9 @@
# how it was set up and how to restore.
#
# Everything people keep is in the database, their files too (Serverpod's database
# cloud storage), so one dump covers the instance; the secrets in deploy/.env are kept
# in Proton Pass instead. restic stores the dump encrypted in a Hetzner Storage Box
# cloud storage), but for the releases of published studysets, which are git
# repositories in the server's volume: a dump and a bundle of each repository cover
# the instance; the secrets in deploy/.env are kept in Proton Pass instead. restic stores the dump encrypted in a Hetzner Storage Box
# and keeps 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots (decided 2026-10-03).
# Hourly, because a restore loses what came after the backup, and for someone who
# started since, that is everything: their app no longer finds its session and starts
@ -44,7 +45,29 @@ dump="$HOME/backups/latest.dump"
)
mv "$dump.partial" "$dump"
restic backup --quiet "$dump"
# Each repository as a git bundle, after the dump: a release the dump lacks is never
# numbered again (the server numbers past the highest tag), while one the bundles
# lacked would be lost. A bundle is whole whenever it is made: git reads the refs, then
# packs what they reach. A repository being created this moment has nothing to bundle
# yet, and comes next hour.
releases="$HOME/backups/repositories.tar"
(
umask 077
docker compose exec -T server sh -c '
out=$(mktemp -d)
for repository in /app/repositories/*.git; do
[ -d "$repository" ] || continue
git -C "$repository" bundle create --quiet \
"$out/$(basename "$repository" .git).bundle" --all ||
echo "Not bundled: $repository" >&2
done
tar -C "$out" -cf - .
rm -rf "$out"
' >"$releases.partial"
)
mv "$releases.partial" "$releases"
restic backup --quiet "$dump" "$releases"
restic forget --quiet --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 3
# Once a day, in the run at 03:00: free the space of forgotten snapshots, and read the

41
deploy/backups.md
View file

@ -1,22 +1,26 @@
# Backups
learn.halkyon.eu backs up its database every hour, encrypted, to a Hetzner Storage Box:
`backup.sh`, started by the systemd timer in `backup/`. The database is all there is to
back up: people's files are in it too (Serverpod's database cloud storage). The other
state of the instance is `deploy/.env`, whose secrets are kept in a password manager,
and Caddy's certificates, which it fetches again.
`backup.sh`, started by the systemd timer in `backup/`. The database holds nearly all
there is to back up, people's files too (Serverpod's database cloud storage); the
releases of published studysets are git repositories in the server's `repositories`
volume (ADR 0005, rule 3), backed up beside it as a git bundle each. The other state of
the instance is `deploy/.env`, whose secrets are kept in a password manager, Caddy's
certificates, which it fetches again, and the dictionary files, which it downloads
again.
| | |
|---|---|
| When | every hour, on the hour; a run missed while the server was off happens when it is back |
| What | `pg_dump` of the whole database, stored by [restic](https://restic.net) (encrypted, deduplicated, compressed) |
| What | `pg_dump` of the whole database, then a `git bundle` of each published studyset's repository in one tar file, stored by [restic](https://restic.net) (encrypted, deduplicated, compressed) |
| Kept | 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots; a deleted account is gone from every backup within about 3 months |
| Where | a sub-account of the Storage Box (SFTP), used for nothing else |
| Protected by | the box's automatic snapshots, daily, which only Hetzner's Console can remove: a taken-over server can delete what restic wrote, but not those |
| Checked | every night at 03:00: forgotten snapshots pruned, the repository's structure read; on Sundays also a random tenth of the data |
| Failures | in the journal only: `systemctl status halkyon-backup`, `journalctl -u halkyon-backup` |
The newest dump also stays on the server, as `~deploy/backups/latest.dump`.
The newest dump also stays on the server, as `~deploy/backups/latest.dump`, and the
newest bundles as `~deploy/backups/repositories.tar`.
**Without the restic password, the backups can't be read by anyone, you included.** Keep
it, with the box's login, somewhere other than the server.
@ -34,6 +38,10 @@ then sync with a server that has forgotten their last hour:
signed in after the backup, though a GitHub account can be signed into again.
- **An account deleted after the backup** comes back. Delete it again if you know which
one it was.
- **A release published after the backup** is gone from the instance; learners who took
it keep their copies. Its version number is given again by the next release only if
the bundles lack it too: the server numbers releases past both the database's and the
repository's highest.
## Setting it up
@ -154,7 +162,26 @@ the way back from a restore that went wrong.
pg_restore -U halkyon_learn -d postgres --clean --create <~/backups/restore.dump
```
4. **Start the server.** It applies the migrations that came after the dump:
4. **Replace the repositories** of published studysets with the bundles of the same
snapshot (or `~/backups/repositories.tar`, beside the newest dump), in a container of
the server's image that does only this:
```sh
restic dump <snapshot> /home/deploy/backups/repositories.tar >~/backups/restore-repositories.tar
docker compose run --rm --no-deps -T --entrypoint sh server -ec '
mkdir /tmp/bundles && tar -C /tmp/bundles -xf -
rm -rf /app/repositories/*.git
for bundle in /tmp/bundles/*.bundle; do
[ -f "$bundle" ] || continue
repository=/app/repositories/$(basename "$bundle" .bundle).git
git init --bare --quiet --initial-branch=main "$repository"
git -C "$repository" config gc.auto 0
git -C "$repository" fetch --quiet "$bundle" "refs/*:refs/*"
done
' <~/backups/restore-repositories.tar
```
5. **Start the server.** It applies the migrations that came after the dump:
```sh
docker compose up -d

6
deploy/compose.yaml
View file

@ -58,12 +58,17 @@ services:
FILE_LIMIT_ANONYMOUS_MB: ${FILE_LIMIT_ANONYMOUS_MB:-}
FILE_LIMIT_IDENTIFIED_MB: ${FILE_LIMIT_IDENTIFIED_MB:-}
FILE_LIMIT_INSTANCE_MB: ${FILE_LIMIT_INSTANCE_MB:-}
# The git repositories of published studysets (ADR 0005, rule 3).
REPOSITORIES_DIR: /app/repositories
# The dictionary files it loads at start and rereads hourly, those that changed
# (see the server's README): those the dictionaries service keeps current, read
# only. To supply files yourself instead, remove that service and mount
# ./dictionaries here (`- ./dictionaries:/app/dictionaries:ro`).
volumes:
- dictionaries:/app/dictionaries:ro
# Every release of every published studyset: backed up beside the database
# (backup.sh), never deleted with `down -v`.
- repositories:/app/repositories
depends_on:
postgres:
condition: service_healthy
@ -112,3 +117,4 @@ volumes:
caddy_config:
dictionaries:
postgres_data:
repositories: