feat(deploy): instances keep their dictionaries current, weekly, by themselves

EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.

An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-09 16:52:37 +02:00
commit 25e01ad128
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
3 changed files with 211 additions and 32 deletions

View file

@ -1,4 +1,5 @@
# The Halkyon Learn instance: Caddy in front, the Serverpod server, PostgreSQL.
# The Halkyon Learn instance: Caddy in front, the Serverpod server, PostgreSQL, and
# the dictionaries kept current beside them.
#
# cd deploy
# cp .env.example .env && chmod 600 .env # then fill in the secrets
@ -57,15 +58,36 @@ services:
FILE_LIMIT_ANONYMOUS_MB: ${FILE_LIMIT_ANONYMOUS_MB:-}
FILE_LIMIT_IDENTIFIED_MB: ${FILE_LIMIT_IDENTIFIED_MB:-}
FILE_LIMIT_INSTANCE_MB: ${FILE_LIMIT_INSTANCE_MB:-}
# The dictionary files it loads at start, those that changed since (see the
# server's README); read only, and empty unless the operator adds some.
# The dictionary files it loads at start and rereads daily, those that changed
# (see the server's README): those the dictionaries service keeps current, read
# only. To supply files yourself instead, remove that service and mount
# ./dictionaries here (`- ./dictionaries:/app/dictionaries:ro`).
volumes:
- ./dictionaries:/app/dictionaries:ro
- dictionaries:/app/dictionaries:ro
depends_on:
postgres:
condition: service_healthy
logging: *logging
# Keeps the dictionary files current from EDRDG's files of the day, once they are a
# week old (EDRDG's licence asks it of servers showing its dictionaries at least
# monthly; the curator's README): in a container of its own, the curator never in
# the server's image or process. It only downloads (some 25 MB a week) and writes
# into the volume; the files are never backed up, as they can be had again.
dictionaries:
build:
context: ..
dockerfile: halkyon_learn_curator/Dockerfile
image: halkyon-learn-dictionaries
restart: unless-stopped
# Docker's stop signal reaches the curator through a small init.
init: true
# A conversion takes some 400 MB for a minute a week.
mem_limit: 1g
volumes:
- dictionaries:/dictionaries
logging: *logging
postgres:
image: postgres:16
restart: unless-stopped
@ -88,4 +110,5 @@ services:
volumes:
caddy_data:
caddy_config:
dictionaries:
postgres_data:

View file

@ -1,5 +1,7 @@
# Dictionary files
Dictionary files (`*.jsonl`, `*.jsonl.gz`) for the server to load at start; see
the server's README. They are downloads turned into files by `halkyon_learn_curator`,
not kept in git.
For an instance that supplies its dictionary files itself, rather than the
`dictionaries` service keeping them current (`compose.yaml`): put the files
(`*.jsonl`, `*.jsonl.gz`, made by `halkyon_learn_curator`) here and mount this
folder into the server as the comment there says. Mind EDRDG's licence: a server
showing its dictionaries must update them at least once a month. Not kept in git.

View file

@ -58,7 +58,7 @@ main() {
trap 'git reset -q --hard "$deployed"; echo "The checkout is back at $deployed." >&2' EXIT
cd deploy
# Build while the running server keeps serving; then dump, migrate, switch.
docker compose build -q server
docker compose build -q server dictionaries
dump
# Migrations as a run of their own, before the switch: Serverpod's maintenance role
# applies them and exits with 0 only if that worked. Each migration is one