feat(deploy): instances keep their dictionaries current, weekly, by themselves

EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.

An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-09 16:52:37 +02:00
commit 25e01ad128
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
13 changed files with 211 additions and 32 deletions

View file

@ -1,5 +1,5 @@
# The Docker build context is the repository root. Nothing secret, local or bulky
# goes into it: the server's Dockerfile copies the whole halkyon_learn_server folder.
# goes into it: the server's and the curator's Dockerfiles copy their whole package folders.
# Secrets and local state
halkyon_learn_server/config/passwords.yaml

View file

@ -144,7 +144,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
- **Tree, tags, modes, settings and files** (ADR 0004): a studyset's `parent` and sibling `position` form the tree, and studysets in a cycle or without their parent show at the top level (`topLevel`); its `archivedAt`, `starredAt` and `updatedAt` are columns, and every change of a studyset writes `updatedAt` with it (the library lists starred studysets first, then the most recently changed); a card's tags are `card_tag` rows, one per assignment, never updated, and a duplicate counts once; `review_log.mode` is `review`, `reverse` (its own schedule), `write` (by hand, its own schedule) or `cram` (never moves a schedule), and a schedule replays the reviews of one mode; `person_settings` is meant as one row per person, and extra rows fold into the one with the smallest id; an `asset` row stands for a file in private cloud storage, found by its hash. **Never derive a row's id from a person or anything else guessable**: ids are unique across the whole table, so someone could claim it first in their own space and break the owner's sync.
- **Files** (ADR 0004, rule 5): `AssetEndpoint` (`requireLogin`) takes uploads into the caller's own space by SHA-256 (`startUpload` gives Serverpod's upload description, or null when the file is there already and claims it anew; `finishUpload` keeps it only if its content has that hash) and serves an asset row's file to its space's owner and members only, `NotFoundException` otherwise, also for a deleted row (the sync engine hides tombstones). Files live in the private database storage at `spaces/<space>/<sha256>`, at most 20 MB each, and within limits (`file_limits.dart`, from the environment: an anonymous person 100 MB, an identified one 500 MB or what a `file_allowance` row grants them, the instance 15 GB), checked when an upload starts (unfinished uploads count, or they could fill the disk unnoticed) and again when it finishes (only what is kept counts), refused with `FileLimitExceededException`; `deletePerson` deletes the person's files and allowance, and `FilePruningFutureCall` deletes daily those no live row refers to (or never finished), 7 days after they were last claimed. In the app, `AssetRepository` (`core/asset/`) keeps files by hash in the person's folder (`files/`, uploads waiting marked in `files/uploads/`), uploads after each sync that went through, and downloads on demand, checking the hash; every hash is checked to be one before it becomes part of a path. Content names a file by the URL `asset:<asset row id>` (`assetUrl`, `RichContent.assetIds`): images are the editor's `image` block, sounds an `audio` block (`core/editor/file_blocks.dart`), and any other URL is never fetched. `StoredFileImage` loads through the repository like Flutter's `NetworkImage`, keyed by the person's repository too, so no image crosses people.
- **PDFs** (`feature/pdf_source/`): a `pdf_source` row per PDF in a studyset (payload: title, file name, the asset of its file, page count, reading deadline; last writer wins) with its `position` among the studyset's files in a column of its own (decided 2026-10-08; the Files tab's default order, Custom order, which Reorder changes; PDFs from before it stand at `a0`, in the order they were added) and an append-only `pdf_reading` log (`reached` with a page, `reset`): the furthest page is the furthest reached since the last reset, so devices never conflict. A PDF's file is an asset of up to 50 MB (other files 20 MB): the server takes up to 50 MB and deletes a stored file over 20 MB that doesn't start with `%PDF-`. Deleting a PDF deletes its asset row too unless another PDF has the same file (asset rows are otherwise never deleted). Cards and notes tied to a page keep it in their payload (`source`, `core/content/pdf_page_ref.dart`; card schema 4, note schema 2); a screenshot is an `image` block with `pdf` and `page` attributes.
- **Dictionaries** (decided 2026-10-08): reference data the server holds for everyone, owned by no one, never synced (`dictionary_entry`, `dictionary_source`, `lib/src/dictionary/`). `halkyon_learn_curator` (AGPL, no Serverpod, so the server never links its code) turns each source's download into one format, a dictionary file (its README is the contract), with an adapter per source; a dictionary is one source in one meaning language (`jmdict` in `en`), of words or of characters (`kind`: JMdict, KANJIDIC2 with the kana), and the endpoints take it by those two, so a search for words never brings kanji; everything specific to a language (kana, romaji, meanings without "to") goes into each entry's `keys` and `looseKeys`, so the server's search knows no language: exact key, exact loose key, key starting with the query, loose key starting with it, each by the source's `rank` (`DictionaryEndpoint.search`, LIKE on barred keys, no extension needed). The server loads the files in its `dictionaries` folder at start, those whose SHA-256 changed, each source replaced in one transaction (`dictionary_loader.dart`). A card made from an entry copies what it needs and keeps `(source, sourceId)` as a soft reference only (`CardContent.entry`, card schema 6), so loading a newer version never touches cards. Every file carries its credits, which the app shows (About, and at the foot of the dictionary). In the app, a studyset's dictionary follows from its front language (`StudysetDictionary`, through `StudysetLanguageScope.of`): no attaching or detaching, and nothing dictionary-shaped shows without one (Add material › From the dictionary; Look up beside a vocabulary card's front). A word's example sentences (Tatoeba, through JMdict's examples edition) become its card's hints, up to three, the word in bold; Look up gives them to a card without hints of its own. `DictionaryAdder` makes the cards, each entry once per studyset, a word tagged `JLPT-NX` and with its word class by the first sense (`Noun`, `Verb`, `na-Adjective`); Look up in the composer gives a card the same tags (`DictionaryItem.tagNames`), and an entry looked up again on that card swaps the tags the one before brought (`isDictionaryTagName`). Where the instance also holds the language's characters, the dictionary screen has a tab for them (Kanji & kana: the kana, the JLPT levels and the school grades to add whole, each under its name), a character becoming a character card tagged `JLPT-NX`, `Grade N` or `Secondary school`, and `Hiragana` or `Katakana`; a character card's Look up searches the characters.
- **Dictionaries** (decided 2026-10-08): reference data the server holds for everyone, owned by no one, never synced (`dictionary_entry`, `dictionary_source`, `lib/src/dictionary/`). `halkyon_learn_curator` (AGPL, no Serverpod, so the server never links its code) turns each source's download into one format, a dictionary file (its README is the contract), with an adapter per source; a dictionary is one source in one meaning language (`jmdict` in `en`), of words or of characters (`kind`: JMdict, KANJIDIC2 with the kana), and the endpoints take it by those two, so a search for words never brings kanji; everything specific to a language (kana, romaji, meanings without "to") goes into each entry's `keys` and `looseKeys`, so the server's search knows no language: exact key, exact loose key, key starting with the query, loose key starting with it, each by the source's `rank` (`DictionaryEndpoint.search`, LIKE on barred keys, no extension needed). The server loads the files in its `dictionaries` folder at start and daily (`DictionaryReloadFutureCall`), those whose SHA-256 changed, each source replaced in one transaction, one load at a time (`dictionary_loader.dart`). EDRDG's licence asks a server showing its dictionaries to update them at least monthly (decided 2026-10-09: instances keep current by themselves, weekly): the `dictionaries` service of `deploy/compose.yaml` runs the curator in a container of its own (`halkyon_learn_curator/Dockerfile`, user ID 10001; the AGPL tool never enters the server's image or process), `curate update --every 7` downloading EDRDG's files of the day (`JMdict_e_examp.gz`, `kanjidic2.xml.gz`; the JLPT lists are static, in the curator's `data/`), converting them and renaming each over its old file in the shared volume, which the server mounts read-only. A card made from an entry copies what it needs and keeps `(source, sourceId)` as a soft reference only (`CardContent.entry`, card schema 6), so loading a newer version never touches cards. Every file carries its credits, which the app shows (About, and at the foot of the dictionary). In the app, a studyset's dictionary follows from its front language (`StudysetDictionary`, through `StudysetLanguageScope.of`): no attaching or detaching, and nothing dictionary-shaped shows without one (Add material › From the dictionary; Look up beside a vocabulary card's front). A word's example sentences (Tatoeba, through JMdict's examples edition) become its card's hints, up to three, the word in bold; Look up gives them to a card without hints of its own. `DictionaryAdder` makes the cards, each entry once per studyset, a word tagged `JLPT-NX` and with its word class by the first sense (`Noun`, `Verb`, `na-Adjective`); Look up in the composer gives a card the same tags (`DictionaryItem.tagNames`), and an entry looked up again on that card swaps the tags the one before brought (`isDictionaryTagName`). Where the instance also holds the language's characters, the dictionary screen has a tab for them (Kanji & kana: the kana, the JLPT levels and the school grades to add whole, each under its name), a character becoming a character card tagged `JLPT-NX`, `Grade N` or `Secondary school`, and `Hiragana` or `Katakana`; a character card's Look up searches the characters.
- **The Pomodoro timer** (`feature/pomodoro/`): the profiles follow the person (`PersonSettings.pomodoro`, settings schema 4), the running timer is the device's own (`PomodoroRun` in `<person folder>/pomodoro.json`, `PomodoroRunStore`). `PomodoroRun` is a pure state machine on the time: a phase that ends leads to the next, running if its profile starts it by itself, else paused at its full length; a timer kept while the app was closed catches up (`caughtUpTo`). `PomodoroBloc` sits above the router (`PersonApp`), and `PomodoroPresenter` hands every change to `PomodoroPlatform` as a `PomodoroNotice` in the app's language, and its buttons back. **Android:** the app's own Kotlin (`PomodoroService.kt`; MethodChannel `app.halkyon.learn/pomodoro`, EventChannel `…/buttons`): a `specialUse` foreground service with a countdown notification (chronometer), Pause/Resume, Skip, Stop, on Android 16 a Live Update (`NotificationCompat.ProgressStyle`, the cycle's phases as segments, `setRequestPromotedOngoing`); an exact alarm (`USE_EXACT_ALARM`) alerts at the phase's end with the app closed; Do Not Disturb through the interruption filter, the one before kept in shared preferences and restored. With the app swiped away its buttons open it (a service may not start an activity from a notification's button since Android 12, so the notification is posted again with activity intents), and Stop works alone, leaving a mark the app takes at its next start (`takeStoppedOutside`) to forget its timer; a profile's next phase only starts by itself while the app runs. Dart's timers sleep with the device, so `PomodoroBloc` also looks every 15 s while a phase runs, and the alarm's receiver wakes it (`phaseEnds`); a kept timer is caught up before it first shows, or its ended phase would alert again. No notification plugin: flutter_local_notifications had Live Updates only in a prerelease needing compileSdk 37 (checked 2026-10-07). **Linux:** `LinuxPomodoroPlatform`, over package:dbus: where the top bar shows icons (GNOME with the AppIndicator extension, built into Ubuntu; KDE) the timer counts down there with its buttons in the icon's menu (`StatusNotifierItem`, written in Dart and laid out as libayatana-appindicator lays itself out, within the snap's rules, `desktop-legacy` of snapd v2.77: the item at `/org/ayatana/NotificationItem/<id>`, the only path the label's signal `XAyatanaNewLabel` may come from, registered with the watcher by that path, since a snap may bind no `org.freedesktop.StatusNotifierItem-*` name; the menu at `/MenuBar`; registered again when the watcher restarts); elsewhere the timer shows in the app alone, and on GNOME the Pomodoro card suggests the extension until dismissed, asking GNOME Shell to install or switch it on (`InstallRemoteExtension`, `EnableExtension`) or opening its page (decided 2026-10-07: no notification stands in; xdg-desktop-portal's Background status is for Flatpaks only). An alert with sound comes through `org.freedesktop.Notifications` when a phase ends while the app runs, and GNOME's `show-banners` goes off while focusing, through `gsettings`; the value before and the dismissed suggestion are kept in `<app support>/`. Calls to the bus go one at a time (`SerialQueue`).
- **Citations** (`core/citation/`): a `Source` is plain text parts as the person typed them (authors as full names, one per line; year; title; `container`, what it is a part of; publisher; ISBN; link; `raw`, a citation a file brought, shown while nothing else is cited), and `formatCitation` cites it in the device's `CitationStyle` (APA 7, MLA 9, Chicago 18 notes and bibliography, decided 2026-10-07): it splits and abbreviates the names itself (inverted names, particles, suffixes; names written with `&` or "et al." stay as written), sets a work of its own in italics and a part of something apart from the whole by `container`, and writes a missing year as "n.d." in APA and Chicago. A step's sources (step schema 2) and a studyset's bibliography (`bibliography`, studyset schema 4, last writer wins like the rest of its payload) are lists of them, shown and changed through `SourceList` and `showSourcesSheet`, in reference-list order (alphabetical, whatever order they were added in); the path import reads them too, text as `raw` and the old format's `citation` as `container`.
- **What a card carries beside its type's fields** (card schema 5): up to six `hints` and an `explanation` (rich text, any type), a `report` that something on it is wrong (category, note, when; studied as before, its badge in the list), and a word's `senses` (glosses, label, JMdict's grammar codes, whose English short forms and meanings are data in `feature/card/model/jmdict_tags.dart`; with senses a word's meanings are their glosses, and `uk` on its first sense shows its reading from the start). Each merges as a field of its own. Deleting a card goes through `CardDeletions` (a `DelayedDeletions`, as notes and studysets), so the list and the session offer Undo. A study session's Undo takes back its steps last first (ratings, burials and deletions alike, across its rounds), so it holds the cards it deletes (`hold`) until it ends, when they go after the usual time to undo; a card deleted in a session that never ended (the app killed) stays. Vocabulary can be quizzed as multiple choice (`quizVocabulary`, `quizAspect` in the study options, settings schema 5; `vocabularyQuiz` draws the wrong answers from the other words studied). **A character card** (`CardType.character`, card schema 7, decided 2026-10-09: one type for kanji and kana) holds the character on its front, its `meanings` (a kana's romaji is its meaning, so kana are studied and written as kanji are) and a kanji's `onReadings` and `kunReadings` as dictionaries write them (katakana; hiragana with the okurigana after a dot); its JLPT level and school grade are tags, its strokes the app's own. **How characters are written** (`feature/kanji/`, decided 2026-10-09: bundled, so writing works offline) is KanjiVG's, in `assets/kanjivg/` (CC BY-SA 3.0, its notice in the licence list): the curator's `curate kanjivg` writes one gzipped JSON file per 256 code points (2.7 MB in all), which `StrokeRepository` (app-wide in `Dependencies`, as the maps) reads when a character of it is first asked for; the strokes are SVG path data in a 109 × 109 square, read with `path_parsing` (pure Dart, so `CharacterStrokes` flattens them into points without `dart:ui`). `StrokeOrderView` traces them in order; `showCharacterSheet` shows a character with its stroke order, meanings, readings and components, from a studied character's answer and from the dictionary's characters. **Writing by hand** (decided 2026-10-09, after Kanji Dojo's practice): the study option Write by hand (`StudyOptions.writeByHand`, settings schema 7) has character cards and words written in the session, mode `write` on a schedule of its own (words so written are neither quizzed nor typed). `WritingCanvas` takes raw pointers and claims every touch on it at once (an `EagerGestureRecognizer`), so a stroke never scrolls or swipes to rate; each stroke is checked as it ends (`checkStroke`, `kanji/model/stroke_check.dart`: both resampled to 22 points, shape compared once centred, position, length and direction, against what a loose hand strays by; a stroke closer to one still to come is out of order), a right one drawn as KanjiVG draws it, and the hand's offset from the middle forgiven in the strokes after. Mistakes are strokes that didn't count and hints asked for; two misses in a row show the next stroke faintly. `WritingBody` writes a word character by character, filling in those without a stroke order, and once written (or given up on, Show it) outlines the rating the mistakes suggest (`suggestedRating`: none Good, one or two OK, more Bad; given up, Bad). `stroke_check_test.dart` holds the checker to real strokes, moved, wavering, larger and smaller, reversed and out of order: change its weights only with that test passing.

View file

@ -1,4 +1,5 @@
# The Halkyon Learn instance: Caddy in front, the Serverpod server, PostgreSQL.
# The Halkyon Learn instance: Caddy in front, the Serverpod server, PostgreSQL, and
# the dictionaries kept current beside them.
#
# cd deploy
# cp .env.example .env && chmod 600 .env # then fill in the secrets
@ -57,15 +58,36 @@ services:
FILE_LIMIT_ANONYMOUS_MB: ${FILE_LIMIT_ANONYMOUS_MB:-}
FILE_LIMIT_IDENTIFIED_MB: ${FILE_LIMIT_IDENTIFIED_MB:-}
FILE_LIMIT_INSTANCE_MB: ${FILE_LIMIT_INSTANCE_MB:-}
# The dictionary files it loads at start, those that changed since (see the
# server's README); read only, and empty unless the operator adds some.
# The dictionary files it loads at start and rereads daily, those that changed
# (see the server's README): those the dictionaries service keeps current, read
# only. To supply files yourself instead, remove that service and mount
# ./dictionaries here (`- ./dictionaries:/app/dictionaries:ro`).
volumes:
- ./dictionaries:/app/dictionaries:ro
- dictionaries:/app/dictionaries:ro
depends_on:
postgres:
condition: service_healthy
logging: *logging
# Keeps the dictionary files current from EDRDG's files of the day, once they are a
# week old (EDRDG's licence asks it of servers showing its dictionaries at least
# monthly; the curator's README): in a container of its own, the curator never in
# the server's image or process. It only downloads (some 25 MB a week) and writes
# into the volume; the files are never backed up, as they can be had again.
dictionaries:
build:
context: ..
dockerfile: halkyon_learn_curator/Dockerfile
image: halkyon-learn-dictionaries
restart: unless-stopped
# Docker's stop signal reaches the curator through a small init.
init: true
# A conversion takes some 400 MB for a minute a week.
mem_limit: 1g
volumes:
- dictionaries:/dictionaries
logging: *logging
postgres:
image: postgres:16
restart: unless-stopped
@ -88,4 +110,5 @@ services:
volumes:
caddy_data:
caddy_config:
dictionaries:
postgres_data:

View file

@ -1,5 +1,7 @@
# Dictionary files
Dictionary files (`*.jsonl`, `*.jsonl.gz`) for the server to load at start; see
the server's README. They are downloads turned into files by `halkyon_learn_curator`,
not kept in git.
For an instance that supplies its dictionary files itself, rather than the
`dictionaries` service keeping them current (`compose.yaml`): put the files
(`*.jsonl`, `*.jsonl.gz`, made by `halkyon_learn_curator`) here and mount this
folder into the server as the comment there says. Mind EDRDG's licence: a server
showing its dictionaries must update them at least once a month. Not kept in git.

View file

@ -58,7 +58,7 @@ main() {
trap 'git reset -q --hard "$deployed"; echo "The checkout is back at $deployed." >&2' EXIT
cd deploy
# Build while the running server keeps serving; then dump, migrate, switch.
docker compose build -q server
docker compose build -q server dictionaries
dump
# Migrations as a run of their own, before the switch: Serverpod's maintenance role
# applies them and exits with 0 only if that worked. Each migration is one

View file

@ -0,0 +1,51 @@
# The dictionaries' image: the curator, keeping a server's dictionary files current
# from EDRDG's files of the day (`curate update`, the README). Built from the
# repository root (the build context), where the workspace's root pubspec and lockfile
# live:
#
# docker build -f halkyon_learn_curator/Dockerfile .
#
# A container of its own beside the server (deploy/compose.yaml, decided 2026-10-09):
# the curator (AGPL) never enters the server's image or process, and a conversion
# that fails can't touch the server. They share only the folder of files.
# ── Build stage: compile the curator to a native executable ───────────────────────
FROM dart:3.13.4 AS build
WORKDIR /app
COPY pubspec.yaml pubspec.lock ./
COPY halkyon_learn_curator halkyon_learn_curator
# The workspace root with only the curator package, as the server's Dockerfile does it
# with the server's. The check fails the build loudly if the member list changes shape.
RUN sed -i -e '/^ - halkyon_learn_client$/d' -e '/^ - halkyon_learn_flutter$/d' \
-e '/^ - halkyon_learn_server$/d' pubspec.yaml \
&& grep -q '^ - halkyon_learn_curator$' pubspec.yaml \
&& ! grep -q -e 'halkyon_learn_client' -e 'halkyon_learn_flutter' \
-e 'halkyon_learn_server' pubspec.yaml
RUN dart pub get
WORKDIR /app/halkyon_learn_curator
RUN dart build cli --target bin/curate.dart --output build
# ── Runtime stage ─────────────────────────────────────────────────────────────────
# Ubuntu 24.04 as the server's, with CA certificates for EDRDG's HTTPS.
FROM ubuntu:24.04
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=build /app/halkyon_learn_curator/build/bundle/ ./
COPY --from=build /app/halkyon_learn_curator/data/ data/
# An unprivileged user of its own: user ID 10001, shared with neither the containers
# of PostgreSQL and the server (999) nor the CI runner on the host (61000). The folder
# it writes into belongs to it, which a new volume mounted there takes over.
RUN useradd --system --uid 10001 --no-create-home --shell /usr/sbin/nologin curator \
&& mkdir /dictionaries \
&& chown curator /dictionaries
USER curator
# Updates the files whenever they are a week old (EDRDG's licence asks at least
# monthly), and sleeps in between.
ENTRYPOINT ["./bin/curate", "update", "--into", "/dictionaries", "--data", "data"]
CMD ["--every", "7"]

View file

@ -14,16 +14,25 @@ When you are finished, you can shut down the running server with `Q`.
The dictionaries people look words up in (JMdict, …) are reference data the
server holds for everyone. They come as dictionary files, which
`halkyon_learn_curator` makes from the sources' downloads (its README says how).
At every start, the server loads the files in its `dictionaries` folder that
changed since they were last loaded (by their SHA-256): in the background, a
source at a time, and all at once, so searches meanwhile still find the old
entries. A file it can't read is logged and left out.
At every start, and daily at 04:07, the server loads the files in its
`dictionaries` folder that changed since they were last loaded (by their
SHA-256): in the background, a source at a time, and all at once, so searches
meanwhile still find the old entries. A file it can't read is logged and left
out. `DICTIONARIES_DIR` names another folder.
- **Development:** put the file into `halkyon_learn_server/dictionaries/` and
restart `serverpod start`.
- **An instance:** put it into `deploy/dictionaries/` beside `compose.yaml`
(mounted read-only as `/app/dictionaries`) and restart the server container.
`DICTIONARIES_DIR` names another folder.
EDRDG's licence asks a server showing its dictionaries to update them at least
once a month:
- **An instance** keeps them current by itself: the `dictionaries` service in
`deploy/compose.yaml` (the curator, in a container of its own) downloads
EDRDG's files of the day once its files are a week old, converts them, and
replaces them in a volume the server reads (`curate update`, the curator's
README); the server takes them up the next morning. To supply files yourself
instead, remove the service and mount `deploy/dictionaries/` into the server
(`compose.yaml` says how), and update them monthly.
- **Development:** `dart run halkyon_learn_curator:curate update --into
../halkyon_learn_server/dictionaries` in the curator's folder (about a
minute), then restart `serverpod start`, or wait for the morning.
JMdict in English, with the JLPT levels and Tatoeba's example sentences, is
about 20 MB gzipped and takes the server half a minute to load (219,000

View file

@ -42,22 +42,25 @@ Future<void> run(List<String> args) async {
.filePruning
.pruneFiles();
// The dictionaries the operator put into the server's folder, those that
// changed since; in the background, as loading takes a minute or two.
// The dictionaries in the server's folder, those that changed since; in
// the background, as loading takes a minute or two. Then again daily, as
// the dictionaries service replaces them weekly.
unawaited(_loadDictionaries(pod));
await pod.futureCalls.cancel(_dictionaryReload);
await pod.futureCalls
.callRecurring(identifier: _dictionaryReload)
.cron('7 4 * * *')
.dictionaryReload
.reloadDictionaries();
}
/// Loads the dictionaries the operator put into the server's folder (or
/// `DICTIONARIES_DIR`). [loadDictionaries] throws nothing; a session that
/// can't be had is reported here, as an error in the background would end
/// the server.
/// Loads the dictionaries in the server's folder ([dictionaryFolder]).
/// [loadDictionaries] throws nothing; a session that can't be had is
/// reported here, as an error in the background would end the server.
Future<void> _loadDictionaries(Serverpod pod) async {
try {
await pod.withSession(
(session) => loadDictionaries(
session,
Directory(Platform.environment['DICTIONARIES_DIR'] ?? 'dictionaries'),
),
(session) => loadDictionaries(session, dictionaryFolder()),
);
} on Object catch (error) {
stderr.writeln('Loading the dictionaries failed: $error');
@ -69,3 +72,6 @@ const _pruning = 'prune-abandoned-anonymous-people';
/// The identifier of the daily pruning of files.
const _filePruning = 'prune-unreferenced-files';
/// The identifier of the daily rereading of the dictionaries.
const _dictionaryReload = 'reload-dictionaries';

View file

@ -15,6 +15,16 @@ const dictionaryFormatVersion = 1;
/// How many entries go into the database at once.
const _batch = 1000;
/// The folder of dictionary files: `DICTIONARIES_DIR`, else `dictionaries`
/// in the server's folder.
Directory dictionaryFolder() =>
Directory(Platform.environment['DICTIONARIES_DIR'] ?? 'dictionaries');
/// The loads asked for, one after another: two at once would both replace
/// a changed source, the second failing on the first's rows (and logging
/// it) for nothing.
Future<void> _loads = Future.value();
/// Loads every dictionary file in [directory] (`*.jsonl`, or gzipped
/// `*.jsonl.gz`) that changed since it was loaded: the operator drops a
/// file the curator wrote into the server's `dictionaries` folder, and the
@ -25,8 +35,11 @@ const _batch = 1000;
/// beside the new one) are refused: neither loads, as which is meant is
/// unclear, and loading both would replace the source twice at every start.
/// Nothing here throws: the server starts it in the background, where an
/// error would end the process.
Future<void> loadDictionaries(Session session, Directory directory) async {
/// error would end the process. Loads run one at a time.
Future<void> loadDictionaries(Session session, Directory directory) =>
_loads = _loads.then((_) => _loadFolder(session, directory));
Future<void> _loadFolder(Session session, Directory directory) async {
void failed(String message, Object error, StackTrace st) => session.log(
message,
level: LogLevel.error,

View file

@ -0,0 +1,11 @@
import 'package:halkyon_learn_server/src/dictionary/dictionary_loader.dart';
import 'package:serverpod/serverpod.dart';
/// Rereads the dictionary folder and loads the files that changed: the
/// dictionaries service replaces them weekly (`deploy/compose.yaml`); the
/// server schedules it daily when it starts (`server.dart`).
class DictionaryReloadFutureCall extends FutureCall {
/// Loads the dictionary files that changed since they were loaded.
Future<void> reloadDictionaries(Session session) =>
loadDictionaries(session, dictionaryFolder());
}

View file

@ -16,6 +16,7 @@ import 'package:clock/clock.dart' as _io0w16m8;
import 'package:serverpod/serverpod.dart' as _is;
import '../asset/file_pruning_future_call.dart' as _ie5h5r65;
import '../auth/pruning_future_call.dart' as _igevg04y;
import '../dictionary/dictionary_reload_future_call.dart' as _ikbvcxb6;
/// Invokes a future call.
typedef _InvokeFutureCall =
@ -62,6 +63,8 @@ class FutureCalls extends _is.FutureCallDispatch<_FutureCallRef> {
'FilePruningPruneFilesFutureCall': FilePruningPruneFilesFutureCall(),
'PruningPruneAnonymousPeopleFutureCall':
PruningPruneAnonymousPeopleFutureCall(),
'DictionaryReloadReloadDictionariesFutureCall':
DictionaryReloadReloadDictionariesFutureCall(),
};
_futureCallManager = futureCallManager;
_serverId = serverId;
@ -185,6 +188,10 @@ class _FutureCallRef {
late final filePruning = _FilePruningFutureCallDispatcher(_invokeFutureCall);
late final pruning = _PruningFutureCallDispatcher(_invokeFutureCall);
late final dictionaryReload = _DictionaryReloadFutureCallDispatcher(
_invokeFutureCall,
);
}
class _FilePruningFutureCallDispatcher {
@ -213,6 +220,19 @@ class _PruningFutureCallDispatcher {
}
}
class _DictionaryReloadFutureCallDispatcher {
_DictionaryReloadFutureCallDispatcher(this._invokeFutureCall);
final _InvokeFutureCall _invokeFutureCall;
Future<void> reloadDictionaries() {
return _invokeFutureCall(
'DictionaryReloadReloadDictionariesFutureCall',
null,
);
}
}
/// Prunes one batch and logs how many files went.
class FilePruningPruneFilesFutureCall extends _is.FutureCall
implements _is.InvokableFutureCall {
@ -236,3 +256,15 @@ class PruningPruneAnonymousPeopleFutureCall extends _is.FutureCall
await _igevg04y.PruningFutureCall().pruneAnonymousPeople(session);
}
}
/// Loads the dictionary files that changed since they were loaded.
class DictionaryReloadReloadDictionariesFutureCall extends _is.FutureCall
implements _is.InvokableFutureCall {
@override
_ida.Future<void> invoke(
_is.Session session,
_is.SerializableModel? object,
) async {
await _ikbvcxb6.DictionaryReloadFutureCall().reloadDictionaries(session);
}
}

View file

@ -395,6 +395,19 @@ void main() {
expect(await search('|'), isEmpty);
});
test('loads asked for at once leave the dictionary loaded once, '
'whole', () async {
write([taberu, terebi]);
await Future.wait([
loadDictionaries(session, folder),
loadDictionaries(session, folder),
]);
expect(await DictionaryEntry.db.count(session), 2);
expect(await DictionarySource.db.count(session), 1);
});
test('two files of one source are refused, and neither loads, as '
'which one is meant is unclear', () async {
write([taberu], name: 'jmdict-en-old.jsonl');

View file

@ -228,6 +228,8 @@ class _FutureCalls {
late final filePruning = _FilePruningFutureCall();
late final pruning = _PruningFutureCall();
late final dictionaryReload = _DictionaryReloadFutureCall();
}
class _AssetEndpoint {
@ -847,3 +849,20 @@ class _PruningFutureCall {
}
}
}
class _DictionaryReloadFutureCall {
Future<void> reloadDictionaries(
_ist.TestSessionBuilder sessionBuilder,
) async {
var _localUniqueSession =
(sessionBuilder as _ist.InternalTestSessionBuilder).internalBuild();
try {
await _iip9210p.DictionaryReloadReloadDictionariesFutureCall().invoke(
_localUniqueSession,
null,
);
} finally {
await _localUniqueSession.close();
}
}
}