feat(server): dictionaries, loaded from files and searched

The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.

DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).

The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-09 00:51:27 +02:00
commit 3013bc04b3
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64

View file

@ -144,6 +144,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
- **Tree, tags, modes, settings and files** (ADR 0004): a studyset's `parent` and sibling `position` form the tree, and studysets in a cycle or without their parent show at the top level (`topLevel`); its `archivedAt`, `starredAt` and `updatedAt` are columns, and every change of a studyset writes `updatedAt` with it (the library lists starred studysets first, then the most recently changed); a card's tags are `card_tag` rows, one per assignment, never updated, and a duplicate counts once; `review_log.mode` is `review`, `reverse` (its own schedule) or `cram` (never moves a schedule), and a schedule replays the reviews of one mode; `person_settings` is meant as one row per person, and extra rows fold into the one with the smallest id; an `asset` row stands for a file in private cloud storage, found by its hash. **Never derive a row's id from a person or anything else guessable**: ids are unique across the whole table, so someone could claim it first in their own space and break the owner's sync.
- **Files** (ADR 0004, rule 5): `AssetEndpoint` (`requireLogin`) takes uploads into the caller's own space by SHA-256 (`startUpload` gives Serverpod's upload description, or null when the file is there already and claims it anew; `finishUpload` keeps it only if its content has that hash) and serves an asset row's file to its space's owner and members only, `NotFoundException` otherwise, also for a deleted row (the sync engine hides tombstones). Files live in the private database storage at `spaces/<space>/<sha256>`, at most 20 MB each, and within limits (`file_limits.dart`, from the environment: an anonymous person 100 MB, an identified one 500 MB or what a `file_allowance` row grants them, the instance 15 GB), checked when an upload starts (unfinished uploads count, or they could fill the disk unnoticed) and again when it finishes (only what is kept counts), refused with `FileLimitExceededException`; `deletePerson` deletes the person's files and allowance, and `FilePruningFutureCall` deletes daily those no live row refers to (or never finished), 7 days after they were last claimed. In the app, `AssetRepository` (`core/asset/`) keeps files by hash in the person's folder (`files/`, uploads waiting marked in `files/uploads/`), uploads after each sync that went through, and downloads on demand, checking the hash; every hash is checked to be one before it becomes part of a path. Content names a file by the URL `asset:<asset row id>` (`assetUrl`, `RichContent.assetIds`): images are the editor's `image` block, sounds an `audio` block (`core/editor/file_blocks.dart`), and any other URL is never fetched. `StoredFileImage` loads through the repository like Flutter's `NetworkImage`, keyed by the person's repository too, so no image crosses people.
- **PDFs** (`feature/pdf_source/`): a `pdf_source` row per PDF in a studyset (payload: title, file name, the asset of its file, page count, reading deadline; last writer wins) with its `position` among the studyset's files in a column of its own (decided 2026-10-08; the Files tab's default order, Custom order, which Reorder changes; PDFs from before it stand at `a0`, in the order they were added) and an append-only `pdf_reading` log (`reached` with a page, `reset`): the furthest page is the furthest reached since the last reset, so devices never conflict. A PDF's file is an asset of up to 50 MB (other files 20 MB): the server takes up to 50 MB and deletes a stored file over 20 MB that doesn't start with `%PDF-`. Deleting a PDF deletes its asset row too unless another PDF has the same file (asset rows are otherwise never deleted). Cards and notes tied to a page keep it in their payload (`source`, `core/content/pdf_page_ref.dart`; card schema 4, note schema 2); a screenshot is an `image` block with `pdf` and `page` attributes.
- **Dictionaries** (decided 2026-10-08): reference data the server holds for everyone, owned by no one, never synced (`dictionary_entry`, `dictionary_source`, `lib/src/dictionary/`). `halkyon_learn_curator` (AGPL, no Serverpod, so the server never links its code) turns each source's download into one format, a dictionary file (its README is the contract), with an adapter per source; everything specific to a language (kana, romaji, meanings without "to") goes into each entry's `keys` and `looseKeys`, so the server's search knows no language: exact key, exact loose key, key starting with the query, loose key starting with it, each by the source's `rank` (`DictionaryEndpoint.search`, LIKE on barred keys, no extension needed). The server loads the files in its `dictionaries` folder at start, those whose SHA-256 changed, each source replaced in one transaction (`dictionary_loader.dart`). A card made from an entry copies what it needs and keeps `(source, sourceId)` as a soft reference only, so loading a newer version never touches cards. Every file carries its credits, which the app shows.
- **The Pomodoro timer** (`feature/pomodoro/`): the profiles follow the person (`PersonSettings.pomodoro`, settings schema 4), the running timer is the device's own (`PomodoroRun` in `<person folder>/pomodoro.json`, `PomodoroRunStore`). `PomodoroRun` is a pure state machine on the time: a phase that ends leads to the next, running if its profile starts it by itself, else paused at its full length; a timer kept while the app was closed catches up (`caughtUpTo`). `PomodoroBloc` sits above the router (`PersonApp`), and `PomodoroPresenter` hands every change to `PomodoroPlatform` as a `PomodoroNotice` in the app's language, and its buttons back. **Android:** the app's own Kotlin (`PomodoroService.kt`; MethodChannel `app.halkyon.learn/pomodoro`, EventChannel `…/buttons`): a `specialUse` foreground service with a countdown notification (chronometer), Pause/Resume, Skip, Stop, on Android 16 a Live Update (`NotificationCompat.ProgressStyle`, the cycle's phases as segments, `setRequestPromotedOngoing`); an exact alarm (`USE_EXACT_ALARM`) alerts at the phase's end with the app closed; Do Not Disturb through the interruption filter, the one before kept in shared preferences and restored. With the app swiped away its buttons open it (a service may not start an activity from a notification's button since Android 12, so the notification is posted again with activity intents), and Stop works alone, leaving a mark the app takes at its next start (`takeStoppedOutside`) to forget its timer; a profile's next phase only starts by itself while the app runs. Dart's timers sleep with the device, so `PomodoroBloc` also looks every 15 s while a phase runs, and the alarm's receiver wakes it (`phaseEnds`); a kept timer is caught up before it first shows, or its ended phase would alert again. No notification plugin: flutter_local_notifications had Live Updates only in a prerelease needing compileSdk 37 (checked 2026-10-07). **Linux:** `LinuxPomodoroPlatform`, over package:dbus: where the top bar shows icons (GNOME with the AppIndicator extension, built into Ubuntu; KDE) the timer counts down there with its buttons in the icon's menu (`StatusNotifierItem`, written in Dart and laid out as libayatana-appindicator lays itself out, within the snap's rules, `desktop-legacy` of snapd v2.77: the item at `/org/ayatana/NotificationItem/<id>`, the only path the label's signal `XAyatanaNewLabel` may come from, registered with the watcher by that path, since a snap may bind no `org.freedesktop.StatusNotifierItem-*` name; the menu at `/MenuBar`; registered again when the watcher restarts); elsewhere the timer shows in the app alone, and on GNOME the Pomodoro card suggests the extension until dismissed, asking GNOME Shell to install or switch it on (`InstallRemoteExtension`, `EnableExtension`) or opening its page (decided 2026-10-07: no notification stands in; xdg-desktop-portal's Background status is for Flatpaks only). An alert with sound comes through `org.freedesktop.Notifications` when a phase ends while the app runs, and GNOME's `show-banners` goes off while focusing, through `gsettings`; the value before and the dismissed suggestion are kept in `<app support>/`. Calls to the bus go one at a time (`SerialQueue`).
- **Citations** (`core/citation/`): a `Source` is plain text parts as the person typed them (authors as full names, one per line; year; title; `container`, what it is a part of; publisher; ISBN; link; `raw`, a citation a file brought, shown while nothing else is cited), and `formatCitation` cites it in the device's `CitationStyle` (APA 7, MLA 9, Chicago 18 notes and bibliography, decided 2026-10-07): it splits and abbreviates the names itself (inverted names, particles, suffixes; names written with `&` or "et al." stay as written), sets a work of its own in italics and a part of something apart from the whole by `container`, and writes a missing year as "n.d." in APA and Chicago. A step's sources (step schema 2) and a studyset's bibliography (`bibliography`, studyset schema 4, last writer wins like the rest of its payload) are lists of them, shown and changed through `SourceList` and `showSourcesSheet`, in reference-list order (alphabetical, whatever order they were added in); the path import reads them too, text as `raw` and the old format's `citation` as `container`.
- **What a card carries beside its type's fields** (card schema 5): up to six `hints` and an `explanation` (rich text, any type), a `report` that something on it is wrong (category, note, when; studied as before, its badge in the list), and a word's `senses` (glosses, label, JMdict's grammar codes, whose English short forms and meanings are data in `feature/card/model/jmdict_tags.dart`; with senses a word's meanings are their glosses, and `uk` on its first sense shows its reading from the start). Each merges as a field of its own. Deleting a card goes through `CardDeletions` (a `DelayedDeletions`, as notes and studysets), so the list and the session offer Undo. A study session's Undo takes back its steps last first (ratings, burials and deletions alike, across its rounds), so it holds the cards it deletes (`hold`) until it ends, when they go after the usual time to undo; a card deleted in a session that never ended (the app killed) stays. Vocabulary can be quizzed as multiple choice (`quizVocabulary`, `quizAspect` in the study options, settings schema 5; `vocabularyQuiz` draws the wrong answers from the other words studied).