2026-09-16 19:24:59 +02:00
|
|
|
# halkyon_learn_server
|
|
|
|
|
|
|
|
|
|
This is the starting point for your Serverpod server.
|
|
|
|
|
|
|
|
|
|
Start your server by running:
|
|
|
|
|
|
|
|
|
|
cd halkyon_learn
|
|
|
|
|
serverpod start
|
|
|
|
|
|
|
|
|
|
When you are finished, you can shut down the running server with `Q`.
|
2026-09-30 12:35:46 +02:00
|
|
|
|
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
## Dictionaries
|
|
|
|
|
|
|
|
|
|
The dictionaries people look words up in (JMdict, …) are reference data the
|
|
|
|
|
server holds for everyone. They come as dictionary files, which
|
|
|
|
|
`halkyon_learn_curator` makes from the sources' downloads (its README says how).
|
feat: a dictionary's corrections reach the cards made from it, by offer
Every dictionary entry now has a version: 16 hex digits of the SHA-256 of
what cards take of it (headword, reading, entry), made by the loader and
carried on every hit. A card keeps the version it took (card schema 8).
DictionaryEndpoint.changed returns, of the entries named with the
versions cards keep (1,000 at most a call; the app asks 500 at a time),
those whose version is another now.
The card browser says how many cards are behind their dictionary and takes
the corrections at a tap (decided 2026-10-09: by offer, never by
themselves). Taking one writes the entry as it is now over the revision
in which the card last took it, beside whatever came after, so the
revisions merge per side as two devices' edits do (ADR 0003): a side only
the dictionary changed takes the correction, a side the person changed
too is a conflict settled in the composer, whose words now name that
case. Cards made before versions are never offered one.
The server rereads its dictionary folder hourly now, not daily: the
dictionaries service fills its volume after an instance's first start,
and an hour with nothing changed costs a hash of each file.
Migration 20261009145434785-dictionary-entry-version: adds
dictionary_entry.version, text NOT NULL DEFAULT '' (rows loaded before
have none and are never reported as corrected), and by hand empties
dictionary_source."fileHash", so every source loads once more and its
entries get their versions; the schema stays as definition.sql says. The
old server goes on working until the switch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 17:32:00 +02:00
|
|
|
At every start, and every hour at seven past, the server loads the files in
|
|
|
|
|
its `dictionaries` folder that changed since they were last loaded (by their
|
feat(deploy): instances keep their dictionaries current, weekly, by themselves
EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.
An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:52:37 +02:00
|
|
|
SHA-256): in the background, a source at a time, and all at once, so searches
|
|
|
|
|
meanwhile still find the old entries. A file it can't read is logged and left
|
|
|
|
|
out. `DICTIONARIES_DIR` names another folder.
|
|
|
|
|
|
|
|
|
|
EDRDG's licence asks a server showing its dictionaries to update them at least
|
|
|
|
|
once a month:
|
|
|
|
|
|
|
|
|
|
- **An instance** keeps them current by itself: the `dictionaries` service in
|
|
|
|
|
`deploy/compose.yaml` (the curator, in a container of its own) downloads
|
|
|
|
|
EDRDG's files of the day once its files are a week old, converts them, and
|
|
|
|
|
replaces them in a volume the server reads (`curate update`, the curator's
|
feat: a dictionary's corrections reach the cards made from it, by offer
Every dictionary entry now has a version: 16 hex digits of the SHA-256 of
what cards take of it (headword, reading, entry), made by the loader and
carried on every hit. A card keeps the version it took (card schema 8).
DictionaryEndpoint.changed returns, of the entries named with the
versions cards keep (1,000 at most a call; the app asks 500 at a time),
those whose version is another now.
The card browser says how many cards are behind their dictionary and takes
the corrections at a tap (decided 2026-10-09: by offer, never by
themselves). Taking one writes the entry as it is now over the revision
in which the card last took it, beside whatever came after, so the
revisions merge per side as two devices' edits do (ADR 0003): a side only
the dictionary changed takes the correction, a side the person changed
too is a conflict settled in the composer, whose words now name that
case. Cards made before versions are never offered one.
The server rereads its dictionary folder hourly now, not daily: the
dictionaries service fills its volume after an instance's first start,
and an hour with nothing changed costs a hash of each file.
Migration 20261009145434785-dictionary-entry-version: adds
dictionary_entry.version, text NOT NULL DEFAULT '' (rows loaded before
have none and are never reported as corrected), and by hand empties
dictionary_source."fileHash", so every source loads once more and its
entries get their versions; the schema stays as definition.sql says. The
old server goes on working until the switch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 17:32:00 +02:00
|
|
|
README); the server takes them up within the hour. To supply files yourself
|
feat(deploy): instances keep their dictionaries current, weekly, by themselves
EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.
An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:52:37 +02:00
|
|
|
instead, remove the service and mount `deploy/dictionaries/` into the server
|
|
|
|
|
(`compose.yaml` says how), and update them monthly.
|
|
|
|
|
- **Development:** `dart run halkyon_learn_curator:curate update --into
|
|
|
|
|
../halkyon_learn_server/dictionaries` in the curator's folder (about a
|
feat: a dictionary's corrections reach the cards made from it, by offer
Every dictionary entry now has a version: 16 hex digits of the SHA-256 of
what cards take of it (headword, reading, entry), made by the loader and
carried on every hit. A card keeps the version it took (card schema 8).
DictionaryEndpoint.changed returns, of the entries named with the
versions cards keep (1,000 at most a call; the app asks 500 at a time),
those whose version is another now.
The card browser says how many cards are behind their dictionary and takes
the corrections at a tap (decided 2026-10-09: by offer, never by
themselves). Taking one writes the entry as it is now over the revision
in which the card last took it, beside whatever came after, so the
revisions merge per side as two devices' edits do (ADR 0003): a side only
the dictionary changed takes the correction, a side the person changed
too is a conflict settled in the composer, whose words now name that
case. Cards made before versions are never offered one.
The server rereads its dictionary folder hourly now, not daily: the
dictionaries service fills its volume after an instance's first start,
and an hour with nothing changed costs a hash of each file.
Migration 20261009145434785-dictionary-entry-version: adds
dictionary_entry.version, text NOT NULL DEFAULT '' (rows loaded before
have none and are never reported as corrected), and by hand empties
dictionary_source."fileHash", so every source loads once more and its
entries get their versions; the schema stays as definition.sql says. The
old server goes on working until the switch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 17:32:00 +02:00
|
|
|
minute); the server takes them up within the hour.
|
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
|
2026-10-09 13:36:16 +02:00
|
|
|
JMdict in English, with the JLPT levels and Tatoeba's example sentences, is
|
|
|
|
|
about 20 MB gzipped and takes the server half a minute to load (219,000
|
2026-10-09 16:05:05 +02:00
|
|
|
entries). KANJIDIC2 in English, with the JLPT levels of the kanji and the kana,
|
|
|
|
|
is a dictionary of characters beside it (`kanjidic-en.jsonl.gz`, 0.7 MB,
|
|
|
|
|
12,800 entries): the app's Kanji & kana tab. Their licences (CC BY-SA 4.0;
|
|
|
|
|
CC BY 2.0 FR for the sentences; CC BY for the JLPT lists) ask for attribution:
|
|
|
|
|
the app lists the credits each file carries.
|
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
|
2026-09-30 12:35:46 +02:00
|
|
|
## Licence
|
|
|
|
|
|
|
|
|
|
Copyright (C) 2026 The Halkyon Learn contributors
|
|
|
|
|
|
|
|
|
|
Licensed under the Server Side Public License, version 1 (`SSPL-1.0`); the full text
|
|
|
|
|
is in `LICENSE`.
|