2026-09-16 19:24:59 +02:00
|
|
|
# halkyon_learn_server
|
|
|
|
|
|
|
|
|
|
This is the starting point for your Serverpod server.
|
|
|
|
|
|
|
|
|
|
Start your server by running:
|
|
|
|
|
|
|
|
|
|
cd halkyon_learn
|
|
|
|
|
serverpod start
|
|
|
|
|
|
|
|
|
|
When you are finished, you can shut down the running server with `Q`.
|
2026-09-30 12:35:46 +02:00
|
|
|
|
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
## Dictionaries
|
|
|
|
|
|
|
|
|
|
The dictionaries people look words up in (JMdict, …) are reference data the
|
|
|
|
|
server holds for everyone. They come as dictionary files, which
|
|
|
|
|
`halkyon_learn_curator` makes from the sources' downloads (its README says how).
|
feat(deploy): instances keep their dictionaries current, weekly, by themselves
EDRDG's licence asks of a server showing its dictionaries that it update
them at least monthly. The instance now does it alone: a `dictionaries`
service in compose, built from the curator (halkyon_learn_curator/
Dockerfile, its own unprivileged user 10001, 1 GB at most), runs `curate
update --every 7`, downloading EDRDG's files of the day once its files are
a week old and replacing them in a volume the server mounts read-only.
The curator never enters the server's image or process (decided
2026-10-09: a container of its own, weekly). The server rereads the
folder daily at 04:07 (DictionaryReloadFutureCall), loading what changed,
and loads now run one at a time. redeploy.sh builds both images.
An operator who'd rather supply the files removes the service and mounts
deploy/dictionaries/ into the server, as compose.yaml says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:52:37 +02:00
|
|
|
At every start, and daily at 04:07, the server loads the files in its
|
|
|
|
|
`dictionaries` folder that changed since they were last loaded (by their
|
|
|
|
|
SHA-256): in the background, a source at a time, and all at once, so searches
|
|
|
|
|
meanwhile still find the old entries. A file it can't read is logged and left
|
|
|
|
|
out. `DICTIONARIES_DIR` names another folder.
|
|
|
|
|
|
|
|
|
|
EDRDG's licence asks a server showing its dictionaries to update them at least
|
|
|
|
|
once a month:
|
|
|
|
|
|
|
|
|
|
- **An instance** keeps them current by itself: the `dictionaries` service in
|
|
|
|
|
`deploy/compose.yaml` (the curator, in a container of its own) downloads
|
|
|
|
|
EDRDG's files of the day once its files are a week old, converts them, and
|
|
|
|
|
replaces them in a volume the server reads (`curate update`, the curator's
|
|
|
|
|
README); the server takes them up the next morning. To supply files yourself
|
|
|
|
|
instead, remove the service and mount `deploy/dictionaries/` into the server
|
|
|
|
|
(`compose.yaml` says how), and update them monthly.
|
|
|
|
|
- **Development:** `dart run halkyon_learn_curator:curate update --into
|
|
|
|
|
../halkyon_learn_server/dictionaries` in the curator's folder (about a
|
|
|
|
|
minute), then restart `serverpod start`, or wait for the morning.
|
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
|
2026-10-09 13:36:16 +02:00
|
|
|
JMdict in English, with the JLPT levels and Tatoeba's example sentences, is
|
|
|
|
|
about 20 MB gzipped and takes the server half a minute to load (219,000
|
2026-10-09 16:05:05 +02:00
|
|
|
entries). KANJIDIC2 in English, with the JLPT levels of the kanji and the kana,
|
|
|
|
|
is a dictionary of characters beside it (`kanjidic-en.jsonl.gz`, 0.7 MB,
|
|
|
|
|
12,800 entries): the app's Kanji & kana tab. Their licences (CC BY-SA 4.0;
|
|
|
|
|
CC BY 2.0 FR for the sentences; CC BY for the JLPT lists) ask for attribution:
|
|
|
|
|
the app lists the credits each file carries.
|
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
|
|
|
|
2026-09-30 12:35:46 +02:00
|
|
|
## Licence
|
|
|
|
|
|
|
|
|
|
Copyright (C) 2026 The Halkyon Learn contributors
|
|
|
|
|
|
|
|
|
|
Licensed under the Server Side Public License, version 1 (`SSPL-1.0`); the full text
|
|
|
|
|
is in `LICENSE`.
|