halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 37 Packages Wiki Activity Actions 1

feat(curator): curate update keeps a server's dictionaries current

Browse source 
EDRDG's licence asks of servers showing its dictionaries that they update
them at least monthly. `curate update --into <folder>` downloads EDRDG's
files of the day (JMdict with Tatoeba's sentences, KANJIDIC2), converts
them with the JLPT levels the curator now brings along (yomitan-jlpt-vocab's
lists, CC BY-SA 4.0, and the kanji levels of kanji-data's kanji.json: both
static, in data/ with their origin), and puts each file in place at once
(written beside it as .part, then renamed over it), so the server never
reads half a file; a failure leaves the files as they were. With --every 7
it keeps running and updates whenever the files are a week old, as their
dates say, so a restart doesn't download again; after a failure it tries
again in six hours. A real update takes some 50 seconds and 400 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-09 16:40:19 +02:00
commit 82b38fadd8
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
1 changed files with 9047 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

20
halkyon_learn_curator/README.md
View file

@ -52,6 +52,26 @@ KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:
Put the files into the server's `dictionaries` folder. The server loads each on
its next start, and again whenever it changes (see the server's README).
## Keeping a server's dictionaries current
EDRDG's licence asks of a server showing its dictionaries that it update them
at least once a month. `curate update` does it from the sources alone:
```sh
dart run halkyon_learn_curator:curate update \
--into ../halkyon_learn_server/dictionaries [--every 7]
```
It downloads EDRDG's files of the day (`JMdict_e_examp.gz`, `kanjidic2.xml.gz`),
converts them with the JLPT levels in `data/` (static lists, see
`data/README.md`), and writes `jmdict-en.jsonl.gz` and `kanjidic-en.jsonl.gz`
into the folder, each beside its place first and then renamed over it, so a
server never reads half a file; should anything fail, the files stay as they
were. Some 50 seconds and 400 MB. With `--every 7` it keeps running and updates
whenever the files are a week old, as their dates say (a restart doesn't
download again), and tries again six hours after a failure. An instance runs it
so in a container of its own (the server's README).
## The format (`halkyon-dictionary`, version 1)
[JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`.