halkyon/halkyon_learn_curator/README.md

halkyon_learn_curator

Turns dictionary downloads into the dictionary files a Halkyon Learn server loads. It reads each source with an adapter of its own and writes one format for all of them, so a new source or language needs a new adapter, never a change to the server or the app. It doesn't depend on Serverpod or the server. It also makes the stroke files the app writes characters with (below).

Making a dictionary file

JMdict in English, with the JLPT levels and example sentences on top:

1. Download JMdict_e_examp.gz from EDRDG, which makes it anew every day: JMdict with Tatoeba's sentences (CC BY 2.0 FR) on the meanings they show. JMdict_e.gz works too, without them, and so does jmdict-simplified's JSON (jmdict-examples-eng-<version>.json, from its releases).

2. Download original_data/n1.csv … n5.csv from yomitan-jlpt-vocab into a folder.

3. Convert, in this package:

   dart run halkyon_learn_curator:curate jmdict \
     --input JMdict_e_examp.gz --jlpt jlpt/ \
     --output jmdict-en.jsonl.gz
   

   The whole of JMdict (about 219,000 entries) takes some 30 seconds and 300 MB of memory: entries are read one at a time. A file whose name holds .json is read as jmdict-simplified's JSON, any other as EDRDG's XML.

KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:

1. Download kanjidic2.xml.gz from the EDRDG.

2. Download kanji.json from kanji-data: Jonathan Waller's JLPT levels by kanji (jlpt_new; nothing else is read).

3. Convert, in this package:

   dart run halkyon_learn_curator:curate kanjidic \
     --input kanjidic2.xml.gz --jlpt kanji.json \
     --output kanjidic-en.jsonl.gz
   

   About 12,800 characters, in some 5 seconds.

Put the files into the server's dictionaries folder. The server loads each on its next start, and again whenever it changes (see the server's README).

Keeping a server's dictionaries current

EDRDG's licence asks of a server showing its dictionaries that it update them at least once a month. curate update does it from the sources alone:

dart run halkyon_learn_curator:curate update \
  --into ../halkyon_learn_server/dictionaries [--every 7]

It downloads EDRDG's files of the day (JMdict_e_examp.gz, kanjidic2.xml.gz), converts them with the JLPT levels in data/ (static lists, see data/README.md), and writes jmdict-en.jsonl.gz and kanjidic-en.jsonl.gz into the folder, each beside its place first and then renamed over it, so a server never reads half a file; should anything fail, the files stay as they were. Some 50 seconds and 400 MB. With --every 7 it keeps running and updates whenever the files are a week old, as their dates say (a restart doesn't download again), and tries again six hours after a failure. An instance runs it so in a container of its own (the server's README). Beside the files, .conversion says how the curator converted them (conversion in lib/src/update.dart): raise it whenever the curator writes something else from the same download (a label, a field), and an instance converts again as soon as the new curator runs, not a week later.

The format (halkyon-dictionary, version 1)

JSON Lines, gzipped when the name ends in .gz.

The first line says what the file holds:

Field
format, formatVersion	"halkyon-dictionary", 1
kind	words (a dictionary of words, JMdict) or characters (of the characters words are written in, KANJIDIC2). The app shows them apart.
source, sourceVersion	The source's name (jmdict) and version (3.6.2 (2026-10-05)). An instance holds one version of a source for each meaning language.
language, glossLanguage	BCP 47 tags: the words' language (ja) and the meanings' (en).
license	The data's licence, an SPDX identifier (CC-BY-SA-4.0).
credits	Everything it is drawn from: name, by, license, url. The app lists them where it credits its data sources.

Every further line is an entry:

Field
id	Its id in the source; with the source and meaning language, it names the entry for good.
headword, reading	How the word is shown, and how it is read; no reading when the headword is its own (a word written in kana).
labels	What it can be picked by: common, jlpt:n5 … jlpt:n1 (later e.g. hsk:3, topik:1); for characters also the school grade (grade:1 … grade:6, grade:secondary for the rest of the Jōyō kanji), a kanji's Kangxi radical (radical:1 … radical:214, KANJIDIC2's classical one) and the script (hiragana, katakana).
rank	Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency.
keys	What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language.
looseKeys	What a search matches after every keys match: spellings people type that aren't quite right (romaji with long vowels typed once, sushi for すうし). Left out when empty.
entry	The entry itself, as cards take it: forms and readings (text, common, the source's tags; for a reading the forms it belongs to), senses (glosses; the source's codes in pos, misc, field, dialect; info; the forms and readings a sense is limited to; examples, sentences using the word in that sense: text, the form it takes there, ja, the sentence, en, its translation, and the source's id, e.g. tatoeba). Lists are left out when empty. A character's: meanings; on and kun, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a ., a dash where it attaches to others); grade, strokes, frequency (its place among the 2,500 most used in newspapers) and jlpt where it has them. A kana's: romaji and script.

The app's stroke files

How every kanji and kana is written comes from KanjiVG and is bundled with the app, so writing works without a server. To update it, download kanjivg-<date>.xml.gz from KanjiVG's releases and, in this package:

dart run halkyon_learn_curator:curate kanjivg \
  --input kanjivg-20260714.xml.gz \
  --output ../halkyon_learn_flutter/assets/kanjivg

It writes one gzipped JSON file for every 256 code points that have any, named by the code point's upper bits in hexadecimal (食, U+98DF, is in 098.json.gz; 94 files, 2.7 MB), so the app reads only the part it needs. Each maps a character to its strokes (s, SVG paths in KanjiVG's 109 × 109 box, in the order they're written) and the components KanjiVG first splits it into (c, when it has any). Then update the release named in the folder's ATTRIBUTION.md.

Adding a source

Write an adapter in lib/src/sources/ that turns the source into the lines above, with its credits and a sample of real data in test/fixtures/ (and where that comes from in test/fixtures/README.md); give it a command in bin/curate.dart. Check the source's licence first, and that the instance may pass its data on to the people using it.