halkyon/halkyon_learn_curator/README.md

halkyon_learn_curator

Turns dictionary downloads into the dictionary files a Halkyon Learn server loads. It reads each source with an adapter of its own and writes one format for all of them, so a new source or language needs a new adapter, never a change to the server or the app. It doesn't depend on Serverpod or the server. It also makes the stroke files the app writes characters with (below).

Making a dictionary file

JMdict in English, with the JLPT levels and example sentences on top:

1. Download JMdict_e_examp.gz from EDRDG, which makes it anew every day: JMdict with Tatoeba's sentences (CC BY 2.0 FR) on the meanings they show. JMdict_e.gz works too, without them, and so does jmdict-simplified's JSON (jmdict-examples-eng-<version>.json, from its releases).

2. Download original_data/n1.csv … n5.csv from yomitan-jlpt-vocab into a folder.

3. Convert, in this package:

   dart run halkyon_learn_curator:curate jmdict \
     --input JMdict_e_examp.gz --jlpt jlpt/ \
     --output jmdict-en.jsonl.gz
   

   The whole of JMdict (about 219,000 entries) takes some 30 seconds and 300 MB of memory: entries are read one at a time. A file whose name holds .json is read as jmdict-simplified's JSON, any other as EDRDG's XML.

KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:

1. Download kanjidic2.xml.gz from the EDRDG.

2. Download kanji.json from kanji-data: Jonathan Waller's JLPT levels by kanji (jlpt_new; nothing else is read).

3. Convert, in this package:

   dart run halkyon_learn_curator:curate kanjidic \
     --input kanjidic2.xml.gz --jlpt kanji.json \
     --output kanjidic-en.jsonl.gz
   

   About 12,800 characters, in some 5 seconds.

Wiktionary, for the languages other than Japanese and Chinese (decided 2026-10-14: German, Greek and English first, then Chinese, then Korean):

1. Download raw-wiktextract-data.jsonl.gz from kaikki.org: the English Wiktionary as Wiktextract reads it, every language in one file (some 3 GB, 24 GB unpacked; kaikki.org's files of single languages are being withdrawn).

2. Convert, in this package, naming the languages:

   dart run halkyon_learn_curator:curate wiktionary \
     --input raw-wiktextract-data.jsonl.gz --languages de,el,en \
     --version 2026-10-03 --output ../halkyon_learn_server/dictionaries
   

   Each language but English becomes its words with Wiktionary's English meanings (wiktionary-de-en.jsonl.gz, source wiktionary-de; a Korean word with its Hanja and romanisation); English becomes its words with what their translation tables give in German (wiktionary-en-de.jsonl.gz), German being the app's other language. An inflected form, and an entry Wiktionary has for one alone ("Häusern: dative plural of Haus"), finds its word rather than being an entry of its own; names and affixes are left out. Word frequencies from data/wordfreq/ rank the words and say which are common (the 15,000 most used). Reading the whole dump takes a while; it is read once, streamed, keeping in memory only which forms belong to which word.

Chinese from CC-CEDICT (decided 2026-10-14: a dictionary of its own, made for learners of Chinese, where Wiktionary's Chinese entries sit at their traditional forms):

1. Download cedict_1_0_ts_utf-8_mdbg.txt.gz from MDBG, which makes it anew every day.

2. Convert, in this package:

   dart run halkyon_learn_curator:curate cedict \
     --input cedict_1_0_ts_utf-8_mdbg.txt.gz \
     --output ../halkyon_learn_server/dictionaries
   

   It writes two dictionaries of the same entries, cc-cedict-hans-en (zh-Hans, headwords in simplified characters) and cc-cedict-hant-en (zh-Hant, in traditional ones), so a studyset finds its words as it writes them; each finds a word by either script, by its pinyin with or without tones (xuéxí, xuexi, xue2xi2) and by its meanings. Readings are pinyin with tone marks, a syllable to a character (xué xí). The HSK levels of the 2025 examination syllabus (data/hsk/) label the words hsk:1 … hsk:6, and hsk:7 for the advanced levels 7 to 9, which the syllabus doesn't tell apart, each given to the entry read as the syllabus reads the word (的 de, not 的 dì), and rank them, HSK 1 first; wordfreq ranks the rest.

Example sentences from Tatoeba for those dictionaries (JMdict brings its own), as hints on the cards made from them:

dart run halkyon_learn_curator:curate tatoeba \
  --input wiktionary-de-en.jsonl.gz --sentences deu_sentences.tsv.bz2 \
  --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
  --output wiktionary-de-en.jsonl.gz

from Tatoeba's weekly exports of the dictionary's language, of its meaning language and of the links between them (CC BY 2.0 FR). Each sentence with a translation goes to the entries its words find by their text keys, as the app finds words in a line of subtitles, up to three for each: those using the word as its headword writes it first, then those nearest seven words long, the shortest then; they come first in an entry's first sense, the word as it stands there beside. Some ten seconds for German.

Put the files into the server's dictionaries folder. The server loads each on its next start, and again whenever it changes (see the server's README).

Keeping a server's dictionaries current

EDRDG's licence asks of a server showing its dictionaries that it update them at least once a month. curate update does it from the sources alone:

dart run halkyon_learn_curator:curate update \
  --into ../halkyon_learn_server/dictionaries [--every 7]

It downloads EDRDG's files of the day (JMdict_e_examp.gz, kanjidic2.xml.gz), converts them with the JLPT levels in data/ (static lists, see data/README.md), and writes jmdict-en.jsonl.gz and kanjidic-en.jsonl.gz into the folder, each beside its place first and then renamed over it, so a server never reads half a file; should anything fail, the files stay as they were. Some 50 seconds and 400 MB. With --every 7 it keeps running and updates whenever the files are a week old, as their dates say (a restart doesn't download again), and tries again six hours after a failure. An instance runs it so in a container of its own (the server's README). Beside the files, .conversion says how the curator converted them (conversion in lib/src/update.dart): raise it whenever the curator writes something else from the same download (a label, a field), and an instance converts again as soon as the new curator runs, not a week later.

With --languages de,el,en,ko,zh it also keeps those languages current, each file with Tatoeba's sentences of the week as examples (a dictionary whose sentences can't be had keeps its words without them): Chinese from MDBG's CC-CEDICT of the day as often as EDRDG's files (below), the rest from Wiktionary: each time it wakes it asks kaikki.org when its dump last changed (Last-Modified, without downloading it), and converts it again only when it is newer than the files' (about monthly), when the languages named changed, or when wiktionaryConversion in lib/src/update.dart was raised; .wiktionary beside the files says which. The dump is converted as it downloads, never stored. Files of a language no longer named stay until removed by hand.

The format (halkyon-dictionary, version 1)

JSON Lines, gzipped when the name ends in .gz.

The first line says what the file holds:

Field
format, formatVersion	"halkyon-dictionary", 1
kind	words (a dictionary of words, JMdict) or characters (of the characters words are written in, KANJIDIC2). The app shows them apart.
source, sourceVersion	The source's name (jmdict) and version (3.6.2 (2026-10-05)). An instance holds one version of a source for each meaning language.
language, glossLanguage	BCP 47 tags: the words' language (ja) and the meanings' (en).
license	The data's licence, an SPDX identifier (CC-BY-SA-4.0).
credits	Everything it is drawn from: name, by, license, url. The app lists them where it credits its data sources.

Every further line is an entry:

Field
id	Its id in the source; with the source and meaning language, it names the entry for good.
headword, reading	How the word is shown, and how it is read; no reading when the headword is its own (a word written in kana).
labels	What it can be picked by: common (JMdict's own; for Wiktionary and CC-CEDICT the 15,000 words wordfreq finds most used), jlpt:n5 … jlpt:n1, hsk:1 … hsk:7 (later e.g. topik:1); for characters also the school grade (grade:1 … grade:6, grade:secondary for the rest of the Jōyō kanji), a kanji's Kangxi radical (radical:1 … radical:214, KANJIDIC2's classical one) and the script (hiragana, katakana).
rank	Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency.
keys	What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language.
looseKeys	What a search matches after every keys match: spellings people type that aren't quite right (romaji with long vowels typed once, sushi for すうし; a word without its accents or umlauts, strasse, haeuser, spiti). Left out when empty.
textKeys	What finds the entry in running text (subtitles, lyrics): its own forms, lower case (haus, häuser, häusern), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's.
entry	The entry itself, as cards take it: forms and readings (text, common, the source's tags; for a reading the forms it belongs to), senses (glosses; the source's codes in pos, misc, field, dialect; info; the forms and readings a sense is limited to; examples, sentences using the word in that sense: text, the form it takes there, ja, the sentence, en, its translation, and the source's id, e.g. tatoeba). Lists are left out when empty. A character's: meanings; on and kun, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a ., a dash where it attaches to others); grade, strokes, frequency (its place among the 2,500 most used in newspapers) and jlpt where it has them. A kana's: romaji and script. A Wiktionary word's: pos (Wiktionary's part of speech for the whole entry: noun, verb, adj), genders (m, f, n of a German or Greek noun), romanization (Greek's), ipa, inflections (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; text and Wiktionary's tags), and senses with glosses, tags (how and where a sense is used: colloquial, Austria), info (for English with German translations, the English meaning a translation is of) and examples keyed by language as JMdict's (de, the sentence; en, its translation; text, the word as it stands there). A CC-CEDICT word's: forms (the word in the other script), readings (pinyin with tone marks), senses (each of CC-CEDICT's meanings, its glosses split at ;) and classifiers (its measure words, 个 gè).

The app's stroke files

How every kanji and kana is written comes from KanjiVG and is bundled with the app, so writing works without a server. To update it, download kanjivg-<date>.xml.gz from KanjiVG's releases and, in this package:

dart run halkyon_learn_curator:curate kanjivg \
  --input kanjivg-20260714.xml.gz \
  --output ../halkyon_learn_flutter/assets/kanjivg

It writes one gzipped JSON file for every 256 code points that have any, named by the code point's upper bits in hexadecimal (食, U+98DF, is in 098.json.gz; 94 files, 2.7 MB), so the app reads only the part it needs. Each maps a character to its strokes (s, SVG paths in KanjiVG's 109 × 109 box, in the order they're written) and the components KanjiVG first splits it into (c, when it has any). Then update the release named in the folder's ATTRIBUTION.md.

Adding a source

Write an adapter in lib/src/sources/ that turns the source into the lines above, with its credits and a sample of real data in test/fixtures/ (and where that comes from in test/fixtures/README.md); give it a command in bin/curate.dart. Check the source's licence first, and that the instance may pass its data on to the people using it.