halkyon/apps/learn/halkyon_learn_curator

halkyon_learn_curator

Turns dictionary downloads into the dictionary files a Halkyon Learn server loads. It reads each source with an adapter of its own and writes one format for all of them, so a new source or language needs a new adapter, never a change to the server or the app. It doesn't depend on Serverpod or the server. It also makes the stroke files the app writes characters with (below).

Making a dictionary file

JMdict in English, with the JLPT levels and example sentences on top:

1. Download JMdict_e_examp.gz from EDRDG, which makes it anew every day: JMdict with Tatoeba's sentences (CC BY 2.0 FR) on the meanings they show. JMdict_e.gz works too, without them, and so does jmdict-simplified's JSON (jmdict-examples-eng-<version>.json, from its releases).

2. Download original_data/n1.csv … n5.csv from yomitan-jlpt-vocab into a folder.

3. Convert, in this package:

   dart run halkyon_learn_curator:curate jmdict \
     --input JMdict_e_examp.gz --jlpt jlpt/ \
     --output jmdict-en.jsonl.gz
   

   The whole of JMdict (about 219,000 entries) takes some 30 seconds and 300 MB of memory: entries are read one at a time. A file whose name holds .json is read as jmdict-simplified's JSON, any other as EDRDG's XML.

KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:

1. Download kanjidic2.xml.gz from the EDRDG.

2. Download kanji.json from kanji-data: Jonathan Waller's JLPT levels by kanji (jlpt_new; nothing else is read).

3. Convert, in this package:

   dart run halkyon_learn_curator:curate kanjidic \
     --input kanjidic2.xml.gz --jlpt kanji.json \
     --output kanjidic-en.jsonl.gz
   

   About 12,800 characters, in some 5 seconds.

Wiktionary, for the words of German, Greek and English (decided 2026-10-14), and for Korean the forms of its words and the native readings of its Hanja:

1. Download raw-wiktextract-data.jsonl.gz from kaikki.org: the English Wiktionary as Wiktextract reads it, every language in one file (some 3 GB, 24 GB unpacked; kaikki.org's files of single languages are being withdrawn).

2. Convert, in this package, naming the languages:

   dart run halkyon_learn_curator:curate wiktionary \
     --input raw-wiktextract-data.jsonl.gz --languages de,el,en \
     --version 2026-10-03 --output ../halkyon_learn_server/dictionaries
   

   Each language but English becomes its words with Wiktionary's English meanings (wiktionary-de-en.jsonl.gz, source wiktionary-de); English becomes its words with what their translation tables give in German (wiktionary-en-de.jsonl.gz), German being the app's other language. An inflected form, and an entry Wiktionary has for one alone ("Häusern: dative plural of Haus"), finds its word rather than being an entry of its own; names and affixes are left out. Word frequencies from data/wordfreq/ rank the words and say which are common (the 15,000 most used). Reading the whole dump takes a while; it is read once, streamed, keeping in memory only which forms belong to which word.

Chinese from CC-CEDICT (decided 2026-10-14: a dictionary of its own, made for learners of Chinese, where Wiktionary's Chinese entries sit at their traditional forms):

1. Download cedict_1_0_ts_utf-8_mdbg.txt.gz from MDBG, which makes it anew every day.

2. Convert, in this package:

   dart run halkyon_learn_curator:curate cedict \
     --input cedict_1_0_ts_utf-8_mdbg.txt.gz \
     --output ../halkyon_learn_server/dictionaries
   

   It writes two dictionaries of the same entries, cc-cedict-hans-en (zh-Hans, headwords in simplified characters) and cc-cedict-hant-en (zh-Hant, in traditional ones), so a studyset finds its words as it writes them; each finds a word by either script, by its pinyin with or without tones (xuéxí, xuexi, xue2xi2) and by its meanings. Readings are pinyin with tone marks, a syllable to a character (xué xí). The HSK levels of the 2025 examination syllabus (data/hsk/) label the words hsk:1 … hsk:6, and hsk:7 for the advanced levels 7 to 9, which the syllabus doesn't tell apart, each given to the entry read as the syllabus reads the word (的 de, not 的 dì), and rank them, HSK 1 first; wordfreq ranks the rest.

Chinese characters and Korean Hanja from Unihan, the Unicode Consortium's database of Han characters (decided 2026-10-14, for parity with KANJIDIC2; Unicode License v3):

1. Download Unihan.zip from unicode.org (8 MB, a new one with each version of Unicode), and CC-CEDICT as above.

2. Convert, naming the languages:

   dart run halkyon_learn_curator:curate unihan --input Unihan.zip \
     --languages ko,zh --cedict cedict_1_0_ts_utf-8_mdbg.txt.gz \
     [--korean .wiktionary-ko.json.gz] \
     --output ../halkyon_learn_server/dictionaries
   

   For Chinese, two dictionaries of characters as for the words: the 8,105 General Standard Chinese Characters of 2013 (unihan-hans-en, zh-Hans), and the traditional characters in use (unihan-hant-en, zh-Hant): each of them as Taiwan and Hong Kong write it, where it is a form of its own or a word of CC-CEDICT writes it so (Unihan calls 这 its own traditional variant, but no word is written with it), with those Hong Kong's primary schools teach. Each with its pinyin (the usual reading first: the mainland's in simplified, Taiwan's in traditional characters), meanings, stroke count and other script's forms; labelled by its HSK level (hsk:1 … hsk:7, from the syllabus's characters in data/hsk/), the 3,500 standard characters in common use (common), a Hong Kong grade (grade:1 … grade:6) and its Kangxi radical (radical:N), and ranked by HSK level, then by the most used word of CC-CEDICT written with it (wordfreq), so 後 ranks as the 后 of 以后 and 后, the queen, by 皇后. For Korean, the 7,800 Hanja with a Korean reading (unihan-ko-en): their sounds (음, the one taught first), their native meanings with their sounds as Wiktionary gives them (훈, 물 수 for 水; from the Wiktionary update's .wiktionary-ko.json.gz), meanings and strokes, labelled education for the 1,800 taught in schools; and the 40 Hangul letters (jamo) as the kana are in KANJIDIC2's, with their sound and name, labelled hangul. Some 3 seconds.

The Greek alphabet, written by the curator itself (greek-alphabet-en, el, CC0): each letter with its capital, its Greek and English names, its transliteration (ELOT 743) and its sound, labelled greek, for a characters tab as kana have.

Korean words from KRDict, the National Institute of Korean Language's Basic Korean Dictionary for learners (decided 2026-10-14: levels, pronunciation and Hanja, which Wiktionary lacks; CC BY-SA 2.0 KR, later versions allowed):

1. Download its JSON export from its page (Json 전체 내려받기; no account needed; some 84 MB, a gigabyte unpacked, renewed about monthly).

2. Convert:

   dart run halkyon_learn_curator:curate krdict \
     --input krdict_json_20260919.zip [--korean .wiktionary-ko.json.gz] \
     --output ../halkyon_learn_server/dictionaries
   

   krdict-ko-en.jsonl.gz holds its 54,870 words, idioms and proverbs, each with its part of speech, its pronunciation as its reading where it differs from the spelling (같이 [가치]), its Hanja, the few forms KRDict gives and its senses (KRDict's English words, its English definition as info); labelled by its learners' level (level:beginner, level:intermediate, level:advanced) and ranked by it, then by wordfreq (a verb, rarely written as the dictionary writes it, by its most used form). With the Wiktionary update's file, every form Wiktionary's tables give (가다's 99) finds a word in running text too. An idiom or proverb carries the id of the word it is filed under, so its own id adds itself (27500:가는 날이 장날). Read an entry at a time, some 25 seconds and 600 MB.

Example sentences from Tatoeba for those dictionaries (JMdict brings its own), as hints on the cards made from them:

dart run halkyon_learn_curator:curate tatoeba \
  --input wiktionary-de-en.jsonl.gz --sentences deu_sentences.tsv.bz2 \
  --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
  --output wiktionary-de-en.jsonl.gz

from Tatoeba's weekly exports of the dictionary's language, of its meaning language and of the links between them (CC BY 2.0 FR). Each sentence with a translation goes to the entries its words find by their text keys, as the app finds words in a line of subtitles, up to three for each: those using the word as its headword writes it first, then those nearest seven words long, the shortest then; they come first in an entry's first sense, the word as it stands there beside. Some ten seconds for German.

Readings over the examples, from Tatoeba's transcriptions of their sentences (decided 9 Nov 2026): furigana over JMdict's Japanese, pinyin over Chinese. update adds them itself; by hand:

dart run halkyon_learn_curator:curate readings \
  --input jmdict-en.jsonl.gz --transcriptions jpn_transcriptions.tsv.bz2 \
  --output jmdict-en.jsonl.gz

from Tatoeba's jpn_transcriptions.tsv.bz2 or cmn_transcriptions.tsv.bz2 (CC BY 2.0 FR, the sentences' own licence). Each example with a Tatoeba ID gets ruby, a list of [start, end, reading] over its sentence (UTF-16 code units), where the transcription is of that very sentence: Japanese's [漢字|かん|じ] one reading for each kanji where there are as many, else one over the word; Chinese's tone numbers as tone marks, each of Tatoeba's words over as many Han characters as it has syllables, none where they don't count up (about 1 % of them). rubyUnreviewed marks those no one wrote or checked, MeCab's furigana and Tatoeba's automatic pinyin, some half of the Japanese.

Put the files into the server's dictionaries folder. The server loads each on its next start, and again whenever it changes (see the server's README).

Keeping a server's dictionaries current

EDRDG's licence asks of a server showing its dictionaries that it update them at least once a month. curate update does it from the sources alone:

dart run halkyon_learn_curator:curate update \
  --into ../halkyon_learn_server/dictionaries [--every 7]

It downloads EDRDG's files of the day (JMdict_e_examp.gz, kanjidic2.xml.gz), converts them with the JLPT levels in data/ (static lists, see data/README.md), and writes jmdict-en.jsonl.gz and kanjidic-en.jsonl.gz into the folder, each beside its place first and then renamed over it, so a server never reads half a file; should anything fail, the files stay as they were. Some 50 seconds and 400 MB. With --every 7 it keeps running and updates whenever the files are a week old, as their dates say (a restart doesn't download again), and tries again six hours after a failure. An instance runs it so in a container of its own (the server's README). Beside the files, .conversion says how the curator converted them (conversion in lib/src/update.dart): raise it whenever the curator writes something else from the same download (a label, a field), and an instance converts again as soon as the new curator runs, not a week later.

With --languages de,el,en,ko,zh it also keeps those languages current, each file of words with Tatoeba's sentences of the week as examples (a dictionary whose sentences can't be had keeps its words without them), in this order: Chinese from MDBG's CC-CEDICT of the day as often as EDRDG's files; the words of the rest from Wiktionary: each time it wakes it asks kaikki.org when its dump last changed (Last-Modified, without downloading it), and converts it again only when it is newer than the files' (about monthly), when the languages named changed, or when wiktionaryConversion in lib/src/update.dart was raised; .wiktionary beside the files says which (for Korean it keeps .wiktionary-ko.json.gz, the forms and Hanja readings). The dump is converted as it downloads, never stored. Korean's words from KRDict: each time it wakes it reads KRDict's page, and converts again only when it offers another export, when Wiktionary's Korean file changed, or when krdictConversion was raised (.krdict). Then the characters (Unihan for Chinese and Korean, with CC-CEDICT's words; the Greek alphabet) as often as EDRDG's files, and at once when Wiktionary's Korean file is newer than the Hanja's. Files the curator wrote for a language no longer named, or for a source replaced (Korean's words from Wiktionary, before KRDict), are removed, and the server unloads them.

Each run also packs MeCab's IPADIC, where the image installed it (Ubuntu's mecab-ipadic-utf8, which the Dockerfile installs), into mecab-ipadic.tar.gz beside the files: the dictionary the app's devices download from their instance to cut Japanese into words themselves (ADR 0012). The same dictionary always packs to the same bytes, and a pack whose content is unchanged is left as it is (.mecab-ipadic.tar.gz.sha256 keeps its hash), so devices download it again only when it changed. The server loads no file but dictionary files, so it leaves the pack be, and serves it. By hand, from any compiled IPADIC:

dart run halkyon_learn_curator:curate ipadic \
  [--input /var/lib/mecab/dic/ipadic-utf8] \
  --output ../halkyon_learn_server/dictionaries

The format (halkyon-dictionary, version 1)

JSON Lines, gzipped when the name ends in .gz.

The first line says what the file holds:

Field
format, formatVersion	"halkyon-dictionary", 1
kind	words (a dictionary of words, JMdict) or characters (of the characters words are written in, KANJIDIC2). The app shows them apart.
source, sourceVersion	The source's name (jmdict) and version (3.6.2 (2026-10-05)). An instance holds one version of a source for each meaning language.
language, glossLanguage	BCP 47 tags: the words' language (ja) and the meanings' (en).
license	The data's licence, an SPDX identifier (CC-BY-SA-4.0).
credits	Everything it is drawn from: name, by, license, url. The app lists them where it credits its data sources.

Every further line is an entry:

Field
id	Its id in the source; with the source and meaning language, it names the entry for good.
headword, reading	How the word is shown, and how it is read; no reading when the headword is its own (a word written in kana).
labels	What it can be picked by: common (JMdict's own; for Wiktionary, CC-CEDICT and KRDict the 15,000 words wordfreq finds most used; for Chinese characters the 3,500 standard ones in common use), jlpt:n5 … jlpt:n1, hsk:1 … hsk:7, KRDict's learners' levels (level:beginner, level:intermediate, level:advanced); for characters also the school grade (grade:1 … grade:6, grade:secondary for the rest of the Jōyō kanji; Hong Kong's for traditional Chinese), Korea's 1,800 Hanja of education (education), a character's Kangxi radical (radical:1 … radical:214, KANJIDIC2's classical one, Unihan's kRSUnicode) and the script (hiragana, katakana, hangul, greek).
rank	Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency.
keys	What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language.
looseKeys	What a search matches after every keys match: spellings people type that aren't quite right (romaji with long vowels typed once, sushi for すうし; a word without its accents or umlauts, strasse, haeuser, spiti). Left out when empty.
textKeys	What finds the entry in running text (subtitles, lyrics): its own forms, lower case (haus, häuser, häusern), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's.
entry	The entry itself, as cards take it: forms and readings (text, common, the source's tags; for a reading the forms it belongs to), senses (glosses; the source's codes in pos, misc, field, dialect; info; the forms and readings a sense is limited to; examples, sentences using the word in that sense: text, the form it takes there, ja, the sentence, en, its translation, the source's id, e.g. tatoeba, and where Tatoeba transcribed the sentence ruby and rubyUnreviewed, the readings over it). Lists are left out when empty. A character's: meanings; on and kun, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a ., a dash where it attaches to others); grade, strokes, frequency (its place among the 2,500 most used in newspapers) and jlpt where it has them. A kana's: romaji and script. A Wiktionary word's: pos (Wiktionary's part of speech for the whole entry: noun, verb, adj), genders (m, f, n of a German or Greek noun), romanization (Greek's), ipa, inflections (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; text and Wiktionary's tags), and senses with glosses, tags (how and where a sense is used: colloquial, Austria), info (for English with German translations, the English meaning a translation is of) and examples keyed by language as JMdict's (de, the sentence; en, its translation; text, the word as it stands there). A CC-CEDICT word's: forms (the word in the other script), readings (pinyin with tone marks), senses (each of CC-CEDICT's meanings, its glosses split at ;) and classifiers (its measure words, 个 gè). A KRDict word's: pos, hanja, inflections (text) and senses (glosses, its English definition as info). A Chinese character's: meanings, pinyin (the usual reading first), strokes, traditional or simplified (its forms in the other script), hsk and frequency (the place of the most used word written with it). A Hanja's: meanings, eum (its sounds in Hangul, the one taught first), hun and eumhun (its native meanings, and each with its sound, 물 수) and strokes. A Hangul letter's: romaji (its sound, g, k), script and name (기역). A Greek letter's: romaji (its transliteration), script, name, englishName and sound (IPA).

The app's stroke files

How every kanji and kana is written comes from KanjiVG and is bundled with the app, so writing works without a server. To update it, download kanjivg-<date>.xml.gz from KanjiVG's releases and, in this package:

dart run halkyon_learn_curator:curate kanjivg \
  --input kanjivg-20260714.xml.gz \
  --output ../halkyon_learn_flutter/assets/kanjivg

It writes one gzipped JSON file for every 256 code points that have any, named by the code point's upper bits in hexadecimal (食, U+98DF, is in 098.json.gz; 94 files, 2.7 MB), so the app reads only the part it needs. Each maps a character to its strokes (s, SVG paths in KanjiVG's 109 × 109 box, in the order they're written) and the components KanjiVG first splits it into (c, when it has any). Then update the release named in the folder's ATTRIBUTION.md.

Chinese characters are written as Make Me a Hanzi draws them (skishore/makemeahanzi, decided 2026-10-14 for parity with KanjiVG: its own shapes and the mainland's stroke order, 9,574 characters of both scripts; the strokes under the Arphic Public License, the components under the LGPL). Download its graphics.txt and dictionary.txt, then:

dart run halkyon_learn_curator:curate makemeahanzi \
  --input graphics.txt --dictionary dictionary.txt \
  --output ../halkyon_learn_flutter/assets/makemeahanzi

It writes the same files as for KanjiVG (99 files, 2.7 MB): each stroke is the line Make Me a Hanzi draws through the middle of its outline (its median), scaled from its 1024 square, whose y axis points up from 124 below the bottom, into KanjiVG's 109 square, y pointing down; the outlines are left out, as the app draws and checks strokes along their middle. A character's components are the first split of its decomposition (海: 氵, 每). Each file says how and when it was changed (notice), as the Arphic Public License asks of a changed file. They stay a data file beside the app, never in cards (the folder's ATTRIBUTION.md).

The Hangul letters are drawn by the curator itself (CC0), as no openly licensed data draws them with their strokes:

dart run halkyon_learn_curator:curate hangul \
  --output ../halkyon_learn_flutter/assets/hangul

writes 031.json.gz: the 40 letters in the order Korean schools teach their strokes (lib/src/sources/hangul_strokes.dart), and the pairs of consonants (ㄲ, and the clusters closing syllables, 닭's ㄺ) as their two letters side by side, so a Korean word is written letter by letter.

Adding a source

Write an adapter in lib/src/sources/ that turns the source into the lines above, with its credits and a sample of real data in test/fixtures/ (and where that comes from in test/fixtures/README.md); give it a command in bin/curate.dart. Check the source's licence first, and that the instance may pass its data on to the people using it.