# halkyon_learn_curator Turns dictionary downloads into the dictionary files a Halkyon Learn server loads. It reads each source with an adapter of its own and writes one format for all of them, so a new source or language needs a new adapter, never a change to the server or the app. It doesn't depend on Serverpod or the server. It also makes the stroke files the app writes characters with (below). ## Making a dictionary file JMdict in English, with the JLPT levels and example sentences on top: 1. Download `JMdict_e_examp.gz` from [EDRDG](https://www.edrdg.org/pub/Nihongo/JMdict_e_examp.gz), which makes it anew every day: JMdict with Tatoeba's sentences (CC BY 2.0 FR) on the meanings they show. `JMdict_e.gz` works too, without them, and so does jmdict-simplified's JSON (`jmdict-examples-eng-.json`, from [its releases](https://github.com/scriptin/jmdict-simplified/releases)). 2. Download `original_data/n1.csv` … `n5.csv` from [yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab) into a folder. 3. Convert, in this package: ```sh dart run halkyon_learn_curator:curate jmdict \ --input JMdict_e_examp.gz --jlpt jlpt/ \ --output jmdict-en.jsonl.gz ``` The whole of JMdict (about 219,000 entries) takes some 30 seconds and 300 MB of memory: entries are read one at a time. A file whose name holds `.json` is read as jmdict-simplified's JSON, any other as EDRDG's XML. KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana: 1. Download `kanjidic2.xml.gz` from [the EDRDG](https://www.edrdg.org/wiki/index.php/KANJIDIC_Project). 2. Download `kanji.json` from [kanji-data](https://github.com/davidluzgouveia/kanji-data): Jonathan Waller's JLPT levels by kanji (`jlpt_new`; nothing else is read). 3. Convert, in this package: ```sh dart run halkyon_learn_curator:curate kanjidic \ --input kanjidic2.xml.gz --jlpt kanji.json \ --output kanjidic-en.jsonl.gz ``` About 12,800 characters, in some 5 seconds. Wiktionary, for the languages other than Japanese and Chinese (decided 2026-10-14: German, Greek and English first, then Chinese, then Korean): 1. Download `raw-wiktextract-data.jsonl.gz` from [kaikki.org](https://kaikki.org/dictionary/rawdata.html): the English Wiktionary as Wiktextract reads it, every language in one file (some 3 GB, 24 GB unpacked; kaikki.org's files of single languages are being withdrawn). 2. Convert, in this package, naming the languages: ```sh dart run halkyon_learn_curator:curate wiktionary \ --input raw-wiktextract-data.jsonl.gz --languages de,el,en \ --version 2026-10-03 --output ../halkyon_learn_server/dictionaries ``` Each language but English becomes its words with Wiktionary's English meanings (`wiktionary-de-en.jsonl.gz`, source `wiktionary-de`; a Korean word with its Hanja and romanisation); English becomes its words with what their translation tables give in German (`wiktionary-en-de.jsonl.gz`), German being the app's other language. An inflected form, and an entry Wiktionary has for one alone ("Häusern: dative plural of Haus"), finds its word rather than being an entry of its own; names and affixes are left out. Word frequencies from `data/wordfreq/` rank the words and say which are `common` (the 15,000 most used). Reading the whole dump takes a while; it is read once, streamed, keeping in memory only which forms belong to which word. Chinese from CC-CEDICT (decided 2026-10-14: a dictionary of its own, made for learners of Chinese, where Wiktionary's Chinese entries sit at their traditional forms): 1. Download `cedict_1_0_ts_utf-8_mdbg.txt.gz` from [MDBG](https://www.mdbg.net/chinese/dictionary?page=cc-cedict), which makes it anew every day. 2. Convert, in this package: ```sh dart run halkyon_learn_curator:curate cedict \ --input cedict_1_0_ts_utf-8_mdbg.txt.gz \ --output ../halkyon_learn_server/dictionaries ``` It writes two dictionaries of the same entries, `cc-cedict-hans-en` (`zh-Hans`, headwords in simplified characters) and `cc-cedict-hant-en` (`zh-Hant`, in traditional ones), so a studyset finds its words as it writes them; each finds a word by either script, by its pinyin with or without tones (`xuéxí`, `xuexi`, `xue2xi2`) and by its meanings. Readings are pinyin with tone marks, a syllable to a character (`xué xí`). The HSK levels of the 2025 examination syllabus (`data/hsk/`) label the words `hsk:1` … `hsk:6`, and `hsk:7` for the advanced levels 7 to 9, which the syllabus doesn't tell apart, each given to the entry read as the syllabus reads the word (的 *de*, not 的 *dì*), and rank them, HSK 1 first; wordfreq ranks the rest. Example sentences from Tatoeba for those dictionaries (JMdict brings its own), as hints on the cards made from them: ```sh dart run halkyon_learn_curator:curate tatoeba \ --input wiktionary-de-en.jsonl.gz --sentences deu_sentences.tsv.bz2 \ --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \ --output wiktionary-de-en.jsonl.gz ``` from [Tatoeba's weekly exports](https://downloads.tatoeba.org/exports/per_language/) of the dictionary's language, of its meaning language and of the links between them (CC BY 2.0 FR). Each sentence with a translation goes to the entries its words find by their text keys, as the app finds words in a line of subtitles, up to three for each: those using the word as its headword writes it first, then those nearest seven words long, the shortest then; they come first in an entry's first sense, the word as it stands there beside. Some ten seconds for German. Put the files into the server's `dictionaries` folder. The server loads each on its next start, and again whenever it changes (see the server's README). ## Keeping a server's dictionaries current EDRDG's licence asks of a server showing its dictionaries that it update them at least once a month. `curate update` does it from the sources alone: ```sh dart run halkyon_learn_curator:curate update \ --into ../halkyon_learn_server/dictionaries [--every 7] ``` It downloads EDRDG's files of the day (`JMdict_e_examp.gz`, `kanjidic2.xml.gz`), converts them with the JLPT levels in `data/` (static lists, see `data/README.md`), and writes `jmdict-en.jsonl.gz` and `kanjidic-en.jsonl.gz` into the folder, each beside its place first and then renamed over it, so a server never reads half a file; should anything fail, the files stay as they were. Some 50 seconds and 400 MB. With `--every 7` it keeps running and updates whenever the files are a week old, as their dates say (a restart doesn't download again), and tries again six hours after a failure. An instance runs it so in a container of its own (the server's README). Beside the files, `.conversion` says how the curator converted them (`conversion` in `lib/src/update.dart`): raise it whenever the curator writes something else from the same download (a label, a field), and an instance converts again as soon as the new curator runs, not a week later. With `--languages de,el,en,ko,zh` it also keeps those languages current, each file with Tatoeba's sentences of the week as examples (a dictionary whose sentences can't be had keeps its words without them): Chinese from MDBG's CC-CEDICT of the day as often as EDRDG's files (below), the rest from Wiktionary: each time it wakes it asks kaikki.org when its dump last changed (`Last-Modified`, without downloading it), and converts it again only when it is newer than the files' (about monthly), when the languages named changed, or when `wiktionaryConversion` in `lib/src/update.dart` was raised; `.wiktionary` beside the files says which. The dump is converted as it downloads, never stored. Files of a language no longer named stay until removed by hand. ## The format (`halkyon-dictionary`, version 1) [JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`. The **first line** says what the file holds: | Field | | |---|---| | `format`, `formatVersion` | `"halkyon-dictionary"`, `1` | | `kind` | `words` (a dictionary of words, JMdict) or `characters` (of the characters words are written in, KANJIDIC2). The app shows them apart. | | `source`, `sourceVersion` | The source's name (`jmdict`) and version (`3.6.2 (2026-10-05)`). An instance holds one version of a source for each meaning language. | | `language`, `glossLanguage` | BCP 47 tags: the words' language (`ja`) and the meanings' (`en`). | | `license` | The data's licence, an SPDX identifier (`CC-BY-SA-4.0`). | | `credits` | Everything it is drawn from: `name`, `by`, `license`, `url`. The app lists them where it credits its data sources. | **Every further line** is an entry: | Field | | |---|---| | `id` | Its id in the source; with the source and meaning language, it names the entry for good. | | `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). | | `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary and CC-CEDICT the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1`, `hsk:1` … `hsk:7` (later e.g. `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). | | `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. | | `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. | | `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. | | `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. | | `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). | ## The app's stroke files How every kanji and kana is written comes from [KanjiVG](https://kanjivg.tagaini.net) and is bundled with the app, so writing works without a server. To update it, download `kanjivg-.xml.gz` from [KanjiVG's releases](https://github.com/KanjiVG/kanjivg/releases) and, in this package: ```sh dart run halkyon_learn_curator:curate kanjivg \ --input kanjivg-20260714.xml.gz \ --output ../halkyon_learn_flutter/assets/kanjivg ``` It writes one gzipped JSON file for every 256 code points that have any, named by the code point's upper bits in hexadecimal (食, U+98DF, is in `098.json.gz`; 94 files, 2.7 MB), so the app reads only the part it needs. Each maps a character to its strokes (`s`, SVG paths in KanjiVG's 109 × 109 box, in the order they're written) and the components KanjiVG first splits it into (`c`, when it has any). Then update the release named in the folder's `ATTRIBUTION.md`. ## Adding a source Write an adapter in `lib/src/sources/` that turns the source into the lines above, with its credits and a sample of real data in `test/fixtures/` (and where that comes from in `test/fixtures/README.md`); give it a command in `bin/curate.dart`. Check the source's licence first, and that the instance may pass its data on to the people using it.