halkyon/halkyon_learn_curator/README.md

# halkyon_learn_curator

Turns dictionary downloads into the dictionary files a Halkyon Learn server
loads. It reads each source with an adapter of its own and writes one format
for all of them, so a new source or language needs a new adapter, never a
change to the server or the app. It doesn't depend on Serverpod or the server.
It also makes the stroke files the app writes characters with (below).

## Making a dictionary file

JMdict in English, with the JLPT levels and example sentences on top:

1. Download `JMdict_e_examp.gz` from
   [EDRDG](https://www.edrdg.org/pub/Nihongo/JMdict_e_examp.gz), which makes
   it anew every day: JMdict with Tatoeba's sentences (CC BY 2.0 FR) on the
   meanings they show. `JMdict_e.gz` works too, without them, and so does
   jmdict-simplified's JSON (`jmdict-examples-eng-<version>.json`, from
   [its releases](https://github.com/scriptin/jmdict-simplified/releases)).
2. Download `original_data/n1.csv` … `n5.csv` from
   [yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab) into a
   folder.
3. Convert, in this package:

   ```sh
   dart run halkyon_learn_curator:curate jmdict \
     --input JMdict_e_examp.gz --jlpt jlpt/ \
     --output jmdict-en.jsonl.gz
   ```

   The whole of JMdict (about 219,000 entries) takes some 30 seconds and
   300 MB of memory: entries are read one at a time. A file whose name
   holds `.json` is read as jmdict-simplified's JSON, any other as EDRDG's
   XML.

KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:

1. Download `kanjidic2.xml.gz` from
   [the EDRDG](https://www.edrdg.org/wiki/index.php/KANJIDIC_Project).
2. Download `kanji.json` from
   [kanji-data](https://github.com/davidluzgouveia/kanji-data): Jonathan
   Waller's JLPT levels by kanji (`jlpt_new`; nothing else is read).
3. Convert, in this package:

   ```sh
   dart run halkyon_learn_curator:curate kanjidic \
     --input kanjidic2.xml.gz --jlpt kanji.json \
     --output kanjidic-en.jsonl.gz
   ```

   About 12,800 characters, in some 5 seconds.

Wiktionary, for the words of German, Greek and English (decided 2026-10-14),
and for Korean the forms of its words and the native readings of its Hanja:

1. Download `raw-wiktextract-data.jsonl.gz` from
   [kaikki.org](https://kaikki.org/dictionary/rawdata.html): the English
   Wiktionary as Wiktextract reads it, every language in one file (some 3 GB,
   24 GB unpacked; kaikki.org's files of single languages are being withdrawn).
2. Convert, in this package, naming the languages:

   ```sh
   dart run halkyon_learn_curator:curate wiktionary \
     --input raw-wiktextract-data.jsonl.gz --languages de,el,en \
     --version 2026-10-03 --output ../halkyon_learn_server/dictionaries
   ```

   Each language but English becomes its words with Wiktionary's English
   meanings (`wiktionary-de-en.jsonl.gz`, source `wiktionary-de`); English
   becomes its words with what their translation tables give in German
   (`wiktionary-en-de.jsonl.gz`), German being the app's other language. An
   inflected form, and an entry Wiktionary has for one alone ("Häusern: dative
   plural of Haus"), finds its word rather than being an entry of its own;
   names and affixes are left out. Word frequencies from `data/wordfreq/` rank
   the words and say which are `common` (the 15,000 most used). Reading the
   whole dump takes a while; it is read once, streamed, keeping in memory only
   which forms belong to which word.

Chinese from CC-CEDICT (decided 2026-10-14: a dictionary of its own, made for
learners of Chinese, where Wiktionary's Chinese entries sit at their traditional
forms):

1. Download `cedict_1_0_ts_utf-8_mdbg.txt.gz` from
   [MDBG](https://www.mdbg.net/chinese/dictionary?page=cc-cedict), which makes it
   anew every day.
2. Convert, in this package:

   ```sh
   dart run halkyon_learn_curator:curate cedict \
     --input cedict_1_0_ts_utf-8_mdbg.txt.gz \
     --output ../halkyon_learn_server/dictionaries
   ```

   It writes two dictionaries of the same entries, `cc-cedict-hans-en`
   (`zh-Hans`, headwords in simplified characters) and `cc-cedict-hant-en`
   (`zh-Hant`, in traditional ones), so a studyset finds its words as it writes
   them; each finds a word by either script, by its pinyin with or without tones
   (`xuéxí`, `xuexi`, `xue2xi2`) and by its meanings. Readings are pinyin with
   tone marks, a syllable to a character (`xué xí`). The HSK levels of the 2025
   examination syllabus (`data/hsk/`) label the words `hsk:1` … `hsk:6`, and
   `hsk:7` for the advanced levels 7 to 9, which the syllabus doesn't tell
   apart, each given to the entry read as the syllabus reads the word (的 *de*,
   not 的 *dì*), and rank them, HSK 1 first; wordfreq ranks the rest.

Chinese characters and Korean Hanja from Unihan, the Unicode Consortium's
database of Han characters (decided 2026-10-14, for parity with KANJIDIC2;
Unicode License v3):

1. Download `Unihan.zip` from
   [unicode.org](https://www.unicode.org/Public/UCD/latest/ucd/Unihan.zip)
   (8 MB, a new one with each version of Unicode), and CC-CEDICT as above.
2. Convert, naming the languages:

   ```sh
   dart run halkyon_learn_curator:curate unihan --input Unihan.zip \
     --languages ko,zh --cedict cedict_1_0_ts_utf-8_mdbg.txt.gz \
     [--korean .wiktionary-ko.json.gz] \
     --output ../halkyon_learn_server/dictionaries
   ```

   For Chinese, two dictionaries of characters as for the words: the 8,105
   General Standard Chinese Characters of 2013 (`unihan-hans-en`, `zh-Hans`),
   and the traditional characters in use (`unihan-hant-en`, `zh-Hant`): each of
   them as Taiwan and Hong Kong write it, where it is a form of its own or a
   word of CC-CEDICT writes it so (Unihan calls 这 its own traditional variant,
   but no word is written with it), with those Hong Kong's primary schools
   teach. Each with its pinyin (the usual reading first: the mainland's in
   simplified, Taiwan's in traditional characters), meanings, stroke count and
   other script's forms; labelled by its HSK level (`hsk:1` … `hsk:7`, from the
   syllabus's characters in `data/hsk/`), the 3,500 standard characters in
   common use (`common`), a Hong Kong grade (`grade:1` … `grade:6`) and its
   Kangxi radical (`radical:N`), and ranked by HSK level, then by the most used
   word of CC-CEDICT written with it (wordfreq), so 後 ranks as the 后 of 以后
   and 后, the queen, by 皇后. For Korean, the 7,800 Hanja with a Korean reading
   (`unihan-ko-en`): their sounds (음, the one taught first), their native
   meanings with their sounds as Wiktionary gives them (훈, `물 수` for 水; from
   the Wiktionary update's `.wiktionary-ko.json.gz`), meanings and strokes,
   labelled `education` for the 1,800 taught in schools; and the 40 Hangul
   letters (jamo) as the kana are in KANJIDIC2's, with their sound and name,
   labelled `hangul`. Some 3 seconds.

The Greek alphabet, written by the curator itself (`greek-alphabet-en`, `el`,
CC0): each letter with its capital, its Greek and English names, its
transliteration (ELOT 743) and its sound, labelled `greek`, for a characters
tab as kana have.

Korean words from KRDict, the National Institute of Korean Language's Basic
Korean Dictionary for learners (decided 2026-10-14: levels, pronunciation and
Hanja, which Wiktionary lacks; CC BY-SA 2.0 KR, later versions allowed):

1. Download its JSON export from
   [its page](https://krdict.korean.go.kr/download/downloadPopup) (Json 전체
   내려받기; no account needed; some 84 MB, a gigabyte unpacked, renewed about
   monthly).
2. Convert:

   ```sh
   dart run halkyon_learn_curator:curate krdict \
     --input krdict_json_20260919.zip [--korean .wiktionary-ko.json.gz] \
     --output ../halkyon_learn_server/dictionaries
   ```

   `krdict-ko-en.jsonl.gz` holds its 54,870 words, idioms and proverbs, each
   with its part of speech, its pronunciation as its reading where it differs
   from the spelling (같이 [가치]), its Hanja, the few forms KRDict gives and
   its senses (KRDict's English words, its English definition as `info`);
   labelled by its learners' level (`level:beginner`, `level:intermediate`,
   `level:advanced`) and ranked by it, then by wordfreq (a verb, rarely written
   as the dictionary writes it, by its most used form). With the Wiktionary
   update's file, every form Wiktionary's tables give (가다's 99) finds a word in
   running text too. An idiom or proverb carries the id of the word it is filed
   under, so its own id adds itself (`27500:가는 날이 장날`). Read an entry at a
   time, some 25 seconds and 600 MB.

Example sentences from Tatoeba for those dictionaries (JMdict brings its own),
as hints on the cards made from them:

```sh
dart run halkyon_learn_curator:curate tatoeba \
  --input wiktionary-de-en.jsonl.gz --sentences deu_sentences.tsv.bz2 \
  --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
  --output wiktionary-de-en.jsonl.gz
```

from [Tatoeba's weekly exports](https://downloads.tatoeba.org/exports/per_language/)
of the dictionary's language, of its meaning language and of the links between
them (CC BY 2.0 FR). Each sentence with a translation goes to the entries its
words find by their text keys, as the app finds words in a line of subtitles,
up to three for each: those using the word as its headword writes it first, then
those nearest seven words long, the shortest then; they come first in an entry's
first sense, the word as it stands there beside. Some ten seconds for German.

Put the files into the server's `dictionaries` folder. The server loads each on
its next start, and again whenever it changes (see the server's README).

## Keeping a server's dictionaries current


EDRDG's licence asks of a server showing its dictionaries that it update them
at least once a month. `curate update` does it from the sources alone:

```sh
dart run halkyon_learn_curator:curate update \
  --into ../halkyon_learn_server/dictionaries [--every 7]
```

It downloads EDRDG's files of the day (`JMdict_e_examp.gz`, `kanjidic2.xml.gz`),
converts them with the JLPT levels in `data/` (static lists, see
`data/README.md`), and writes `jmdict-en.jsonl.gz` and `kanjidic-en.jsonl.gz`
into the folder, each beside its place first and then renamed over it, so a
server never reads half a file; should anything fail, the files stay as they
were. Some 50 seconds and 400 MB. With `--every 7` it keeps running and updates
whenever the files are a week old, as their dates say (a restart doesn't
download again), and tries again six hours after a failure. An instance runs it
so in a container of its own (the server's README). Beside the files,
`.conversion` says how the curator converted them (`conversion` in
`lib/src/update.dart`): raise it whenever the curator writes something else
from the same download (a label, a field), and an instance converts again as
soon as the new curator runs, not a week later.

With `--languages de,el,en,ko,zh` it also keeps those languages current, each
file of words with Tatoeba's sentences of the week as examples (a dictionary
whose sentences can't be had keeps its words without them), in this order:
Chinese from MDBG's CC-CEDICT of the day as often as EDRDG's files; the words of
the rest from Wiktionary: each time it wakes it asks kaikki.org when its dump
last changed (`Last-Modified`, without downloading it), and converts it again
only when it is newer than the files' (about monthly), when the languages named
changed, or when `wiktionaryConversion` in `lib/src/update.dart` was raised;
`.wiktionary` beside the files says which (for Korean it keeps
`.wiktionary-ko.json.gz`, the forms and Hanja readings). The dump is converted
as it downloads, never stored. Korean's words from KRDict: each time it wakes
it reads KRDict's page, and converts again only when it offers another export,
when Wiktionary's Korean file changed, or when `krdictConversion` was raised
(`.krdict`). Then the characters (Unihan for Chinese and Korean, with
CC-CEDICT's words; the Greek alphabet) as often as EDRDG's files, and at once
when Wiktionary's Korean file is newer than the Hanja's. Files the curator wrote
for a language no longer named, or for a source replaced (Korean's words from
Wiktionary, before KRDict), are removed, and the server unloads them.

## The format (`halkyon-dictionary`, version 1)

[JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`.

The **first line** says what the file holds:

| Field | |
|---|---|
| `format`, `formatVersion` | `"halkyon-dictionary"`, `1` |
| `kind` | `words` (a dictionary of words, JMdict) or `characters` (of the characters words are written in, KANJIDIC2). The app shows them apart. |
| `source`, `sourceVersion` | The source's name (`jmdict`) and version (`3.6.2 (2026-10-05)`). An instance holds one version of a source for each meaning language. |
| `language`, `glossLanguage` | BCP 47 tags: the words' language (`ja`) and the meanings' (`en`). |
| `license` | The data's licence, an SPDX identifier (`CC-BY-SA-4.0`). |
| `credits` | Everything it is drawn from: `name`, `by`, `license`, `url`. The app lists them where it credits its data sources. |

**Every further line** is an entry:

| Field | |
|---|---|
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary, CC-CEDICT and KRDict the 15,000 words wordfreq finds most used; for Chinese characters the 3,500 standard ones in common use), `jlpt:n5` … `jlpt:n1`, `hsk:1` … `hsk:7`, KRDict's learners' levels (`level:beginner`, `level:intermediate`, `level:advanced`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji; Hong Kong's for traditional Chinese), Korea's 1,800 Hanja of education (`education`), a character's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one, Unihan's `kRSUnicode`) and the script (`hiragana`, `katakana`, `hangul`, `greek`). |
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |

## The app's stroke files

How every kanji and kana is written comes from
[KanjiVG](https://kanjivg.tagaini.net) and is bundled with the app, so writing
works without a server. To update it, download `kanjivg-<date>.xml.gz` from
[KanjiVG's releases](https://github.com/KanjiVG/kanjivg/releases) and, in this
package:

```sh
dart run halkyon_learn_curator:curate kanjivg \
  --input kanjivg-20260714.xml.gz \
  --output ../halkyon_learn_flutter/assets/kanjivg
```

It writes one gzipped JSON file for every 256 code points that have any,
named by the code point's upper bits in hexadecimal (食, U+98DF, is in
`098.json.gz`; 94 files, 2.7 MB), so the app reads only the part it needs.
Each maps a character to its strokes (`s`, SVG paths in KanjiVG's 109 × 109
box, in the order they're written) and the components KanjiVG first splits it
into (`c`, when it has any). Then update the release named in the folder's
`ATTRIBUTION.md`.

## Adding a source

Write an adapter in `lib/src/sources/` that turns the source into the lines
above, with its credits and a sample of real data in `test/fixtures/` (and
where that comes from in `test/fixtures/README.md`); give it a command in
`bin/curate.dart`. Check the source's licence first, and that the instance may
pass its data on to the people using it.