feat(curator): Chinese from CC-CEDICT, with the HSK levels of 2025

Chinese comes from CC-CEDICT (CC BY-SA 4.0), made for learners of
Chinese: `curate cedict` writes two dictionaries of its entries,
`cc-cedict-hans-en` (zh-Hans) and `cc-cedict-hant-en` (zh-Hant), so a
studyset finds its words as it writes them, each finding a word by
either script, by pinyin with or without tones and by its meanings.
Readings are pinyin with tone marks, a syllable to a character; measure
words are a remark beside the meanings.

The HSK levels are those of the 2025 examination syllabus, which HSK
3.0's examination follows since July 2026 (decided 2026-10-14: level
tags, credited), as hsk30 extracted them (MIT): `hsk:1` … `hsk:6`, and
`hsk:7` for 7 to 9, each given to the entry read as the syllabus reads
the word (的 de, not 的 dì; 书 shū, not the Book of Documents, Shū),
ranking the words, HSK 1 first.

`curate update --languages de,el,en,zh` takes over from `--wiktionary`:
Chinese from MDBG's file of the day as often as EDRDG's files, the rest
from Wiktionary's dump.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-14 07:27:02 +02:00
commit 9d21f7f20e
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64

View file

@ -75,6 +75,32 @@ Greek and English first, then Chinese, then Korean):
whole dump takes a while; it is read once, streamed, keeping in memory only
which forms belong to which word.
Chinese from CC-CEDICT (decided 2026-10-14: a dictionary of its own, made for
learners of Chinese, where Wiktionary's Chinese entries sit at their traditional
forms):
1. Download `cedict_1_0_ts_utf-8_mdbg.txt.gz` from
[MDBG](https://www.mdbg.net/chinese/dictionary?page=cc-cedict), which makes it
anew every day.
2. Convert, in this package:
```sh
dart run halkyon_learn_curator:curate cedict \
--input cedict_1_0_ts_utf-8_mdbg.txt.gz \
--output ../halkyon_learn_server/dictionaries
```
It writes two dictionaries of the same entries, `cc-cedict-hans-en`
(`zh-Hans`, headwords in simplified characters) and `cc-cedict-hant-en`
(`zh-Hant`, in traditional ones), so a studyset finds its words as it writes
them; each finds a word by either script, by its pinyin with or without tones
(`xuéxí`, `xuexi`, `xue2xi2`) and by its meanings. Readings are pinyin with
tone marks, a syllable to a character (`xué xí`). The HSK levels of the 2025
examination syllabus (`data/hsk/`) label the words `hsk:1` … `hsk:6`, and
`hsk:7` for the advanced levels 7 to 9, which the syllabus doesn't tell
apart, each given to the entry read as the syllabus reads the word (的 *de*,
not 的 *dì*), and rank them, HSK 1 first; wordfreq ranks the rest.
Put the files into the server's `dictionaries` folder. The server loads each on
its next start, and again whenever it changes (see the server's README).
@ -103,7 +129,8 @@ so in a container of its own (the server's README). Beside the files,
from the same download (a label, a field), and an instance converts again as
soon as the new curator runs, not a week later.
With `--wiktionary de,el,en` it also keeps those languages current from
With `--languages de,el,en,zh` it also keeps those languages current: Chinese
from MDBG's CC-CEDICT of the day as often as EDRDG's files (below), the rest from
Wiktionary: each time it wakes it asks kaikki.org when its dump last changed
(`Last-Modified`, without downloading it), and converts it again only when it is
newer than the files' (about monthly), when the languages named changed, or when
@ -132,12 +159,12 @@ The **first line** says what the file holds:
|---|---|
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary and CC-CEDICT the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1`, `hsk:1` … `hsk:7` (later e.g. `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). |
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). |
## The app's stroke files