feat(curator): characters of Chinese and Korean, the Greek alphabet, Korean words from KRDict
For parity with Japanese's KANJIDIC2 and kana: - Unihan (Unicode License v3) makes characters' dictionaries: the 8,105 standard simplified characters (unihan-hans) and the traditional ones in use (unihan-hant), told apart and ranked by CC-CEDICT's words (Unihan calls 这 its own traditional variant; 後 ranks as the 后 of 以后), with pinyin, meanings, strokes, Kangxi radical, HSK 2025 character levels, the 3,500 common ones and Hong Kong's grades; and Korean Hanja (unihan-ko) with their sounds, the 1,800 of education, and their native readings from Wiktionary (물 수), with the 40 Hangul letters as kana are. - The Greek alphabet, the curator's own (CC0): letters with names, transliteration and sound. - KRDict, the Basic Korean Dictionary for learners (CC BY-SA 2.0 KR, downloaded without an account), replaces Wiktionary for Korean's words: learners' levels, pronunciation where it differs from the spelling (같이 [가치]), Hanja, English senses; Wiktionary's update keeps the words' forms (가다's 99) and the Hanja's readings beside it for both. - The update converts KRDict when its page offers another export and the characters weekly, and removes the files of languages no longer kept or of sources replaced. Measured on the real data: Unihan 3 s; KRDict 54,870 entries in 24 s and under 600 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
4e6581b9a5
commit
8820cefde3
GPG key ID:
07F6FCF405488B64
1 changed files with 6997 additions and 85 deletions
|
|
@ -49,8 +49,8 @@ KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:
|
|||
|
||||
About 12,800 characters, in some 5 seconds.
|
||||
|
||||
Wiktionary, for the languages other than Japanese and Chinese (decided
|
||||
2026-10-14: German, Greek and English first, then Chinese, then Korean):
|
||||
Wiktionary, for the words of German, Greek and English (decided 2026-10-14),
|
||||
and for Korean the forms of its words and the native readings of its Hanja:
|
||||
|
||||
1. Download `raw-wiktextract-data.jsonl.gz` from
|
||||
[kaikki.org](https://kaikki.org/dictionary/rawdata.html): the English
|
||||
|
|
@ -65,8 +65,7 @@ Wiktionary, for the languages other than Japanese and Chinese (decided
|
|||
```
|
||||
|
||||
Each language but English becomes its words with Wiktionary's English
|
||||
meanings (`wiktionary-de-en.jsonl.gz`, source `wiktionary-de`; a Korean word
|
||||
with its Hanja and romanisation); English
|
||||
meanings (`wiktionary-de-en.jsonl.gz`, source `wiktionary-de`); English
|
||||
becomes its words with what their translation tables give in German
|
||||
(`wiktionary-en-de.jsonl.gz`), German being the app's other language. An
|
||||
inflected form, and an entry Wiktionary has for one alone ("Häusern: dative
|
||||
|
|
@ -102,6 +101,76 @@ forms):
|
|||
apart, each given to the entry read as the syllabus reads the word (的 *de*,
|
||||
not 的 *dì*), and rank them, HSK 1 first; wordfreq ranks the rest.
|
||||
|
||||
Chinese characters and Korean Hanja from Unihan, the Unicode Consortium's
|
||||
database of Han characters (decided 2026-10-14, for parity with KANJIDIC2;
|
||||
Unicode License v3):
|
||||
|
||||
1. Download `Unihan.zip` from
|
||||
[unicode.org](https://www.unicode.org/Public/UCD/latest/ucd/Unihan.zip)
|
||||
(8 MB, a new one with each version of Unicode), and CC-CEDICT as above.
|
||||
2. Convert, naming the languages:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate unihan --input Unihan.zip \
|
||||
--languages ko,zh --cedict cedict_1_0_ts_utf-8_mdbg.txt.gz \
|
||||
[--korean .wiktionary-ko.json.gz] \
|
||||
--output ../halkyon_learn_server/dictionaries
|
||||
```
|
||||
|
||||
For Chinese, two dictionaries of characters as for the words: the 8,105
|
||||
General Standard Chinese Characters of 2013 (`unihan-hans-en`, `zh-Hans`),
|
||||
and the traditional characters in use (`unihan-hant-en`, `zh-Hant`): each of
|
||||
them as Taiwan and Hong Kong write it, where it is a form of its own or a
|
||||
word of CC-CEDICT writes it so (Unihan calls 这 its own traditional variant,
|
||||
but no word is written with it), with those Hong Kong's primary schools
|
||||
teach. Each with its pinyin (the usual reading first: the mainland's in
|
||||
simplified, Taiwan's in traditional characters), meanings, stroke count and
|
||||
other script's forms; labelled by its HSK level (`hsk:1` … `hsk:7`, from the
|
||||
syllabus's characters in `data/hsk/`), the 3,500 standard characters in
|
||||
common use (`common`), a Hong Kong grade (`grade:1` … `grade:6`) and its
|
||||
Kangxi radical (`radical:N`), and ranked by HSK level, then by the most used
|
||||
word of CC-CEDICT written with it (wordfreq), so 後 ranks as the 后 of 以后
|
||||
and 后, the queen, by 皇后. For Korean, the 7,800 Hanja with a Korean reading
|
||||
(`unihan-ko-en`): their sounds (음, the one taught first), their native
|
||||
meanings with their sounds as Wiktionary gives them (훈, `물 수` for 水; from
|
||||
the Wiktionary update's `.wiktionary-ko.json.gz`), meanings and strokes,
|
||||
labelled `education` for the 1,800 taught in schools; and the 40 Hangul
|
||||
letters (jamo) as the kana are in KANJIDIC2's, with their sound and name,
|
||||
labelled `hangul`. Some 3 seconds.
|
||||
|
||||
The Greek alphabet, written by the curator itself (`greek-alphabet-en`, `el`,
|
||||
CC0): each letter with its capital, its Greek and English names, its
|
||||
transliteration (ELOT 743) and its sound, labelled `greek`, for a characters
|
||||
tab as kana have.
|
||||
|
||||
Korean words from KRDict, the National Institute of Korean Language's Basic
|
||||
Korean Dictionary for learners (decided 2026-10-14: levels, pronunciation and
|
||||
Hanja, which Wiktionary lacks; CC BY-SA 2.0 KR, later versions allowed):
|
||||
|
||||
1. Download its JSON export from
|
||||
[its page](https://krdict.korean.go.kr/download/downloadPopup) (Json 전체
|
||||
내려받기; no account needed; some 84 MB, a gigabyte unpacked, renewed about
|
||||
monthly).
|
||||
2. Convert:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate krdict \
|
||||
--input krdict_json_20260919.zip [--korean .wiktionary-ko.json.gz] \
|
||||
--output ../halkyon_learn_server/dictionaries
|
||||
```
|
||||
|
||||
`krdict-ko-en.jsonl.gz` holds its 54,870 words, idioms and proverbs, each
|
||||
with its part of speech, its pronunciation as its reading where it differs
|
||||
from the spelling (같이 [가치]), its Hanja, the few forms KRDict gives and
|
||||
its senses (KRDict's English words, its English definition as `info`);
|
||||
labelled by its learners' level (`level:beginner`, `level:intermediate`,
|
||||
`level:advanced`) and ranked by it, then by wordfreq (a verb, rarely written
|
||||
as the dictionary writes it, by its most used form). With the Wiktionary
|
||||
update's file, every form Wiktionary's tables give (가다's 99) finds a word in
|
||||
running text too. An idiom or proverb carries the id of the word it is filed
|
||||
under, so its own id adds itself (`27500:가는 날이 장날`). Read an entry at a
|
||||
time, some 25 seconds and 600 MB.
|
||||
|
||||
Example sentences from Tatoeba for those dictionaries (JMdict brings its own),
|
||||
as hints on the cards made from them:
|
||||
|
||||
|
|
@ -149,15 +218,23 @@ from the same download (a label, a field), and an instance converts again as
|
|||
soon as the new curator runs, not a week later.
|
||||
|
||||
With `--languages de,el,en,ko,zh` it also keeps those languages current, each
|
||||
file with Tatoeba's sentences of the week as examples (a dictionary whose
|
||||
sentences can't be had keeps its words without them): Chinese from MDBG's
|
||||
CC-CEDICT of the day as often as EDRDG's files (below), the rest from
|
||||
Wiktionary: each time it wakes it asks kaikki.org when its dump last changed
|
||||
(`Last-Modified`, without downloading it), and converts it again only when it is
|
||||
newer than the files' (about monthly), when the languages named changed, or when
|
||||
`wiktionaryConversion` in `lib/src/update.dart` was raised; `.wiktionary` beside
|
||||
the files says which. The dump is converted as it downloads, never stored.
|
||||
Files of a language no longer named stay until removed by hand.
|
||||
file of words with Tatoeba's sentences of the week as examples (a dictionary
|
||||
whose sentences can't be had keeps its words without them), in this order:
|
||||
Chinese from MDBG's CC-CEDICT of the day as often as EDRDG's files; the words of
|
||||
the rest from Wiktionary: each time it wakes it asks kaikki.org when its dump
|
||||
last changed (`Last-Modified`, without downloading it), and converts it again
|
||||
only when it is newer than the files' (about monthly), when the languages named
|
||||
changed, or when `wiktionaryConversion` in `lib/src/update.dart` was raised;
|
||||
`.wiktionary` beside the files says which (for Korean it keeps
|
||||
`.wiktionary-ko.json.gz`, the forms and Hanja readings). The dump is converted
|
||||
as it downloads, never stored. Korean's words from KRDict: each time it wakes
|
||||
it reads KRDict's page, and converts again only when it offers another export,
|
||||
when Wiktionary's Korean file changed, or when `krdictConversion` was raised
|
||||
(`.krdict`). Then the characters (Unihan for Chinese and Korean, with
|
||||
CC-CEDICT's words; the Greek alphabet) as often as EDRDG's files, and at once
|
||||
when Wiktionary's Korean file is newer than the Hanja's. Files the curator wrote
|
||||
for a language no longer named, or for a source replaced (Korean's words from
|
||||
Wiktionary, before KRDict), are removed, and the server unloads them.
|
||||
|
||||
## The format (`halkyon-dictionary`, version 1)
|
||||
|
||||
|
|
@ -180,12 +257,12 @@ The **first line** says what the file holds:
|
|||
|---|---|
|
||||
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
|
||||
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
|
||||
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary and CC-CEDICT the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1`, `hsk:1` … `hsk:7` (later e.g. `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
|
||||
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary, CC-CEDICT and KRDict the 15,000 words wordfreq finds most used; for Chinese characters the 3,500 standard ones in common use), `jlpt:n5` … `jlpt:n1`, `hsk:1` … `hsk:7`, KRDict's learners' levels (`level:beginner`, `level:intermediate`, `level:advanced`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji; Hong Kong's for traditional Chinese), Korea's 1,800 Hanja of education (`education`), a character's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one, Unihan's `kRSUnicode`) and the script (`hiragana`, `katakana`, `hangul`, `greek`). |
|
||||
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
|
||||
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
||||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
|
||||
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |
|
||||
|
||||
## The app's stroke files
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue