feat(curator): characters of Chinese and Korean, the Greek alphabet, Korean words from KRDict
For parity with Japanese's KANJIDIC2 and kana: - Unihan (Unicode License v3) makes characters' dictionaries: the 8,105 standard simplified characters (unihan-hans) and the traditional ones in use (unihan-hant), told apart and ranked by CC-CEDICT's words (Unihan calls 这 its own traditional variant; 後 ranks as the 后 of 以后), with pinyin, meanings, strokes, Kangxi radical, HSK 2025 character levels, the 3,500 common ones and Hong Kong's grades; and Korean Hanja (unihan-ko) with their sounds, the 1,800 of education, and their native readings from Wiktionary (물 수), with the 40 Hangul letters as kana are. - The Greek alphabet, the curator's own (CC0): letters with names, transliteration and sound. - KRDict, the Basic Korean Dictionary for learners (CC BY-SA 2.0 KR, downloaded without an account), replaces Wiktionary for Korean's words: learners' levels, pronunciation where it differs from the spelling (같이 [가치]), Hanja, English senses; Wiktionary's update keeps the words' forms (가다's 99) and the Hanja's readings beside it for both. - The update converts KRDict when its page offers another export and the characters weekly, and removes the files of languages no longer kept or of sources replaced. Measured on the real data: Unihan 3 s; KRDict 54,870 entries in 24 s and under 600 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
4e6581b9a5
commit
8820cefde3
1 changed files with 6997 additions and 85 deletions
23
halkyon_learn_curator/test/fixtures/README.md
vendored
23
halkyon_learn_curator/test/fixtures/README.md
vendored
|
|
@ -30,11 +30,12 @@ declarations of the entities they use: JMdict is © the Electronic Dictionary
|
|||
Research and Development Group, used under its licence, CC BY-SA 4.0
|
||||
(<https://www.edrdg.org/edrdg/licence.html>).
|
||||
|
||||
`wiktionary-sample.jsonl` holds 30 lines of kaikki.org's Wiktextract data of the
|
||||
`wiktionary-sample.jsonl` holds 31 lines of kaikki.org's Wiktextract data of the
|
||||
English Wiktionary (the dump of 2026-10-03, read per word from kaikki.org): the
|
||||
German Haus, Häuser, Häusern, essen, aß, gegessen, Straße and im, the Greek σπίτι,
|
||||
σπιτιού and γράφω, the English house, houses, go and went, the French maison,
|
||||
and the Korean 먹다, 집 and 학생, each shortened to the fields the curator reads (no categories,
|
||||
the Korean 먹다, 집 and 학생, and the Korean entry of the Han character 水 (its readings,
|
||||
물 수), each shortened to the fields the curator reads (no categories,
|
||||
etymologies or related words; at most six senses, two examples, 40 forms, two
|
||||
sounds, and 14 translations, those into German and Greek). Wiktionary's text is
|
||||
by its contributors, CC BY-SA 4.0 (<https://en.wiktionary.org/wiki/Wiktionary:Copyrights>);
|
||||
|
|
@ -51,3 +52,21 @@ Korean sentences using Haus, aß, Straße, 学习, 집에서 and 학생이 (and
|
|||
without an English translation), the links to their English translations, and
|
||||
those translations. The sentences are by Tatoeba's contributors, CC BY 2.0 FR
|
||||
(<https://tatoeba.org/en/terms_of_use>).
|
||||
|
||||
`unihan-sample.txt` holds the lines of Unihan 18.0.0 (2026-07-31,
|
||||
<https://www.unicode.org/Public/UCD/latest/ucd/Unihan.zip>) the curator reads for
|
||||
水, 说, 說, 説, 這, 这, 了, 瞭, 后, 後, 行, 们, 們, 学 and 學, with a header like its
|
||||
files'. The Unihan Database is © Unicode, Inc., under the Unicode License v3
|
||||
(<https://www.unicode.org/license.txt>).
|
||||
|
||||
`cedict-characters-sample.txt` holds 17 lines of CC-CEDICT (MDBG's file of
|
||||
2026-10-14) for the words those characters write (這, 了, 瞭, 瞭解, 為了, 後, 后,
|
||||
以後, 皇后, 們, 我們, 說, 水, 行, 銀行, 學 and 學習), CC BY-SA 4.0 as above.
|
||||
|
||||
`krdict-sample.json` holds six entries of KRDict's JSON export of 2026-09-19
|
||||
(<https://krdict.korean.go.kr/download/downloadPopup>): 같이 (the adverb and the
|
||||
particle), 가다 with its proverb 가는 날이 장날, 읽다 and 학교, each cut to its first
|
||||
three senses, their English and Japanese equivalents and two examples, without
|
||||
sound files or related words. 한국어기초사전 is by the National Institute of
|
||||
Korean Language (국립국어원), CC BY-SA 2.0 KR
|
||||
(<https://krdict.korean.go.kr/kboardPolicy/copyRightTermsInfo>).
|
||||
|
|
|
|||
Loading…
Reference in a new issue