halkyon/apps/learn/halkyon_learn_curator/test/fixtures/README.md

80 lines
4.8 KiB
Markdown
Raw Normal View History

# Test data
`jmdict-eng-sample.json` holds six entries of JMdict 3.6.2 (2026-10-05) as
jmdict-simplified writes it (<https://github.com/scriptin/jmdict-simplified>):
JMdict is © the Electronic Dictionary Research and Development Group, used under
its licence, CC BY-SA 4.0 (<https://www.edrdg.org/edrdg/licence.html>).
`jlpt/` holds a few rows of the JLPT lists of yomitan-jlpt-vocab
(<https://github.com/stephenmk/yomitan-jlpt-vocab>), CC BY-SA 4.0, from Jonathan
Waller's lists (<http://www.tanos.co.uk/jlpt/>), CC BY; one row without an id is
made up.
`kanjidic2-sample.xml` holds five characters of KANJIDIC2 (database version
2026-282) with its header and a shortened DTD: KANJIDIC2 is © the Electronic
Dictionary Research and Development Group, used under its licence, CC BY-SA 4.0
(<https://www.edrdg.org/edrdg/licence.html>).
`kanji-data-sample.json` holds four entries of kanji-data's `kanji.json`
(<https://github.com/davidluzgouveia/kanji-data>), cut to their numbers, from
KANJIDIC2 and from Jonathan Waller's JLPT levels (<http://www.tanos.co.uk/jlpt/>),
CC BY.
`kanjivg-sample.xml` holds four characters of KanjiVG (the release of
2026-07-14) with its notice, among them 橅, whose entry names itself 機:
KanjiVG is © Ulrich Apel, CC BY-SA 3.0 (<https://kanjivg.tagaini.net>).
`jmdict-e-examp-sample.xml` holds five entries of EDRDG's `JMdict_e_examp.gz`
of 2026-10-09 (JMdict with Tatoeba's sentences, CC BY 2.0 FR) and the
declarations of the entities they use: JMdict is © the Electronic Dictionary
Research and Development Group, used under its licence, CC BY-SA 4.0
(<https://www.edrdg.org/edrdg/licence.html>).
feat(curator): dictionaries of German, Greek and English from Wiktionary The languages besides Japanese come from the English Wiktionary as Wiktextract reads it (decided 2026-10-14: German, Greek and English first, then Chinese, then Korean): kaikki.org's raw dump, every language in one file, as its per-language files are being withdrawn. `curate wiktionary --languages de,el,en` reads it once, streamed, and writes German and Greek with Wiktionary's English meanings and English with what its translation tables give in German, the app's other language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources `wiktionary-<language>`, as the server holds one source per meaning language). Inflected forms, and Wiktionary's entries for forms alone ("Häusern: dative plural of Haus"), find their word instead of being entries; names and affixes are left out, as are articles that tables give cells of their own, abbreviations and rare spellings, which would match common words. Entries keep their part of speech, genders, IPA, principal forms and examples keyed by language as JMdict's. wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank the words by how often they are used and label the 15,000 most used `common`. Typing without accents or umlauts finds a word (`strasse`, `haeuser`, `spiti`). The format gains `textKeys`: what finds an entry in running text, its own forms, never its meanings, which would match words of the meaning language in a subtitle. `curate update --wiktionary de,el,en` keeps the languages current: it asks kaikki.org when the dump last changed and converts it again only when it is newer, or the languages or the conversion changed. The whole dump converts in about ten minutes in under 300 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 07:19:16 +02:00
feat(curator): characters of Chinese and Korean, the Greek alphabet, Korean words from KRDict For parity with Japanese's KANJIDIC2 and kana: - Unihan (Unicode License v3) makes characters' dictionaries: the 8,105 standard simplified characters (unihan-hans) and the traditional ones in use (unihan-hant), told apart and ranked by CC-CEDICT's words (Unihan calls 这 its own traditional variant; 後 ranks as the 后 of 以后), with pinyin, meanings, strokes, Kangxi radical, HSK 2025 character levels, the 3,500 common ones and Hong Kong's grades; and Korean Hanja (unihan-ko) with their sounds, the 1,800 of education, and their native readings from Wiktionary (물 수), with the 40 Hangul letters as kana are. - The Greek alphabet, the curator's own (CC0): letters with names, transliteration and sound. - KRDict, the Basic Korean Dictionary for learners (CC BY-SA 2.0 KR, downloaded without an account), replaces Wiktionary for Korean's words: learners' levels, pronunciation where it differs from the spelling (같이 [가치]), Hanja, English senses; Wiktionary's update keeps the words' forms (가다's 99) and the Hanja's readings beside it for both. - The update converts KRDict when its page offers another export and the characters weekly, and removes the files of languages no longer kept or of sources replaced. Measured on the real data: Unihan 3 s; KRDict 54,870 entries in 24 s and under 600 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 12:08:10 +02:00
`wiktionary-sample.jsonl` holds 31 lines of kaikki.org's Wiktextract data of the
feat(curator): dictionaries of German, Greek and English from Wiktionary The languages besides Japanese come from the English Wiktionary as Wiktextract reads it (decided 2026-10-14: German, Greek and English first, then Chinese, then Korean): kaikki.org's raw dump, every language in one file, as its per-language files are being withdrawn. `curate wiktionary --languages de,el,en` reads it once, streamed, and writes German and Greek with Wiktionary's English meanings and English with what its translation tables give in German, the app's other language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources `wiktionary-<language>`, as the server holds one source per meaning language). Inflected forms, and Wiktionary's entries for forms alone ("Häusern: dative plural of Haus"), find their word instead of being entries; names and affixes are left out, as are articles that tables give cells of their own, abbreviations and rare spellings, which would match common words. Entries keep their part of speech, genders, IPA, principal forms and examples keyed by language as JMdict's. wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank the words by how often they are used and label the 15,000 most used `common`. Typing without accents or umlauts finds a word (`strasse`, `haeuser`, `spiti`). The format gains `textKeys`: what finds an entry in running text, its own forms, never its meanings, which would match words of the meaning language in a subtitle. `curate update --wiktionary de,el,en` keeps the languages current: it asks kaikki.org when the dump last changed and converts it again only when it is newer, or the languages or the conversion changed. The whole dump converts in about ten minutes in under 300 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 07:19:16 +02:00
English Wiktionary (the dump of 2026-10-03, read per word from kaikki.org): the
German Haus, Häuser, Häusern, essen, aß, gegessen, Straße and im, the Greek σπίτι,
σπιτιού and γράφω, the English house, houses, go and went, the French maison,
feat(curator): characters of Chinese and Korean, the Greek alphabet, Korean words from KRDict For parity with Japanese's KANJIDIC2 and kana: - Unihan (Unicode License v3) makes characters' dictionaries: the 8,105 standard simplified characters (unihan-hans) and the traditional ones in use (unihan-hant), told apart and ranked by CC-CEDICT's words (Unihan calls 这 its own traditional variant; 後 ranks as the 后 of 以后), with pinyin, meanings, strokes, Kangxi radical, HSK 2025 character levels, the 3,500 common ones and Hong Kong's grades; and Korean Hanja (unihan-ko) with their sounds, the 1,800 of education, and their native readings from Wiktionary (물 수), with the 40 Hangul letters as kana are. - The Greek alphabet, the curator's own (CC0): letters with names, transliteration and sound. - KRDict, the Basic Korean Dictionary for learners (CC BY-SA 2.0 KR, downloaded without an account), replaces Wiktionary for Korean's words: learners' levels, pronunciation where it differs from the spelling (같이 [가치]), Hanja, English senses; Wiktionary's update keeps the words' forms (가다's 99) and the Hanja's readings beside it for both. - The update converts KRDict when its page offers another export and the characters weekly, and removes the files of languages no longer kept or of sources replaced. Measured on the real data: Unihan 3 s; KRDict 54,870 entries in 24 s and under 600 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 12:08:10 +02:00
the Korean 먹다, 집 and 학생, and the Korean entry of the Han character 水 (its readings,
물 수), each shortened to the fields the curator reads (no categories,
feat(curator): dictionaries of German, Greek and English from Wiktionary The languages besides Japanese come from the English Wiktionary as Wiktextract reads it (decided 2026-10-14: German, Greek and English first, then Chinese, then Korean): kaikki.org's raw dump, every language in one file, as its per-language files are being withdrawn. `curate wiktionary --languages de,el,en` reads it once, streamed, and writes German and Greek with Wiktionary's English meanings and English with what its translation tables give in German, the app's other language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources `wiktionary-<language>`, as the server holds one source per meaning language). Inflected forms, and Wiktionary's entries for forms alone ("Häusern: dative plural of Haus"), find their word instead of being entries; names and affixes are left out, as are articles that tables give cells of their own, abbreviations and rare spellings, which would match common words. Entries keep their part of speech, genders, IPA, principal forms and examples keyed by language as JMdict's. wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank the words by how often they are used and label the 15,000 most used `common`. Typing without accents or umlauts finds a word (`strasse`, `haeuser`, `spiti`). The format gains `textKeys`: what finds an entry in running text, its own forms, never its meanings, which would match words of the meaning language in a subtitle. `curate update --wiktionary de,el,en` keeps the languages current: it asks kaikki.org when the dump last changed and converts it again only when it is newer, or the languages or the conversion changed. The whole dump converts in about ten minutes in under 300 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 07:19:16 +02:00
etymologies or related words; at most six senses, two examples, 40 forms, two
sounds, and 14 translations, those into German and Greek). Wiktionary's text is
by its contributors, CC BY-SA 4.0 (<https://en.wiktionary.org/wiki/Wiktionary:Copyrights>);
the extraction is Tatu Ylonen's Wiktextract (<https://kaikki.org>).
`cedict-sample.txt` holds CC-CEDICT's header and 23 of its lines (MDBG's file of
2026-10-13): 中国, 了, 女儿, 学习, 书, 熊猫, 略, 的, 说, 银行, 和 and 咊, an old
variant of 和, with their readings. CC-CEDICT is by MDBG and its contributors, CC BY-SA 4.0
(<https://www.mdbg.net/chinese/dictionary?page=cc-cedict>).
`tatoeba-*.tsv` hold a few lines of Tatoeba's exports of 2026-10-10
(<https://downloads.tatoeba.org/exports/per_language/>): German, Chinese and
Korean sentences using Haus, aß, Straße, 学习, 집에서 and 학생이 (and one of each
without an English translation), the links to their English translations, and
those translations. The sentences are by Tatoeba's contributors, CC BY 2.0 FR
(<https://tatoeba.org/en/terms_of_use>).
feat(curator): characters of Chinese and Korean, the Greek alphabet, Korean words from KRDict For parity with Japanese's KANJIDIC2 and kana: - Unihan (Unicode License v3) makes characters' dictionaries: the 8,105 standard simplified characters (unihan-hans) and the traditional ones in use (unihan-hant), told apart and ranked by CC-CEDICT's words (Unihan calls 这 its own traditional variant; 後 ranks as the 后 of 以后), with pinyin, meanings, strokes, Kangxi radical, HSK 2025 character levels, the 3,500 common ones and Hong Kong's grades; and Korean Hanja (unihan-ko) with their sounds, the 1,800 of education, and their native readings from Wiktionary (물 수), with the 40 Hangul letters as kana are. - The Greek alphabet, the curator's own (CC0): letters with names, transliteration and sound. - KRDict, the Basic Korean Dictionary for learners (CC BY-SA 2.0 KR, downloaded without an account), replaces Wiktionary for Korean's words: learners' levels, pronunciation where it differs from the spelling (같이 [가치]), Hanja, English senses; Wiktionary's update keeps the words' forms (가다's 99) and the Hanja's readings beside it for both. - The update converts KRDict when its page offers another export and the characters weekly, and removes the files of languages no longer kept or of sources replaced. Measured on the real data: Unihan 3 s; KRDict 54,870 entries in 24 s and under 600 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 12:08:10 +02:00
`unihan-sample.txt` holds the lines of Unihan 18.0.0 (2026-07-31,
<https://www.unicode.org/Public/UCD/latest/ucd/Unihan.zip>) the curator reads for
水, 说, 說, 説, 這, 这, 了, 瞭, 后, 後, 行, 们, 們, 学 and 學, with a header like its
files'. The Unihan Database is © Unicode, Inc., under the Unicode License v3
(<https://www.unicode.org/license.txt>).
`cedict-characters-sample.txt` holds 17 lines of CC-CEDICT (MDBG's file of
2026-10-14) for the words those characters write (這, 了, 瞭, 瞭解, 為了, 後, 后,
以後, 皇后, 們, 我們, 說, 水, 行, 銀行, 學 and 學習), CC BY-SA 4.0 as above.
`krdict-sample.json` holds six entries of KRDict's JSON export of 2026-09-19
(<https://krdict.korean.go.kr/download/downloadPopup>): 같이 (the adverb and the
particle), 가다 with its proverb 가는 날이 장날, 읽다 and 학교, each cut to its first
three senses, their English and Japanese equivalents and two examples, without
sound files or related words. 한국어기초사전 is by the National Institute of
Korean Language (국립국어원), CC BY-SA 2.0 KR
(<https://krdict.korean.go.kr/kboardPolicy/copyRightTermsInfo>).
`makemeahanzi-graphics-sample.txt` and `makemeahanzi-dictionary-sample.txt` hold
the lines of Make Me a Hanzi's `graphics.txt` and `dictionary.txt` (commit
`618dbab`, 2018-10-16, <https://github.com/skishore/makemeahanzi>) for 水 and 海.
The strokes derive from the fonts Arphic PL KaitiM GB and Arphic PL UKai, © 1999
Arphic Technology Co., Ltd., under the Arphic Public License (its text is in the
app's `assets/makemeahanzi/ARPHICPL.TXT`); the dictionary's lines are under the
LGPL, version 3 or later (`assets/makemeahanzi/LGPL.txt`).