halkyon/halkyon_learn_curator/test/fixtures/README.md

46 lines
2.6 KiB
Markdown
Raw Normal View History

# Test data
`jmdict-eng-sample.json` holds six entries of JMdict 3.6.2 (2026-10-05) as
jmdict-simplified writes it (<https://github.com/scriptin/jmdict-simplified>):
JMdict is © the Electronic Dictionary Research and Development Group, used under
its licence, CC BY-SA 4.0 (<https://www.edrdg.org/edrdg/licence.html>).
`jlpt/` holds a few rows of the JLPT lists of yomitan-jlpt-vocab
(<https://github.com/stephenmk/yomitan-jlpt-vocab>), CC BY-SA 4.0, from Jonathan
Waller's lists (<http://www.tanos.co.uk/jlpt/>), CC BY; one row without an id is
made up.
`kanjidic2-sample.xml` holds five characters of KANJIDIC2 (database version
2026-282) with its header and a shortened DTD: KANJIDIC2 is © the Electronic
Dictionary Research and Development Group, used under its licence, CC BY-SA 4.0
(<https://www.edrdg.org/edrdg/licence.html>).
`kanji-data-sample.json` holds four entries of kanji-data's `kanji.json`
(<https://github.com/davidluzgouveia/kanji-data>), cut to their numbers, from
KANJIDIC2 and from Jonathan Waller's JLPT levels (<http://www.tanos.co.uk/jlpt/>),
CC BY.
`kanjivg-sample.xml` holds four characters of KanjiVG (the release of
2026-07-14) with its notice, among them 橅, whose entry names itself 機:
KanjiVG is © Ulrich Apel, CC BY-SA 3.0 (<https://kanjivg.tagaini.net>).
`jmdict-e-examp-sample.xml` holds five entries of EDRDG's `JMdict_e_examp.gz`
of 2026-10-09 (JMdict with Tatoeba's sentences, CC BY 2.0 FR) and the
declarations of the entities they use: JMdict is © the Electronic Dictionary
Research and Development Group, used under its licence, CC BY-SA 4.0
(<https://www.edrdg.org/edrdg/licence.html>).
feat(curator): dictionaries of German, Greek and English from Wiktionary The languages besides Japanese come from the English Wiktionary as Wiktextract reads it (decided 2026-10-14: German, Greek and English first, then Chinese, then Korean): kaikki.org's raw dump, every language in one file, as its per-language files are being withdrawn. `curate wiktionary --languages de,el,en` reads it once, streamed, and writes German and Greek with Wiktionary's English meanings and English with what its translation tables give in German, the app's other language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources `wiktionary-<language>`, as the server holds one source per meaning language). Inflected forms, and Wiktionary's entries for forms alone ("Häusern: dative plural of Haus"), find their word instead of being entries; names and affixes are left out, as are articles that tables give cells of their own, abbreviations and rare spellings, which would match common words. Entries keep their part of speech, genders, IPA, principal forms and examples keyed by language as JMdict's. wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank the words by how often they are used and label the 15,000 most used `common`. Typing without accents or umlauts finds a word (`strasse`, `haeuser`, `spiti`). The format gains `textKeys`: what finds an entry in running text, its own forms, never its meanings, which would match words of the meaning language in a subtitle. `curate update --wiktionary de,el,en` keeps the languages current: it asks kaikki.org when the dump last changed and converts it again only when it is newer, or the languages or the conversion changed. The whole dump converts in about ten minutes in under 300 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 07:19:16 +02:00
`wiktionary-sample.jsonl` holds 25 lines of kaikki.org's Wiktextract data of the
English Wiktionary (the dump of 2026-10-03, read per word from kaikki.org): the
German Haus, Häuser, Häusern, essen, aß, gegessen and Straße, the Greek σπίτι,
σπιτιού and γράφω, the English house, houses, go and went, and the French
maison, each shortened to the fields the curator reads (no categories,
etymologies or related words; at most six senses, two examples, 40 forms, two
sounds, and 14 translations, those into German and Greek). Wiktionary's text is
by its contributors, CC BY-SA 4.0 (<https://en.wiktionary.org/wiki/Wiktionary:Copyrights>);
the extraction is Tatu Ylonen's Wiktextract (<https://kaikki.org>).
`cedict-sample.txt` holds CC-CEDICT's header and 16 of its lines (MDBG's file of
2026-10-13): 中国, 了, 女儿, 学习, 书, 熊猫, 略, 的, 说 and 银行, with their
readings. CC-CEDICT is by MDBG and its contributors, CC BY-SA 4.0
(<https://www.mdbg.net/chinese/dictionary?page=cc-cedict>).