halkyon/docs/adr/0008-languages-beyond-japanese.md

ADR 0008: Dictionaries and word cutting for languages beyond Japanese

• Status: accepted
• Date: 2026-10-14

Context

Halkyon Learn's language support was built for Japanese: JMdict and KANJIDIC2 as dictionaries, a longest-match splitter with Japanese conjugations for subtitles, furigana, JLPT levels, Jisho links. Studysets can already be in 22 languages, and the app speaks English and German, so learners of German, Greek, English, Chinese and Korean find no dictionary to add words from, and their subtitles aren't cut into words.

What a dictionary for learners needs, per language: meanings in the learner's language, the forms a word takes in running text, readings where the script doesn't say them, some order of usefulness (frequency, exam levels), and a licence that lets the cards made from it be published under CC BY-SA, as studysets are.

Decision

1. Order (the developer's choice): German, Greek and English first, sharing one source and one way of cutting text; then Chinese; then Korean, which needs morphology (from Wiktionary for now, with its particles taken off by rule).
2. Wiktionary for the languages without a dedicated dictionary, as Wiktextract reads the English Wiktionary (kaikki.org, CC BY-SA 4.0 and GFDL; Wiktextract's code MIT). Its per-language files are being withdrawn, so the curator reads the raw dump of every language (some 3 GB), streamed, once. German and Greek get Wiktionary's English meanings; English gets German ones from the translation tables of its entries, which name the English meaning each translation is of (the German Wiktionary has German definitions of English words, worse for cards than translations). German glosses for Greek, Chinese or Korean would only come from inverted translation tables, weak and noisy: those studysets fall back to English meanings, as the app's dictionary choice already does.
3. Each instance converts the dump itself (the developer's choice, as it does EDRDG's files), whenever kaikki.org makes a new one (about monthly): the curator asks for its date first and downloads only when it is newer. Ten minutes in under 300 MB on learn.halkyon.eu's server. DICTIONARY_LANGUAGES (unset: de,el,en,ko,zh) names the languages.
4. Forms find their word; they aren't entries. Wiktionary has entries for inflected forms alone ("Häusern: dative plural of Haus"); they become keys of their word, as do the forms of a word's inflection tables. Left out of the keys: articles that tables give cells of their own (Greek's ο, του), abbreviations (TO for turnover) and rare or obsolete spellings (the for thee), as they would match common words.
5. Text keys (textKeys, a field of the dictionary format): what finds an entry in running text, its own forms, never its meanings, which would match words of the meaning language in a subtitle. JMdict's files name none and stay as they were.
6. Frequency from wordfreq (its small lists, CC BY-SA 4.0, crediting SUBTLEX): Wiktionary and CC-CEDICT have no commonness of their own; the 15,000 most used words are common, and words rank by their own frequency (a form's only when the word has none, at half its rank), so a rare word never ranks by a frequent form.
7. Chinese from CC-CEDICT (CC BY-SA 4.0, made for learners, daily), as two dictionaries of the same entries: simplified (zh-Hans) and traditional (zh-Hant), so a studyset finds words as it writes them; pinyin with tone marks, a syllable to a character; entries that are only variants find their word instead of repeating it.
8. HSK levels of the 2025 syllabus (the developer's choice: level tags, credited), which HSK 3.0's examination follows since July 2026, as hsk30 extracted them (MIT). The Ministry claims copyright on the standard's documents; only each word's level reaches the dictionaries, a fact about it. Levels 7 to 9 are one, as the syllabus has them. A word's level goes to the entry read as the syllabus reads it, with its tones (和 hé, not hè), else by its letters (the syllabus marks the tone changes of 一 and 不).
9. Text is cut by its language (WordCutting): Japanese by its longest words with conjugations, Chinese by its longest words as they stand (with the longer words running over each cut offered beside it, as for Japanese), Korean at its spaces, a word the dictionary lacks as it stands taken without the particle or copula ending it ends with (집에서 is 집, 아이 stays a child; Korean's own grammar, no analyser: a verb's endings are among its forms in Wiktionary), every other language at its spaces, each word looked up as it stands, an entry written exactly so first (German's noun Essen before its verb essen).
10. A noun goes onto its card with its article where the language has gendered articles and the noun one gender (das Haus, το σπίτι): learners learn the gender with the word, and the article says it.
11. Tatoeba's sentences are the examples (CC BY 2.0 FR, weekly per-language exports), as JMdict's examples edition brings them for Japanese: each sentence with a translation into the meaning language goes to the entries its words find, cut as the app cuts subtitles, up to three for each, those using the word as its headword writes it first (Haus, not zu Hause; simplified characters in the simplified dictionary), then those nearest seven words long. Wiktionary's own examples, often quotations, follow them. A dictionary whose sentences can't be had keeps its words.
12. Words are looked up on the web on Jisho in Japanese, built on JMdict, and on Wiktionary at the language's section otherwise, which the dictionaries are drawn from.

Consequences

• An instance downloads some 3 GB a month more, and holds some 210,000 entries more (German 84,000, English 66,000, Korean 35,000, Greek 27,000) besides CC-CEDICT's 122,000 in each script. Measured: the whole dump converts in nine minutes in under 300 MB; the server loads German in 17 s and CC-CEDICT in 12 s, and finds a line's words in 2 to 24 ms.
• Wiktionary has no stable ids: an entry's id is its word, part of speech and etymology number (Haus/noun). Cards keep it as a soft reference, as they keep JMdict's; an entry Wiktionary splits or renames is no longer found by a card made from it, which keeps what it copied.
• German compounds Wiktionary lacks (Haustürschlüssel) and separable verbs split in a sentence (rufe … an) aren't found; a Korean verb in a form Wiktionary's table lacks isn't either. KRDict (the National Institute of Korean Language's learners' dictionary, CC BY-SA 2.0 KR, with levels and Hanja) would do better for Korean; its download needs a person on its site, or an API key.
• Not bundled, as their licences forbid what studysets need: Ding and FreeDict's German (GPL), the Goethe-Institut's and the English Vocabulary Profile's level lists (copyrighted), the Kelly lists (non-commercial), KRDict's sound files.