halkyon/docs/adr/0008-languages-beyond-japanese.md

# ADR 0008: Dictionaries and word cutting for languages beyond Japanese

- **Status:** accepted
- **Date:** 2026-10-14

## Context

Halkyon Learn's language support was built for Japanese: JMdict and KANJIDIC2 as
dictionaries, a longest-match splitter with Japanese conjugations for subtitles,
furigana, JLPT levels, Jisho links. Studysets can already be in 22 languages, and the
app speaks English and German, so learners of German, Greek, English, Chinese and Korean
find no dictionary to add words from, and their subtitles aren't cut into words.

What a dictionary for learners needs, per language: meanings in the learner's language,
the forms a word takes in running text, readings where the script doesn't say them,
some order of usefulness (frequency, exam levels), and a licence that lets the cards
made from it be published under CC BY-SA, as studysets are.

## Decision

1. **Order** (the developer's choice): German, Greek and English first, sharing one
   source and one way of cutting text; then Chinese; then Korean, which needs morphology
   (from Wiktionary for now, with its particles taken off by rule).
2. **Wiktionary for the languages without a dedicated dictionary**, as Wiktextract reads
   the English Wiktionary (kaikki.org, CC BY-SA 4.0 and GFDL; Wiktextract's code MIT). Its
   per-language files are being withdrawn, so the curator reads the raw dump of every
   language (some 3 GB), streamed, once. German and Greek get Wiktionary's English
   meanings; English gets German ones from the translation tables of its entries, which
   name the English meaning each translation is of (the German Wiktionary has German
   *definitions* of English words, worse for cards than translations). German glosses for
   Greek, Chinese or Korean would only come from inverted translation tables, weak and
   noisy: those studysets fall back to English meanings, as the app's dictionary choice
   already does.
3. **Each instance converts the dump itself** (the developer's choice, as it does EDRDG's
   files), whenever kaikki.org makes a new one (about monthly): the curator asks for its
   date first and downloads only when it is newer. Ten minutes in under 300 MB on
   learn.halkyon.eu's server. `DICTIONARY_LANGUAGES` (unset: `de,el,en,ko,zh`) names
   the languages.
4. **Forms find their word; they aren't entries.** Wiktionary has entries for inflected
   forms alone ("Häusern: dative plural of Haus"); they become keys of their word, as do
   the forms of a word's inflection tables. Left out of the keys: articles that tables
   give cells of their own (Greek's `ο`, `του`), abbreviations (`TO` for turnover) and
   rare or obsolete spellings (`the` for thee), as they would match common words.
5. **Text keys** (`textKeys`, a field of the dictionary format): what finds an entry in
   running text, its own forms, never its meanings, which would match words of the meaning
   language in a subtitle. JMdict's files name none and stay as they were.
6. **Frequency from wordfreq** (its small lists, CC BY-SA 4.0, crediting SUBTLEX):
   Wiktionary and CC-CEDICT have no commonness of their own; the 15,000 most used words
   are `common`, and words rank by their own frequency (a form's only when the word has
   none, at half its rank), so a rare word never ranks by a frequent form.
7. **Chinese from CC-CEDICT** (CC BY-SA 4.0, made for learners, daily), as two
   dictionaries of the same entries: simplified (`zh-Hans`) and traditional (`zh-Hant`), so
   a studyset finds words as it writes them; pinyin with tone marks, a syllable to a
   character; entries that are only variants find their word instead of repeating it.
8. **HSK levels of the 2025 syllabus** (the developer's choice: level tags, credited),
   which HSK 3.0's examination follows since July 2026, as hsk30 extracted them (MIT). The
   Ministry claims copyright on the standard's documents; only each word's level reaches
   the dictionaries, a fact about it. Levels 7 to 9 are one, as the syllabus has them. A
   word's level goes to the entry read as the syllabus reads it, with its tones (和 hé, not
   hè), else by its letters (the syllabus marks the tone changes of 一 and 不).
9. **Text is cut by its language** (`WordCutting`): Japanese by its longest words with
   conjugations, Chinese by its longest words as they stand (with the longer words
   running over each cut offered beside it, as for Japanese), Korean at its spaces, a word
   the dictionary lacks as it stands taken without the particle or copula ending it ends
   with (집에서 is 집, 아이 stays a child; Korean's own grammar, no analyser: a verb's
   endings are among its forms in Wiktionary), every other language at its spaces, each
   word looked up as it stands, an entry written exactly so first (German's noun `Essen`
   before its verb `essen`).
10. **A noun goes onto its card with its article** where the language has gendered
    articles and the noun one gender (`das Haus`, `το σπίτι`): learners learn the gender
    with the word, and the article says it.
11. **Tatoeba's sentences are the examples** (CC BY 2.0 FR, weekly per-language
    exports), as JMdict's examples edition brings them for Japanese: each sentence with
    a translation into the meaning language goes to the entries its words find, cut as
    the app cuts subtitles, up to three for each, those using the word as its headword
    writes it first (Haus, not zu Hause; simplified characters in the simplified
    dictionary), then those nearest seven words long. Wiktionary's own examples, often
    quotations, follow them. A dictionary whose sentences can't be had keeps its words.
12. **Words are looked up on the web** on Jisho in Japanese, built on JMdict, and on
    Wiktionary at the language's section otherwise, which the dictionaries are drawn from.

## Consequences

- An instance downloads some 3 GB a month more, and holds some 210,000 entries more
  (German 84,000, English 66,000, Korean 35,000, Greek 27,000) besides CC-CEDICT's
  122,000 in each script. Measured: the whole dump converts in nine minutes in under
  300 MB; the server loads German in 17 s and CC-CEDICT in 12 s, and finds a line's words
  in 2 to 24 ms.
- Wiktionary has no stable ids: an entry's id is its word, part of speech and etymology
  number (`Haus/noun`). Cards keep it as a soft reference, as they keep JMdict's; an entry
  Wiktionary splits or renames is no longer found by a card made from it, which keeps
  what it copied.
- German compounds Wiktionary lacks (`Haustürschlüssel`) and separable verbs split in a
  sentence (`rufe … an`) aren't found; a Korean verb in a form Wiktionary's table lacks
  isn't either. KRDict (the National Institute of Korean Language's learners'
  dictionary, CC BY-SA 2.0 KR, with levels and Hanja) would do better for Korean; its
  download needs a person on its site, or an API key.
- Not bundled, as their licences forbid what studysets need: Ding and FreeDict's German
  (GPL), the Goethe-Institut's and the English Vocabulary Profile's level lists
  (copyrighted), the Kelly lists (non-commercial), KRDict's sound files.