feat(learn): furigana and pinyin over hints, from Tatoeba's transcriptions

The curator puts Tatoeba's readings on a dictionary's examples by their
Tatoeba ID (`withTatoebaReadings`, `curate readings`, and in `update`):
furigana over JMdict's Japanese from its jpn transcriptions, pinyin with
tone marks over Chinese from its cmn ones, each word over as many Han
characters as it has syllables. An example gets `ruby`, `[start, end,
reading]` each, and `rubyUnreviewed` where no one wrote or checked them
(MeCab's furigana, the automatic pinyin). JMdict's conversion goes to 3,
so instances convert again at once.

A word's example hints carry them as ruby marks, the word still in bold.
The hints sheet shows them from the start with a switch above
(Furigana, Pinyin), kept on the device (`DeviceSettings.hintReadings`),
and below a hint whose readings no one reviewed says so.

Whats-New: Example sentences in a word's hints now show furigana or pinyin over them, with a switch to hide them.
Whats-New-De: Beispielsätze in den Hinweisen eines Worts zeigen jetzt Furigana oder Pinyin darüber, mit einem Schalter zum Ausblenden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-11-10 20:28:30 +01:00
commit 3257905536
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64

View file

@ -189,6 +189,26 @@ up to three for each: those using the word as its headword writes it first, then
those nearest seven words long, the shortest then; they come first in an entry's
first sense, the word as it stands there beside. Some ten seconds for German.
Readings over the examples, from Tatoeba's transcriptions of their sentences
(decided 9 Nov 2026): furigana over JMdict's Japanese, pinyin over Chinese.
`update` adds them itself; by hand:
```sh
dart run halkyon_learn_curator:curate readings \
--input jmdict-en.jsonl.gz --transcriptions jpn_transcriptions.tsv.bz2 \
--output jmdict-en.jsonl.gz
```
from Tatoeba's `jpn_transcriptions.tsv.bz2` or `cmn_transcriptions.tsv.bz2`
(CC BY 2.0 FR, the sentences' own licence). Each example with a Tatoeba ID
gets `ruby`, a list of `[start, end, reading]` over its sentence (UTF-16 code
units), where the transcription is of that very sentence: Japanese's
`[漢字|かん|じ]` one reading for each kanji where there are as many, else one
over the word; Chinese's tone numbers as tone marks, each of Tatoeba's words
over as many Han characters as it has syllables, none where they don't count
up (about 1 % of them). `rubyUnreviewed` marks those no one wrote or checked,
MeCab's furigana and Tatoeba's automatic pinyin, some half of the Japanese.
Put the files into the server's `dictionaries` folder. The server loads each on
its next start, and again whenever it changes (see the server's README).
@ -277,7 +297,7 @@ The **first line** says what the file holds:
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, the source's id, e.g. `tatoeba`, and where Tatoeba transcribed the sentence `ruby` and `rubyUnreviewed`, the readings over it). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |
## The app's stroke files