feat(curator): dictionaries of German, Greek and English from Wiktionary
The languages besides Japanese come from the English Wiktionary as
Wiktextract reads it (decided 2026-10-14: German, Greek and English
first, then Chinese, then Korean): kaikki.org's raw dump, every language
in one file, as its per-language files are being withdrawn.
`curate wiktionary --languages de,el,en` reads it once, streamed, and
writes German and Greek with Wiktionary's English meanings and English
with what its translation tables give in German, the app's other
language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources
`wiktionary-<language>`, as the server holds one source per meaning
language). Inflected forms, and Wiktionary's entries for forms alone
("Häusern: dative plural of Haus"), find their word instead of being
entries; names and affixes are left out, as are articles that tables
give cells of their own, abbreviations and rare spellings, which would
match common words. Entries keep their part of speech, genders, IPA,
principal forms and examples keyed by language as JMdict's.
wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank
the words by how often they are used and label the 15,000 most used
`common`. Typing without accents or umlauts finds a word (`strasse`,
`haeuser`, `spiti`).
The format gains `textKeys`: what finds an entry in running text, its
own forms, never its meanings, which would match words of the meaning
language in a subtitle. `curate update --wiktionary de,el,en` keeps the
languages current: it asks kaikki.org when the dump last changed and
converts it again only when it is newer, or the languages or the
conversion changed. The whole dump converts in about ten minutes in
under 300 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
5683a1f33e
commit
6d57f50b16
GPG key ID:
07F6FCF405488B64
1 changed files with 1495 additions and 22 deletions
|
|
@ -49,11 +49,38 @@ KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:
|
|||
|
||||
About 12,800 characters, in some 5 seconds.
|
||||
|
||||
Wiktionary, for the languages other than Japanese (decided 2026-10-14: German,
|
||||
Greek and English first, then Chinese, then Korean):
|
||||
|
||||
1. Download `raw-wiktextract-data.jsonl.gz` from
|
||||
[kaikki.org](https://kaikki.org/dictionary/rawdata.html): the English
|
||||
Wiktionary as Wiktextract reads it, every language in one file (some 3 GB,
|
||||
24 GB unpacked; kaikki.org's files of single languages are being withdrawn).
|
||||
2. Convert, in this package, naming the languages:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate wiktionary \
|
||||
--input raw-wiktextract-data.jsonl.gz --languages de,el,en \
|
||||
--version 2026-10-03 --output ../halkyon_learn_server/dictionaries
|
||||
```
|
||||
|
||||
Each language but English becomes its words with Wiktionary's English
|
||||
meanings (`wiktionary-de-en.jsonl.gz`, source `wiktionary-de`); English
|
||||
becomes its words with what their translation tables give in German
|
||||
(`wiktionary-en-de.jsonl.gz`), German being the app's other language. An
|
||||
inflected form, and an entry Wiktionary has for one alone ("Häusern: dative
|
||||
plural of Haus"), finds its word rather than being an entry of its own;
|
||||
names and affixes are left out. Word frequencies from `data/wordfreq/` rank
|
||||
the words and say which are `common` (the 15,000 most used). Reading the
|
||||
whole dump takes a while; it is read once, streamed, keeping in memory only
|
||||
which forms belong to which word.
|
||||
|
||||
Put the files into the server's `dictionaries` folder. The server loads each on
|
||||
its next start, and again whenever it changes (see the server's README).
|
||||
|
||||
## Keeping a server's dictionaries current
|
||||
|
||||
|
||||
EDRDG's licence asks of a server showing its dictionaries that it update them
|
||||
at least once a month. `curate update` does it from the sources alone:
|
||||
|
||||
|
|
@ -76,6 +103,14 @@ so in a container of its own (the server's README). Beside the files,
|
|||
from the same download (a label, a field), and an instance converts again as
|
||||
soon as the new curator runs, not a week later.
|
||||
|
||||
With `--wiktionary de,el,en` it also keeps those languages current from
|
||||
Wiktionary: each time it wakes it asks kaikki.org when its dump last changed
|
||||
(`Last-Modified`, without downloading it), and converts it again only when it is
|
||||
newer than the files' (about monthly), when the languages named changed, or when
|
||||
`wiktionaryConversion` in `lib/src/update.dart` was raised; `.wiktionary` beside
|
||||
the files says which. The dump is converted as it downloads, never stored.
|
||||
Files of a language no longer named stay until removed by hand.
|
||||
|
||||
## The format (`halkyon-dictionary`, version 1)
|
||||
|
||||
[JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`.
|
||||
|
|
@ -97,11 +132,12 @@ The **first line** says what the file holds:
|
|||
|---|---|
|
||||
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
|
||||
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
|
||||
| `labels` | What it can be picked by: `common`, `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
|
||||
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
|
||||
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
|
||||
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
||||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし). Left out when empty. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. |
|
||||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
|
||||
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). |
|
||||
|
||||
## The app's stroke files
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue