feat(curator): curate update keeps a server's dictionaries current
EDRDG's licence asks of servers showing its dictionaries that they update
them at least monthly. `curate update --into <folder>` downloads EDRDG's
files of the day (JMdict with Tatoeba's sentences, KANJIDIC2), converts
them with the JLPT levels the curator now brings along (yomitan-jlpt-vocab's
lists, CC BY-SA 4.0, and the kanji levels of kanji-data's kanji.json: both
static, in data/ with their origin), and puts each file in place at once
(written beside it as .part, then renamed over it), so the server never
reads half a file; a failure leaves the files as they were. With --every 7
it keeps running and updates whenever the files are a week old, as their
dates say, so a restart doesn't download again; after a failure it tries
again in six hours. A real update takes some 50 seconds and 400 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:40:19 +02:00
|
|
|
# Data the curator brings along
|
|
|
|
|
|
|
|
|
|
What the dictionaries draw on besides their sources, and never changes, so
|
feat(curator): dictionaries of German, Greek and English from Wiktionary
The languages besides Japanese come from the English Wiktionary as
Wiktextract reads it (decided 2026-10-14: German, Greek and English
first, then Chinese, then Korean): kaikki.org's raw dump, every language
in one file, as its per-language files are being withdrawn.
`curate wiktionary --languages de,el,en` reads it once, streamed, and
writes German and Greek with Wiktionary's English meanings and English
with what its translation tables give in German, the app's other
language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources
`wiktionary-<language>`, as the server holds one source per meaning
language). Inflected forms, and Wiktionary's entries for forms alone
("Häusern: dative plural of Haus"), find their word instead of being
entries; names and affixes are left out, as are articles that tables
give cells of their own, abbreviations and rare spellings, which would
match common words. Entries keep their part of speech, genders, IPA,
principal forms and examples keyed by language as JMdict's.
wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank
the words by how often they are used and label the 15,000 most used
`common`. Typing without accents or umlauts finds a word (`strasse`,
`haeuser`, `spiti`).
The format gains `textKeys`: what finds an entry in running text, its
own forms, never its meanings, which would match words of the meaning
language in a subtitle. `curate update --wiktionary de,el,en` keeps the
languages current: it asks kaikki.org when the dump last changed and
converts it again only when it is newer, or the languages or the
conversion changed. The whole dump converts in about ten minutes in
under 300 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 07:19:16 +02:00
|
|
|
`curate update` needs only EDRDG's files of the day and Wiktionary's dump.
|
feat(curator): curate update keeps a server's dictionaries current
EDRDG's licence asks of servers showing its dictionaries that they update
them at least monthly. `curate update --into <folder>` downloads EDRDG's
files of the day (JMdict with Tatoeba's sentences, KANJIDIC2), converts
them with the JLPT levels the curator now brings along (yomitan-jlpt-vocab's
lists, CC BY-SA 4.0, and the kanji levels of kanji-data's kanji.json: both
static, in data/ with their origin), and puts each file in place at once
(written beside it as .part, then renamed over it), so the server never
reads half a file; a failure leaves the files as they were. With --every 7
it keeps running and updates whenever the files are a week old, as their
dates say, so a restart doesn't download again; after a failure it tries
again in six hours. A real update takes some 50 seconds and 400 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 16:40:19 +02:00
|
|
|
|
|
|
|
|
`jlpt/n1.csv` … `n5.csv` are the JLPT vocabulary lists of
|
|
|
|
|
[yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab)
|
|
|
|
|
(`original_data`, as of 2026-10-09), CC BY-SA 4.0 (`jlpt/LICENSE.txt`): Jonathan
|
|
|
|
|
Waller's lists ([tanos.co.uk](http://www.tanos.co.uk/jlpt/), CC BY) with each
|
|
|
|
|
word's JMdict entry and its most common spelling, by stephenmk.
|
|
|
|
|
|
|
|
|
|
`kanji-jlpt.json` maps each kanji on Jonathan Waller's JLPT kanji lists
|
|
|
|
|
([tanos.co.uk](http://www.tanos.co.uk/jlpt/), CC BY) to its level, 5 for N5 to 1
|
|
|
|
|
for N1: the `jlpt_new` of
|
|
|
|
|
[kanji-data](https://github.com/davidluzgouveia/kanji-data)'s `kanji.json`
|
|
|
|
|
(MIT, as of 2026-10-09), nothing else of it.
|
|
|
|
|
|
|
|
|
|
The lists are over ten years old and won't change; replace them only for a list
|
|
|
|
|
of the same shape.
|
feat(curator): dictionaries of German, Greek and English from Wiktionary
The languages besides Japanese come from the English Wiktionary as
Wiktextract reads it (decided 2026-10-14: German, Greek and English
first, then Chinese, then Korean): kaikki.org's raw dump, every language
in one file, as its per-language files are being withdrawn.
`curate wiktionary --languages de,el,en` reads it once, streamed, and
writes German and Greek with Wiktionary's English meanings and English
with what its translation tables give in German, the app's other
language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources
`wiktionary-<language>`, as the server holds one source per meaning
language). Inflected forms, and Wiktionary's entries for forms alone
("Häusern: dative plural of Haus"), find their word instead of being
entries; names and affixes are left out, as are articles that tables
give cells of their own, abbreviations and rare spellings, which would
match common words. Entries keep their part of speech, genders, IPA,
principal forms and examples keyed by language as JMdict's.
wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank
the words by how often they are used and label the 15,000 most used
`common`. Typing without accents or umlauts finds a word (`strasse`,
`haeuser`, `spiti`).
The format gains `textKeys`: what finds an entry in running text, its
own forms, never its meanings, which would match words of the meaning
language in a subtitle. `curate update --wiktionary de,el,en` keeps the
languages current: it asks kaikki.org when the dump last changed and
converts it again only when it is newer, or the languages or the
conversion changed. The whole dump converts in about ten minutes in
under 300 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 07:19:16 +02:00
|
|
|
|
|
|
|
|
`wordfreq/small_<language>.msgpack.gz` are wordfreq's lists of how often words
|
|
|
|
|
are used ([wordfreq](https://github.com/rspeer/wordfreq) v3.0, as of 2021-03-29,
|
|
|
|
|
the last change to its data), unchanged: CC BY-SA 4.0, by Robyn Speer, drawn from
|
|
|
|
|
SUBTLEX (Marc Brysbaert et al., whose authors wordfreq's licence asks to be
|
|
|
|
|
credited, and who made SUBTLEX freely available), OpenSubtitles, Wikipedia,
|
|
|
|
|
Google Books Ngrams and others, as wordfreq's README lists them. The Wiktionary
|
|
|
|
|
dictionaries take their ranks and their `common` label from them and credit
|
|
|
|
|
both. wordfreq's data won't change: it was sunset in 2024.
|