feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
|
|
|
# Test data
|
|
|
|
|
|
|
|
|
|
`jmdict-eng-sample.json` holds six entries of JMdict 3.6.2 (2026-10-05) as
|
|
|
|
|
jmdict-simplified writes it (<https://github.com/scriptin/jmdict-simplified>):
|
|
|
|
|
JMdict is © the Electronic Dictionary Research and Development Group, used under
|
|
|
|
|
its licence, CC BY-SA 4.0 (<https://www.edrdg.org/edrdg/licence.html>).
|
|
|
|
|
|
|
|
|
|
`jlpt/` holds a few rows of the JLPT lists of yomitan-jlpt-vocab
|
|
|
|
|
(<https://github.com/stephenmk/yomitan-jlpt-vocab>), CC BY-SA 4.0, from Jonathan
|
|
|
|
|
Waller's lists (<http://www.tanos.co.uk/jlpt/>), CC BY; one row without an id is
|
|
|
|
|
made up.
|
feat(curator): KANJIDIC2 and the kana as a dictionary of characters; KanjiVG's strokes for the app
A dictionary file now says whether it holds words or characters (`kind`).
The kanjidic adapter turns KANJIDIC2 into one entry per kanji that has a
Japanese reading or an English meaning (12,633), with its readings,
meanings, school grade, strokes, frequency and JLPT level (Jonathan
Waller's, through kanji-data's kanji.json), labelled `jlpt:nX`,
`grade:N`/`grade:secondary`, followed by the 71 hiragana and katakana of
the kana table with their romaji. A kun reading is found whole; its kanji
part only as a loose key, so typing "ta" brings た before every kanji read
so.
`curate kanjivg` writes KanjiVG's strokes and components as the app's
stroke files, one gzipped JSON file per 256 code points, keyed by the code
point in each entry's id (KanjiVG's 橅 names itself 機).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 14:37:22 +02:00
|
|
|
|
|
|
|
|
`kanjidic2-sample.xml` holds five characters of KANJIDIC2 (database version
|
|
|
|
|
2026-282) with its header and a shortened DTD: KANJIDIC2 is © the Electronic
|
|
|
|
|
Dictionary Research and Development Group, used under its licence, CC BY-SA 4.0
|
|
|
|
|
(<https://www.edrdg.org/edrdg/licence.html>).
|
|
|
|
|
|
|
|
|
|
`kanji-data-sample.json` holds four entries of kanji-data's `kanji.json`
|
|
|
|
|
(<https://github.com/davidluzgouveia/kanji-data>), cut to their numbers, from
|
|
|
|
|
KANJIDIC2 and from Jonathan Waller's JLPT levels (<http://www.tanos.co.uk/jlpt/>),
|
|
|
|
|
CC BY.
|
|
|
|
|
|
|
|
|
|
`kanjivg-sample.xml` holds four characters of KanjiVG (the release of
|
|
|
|
|
2026-07-14) with its notice, among them 橅, whose entry names itself 機:
|
|
|
|
|
KanjiVG is © Ulrich Apel, CC BY-SA 3.0 (<https://kanjivg.tagaini.net>).
|
2026-10-09 16:33:48 +02:00
|
|
|
|
|
|
|
|
`jmdict-e-examp-sample.xml` holds five entries of EDRDG's `JMdict_e_examp.gz`
|
|
|
|
|
of 2026-10-09 (JMdict with Tatoeba's sentences, CC BY 2.0 FR) and the
|
|
|
|
|
declarations of the entities they use: JMdict is © the Electronic Dictionary
|
|
|
|
|
Research and Development Group, used under its licence, CC BY-SA 4.0
|
|
|
|
|
(<https://www.edrdg.org/edrdg/licence.html>).
|
feat(curator): dictionaries of German, Greek and English from Wiktionary
The languages besides Japanese come from the English Wiktionary as
Wiktextract reads it (decided 2026-10-14: German, Greek and English
first, then Chinese, then Korean): kaikki.org's raw dump, every language
in one file, as its per-language files are being withdrawn.
`curate wiktionary --languages de,el,en` reads it once, streamed, and
writes German and Greek with Wiktionary's English meanings and English
with what its translation tables give in German, the app's other
language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources
`wiktionary-<language>`, as the server holds one source per meaning
language). Inflected forms, and Wiktionary's entries for forms alone
("Häusern: dative plural of Haus"), find their word instead of being
entries; names and affixes are left out, as are articles that tables
give cells of their own, abbreviations and rare spellings, which would
match common words. Entries keep their part of speech, genders, IPA,
principal forms and examples keyed by language as JMdict's.
wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank
the words by how often they are used and label the 15,000 most used
`common`. Typing without accents or umlauts finds a word (`strasse`,
`haeuser`, `spiti`).
The format gains `textKeys`: what finds an entry in running text, its
own forms, never its meanings, which would match words of the meaning
language in a subtitle. `curate update --wiktionary de,el,en` keeps the
languages current: it asks kaikki.org when the dump last changed and
converts it again only when it is newer, or the languages or the
conversion changed. The whole dump converts in about ten minutes in
under 300 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-14 07:19:16 +02:00
|
|
|
|
|
|
|
|
`wiktionary-sample.jsonl` holds 25 lines of kaikki.org's Wiktextract data of the
|
|
|
|
|
English Wiktionary (the dump of 2026-10-03, read per word from kaikki.org): the
|
|
|
|
|
German Haus, Häuser, Häusern, essen, aß, gegessen and Straße, the Greek σπίτι,
|
|
|
|
|
σπιτιού and γράφω, the English house, houses, go and went, and the French
|
|
|
|
|
maison, each shortened to the fields the curator reads (no categories,
|
|
|
|
|
etymologies or related words; at most six senses, two examples, 40 forms, two
|
|
|
|
|
sounds, and 14 translations, those into German and Greek). Wiktionary's text is
|
|
|
|
|
by its contributors, CC BY-SA 4.0 (<https://en.wiktionary.org/wiki/Wiktionary:Copyrights>);
|
|
|
|
|
the extraction is Tatu Ylonen's Wiktextract (<https://kaikki.org>).
|