feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
|
|
|
# Test data
|
|
|
|
|
|
|
|
|
|
`jmdict-eng-sample.json` holds six entries of JMdict 3.6.2 (2026-10-05) as
|
|
|
|
|
jmdict-simplified writes it (<https://github.com/scriptin/jmdict-simplified>):
|
|
|
|
|
JMdict is © the Electronic Dictionary Research and Development Group, used under
|
|
|
|
|
its licence, CC BY-SA 4.0 (<https://www.edrdg.org/edrdg/licence.html>).
|
|
|
|
|
|
|
|
|
|
`jlpt/` holds a few rows of the JLPT lists of yomitan-jlpt-vocab
|
|
|
|
|
(<https://github.com/stephenmk/yomitan-jlpt-vocab>), CC BY-SA 4.0, from Jonathan
|
|
|
|
|
Waller's lists (<http://www.tanos.co.uk/jlpt/>), CC BY; one row without an id is
|
|
|
|
|
made up.
|
feat(curator): KANJIDIC2 and the kana as a dictionary of characters; KanjiVG's strokes for the app
A dictionary file now says whether it holds words or characters (`kind`).
The kanjidic adapter turns KANJIDIC2 into one entry per kanji that has a
Japanese reading or an English meaning (12,633), with its readings,
meanings, school grade, strokes, frequency and JLPT level (Jonathan
Waller's, through kanji-data's kanji.json), labelled `jlpt:nX`,
`grade:N`/`grade:secondary`, followed by the 71 hiragana and katakana of
the kana table with their romaji. A kun reading is found whole; its kanji
part only as a loose key, so typing "ta" brings た before every kanji read
so.
`curate kanjivg` writes KanjiVG's strokes and components as the app's
stroke files, one gzipped JSON file per 256 code points, keyed by the code
point in each entry's id (KanjiVG's 橅 names itself 機).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 14:37:22 +02:00
|
|
|
|
|
|
|
|
`kanjidic2-sample.xml` holds five characters of KANJIDIC2 (database version
|
|
|
|
|
2026-282) with its header and a shortened DTD: KANJIDIC2 is © the Electronic
|
|
|
|
|
Dictionary Research and Development Group, used under its licence, CC BY-SA 4.0
|
|
|
|
|
(<https://www.edrdg.org/edrdg/licence.html>).
|
|
|
|
|
|
|
|
|
|
`kanji-data-sample.json` holds four entries of kanji-data's `kanji.json`
|
|
|
|
|
(<https://github.com/davidluzgouveia/kanji-data>), cut to their numbers, from
|
|
|
|
|
KANJIDIC2 and from Jonathan Waller's JLPT levels (<http://www.tanos.co.uk/jlpt/>),
|
|
|
|
|
CC BY.
|
|
|
|
|
|
|
|
|
|
`kanjivg-sample.xml` holds four characters of KanjiVG (the release of
|
|
|
|
|
2026-07-14) with its notice, among them 橅, whose entry names itself 機:
|
|
|
|
|
KanjiVG is © Ulrich Apel, CC BY-SA 3.0 (<https://kanjivg.tagaini.net>).
|