feat(curator): turn JMdict and the JLPT lists into dictionary files

A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.

JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.

The server's image leaves the package out, as it does the client's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-09 00:25:16 +02:00
commit a875a91f40
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
5 changed files with 1664 additions and 7 deletions

View file

@ -0,0 +1,2 @@
jmdict_seq,kana,kanji,waller_definition
1601040,はみがき,歯磨,"dentifrice, toothpaste"
1 jmdict_seq kana kanji waller_definition
2 1601040 はみがき 歯磨 dentifrice, toothpaste

View file

@ -0,0 +1,2 @@
jmdict_seq,kana,kanji,waller_definition
1601040,はみがき,歯磨き,"dentifrice,toothpaste"
1 jmdict_seq kana kanji waller_definition
2 1601040 はみがき 歯磨き dentifrice,toothpaste

View file

@ -0,0 +1 @@
jmdict_seq,kana,kanji,waller_definition
1 jmdict_seq kana kanji waller_definition

View file

@ -0,0 +1,2 @@
jmdict_seq,kana,kanji,waller_definition
,ない,,a row without an id
1 jmdict_seq kana kanji waller_definition
2 ない a row without an id

View file

@ -0,0 +1,3 @@
jmdict_seq,kana,kanji,waller_definition
1358280,たべる,食べる,to eat
1080510,テレビ,,television
1 jmdict_seq kana kanji waller_definition
2 1358280 たべる 食べる to eat
3 1080510 テレビ television