Beside Words and Kanji & kana, a Radicals tab lists the 214 Kangxi
radicals by their strokes, with the forms Japanese writes them as inside
kanji (亻, 氵, 艹, …). One chosen lists the kanji KANJIDIC2 files under
it, in the dictionary's order, to add or to see how one is written; the
tab shares the characters' bloc, so a kanji added there shows as held in
a search too.
The curator labels each kanji with its classical radical (`radical:N`),
so the server's existing label queries serve the tab unchanged. Its
files now say which conversion made them (`.conversion`): an instance
running a curator that converts differently converts again at once,
rather than at its weekly update, so the radicals appear right after a
deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
EDRDG's licence asks of servers showing its dictionaries that they update
them at least monthly. `curate update --into <folder>` downloads EDRDG's
files of the day (JMdict with Tatoeba's sentences, KANJIDIC2), converts
them with the JLPT levels the curator now brings along (yomitan-jlpt-vocab's
lists, CC BY-SA 4.0, and the kanji levels of kanji-data's kanji.json: both
static, in data/ with their origin), and puts each file in place at once
(written beside it as .part, then renamed over it), so the server never
reads half a file; a failure leaves the files as they were. With --every 7
it keeps running and updates whenever the files are a week old, as their
dates say, so a restart doesn't download again; after a failure it tries
again in six hours. A real update takes some 50 seconds and 400 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
EDRDG makes JMdict_e_examp.gz anew every day: JMdict in English with
Tatoeba's sentences on the meanings, the file jmdict-simplified's examples
edition is made from once a week. The curator now reads it directly (a
file not named .json is read as XML), streaming 72 MB of XML in some 30
seconds and 300 MB, each entry put into the shape jmdict-simplified gives
it and made an entry by the same code: against jmdict-simplified's
conversion of four days before, the entries differ only by JMdict's edits
since. JMdict's entities stay codes (`&v1;` is `v1`), text cut between two
chunks is joined before its entities are read (the xml package's
normalizeEvents), and a sense without parts of speech has the previous
sense's, as JMdict's DTD says. Its version is the day it was made.
So an instance can keep its dictionaries current from the source alone,
as EDRDG's licence asks of servers (at least monthly).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
A dictionary file now says whether it holds words or characters (`kind`).
The kanjidic adapter turns KANJIDIC2 into one entry per kanji that has a
Japanese reading or an English meaning (12,633), with its readings,
meanings, school grade, strokes, frequency and JLPT level (Jonathan
Waller's, through kanji-data's kanji.json), labelled `jlpt:nX`,
`grade:N`/`grade:secondary`, followed by the 71 hiragana and katakana of
the kana table with their romaji. A kun reading is found whole; its kanji
part only as a loose key, so typing "ta" brings た before every kanji read
so.
`curate kanjivg` writes KanjiVG's strokes and components as the app's
stroke files, one gzipped JSON file per 256 code points, keyed by the code
point in each entry's id (KanjiVG's 橅 names itself 機).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The curator reads JMdict's examples edition (jmdict-simplified), which
ties Tatoeba's sentences to the meanings they show, and keeps them on
each sense of an entry (the form the word takes, the sentence, its
English, its Tatoeba id), crediting Tatoeba (CC BY 2.0 FR). A card made
from a word takes its first three as hints: the sentence with the word in
bold, the English below; Look up gives them to a card without hints of
its own. 98% of N5 and N4 words have one, 82% of N1. No new card format:
hints exist since card schema 5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The curator:
- A word JMdict says is usually written in kana gets a kana headword,
even where the JLPT lists give kanji (その, not 其の), and a rare or
search-only spelling is never the headword while a usual reading
exists (そして, not 而して); 9,611 headwords change, 96 of them N5.
- Keys hold full-width letters as typed (Ｔシャツ as tシャツ), デュ/テュ/
フュ/イェ romanised, long a, i and e typed once as loose keys (kado for
カード), and a blank JLPT line is skipped.
The server:
- A search is one query ordered by tier, then rank: a loose or prefix
match no longer pushes an exact one out of the list (75-150 ms over
all of JMdict, before 150-450).
- Queries find long vowels with macrons spelt おお or ー and typed once
(ōkii, kōhī), ん typed n' or nn, and full-width letters.
- Pages of a label are ordered by rank and id, so entries that share a
rank are neither repeated nor skipped.
- The loader never throws (an unreadable folder ended the server from
the background), refuses two files of one source rather than loading
both at every start, reads no rows back when replacing a source, and
takes its time from clock.
- Dictionary files stay out of the Docker build context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
