For parity with Japanese's KANJIDIC2 and kana:
- Unihan (Unicode License v3) makes characters' dictionaries: the 8,105
standard simplified characters (unihan-hans) and the traditional ones in
use (unihan-hant), told apart and ranked by CC-CEDICT's words (Unihan
calls 这 its own traditional variant; 後 ranks as the 后 of 以后), with
pinyin, meanings, strokes, Kangxi radical, HSK 2025 character levels,
the 3,500 common ones and Hong Kong's grades; and Korean Hanja
(unihan-ko) with their sounds, the 1,800 of education, and their native
readings from Wiktionary (물 수), with the 40 Hangul letters as kana are.
- The Greek alphabet, the curator's own (CC0): letters with names,
transliteration and sound.
- KRDict, the Basic Korean Dictionary for learners (CC BY-SA 2.0 KR,
downloaded without an account), replaces Wiktionary for Korean's words:
learners' levels, pronunciation where it differs from the spelling
(같이 [가치]), Hanja, English senses; Wiktionary's update keeps the
words' forms (가다's 99) and the Hanja's readings beside it for both.
- The update converts KRDict when its page offers another export and the
characters weekly, and removes the files of languages no longer kept
or of sources replaced.
Measured on the real data: Unihan 3 s; KRDict 54,870 entries in 24 s
and under 600 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
JMdict brings Tatoeba's sentences for Japanese; the other languages'
dictionaries had none (CC-CEDICT) or Wiktionary's own, often
quotations. Each file now takes Tatoeba's sentences of the week (its
per-language exports, CC BY 2.0 FR) as its entries' first examples,
which become a card's hints: each sentence with a translation into the
dictionary's meaning language goes to the entries its words find, cut
as the app cuts a line of subtitles (at spaces, Korean's particles off,
Chinese by its longest words), up to three for each, those using the
word as its headword writes it first (Haus rather than zu Hause; 学习
rather than 學習 in the simplified dictionary), then those nearest seven
words long. German's common words nearly all get some (8,849 of
9,292), in ten seconds and under 300 MB; a dictionary whose sentences
can't be had keeps its words. `curate tatoeba` does the same for a file
made by hand.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Chinese comes from CC-CEDICT (CC BY-SA 4.0), made for learners of
Chinese: `curate cedict` writes two dictionaries of its entries,
`cc-cedict-hans-en` (zh-Hans) and `cc-cedict-hant-en` (zh-Hant), so a
studyset finds its words as it writes them, each finding a word by
either script, by pinyin with or without tones and by its meanings.
Readings are pinyin with tone marks, a syllable to a character; measure
words are a remark beside the meanings.
The HSK levels are those of the 2025 examination syllabus, which HSK
3.0's examination follows since July 2026 (decided 2026-10-14: level
tags, credited), as hsk30 extracted them (MIT): `hsk:1` … `hsk:6`, and
`hsk:7` for 7 to 9, each given to the entry read as the syllabus reads
the word (的 de, not 的 dì; 书 shū, not the Book of Documents, Shū),
ranking the words, HSK 1 first.
`curate update --languages de,el,en,zh` takes over from `--wiktionary`:
Chinese from MDBG's file of the day as often as EDRDG's files, the rest
from Wiktionary's dump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The languages besides Japanese come from the English Wiktionary as
Wiktextract reads it (decided 2026-10-14: German, Greek and English
first, then Chinese, then Korean): kaikki.org's raw dump, every language
in one file, as its per-language files are being withdrawn.
`curate wiktionary --languages de,el,en` reads it once, streamed, and
writes German and Greek with Wiktionary's English meanings and English
with what its translation tables give in German, the app's other
language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources
`wiktionary-<language>`, as the server holds one source per meaning
language). Inflected forms, and Wiktionary's entries for forms alone
("Häusern: dative plural of Haus"), find their word instead of being
entries; names and affixes are left out, as are articles that tables
give cells of their own, abbreviations and rare spellings, which would
match common words. Entries keep their part of speech, genders, IPA,
principal forms and examples keyed by language as JMdict's.
wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank
the words by how often they are used and label the 15,000 most used
`common`. Typing without accents or umlauts finds a word (`strasse`,
`haeuser`, `spiti`).
The format gains `textKeys`: what finds an entry in running text, its
own forms, never its meanings, which would match words of the meaning
language in a subtitle. `curate update --wiktionary de,el,en` keeps the
languages current: it asks kaikki.org when the dump last changed and
converts it again only when it is newer, or the languages or the
conversion changed. The whole dump converts in about ten minutes in
under 300 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
EDRDG's licence asks of servers showing its dictionaries that they update
them at least monthly. `curate update --into <folder>` downloads EDRDG's
files of the day (JMdict with Tatoeba's sentences, KANJIDIC2), converts
them with the JLPT levels the curator now brings along (yomitan-jlpt-vocab's
lists, CC BY-SA 4.0, and the kanji levels of kanji-data's kanji.json: both
static, in data/ with their origin), and puts each file in place at once
(written beside it as .part, then renamed over it), so the server never
reads half a file; a failure leaves the files as they were. With --every 7
it keeps running and updates whenever the files are a week old, as their
dates say, so a restart doesn't download again; after a failure it tries
again in six hours. A real update takes some 50 seconds and 400 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
EDRDG makes JMdict_e_examp.gz anew every day: JMdict in English with
Tatoeba's sentences on the meanings, the file jmdict-simplified's examples
edition is made from once a week. The curator now reads it directly (a
file not named .json is read as XML), streaming 72 MB of XML in some 30
seconds and 300 MB, each entry put into the shape jmdict-simplified gives
it and made an entry by the same code: against jmdict-simplified's
conversion of four days before, the entries differ only by JMdict's edits
since. JMdict's entities stay codes (`&v1;` is `v1`), text cut between two
chunks is joined before its entities are read (the xml package's
normalizeEvents), and a sense without parts of speech has the previous
sense's, as JMdict's DTD says. Its version is the day it was made.
So an instance can keep its dictionaries current from the source alone,
as EDRDG's licence asks of servers (at least monthly).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
A dictionary file now says whether it holds words or characters (`kind`).
The kanjidic adapter turns KANJIDIC2 into one entry per kanji that has a
Japanese reading or an English meaning (12,633), with its readings,
meanings, school grade, strokes, frequency and JLPT level (Jonathan
Waller's, through kanji-data's kanji.json), labelled `jlpt:nX`,
`grade:N`/`grade:secondary`, followed by the 71 hiragana and katakana of
the kana table with their romaji. A kun reading is found whole; its kanji
part only as a loose key, so typing "ta" brings た before every kanji read
so.
`curate kanjivg` writes KanjiVG's strokes and components as the app's
stroke files, one gzipped JSON file per 256 code points, keyed by the code
point in each entry's id (KanjiVG's 橅 names itself 機).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The curator reads JMdict's examples edition (jmdict-simplified), which
ties Tatoeba's sentences to the meanings they show, and keeps them on
each sense of an entry (the form the word takes, the sentence, its
English, its Tatoeba id), crediting Tatoeba (CC BY 2.0 FR). A card made
from a word takes its first three as hints: the sentence with the word in
bold, the English below; Look up gives them to a card without hints of
its own. 98% of N5 and N4 words have one, 82% of N1. No new card format:
hints exist since card schema 5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
