For parity with Japanese's KANJIDIC2 and kana:
- Unihan (Unicode License v3) makes characters' dictionaries: the 8,105
standard simplified characters (unihan-hans) and the traditional ones in
use (unihan-hant), told apart and ranked by CC-CEDICT's words (Unihan
calls 这 its own traditional variant; 後 ranks as the 后 of 以后), with
pinyin, meanings, strokes, Kangxi radical, HSK 2025 character levels,
the 3,500 common ones and Hong Kong's grades; and Korean Hanja
(unihan-ko) with their sounds, the 1,800 of education, and their native
readings from Wiktionary (물 수), with the 40 Hangul letters as kana are.
- The Greek alphabet, the curator's own (CC0): letters with names,
transliteration and sound.
- KRDict, the Basic Korean Dictionary for learners (CC BY-SA 2.0 KR,
downloaded without an account), replaces Wiktionary for Korean's words:
learners' levels, pronunciation where it differs from the spelling
(같이 [가치]), Hanja, English senses; Wiktionary's update keeps the
words' forms (가다's 99) and the Hanja's readings beside it for both.
- The update converts KRDict when its page offers another export and the
characters weekly, and removes the files of languages no longer kept
or of sources replaced.
Measured on the real data: Unihan 3 s; KRDict 54,870 entries in 24 s
and under 600 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Wiktionary tags a contraction as an abbreviation and an alternative of
its first word (im: in + dem), so German's im, zum and ins, which every
subtitle has, were neither entries nor found as forms. A contraction is
now an entry of its own, as text writes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
JMdict brings Tatoeba's sentences for Japanese; the other languages'
dictionaries had none (CC-CEDICT) or Wiktionary's own, often
quotations. Each file now takes Tatoeba's sentences of the week (its
per-language exports, CC BY 2.0 FR) as its entries' first examples,
which become a card's hints: each sentence with a translation into the
dictionary's meaning language goes to the entries its words find, cut
as the app cuts a line of subtitles (at spaces, Korean's particles off,
Chinese by its longest words), up to three for each, those using the
word as its headword writes it first (Haus rather than zu Hause; 学习
rather than 學習 in the simplified dictionary), then those nearest seven
words long. German's common words nearly all get some (8,849 of
9,292), in ten seconds and under 300 MB; a dictionary whose sentences
can't be had keeps its words. `curate tatoeba` does the same for a file
made by hand.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Korean comes from the same dump as German and Greek, with English
meanings (decided 2026-10-14: Korean last, from Wiktionary for now): a
verb's conjugations are among its forms, so text finds them; a word's
Hanja and romanisation find it in a search and show beside it. The
counter words Wiktionary's tables name and its notes on what it couldn't
read are no forms of a word.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The syllabus's reading is matched with its tones first, so 和 hé gets
HSK 1 and 和 hè doesn't; only where no entry of a word reads it so (the
syllabus marks the tone changes of 一 and 不, 一下 yíxià, CC-CEDICT
doesn't) are the letters alone compared. CC-CEDICT's 2,885 entries that
are only "variant of …" (咊, an old variant of 和) are no entries of
their own any more, which repeated their word in the simplified file,
but written forms of that word, finding it in text and searches.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
Chinese comes from CC-CEDICT (CC BY-SA 4.0), made for learners of
Chinese: `curate cedict` writes two dictionaries of its entries,
`cc-cedict-hans-en` (zh-Hans) and `cc-cedict-hant-en` (zh-Hant), so a
studyset finds its words as it writes them, each finding a word by
either script, by pinyin with or without tones and by its meanings.
Readings are pinyin with tone marks, a syllable to a character; measure
words are a remark beside the meanings.
The HSK levels are those of the 2025 examination syllabus, which HSK
3.0's examination follows since July 2026 (decided 2026-10-14: level
tags, credited), as hsk30 extracted them (MIT): `hsk:1` … `hsk:6`, and
`hsk:7` for 7 to 9, each given to the entry read as the syllabus reads
the word (的 de, not 的 dì; 书 shū, not the Book of Documents, Shū),
ranking the words, HSK 1 first.
`curate update --languages de,el,en,zh` takes over from `--wiktionary`:
Chinese from MDBG's file of the day as often as EDRDG's files, the rest
from Wiktionary's dump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
The languages besides Japanese come from the English Wiktionary as
Wiktextract reads it (decided 2026-10-14: German, Greek and English
first, then Chinese, then Korean): kaikki.org's raw dump, every language
in one file, as its per-language files are being withdrawn.
`curate wiktionary --languages de,el,en` reads it once, streamed, and
writes German and Greek with Wiktionary's English meanings and English
with what its translation tables give in German, the app's other
language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources
`wiktionary-<language>`, as the server holds one source per meaning
language). Inflected forms, and Wiktionary's entries for forms alone
("Häusern: dative plural of Haus"), find their word instead of being
entries; names and affixes are left out, as are articles that tables
give cells of their own, abbreviations and rare spellings, which would
match common words. Entries keep their part of speech, genders, IPA,
principal forms and examples keyed by language as JMdict's.
wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank
the words by how often they are used and label the 15,000 most used
`common`. Typing without accents or umlauts finds a word (`strasse`,
`haeuser`, `spiti`).
The format gains `textKeys`: what finds an entry in running text, its
own forms, never its meanings, which would match words of the meaning
language in a subtitle. `curate update --wiktionary de,el,en` keeps the
languages current: it asks kaikki.org when the dump last changed and
converts it again only when it is newer, or the languages or the
conversion changed. The whole dump converts in about ten minutes in
under 300 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
EDRDG makes JMdict_e_examp.gz anew every day: JMdict in English with
Tatoeba's sentences on the meanings, the file jmdict-simplified's examples
edition is made from once a week. The curator now reads it directly (a
file not named .json is read as XML), streaming 72 MB of XML in some 30
seconds and 300 MB, each entry put into the shape jmdict-simplified gives
it and made an entry by the same code: against jmdict-simplified's
conversion of four days before, the entries differ only by JMdict's edits
since. JMdict's entities stay codes (`&v1;` is `v1`), text cut between two
chunks is joined before its entities are read (the xml package's
normalizeEvents), and a sense without parts of speech has the previous
sense's, as JMdict's DTD says. Its version is the day it was made.
So an instance can keep its dictionaries current from the source alone,
as EDRDG's licence asks of servers (at least monthly).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
A dictionary file now says whether it holds words or characters (`kind`).
The kanjidic adapter turns KANJIDIC2 into one entry per kanji that has a
Japanese reading or an English meaning (12,633), with its readings,
meanings, school grade, strokes, frequency and JLPT level (Jonathan
Waller's, through kanji-data's kanji.json), labelled `jlpt:nX`,
`grade:N`/`grade:secondary`, followed by the 71 hiragana and katakana of
the kana table with their romaji. A kun reading is found whole; its kanji
part only as a loose key, so typing "ta" brings た before every kanji read
so.
`curate kanjivg` writes KanjiVG's strokes and components as the app's
stroke files, one gzipped JSON file per 256 code points, keyed by the code
point in each entry's id (KanjiVG's 橅 names itself 機).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
