Tizian 「ティツィアーン」
fb0ef7cdab
Beside Words and Kanji & kana, a Radicals tab lists the 214 Kangxi radicals by their strokes, with the forms Japanese writes them as inside kanji (亻, 氵, 艹, …). One chosen lists the kanji KANJIDIC2 files under it, in the dictionary's order, to add or to see how one is written; the tab shares the characters' bloc, so a kanji added there shows as held in a search too. The curator labels each kanji with its classical radical (`radical:N`), so the server's existing label queries serve the tab unchanged. Its files now say which conversion made them (`.conversion`): an instance running a curator that converts differently converts again at once, rather than at its weekly update, so the radicals appear right after a deploy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
134 lines
7.6 KiB
Markdown
134 lines
7.6 KiB
Markdown
# halkyon_learn_curator
|
||
|
||
Turns dictionary downloads into the dictionary files a Halkyon Learn server
|
||
loads. It reads each source with an adapter of its own and writes one format
|
||
for all of them, so a new source or language needs a new adapter, never a
|
||
change to the server or the app. It doesn't depend on Serverpod or the server.
|
||
It also makes the stroke files the app writes characters with (below).
|
||
|
||
## Making a dictionary file
|
||
|
||
JMdict in English, with the JLPT levels and example sentences on top:
|
||
|
||
1. Download `JMdict_e_examp.gz` from
|
||
[EDRDG](https://www.edrdg.org/pub/Nihongo/JMdict_e_examp.gz), which makes
|
||
it anew every day: JMdict with Tatoeba's sentences (CC BY 2.0 FR) on the
|
||
meanings they show. `JMdict_e.gz` works too, without them, and so does
|
||
jmdict-simplified's JSON (`jmdict-examples-eng-<version>.json`, from
|
||
[its releases](https://github.com/scriptin/jmdict-simplified/releases)).
|
||
2. Download `original_data/n1.csv` … `n5.csv` from
|
||
[yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab) into a
|
||
folder.
|
||
3. Convert, in this package:
|
||
|
||
```sh
|
||
dart run halkyon_learn_curator:curate jmdict \
|
||
--input JMdict_e_examp.gz --jlpt jlpt/ \
|
||
--output jmdict-en.jsonl.gz
|
||
```
|
||
|
||
The whole of JMdict (about 219,000 entries) takes some 30 seconds and
|
||
300 MB of memory: entries are read one at a time. A file whose name
|
||
holds `.json` is read as jmdict-simplified's JSON, any other as EDRDG's
|
||
XML.
|
||
|
||
KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:
|
||
|
||
1. Download `kanjidic2.xml.gz` from
|
||
[the EDRDG](https://www.edrdg.org/wiki/index.php/KANJIDIC_Project).
|
||
2. Download `kanji.json` from
|
||
[kanji-data](https://github.com/davidluzgouveia/kanji-data): Jonathan
|
||
Waller's JLPT levels by kanji (`jlpt_new`; nothing else is read).
|
||
3. Convert, in this package:
|
||
|
||
```sh
|
||
dart run halkyon_learn_curator:curate kanjidic \
|
||
--input kanjidic2.xml.gz --jlpt kanji.json \
|
||
--output kanjidic-en.jsonl.gz
|
||
```
|
||
|
||
About 12,800 characters, in some 5 seconds.
|
||
|
||
Put the files into the server's `dictionaries` folder. The server loads each on
|
||
its next start, and again whenever it changes (see the server's README).
|
||
|
||
## Keeping a server's dictionaries current
|
||
|
||
EDRDG's licence asks of a server showing its dictionaries that it update them
|
||
at least once a month. `curate update` does it from the sources alone:
|
||
|
||
```sh
|
||
dart run halkyon_learn_curator:curate update \
|
||
--into ../halkyon_learn_server/dictionaries [--every 7]
|
||
```
|
||
|
||
It downloads EDRDG's files of the day (`JMdict_e_examp.gz`, `kanjidic2.xml.gz`),
|
||
converts them with the JLPT levels in `data/` (static lists, see
|
||
`data/README.md`), and writes `jmdict-en.jsonl.gz` and `kanjidic-en.jsonl.gz`
|
||
into the folder, each beside its place first and then renamed over it, so a
|
||
server never reads half a file; should anything fail, the files stay as they
|
||
were. Some 50 seconds and 400 MB. With `--every 7` it keeps running and updates
|
||
whenever the files are a week old, as their dates say (a restart doesn't
|
||
download again), and tries again six hours after a failure. An instance runs it
|
||
so in a container of its own (the server's README). Beside the files,
|
||
`.conversion` says how the curator converted them (`conversion` in
|
||
`lib/src/update.dart`): raise it whenever the curator writes something else
|
||
from the same download (a label, a field), and an instance converts again as
|
||
soon as the new curator runs, not a week later.
|
||
|
||
## The format (`halkyon-dictionary`, version 1)
|
||
|
||
[JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`.
|
||
|
||
The **first line** says what the file holds:
|
||
|
||
| Field | |
|
||
|---|---|
|
||
| `format`, `formatVersion` | `"halkyon-dictionary"`, `1` |
|
||
| `kind` | `words` (a dictionary of words, JMdict) or `characters` (of the characters words are written in, KANJIDIC2). The app shows them apart. |
|
||
| `source`, `sourceVersion` | The source's name (`jmdict`) and version (`3.6.2 (2026-10-05)`). An instance holds one version of a source for each meaning language. |
|
||
| `language`, `glossLanguage` | BCP 47 tags: the words' language (`ja`) and the meanings' (`en`). |
|
||
| `license` | The data's licence, an SPDX identifier (`CC-BY-SA-4.0`). |
|
||
| `credits` | Everything it is drawn from: `name`, `by`, `license`, `url`. The app lists them where it credits its data sources. |
|
||
|
||
**Every further line** is an entry:
|
||
|
||
| Field | |
|
||
|---|---|
|
||
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
|
||
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
|
||
| `labels` | What it can be picked by: `common`, `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
|
||
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
|
||
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし). Left out when empty. |
|
||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. |
|
||
|
||
## The app's stroke files
|
||
|
||
How every kanji and kana is written comes from
|
||
[KanjiVG](https://kanjivg.tagaini.net) and is bundled with the app, so writing
|
||
works without a server. To update it, download `kanjivg-<date>.xml.gz` from
|
||
[KanjiVG's releases](https://github.com/KanjiVG/kanjivg/releases) and, in this
|
||
package:
|
||
|
||
```sh
|
||
dart run halkyon_learn_curator:curate kanjivg \
|
||
--input kanjivg-20260714.xml.gz \
|
||
--output ../halkyon_learn_flutter/assets/kanjivg
|
||
```
|
||
|
||
It writes one gzipped JSON file for every 256 code points that have any,
|
||
named by the code point's upper bits in hexadecimal (食, U+98DF, is in
|
||
`098.json.gz`; 94 files, 2.7 MB), so the app reads only the part it needs.
|
||
Each maps a character to its strokes (`s`, SVG paths in KanjiVG's 109 × 109
|
||
box, in the order they're written) and the components KanjiVG first splits it
|
||
into (`c`, when it has any). Then update the release named in the folder's
|
||
`ATTRIBUTION.md`.
|
||
|
||
## Adding a source
|
||
|
||
Write an adapter in `lib/src/sources/` that turns the source into the lines
|
||
above, with its credits and a sample of real data in `test/fixtures/` (and
|
||
where that comes from in `test/fixtures/README.md`); give it a command in
|
||
`bin/curate.dart`. Check the source's licence first, and that the instance may
|
||
pass its data on to the people using it.
|