Tizian 「ティツィアーン」
3013bc04b3
The server now holds dictionaries for everyone: `dictionary_entry` and `dictionary_source`, reference data owned by no one, never synced. It loads the dictionary files in its `dictionaries` folder at every start, those whose SHA-256 changed: in the background, each source replaced in one transaction, so searches meanwhile find the old entries; a file it can't read is logged and left out. All of JMdict loads in under half a minute. On an instance the folder is deploy/dictionaries, mounted read-only (compose.yaml); the server's README says how. DictionaryEndpoint (signed in): the dictionaries with their credits; a search (exact key, exact loose key, key starting with the query, loose key starting with it, each in the source's order; katakana as hiragana, macrons written out), some 150-450 ms over all of JMdict; and the entries with a label, counted and a page at a time (all JLPT N5 words). The curator now keeps spellings that aren't quite right apart, as `looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司 before 数詞 (すうし). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
63 lines
3.4 KiB
Markdown
63 lines
3.4 KiB
Markdown
# halkyon_learn_curator
|
|
|
|
Turns dictionary downloads into the dictionary files a Halkyon Learn server
|
|
loads. It reads each source with an adapter of its own and writes one format
|
|
for all of them, so a new source or language needs a new adapter, never a
|
|
change to the server or the app. It doesn't depend on Serverpod or the server.
|
|
|
|
## Making a dictionary file
|
|
|
|
JMdict in English, with the JLPT levels on top:
|
|
|
|
1. Download `jmdict-eng-<version>.json.tgz` from
|
|
[jmdict-simplified's releases](https://github.com/scriptin/jmdict-simplified/releases)
|
|
and unpack it.
|
|
2. Download `original_data/n1.csv` … `n5.csv` from
|
|
[yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab) into a
|
|
folder.
|
|
3. Convert, in this package:
|
|
|
|
```sh
|
|
dart run halkyon_learn_curator:curate jmdict \
|
|
--input jmdict-eng-3.6.2.json --jlpt jlpt/ --output jmdict-en.jsonl.gz
|
|
```
|
|
|
|
The whole of JMdict (about 219,000 entries) takes some 20 seconds and
|
|
300 MB of memory: entries are read one at a time.
|
|
|
|
Put the file into the server's `dictionaries` folder. The server loads it on
|
|
its next start, and again whenever the file changes (see the server's README).
|
|
|
|
## The format (`halkyon-dictionary`, version 1)
|
|
|
|
[JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`.
|
|
|
|
The **first line** says what the file holds:
|
|
|
|
| Field | |
|
|
|---|---|
|
|
| `format`, `formatVersion` | `"halkyon-dictionary"`, `1` |
|
|
| `source`, `sourceVersion` | The source's name (`jmdict`) and version (`3.6.2 (2026-10-05)`). An instance holds one version of a source for each meaning language. |
|
|
| `language`, `glossLanguage` | BCP 47 tags: the words' language (`ja`) and the meanings' (`en`). |
|
|
| `license` | The data's licence, an SPDX identifier (`CC-BY-SA-4.0`). |
|
|
| `credits` | Everything it is drawn from: `name`, `by`, `license`, `url`. The app lists them where it credits its data sources. |
|
|
|
|
**Every further line** is an entry:
|
|
|
|
| Field | |
|
|
|---|---|
|
|
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
|
|
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
|
|
| `labels` | What it can be picked by: `common`, `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`). |
|
|
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest. |
|
|
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
|
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし). Left out when empty. |
|
|
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to). Lists are left out when empty. |
|
|
|
|
## Adding a source
|
|
|
|
Write an adapter in `lib/src/sources/` that turns the source into the lines
|
|
above, with its credits and a sample of real data in `test/fixtures/` (and
|
|
where that comes from in `test/fixtures/README.md`); give it a command in
|
|
`bin/curate.dart`. Check the source's licence first, and that the instance may
|
|
pass its data on to the people using it.
|