Tizian 「ティツィアーン」
a875a91f40
A package of its own in the workspace, without Serverpod (AGPL, like the tooling): an adapter per source writes one format for all, documented in its README, which the server will load. The format is JSON Lines: a header (source, version, languages, licence, credits), then entries with a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a search matches, and the entry as cards take it. Everything specific to a language happens in its adapter (Japanese: forms, readings in kana and Hepburn romaji, long vowels typed once, meanings without "to" and articles), so the server's search knows no language. JMdict comes as jmdict-simplified's JSON, read one entry at a time: all 218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add levels by JMdict entry, the easiest when a word is on several, and give the headword its common spelling. Words rank N5 to N1, then the other common words, then the rest. The server's image leaves the package out, as it does the client's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
62 lines
3.3 KiB
Markdown
62 lines
3.3 KiB
Markdown
# halkyon_learn_curator
|
|
|
|
Turns dictionary downloads into the dictionary files a Halkyon Learn server
|
|
loads. It reads each source with an adapter of its own and writes one format
|
|
for all of them, so a new source or language needs a new adapter, never a
|
|
change to the server or the app. It doesn't depend on Serverpod or the server.
|
|
|
|
## Making a dictionary file
|
|
|
|
JMdict in English, with the JLPT levels on top:
|
|
|
|
1. Download `jmdict-eng-<version>.json.tgz` from
|
|
[jmdict-simplified's releases](https://github.com/scriptin/jmdict-simplified/releases)
|
|
and unpack it.
|
|
2. Download `original_data/n1.csv` … `n5.csv` from
|
|
[yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab) into a
|
|
folder.
|
|
3. Convert, in this package:
|
|
|
|
```sh
|
|
dart run halkyon_learn_curator:curate jmdict \
|
|
--input jmdict-eng-3.6.2.json --jlpt jlpt/ --output jmdict-en.jsonl.gz
|
|
```
|
|
|
|
The whole of JMdict (about 219,000 entries) takes some 20 seconds and
|
|
300 MB of memory: entries are read one at a time.
|
|
|
|
Put the file into the server's `dictionaries` folder. The server loads it on
|
|
its next start, and again whenever the file changes (see the server's README).
|
|
|
|
## The format (`halkyon-dictionary`, version 1)
|
|
|
|
[JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`.
|
|
|
|
The **first line** says what the file holds:
|
|
|
|
| Field | |
|
|
|---|---|
|
|
| `format`, `formatVersion` | `"halkyon-dictionary"`, `1` |
|
|
| `source`, `sourceVersion` | The source's name (`jmdict`) and version (`3.6.2 (2026-10-05)`). An instance holds one version of a source for each meaning language. |
|
|
| `language`, `glossLanguage` | BCP 47 tags: the words' language (`ja`) and the meanings' (`en`). |
|
|
| `license` | The data's licence, an SPDX identifier (`CC-BY-SA-4.0`). |
|
|
| `credits` | Everything it is drawn from: `name`, `by`, `license`, `url`. The app lists them where it credits its data sources. |
|
|
|
|
**Every further line** is an entry:
|
|
|
|
| Field | |
|
|
|---|---|
|
|
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
|
|
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
|
|
| `labels` | What it can be picked by: `common`, `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`). |
|
|
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest. |
|
|
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
|
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to). Lists are left out when empty. |
|
|
|
|
## Adding a source
|
|
|
|
Write an adapter in `lib/src/sources/` that turns the source into the lines
|
|
above, with its credits and a sample of real data in `test/fixtures/` (and
|
|
where that comes from in `test/fixtures/README.md`); give it a command in
|
|
`bin/curate.dart`. Check the source's licence first, and that the instance may
|
|
pass its data on to the people using it.
|