halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 37 Packages Wiki Activity Actions
halkyon/halkyon_learn_curator/README.md

63 lines
3.4 KiB
Markdown
Raw Normal View History

feat(curator): turn JMdict and the JLPT lists into dictionary files A package of its own in the workspace, without Serverpod (AGPL, like the tooling): an adapter per source writes one format for all, documented in its README, which the server will load. The format is JSON Lines: a header (source, version, languages, licence, credits), then entries with a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a search matches, and the entry as cards take it. Everything specific to a language happens in its adapter (Japanese: forms, readings in kana and Hepburn romaji, long vowels typed once, meanings without "to" and articles), so the server's search knows no language. JMdict comes as jmdict-simplified's JSON, read one entry at a time: all 218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add levels by JMdict entry, the easiest when a word is on several, and give the headword its common spelling. Words rank N5 to N1, then the other common words, then the rest. The server's image leaves the package out, as it does the client's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
# halkyon_learn_curator
Turns dictionary downloads into the dictionary files a Halkyon Learn server
loads. It reads each source with an adapter of its own and writes one format
for all of them, so a new source or language needs a new adapter, never a
change to the server or the app. It doesn't depend on Serverpod or the server.
## Making a dictionary file
JMdict in English, with the JLPT levels on top:
1. Download `jmdict-eng-<version>.json.tgz` from
[jmdict-simplified's releases](https://github.com/scriptin/jmdict-simplified/releases)
and unpack it.
2. Download `original_data/n1.csv` … `n5.csv` from
[yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab) into a
folder.
3. Convert, in this package:
```sh
dart run halkyon_learn_curator:curate jmdict \
--input jmdict-eng-3.6.2.json --jlpt jlpt/ --output jmdict-en.jsonl.gz
```
The whole of JMdict (about 219,000 entries) takes some 20 seconds and
300 MB of memory: entries are read one at a time.
Put the file into the server's `dictionaries` folder. The server loads it on
its next start, and again whenever the file changes (see the server's README).
## The format (`halkyon-dictionary`, version 1)
[JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`.
The **first line** says what the file holds:
| Field | |
|---|---|
| `format`, `formatVersion` | `"halkyon-dictionary"`, `1` |
| `source`, `sourceVersion` | The source's name (`jmdict`) and version (`3.6.2 (2026-10-05)`). An instance holds one version of a source for each meaning language. |
| `language`, `glossLanguage` | BCP 47 tags: the words' language (`ja`) and the meanings' (`en`). |
| `license` | The data's licence, an SPDX identifier (`CC-BY-SA-4.0`). |
| `credits` | Everything it is drawn from: `name`, `by`, `license`, `url`. The app lists them where it credits its data sources. |
**Every further line** is an entry:
| Field | |
|---|---|
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
| `labels` | What it can be picked by: `common`, `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`). |
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest. |
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
feat(server): dictionaries, loaded from files and searched The server now holds dictionaries for everyone: `dictionary_entry` and `dictionary_source`, reference data owned by no one, never synced. It loads the dictionary files in its `dictionaries` folder at every start, those whose SHA-256 changed: in the background, each source replaced in one transaction, so searches meanwhile find the old entries; a file it can't read is logged and left out. All of JMdict loads in under half a minute. On an instance the folder is deploy/dictionaries, mounted read-only (compose.yaml); the server's README says how. DictionaryEndpoint (signed in): the dictionaries with their credits; a search (exact key, exact loose key, key starting with the query, loose key starting with it, each in the source's order; katakana as hiragana, macrons written out), some 150-450 ms over all of JMdict; and the entries with a label, counted and a page at a time (all JLPT N5 words). The curator now keeps spellings that aren't quite right apart, as `looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司 before 数詞 (すうし). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし). Left out when empty. |
feat(curator): turn JMdict and the JLPT lists into dictionary files A package of its own in the workspace, without Serverpod (AGPL, like the tooling): an adapter per source writes one format for all, documented in its README, which the server will load. The format is JSON Lines: a header (source, version, languages, licence, credits), then entries with a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a search matches, and the entry as cards take it. Everything specific to a language happens in its adapter (Japanese: forms, readings in kana and Hepburn romaji, long vowels typed once, meanings without "to" and articles), so the server's search knows no language. JMdict comes as jmdict-simplified's JSON, read one entry at a time: all 218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add levels by JMdict entry, the easiest when a word is on several, and give the headword its common spelling. Words rank N5 to N1, then the other common words, then the rest. The server's image leaves the package out, as it does the client's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to). Lists are left out when empty. |
## Adding a source
Write an adapter in `lib/src/sources/` that turns the source into the lines
above, with its credits and a sample of real data in `test/fixtures/` (and
where that comes from in `test/fixtures/README.md`); give it a command in
`bin/curate.dart`. Check the source's licence first, and that the instance may
pass its data on to the people using it.
Reference in a new issue Copy permalink