feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
# halkyon_learn_curator
Turns dictionary downloads into the dictionary files a Halkyon Learn server
loads. It reads each source with an adapter of its own and writes one format
for all of them, so a new source or language needs a new adapter, never a
change to the server or the app. It doesn't depend on Serverpod or the server.
## Making a dictionary file
2026-10-09 13:36:16 +02:00
JMdict in English, with the JLPT levels and example sentences on top:
feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
2026-10-09 13:36:16 +02:00
1. Download `jmdict-examples-eng-<version>.json.tgz` from
feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
[jmdict-simplified's releases ](https://github.com/scriptin/jmdict-simplified/releases )
2026-10-09 13:36:16 +02:00
and unpack it: JMdict with Tatoeba's sentences (CC BY 2.0 FR) on the
meanings they show. `jmdict-eng-<version>.json` works too, without them.
feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
2. Download `original_data/n1.csv` … `n5.csv` from
[yomitan-jlpt-vocab ](https://github.com/stephenmk/yomitan-jlpt-vocab ) into a
folder.
3. Convert, in this package:
```sh
dart run halkyon_learn_curator:curate jmdict \
2026-10-09 13:36:16 +02:00
--input jmdict-examples-eng-3.6.2.json --jlpt jlpt/ \
--output jmdict-en.jsonl.gz
feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
```
The whole of JMdict (about 219,000 entries) takes some 20 seconds and
300 MB of memory: entries are read one at a time.
Put the file into the server's `dictionaries` folder. The server loads it on
its next start, and again whenever the file changes (see the server's README).
## The format (`halkyon-dictionary`, version 1)
[JSON Lines ](https://jsonlines.org/ ), gzipped when the name ends in `.gz` .
The **first line ** says what the file holds:
| Field | |
|---|---|
| `format` , `formatVersion` | `"halkyon-dictionary"` , `1` |
| `source` , `sourceVersion` | The source's name (`jmdict` ) and version (`3.6.2 (2026-10-05)` ). An instance holds one version of a source for each meaning language. |
| `language` , `glossLanguage` | BCP 47 tags: the words' language (`ja` ) and the meanings' (`en` ). |
| `license` | The data's licence, an SPDX identifier (`CC-BY-SA-4.0` ). |
| `credits` | Everything it is drawn from: `name` , `by` , `license` , `url` . The app lists them where it credits its data sources. |
**Every further line** is an entry:
| Field | |
|---|---|
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
| `headword` , `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
| `labels` | What it can be picked by: `common` , `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3` , `topik:1` ). |
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest. |
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
feat(server): dictionaries, loaded from files and searched
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.
DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).
The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:51:27 +02:00
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし). Left out when empty. |
2026-10-09 13:36:16 +02:00
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text` , `common` , the source's `tags` ; for a reading the `forms` it belongs to), `senses` (`glosses` ; the source's codes in `pos` , `misc` , `field` , `dialect` ; `info` ; the `forms` and `readings` a sense is limited to; `examples` , sentences using the word in that sense: `text` , the form it takes there, `ja` , the sentence, `en` , its translation, and the source's id, e.g. `tatoeba` ). Lists are left out when empty. |
feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.
JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.
The server's image leaves the package out, as it does the client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 00:25:16 +02:00
## Adding a source
Write an adapter in `lib/src/sources/` that turns the source into the lines
above, with its credits and a sample of real data in `test/fixtures/` (and
where that comes from in `test/fixtures/README.md` ); give it a command in
`bin/curate.dart` . Check the source's licence first, and that the instance may
pass its data on to the people using it.
