halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 37 Packages Wiki Activity Actions
halkyon/halkyon_learn_curator
History
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Tizian 「ティツィアーン」 3013bc04b3
 feat(server): dictionaries, loaded from files and searched 
The server now holds dictionaries for everyone: `dictionary_entry` and
`dictionary_source`, reference data owned by no one, never synced. It
loads the dictionary files in its `dictionaries` folder at every start,
those whose SHA-256 changed: in the background, each source replaced in
one transaction, so searches meanwhile find the old entries; a file it
can't read is logged and left out. All of JMdict loads in under half a
minute. On an instance the folder is deploy/dictionaries, mounted
read-only (compose.yaml); the server's README says how.

DictionaryEndpoint (signed in): the dictionaries with their credits; a
search (exact key, exact loose key, key starting with the query, loose
key starting with it, each in the source's order; katakana as hiragana,
macrons written out), some 150-450 ms over all of JMdict; and the
entries with a label, counted and a page at a time (all JLPT N5 words).

The curator now keeps spellings that aren't quite right apart, as
`looseKeys` (romaji with long vowels typed once): "sushi" finds 寿司
before 数詞 (すうし).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 18:09:25 +02:00
..
bin feat(curator): turn JMdict and the JLPT lists into dictionary files 2026-10-09 18:09:25 +02:00
lib/src feat(server): dictionaries, loaded from files and searched 2026-10-09 18:09:25 +02:00
test feat(server): dictionaries, loaded from files and searched 2026-10-09 18:09:25 +02:00
analysis_options.yaml feat(curator): turn JMdict and the JLPT lists into dictionary files 2026-10-09 18:09:25 +02:00
LICENSE feat(curator): turn JMdict and the JLPT lists into dictionary files 2026-10-09 18:09:25 +02:00
pubspec.yaml feat(curator): turn JMdict and the JLPT lists into dictionary files 2026-10-09 18:09:25 +02:00
README.md feat(server): dictionaries, loaded from files and searched 2026-10-09 18:09:25 +02:00

README.md

halkyon_learn_curator

Turns dictionary downloads into the dictionary files a Halkyon Learn server loads. It reads each source with an adapter of its own and writes one format for all of them, so a new source or language needs a new adapter, never a change to the server or the app. It doesn't depend on Serverpod or the server.

Making a dictionary file

JMdict in English, with the JLPT levels on top:

  1. Download jmdict-eng-<version>.json.tgz from jmdict-simplified's releases and unpack it.

  2. Download original_data/n1.csv … n5.csv from yomitan-jlpt-vocab into a folder.

  3. Convert, in this package:

    dart run halkyon_learn_curator:curate jmdict \
  --input jmdict-eng-3.6.2.json --jlpt jlpt/ --output jmdict-en.jsonl.gz

    The whole of JMdict (about 219,000 entries) takes some 20 seconds and 300 MB of memory: entries are read one at a time.

Put the file into the server's dictionaries folder. The server loads it on its next start, and again whenever the file changes (see the server's README).

The format (halkyon-dictionary, version 1)

JSON Lines, gzipped when the name ends in .gz.

The first line says what the file holds:

Field
format, formatVersion "halkyon-dictionary", 1
source, sourceVersion The source's name (jmdict) and version (3.6.2 (2026-10-05)). An instance holds one version of a source for each meaning language.
language, glossLanguage BCP 47 tags: the words' language (ja) and the meanings' (en).
license The data's licence, an SPDX identifier (CC-BY-SA-4.0).
credits Everything it is drawn from: name, by, license, url. The app lists them where it credits its data sources.

Every further line is an entry:

Field
id Its id in the source; with the source and meaning language, it names the entry for good.
headword, reading How the word is shown, and how it is read; no reading when the headword is its own (a word written in kana).
labels What it can be picked by: common, jlpt:n5 … jlpt:n1 (later e.g. hsk:3, topik:1).
rank Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest.
keys What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language.
looseKeys What a search matches after every keys match: spellings people type that aren't quite right (romaji with long vowels typed once, sushi for すうし). Left out when empty.
entry The entry itself, as cards take it: forms and readings (text, common, the source's tags; for a reading the forms it belongs to), senses (glosses; the source's codes in pos, misc, field, dialect; info; the forms and readings a sense is limited to). Lists are left out when empty.

Adding a source

Write an adapter in lib/src/sources/ that turns the source into the lines above, with its credits and a sample of real data in test/fixtures/ (and where that comes from in test/fixtures/README.md); give it a command in bin/curate.dart. Check the source's licence first, and that the instance may pass its data on to the people using it.