halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 37 Packages Wiki Activity Actions
halkyon/halkyon_learn_curator/README.md
Tizian 「ティツィアーン」 a875a91f40
 feat(curator): turn JMdict and the JLPT lists into dictionary files 
A package of its own in the workspace, without Serverpod (AGPL, like the
tooling): an adapter per source writes one format for all, documented in
its README, which the server will load. The format is JSON Lines: a
header (source, version, languages, licence, credits), then entries with
a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a
search matches, and the entry as cards take it. Everything specific to a
language happens in its adapter (Japanese: forms, readings in kana and
Hepburn romaji, long vowels typed once, meanings without "to" and
articles), so the server's search knows no language.

JMdict comes as jmdict-simplified's JSON, read one entry at a time: all
218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT
lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add
levels by JMdict entry, the easiest when a word is on several, and give
the headword its common spelling. Words rank N5 to N1, then the other
common words, then the rest.

The server's image leaves the package out, as it does the client's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 18:09:25 +02:00

3.3 KiB
Raw Blame History

halkyon_learn_curator

Turns dictionary downloads into the dictionary files a Halkyon Learn server loads. It reads each source with an adapter of its own and writes one format for all of them, so a new source or language needs a new adapter, never a change to the server or the app. It doesn't depend on Serverpod or the server.

Making a dictionary file

JMdict in English, with the JLPT levels on top:

  1. Download jmdict-eng-<version>.json.tgz from jmdict-simplified's releases and unpack it.

  2. Download original_data/n1.csv … n5.csv from yomitan-jlpt-vocab into a folder.

  3. Convert, in this package:

    dart run halkyon_learn_curator:curate jmdict \
  --input jmdict-eng-3.6.2.json --jlpt jlpt/ --output jmdict-en.jsonl.gz

    The whole of JMdict (about 219,000 entries) takes some 20 seconds and 300 MB of memory: entries are read one at a time.

Put the file into the server's dictionaries folder. The server loads it on its next start, and again whenever the file changes (see the server's README).

The format (halkyon-dictionary, version 1)

JSON Lines, gzipped when the name ends in .gz.

The first line says what the file holds:

Field
format, formatVersion "halkyon-dictionary", 1
source, sourceVersion The source's name (jmdict) and version (3.6.2 (2026-10-05)). An instance holds one version of a source for each meaning language.
language, glossLanguage BCP 47 tags: the words' language (ja) and the meanings' (en).
license The data's licence, an SPDX identifier (CC-BY-SA-4.0).
credits Everything it is drawn from: name, by, license, url. The app lists them where it credits its data sources.

Every further line is an entry:

Field
id Its id in the source; with the source and meaning language, it names the entry for good.
headword, reading How the word is shown, and how it is read; no reading when the headword is its own (a word written in kana).
labels What it can be picked by: common, jlpt:n5 … jlpt:n1 (later e.g. hsk:3, topik:1).
rank Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest.
keys What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language.
entry The entry itself, as cards take it: forms and readings (text, common, the source's tags; for a reading the forms it belongs to), senses (glosses; the source's codes in pos, misc, field, dialect; info; the forms and readings a sense is limited to). Lists are left out when empty.

Adding a source

Write an adapter in lib/src/sources/ that turns the source into the lines above, with its credits and a sample of real data in test/fixtures/ (and where that comes from in test/fixtures/README.md); give it a command in bin/curate.dart. Check the source's licence first, and that the instance may pass its data on to the people using it.