halkyon/halkyon_learn_curator/test/fixtures/README.md
Tizian 「ティツィアーン」 2d0ba20eb7
feat(curator): read JMdict as EDRDG publishes it, the XML of every day
EDRDG makes JMdict_e_examp.gz anew every day: JMdict in English with
Tatoeba's sentences on the meanings, the file jmdict-simplified's examples
edition is made from once a week. The curator now reads it directly (a
file not named .json is read as XML), streaming 72 MB of XML in some 30
seconds and 300 MB, each entry put into the shape jmdict-simplified gives
it and made an entry by the same code: against jmdict-simplified's
conversion of four days before, the entries differ only by JMdict's edits
since. JMdict's entities stay codes (`&v1;` is `v1`), text cut between two
chunks is joined before its entities are read (the xml package's
normalizeEvents), and a sense without parts of speech has the previous
sense's, as JMdict's DTD says. Its version is the day it was made.

So an instance can keep its dictionaries current from the source alone,
as EDRDG's licence asks of servers (at least monthly).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-09 18:09:36 +02:00

1.6 KiB

Test data

jmdict-eng-sample.json holds six entries of JMdict 3.6.2 (2026-10-05) as jmdict-simplified writes it (https://github.com/scriptin/jmdict-simplified): JMdict is © the Electronic Dictionary Research and Development Group, used under its licence, CC BY-SA 4.0 (https://www.edrdg.org/edrdg/licence.html).

jlpt/ holds a few rows of the JLPT lists of yomitan-jlpt-vocab (https://github.com/stephenmk/yomitan-jlpt-vocab), CC BY-SA 4.0, from Jonathan Waller's lists (http://www.tanos.co.uk/jlpt/), CC BY; one row without an id is made up.

kanjidic2-sample.xml holds five characters of KANJIDIC2 (database version 2026-282) with its header and a shortened DTD: KANJIDIC2 is © the Electronic Dictionary Research and Development Group, used under its licence, CC BY-SA 4.0 (https://www.edrdg.org/edrdg/licence.html).

kanji-data-sample.json holds four entries of kanji-data's kanji.json (https://github.com/davidluzgouveia/kanji-data), cut to their numbers, from KANJIDIC2 and from Jonathan Waller's JLPT levels (http://www.tanos.co.uk/jlpt/), CC BY.

kanjivg-sample.xml holds four characters of KanjiVG (the release of 2026-07-14) with its notice, among them 橅, whose entry names itself 機: KanjiVG is © Ulrich Apel, CC BY-SA 3.0 (https://kanjivg.tagaini.net).

jmdict-e-examp-sample.xml holds five entries of EDRDG's JMdict_e_examp.gz of 2026-10-09 (JMdict with Tatoeba's sentences, CC BY 2.0 FR) and the declarations of the entities they use: JMdict is © the Electronic Dictionary Research and Development Group, used under its licence, CC BY-SA 4.0 (https://www.edrdg.org/edrdg/licence.html).