halkyon/apps/learn/halkyon_learn_curator/data/README.md

# Data the curator brings along

What the dictionaries draw on besides their sources, and never changes, so
`curate update` needs only EDRDG's files of the day, MDBG's CC-CEDICT and
Wiktionary's dump.

`jlpt/n1.csv` … `n5.csv` are the JLPT vocabulary lists of
[yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab)
(`original_data`, as of 2026-10-09), CC BY-SA 4.0 (`jlpt/LICENSE.txt`): Jonathan
Waller's lists ([tanos.co.uk](http://www.tanos.co.uk/jlpt/), CC BY) with each
word's JMdict entry and its most common spelling, by stephenmk.

`kanji-jlpt.json` maps each kanji on Jonathan Waller's JLPT kanji lists
([tanos.co.uk](http://www.tanos.co.uk/jlpt/), CC BY) to its level, 5 for N5 to 1
for N1: the `jlpt_new` of
[kanji-data](https://github.com/davidluzgouveia/kanji-data)'s `kanji.json`
(MIT, as of 2026-10-09), nothing else of it.

The lists are over ten years old and won't change; replace them only for a list
of the same shape.

`wordfreq/small_<language>.msgpack.gz` are wordfreq's lists of how often words
are used ([wordfreq](https://github.com/rspeer/wordfreq) v3.0, as of 2021-03-29,
the last change to its data), unchanged: CC BY-SA 4.0, by Robyn Speer, drawn from
SUBTLEX (Marc Brysbaert et al., whose authors wordfreq's licence asks to be
credited, and who made SUBTLEX freely available), OpenSubtitles, Wikipedia,
Google Books Ngrams and others, as wordfreq's README lists them. The Wiktionary
dictionaries take their ranks and their `common` label from them and credit
both. wordfreq's data won't change: it was sunset in 2024.

`hsk/syllabus2025_words.tsv` lists the 10,896 words of the HSK examination
syllabus published in November 2025 by the Center for Language Education and
Cooperation (中外语言交流合作中心), the one HSK 3.0's examination follows since
July 2026, each with its level (1 to 6, 7 for 7 to 9), the level of a homograph
graded twice, its pinyin and its part of speech: as extracted from the
syllabus's PDF by Alvaro Serrano's [hsk30](https://github.com/harukicoder/hsk30)
(`corpus/syllabus2025/`, as of 2026-09-06), MIT (`hsk/LICENSE.txt`), unchanged.
Only each word's level reaches the dictionaries (`hsk:N`), with the pinyin to
tell CC-CEDICT's readings of one word apart; the curator credits both. The
Ministry claims copyright on the standard's documents; a word's level is a fact
about it, which the dictionaries keep with attribution (decided 2026-10-14).
`hsk/syllabus2025_chars.tsv` lists the syllabus's 3,088 characters to recognise
(认读字) with their levels (246, 125, 284, 441, 431 and 413 for levels 1 to 6,
1,148 for 7 to 9), from the same repository and commit (`corpus/syllabus2025/`),
MIT, unchanged; they label the Chinese characters (`hsk:N`) as the words' label
the words.