feat(curator): Chinese from CC-CEDICT, with the HSK levels of 2025

Chinese comes from CC-CEDICT (CC BY-SA 4.0), made for learners of
Chinese: `curate cedict` writes two dictionaries of its entries,
`cc-cedict-hans-en` (zh-Hans) and `cc-cedict-hant-en` (zh-Hant), so a
studyset finds its words as it writes them, each finding a word by
either script, by pinyin with or without tones and by its meanings.
Readings are pinyin with tone marks, a syllable to a character; measure
words are a remark beside the meanings.

The HSK levels are those of the 2025 examination syllabus, which HSK
3.0's examination follows since July 2026 (decided 2026-10-14: level
tags, credited), as hsk30 extracted them (MIT): `hsk:1` … `hsk:6`, and
`hsk:7` for 7 to 9, each given to the entry read as the syllabus reads
the word (的 de, not 的 dì; 书 shū, not the Book of Documents, Shū),
ranking the words, HSK 1 first.

`curate update --languages de,el,en,zh` takes over from `--wiktionary`:
Chinese from MDBG's file of the day as often as EDRDG's files, the rest
from Wiktionary's dump.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-14 07:27:02 +02:00
commit 9d21f7f20e
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64

View file

@ -178,4 +178,39 @@ void main() {
expect(opened, 0);
});
});
test('CC-CEDICT is converted in both scripts with the HSK levels, and '
'said to be up to date by its files', () async {
expect(cedictUpdated(into), isNull);
final written = await updateCedict(
into: into,
data: Directory('data'),
fetch: (uri, file) async {
expect(uri, cedictSource);
await file.writeAsBytes(
gzip.encode(
File('test/fixtures/cedict-sample.txt').readAsBytesSync(),
),
);
},
);
expect(
[for (final file in written) file.uri.pathSegments.last],
['cc-cedict-hans-en.jsonl.gz', 'cc-cedict-hant-en.jsonl.gz'],
);
expect(header('cc-cedict-hant-en.jsonl.gz')['language'], 'zh-Hant');
expect(cedictUpdated(into), isNotNull);
});
test('Chinese comes from CC-CEDICT, the other languages from Wiktionary', () {
final all = dictionarySources({'de', 'el', 'en', 'zh'});
expect(all.wiktionary, {'de', 'el', 'en'});
expect(all.chinese, isTrue);
final traditional = dictionarySources({'zh-Hant'});
expect(traditional.wiktionary, isEmpty);
expect(traditional.chinese, isTrue);
expect(dictionarySources({}).chinese, isFalse);
});
}