feat(curator): Chinese from CC-CEDICT, with the HSK levels of 2025
Chinese comes from CC-CEDICT (CC BY-SA 4.0), made for learners of Chinese: `curate cedict` writes two dictionaries of its entries, `cc-cedict-hans-en` (zh-Hans) and `cc-cedict-hant-en` (zh-Hant), so a studyset finds its words as it writes them, each finding a word by either script, by pinyin with or without tones and by its meanings. Readings are pinyin with tone marks, a syllable to a character; measure words are a remark beside the meanings. The HSK levels are those of the 2025 examination syllabus, which HSK 3.0's examination follows since July 2026 (decided 2026-10-14: level tags, credited), as hsk30 extracted them (MIT): `hsk:1` … `hsk:6`, and `hsk:7` for 7 to 9, each given to the entry read as the syllabus reads the word (的 de, not 的 dì; 书 shū, not the Book of Documents, Shū), ranking the words, HSK 1 first. `curate update --languages de,el,en,zh` takes over from `--wiktionary`: Chinese from MDBG's file of the day as often as EDRDG's files, the rest from Wiktionary's dump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
6d57f50b16
commit
9d21f7f20e
1 changed files with 11610 additions and 22 deletions
|
|
@ -178,4 +178,39 @@ void main() {
|
|||
expect(opened, 0);
|
||||
});
|
||||
});
|
||||
|
||||
test('CC-CEDICT is converted in both scripts with the HSK levels, and '
|
||||
'said to be up to date by its files', () async {
|
||||
expect(cedictUpdated(into), isNull);
|
||||
|
||||
final written = await updateCedict(
|
||||
into: into,
|
||||
data: Directory('data'),
|
||||
fetch: (uri, file) async {
|
||||
expect(uri, cedictSource);
|
||||
await file.writeAsBytes(
|
||||
gzip.encode(
|
||||
File('test/fixtures/cedict-sample.txt').readAsBytesSync(),
|
||||
),
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
expect(
|
||||
[for (final file in written) file.uri.pathSegments.last],
|
||||
['cc-cedict-hans-en.jsonl.gz', 'cc-cedict-hant-en.jsonl.gz'],
|
||||
);
|
||||
expect(header('cc-cedict-hant-en.jsonl.gz')['language'], 'zh-Hant');
|
||||
expect(cedictUpdated(into), isNotNull);
|
||||
});
|
||||
|
||||
test('Chinese comes from CC-CEDICT, the other languages from Wiktionary', () {
|
||||
final all = dictionarySources({'de', 'el', 'en', 'zh'});
|
||||
expect(all.wiktionary, {'de', 'el', 'en'});
|
||||
expect(all.chinese, isTrue);
|
||||
final traditional = dictionarySources({'zh-Hant'});
|
||||
expect(traditional.wiktionary, isEmpty);
|
||||
expect(traditional.chinese, isTrue);
|
||||
expect(dictionarySources({}).chinese, isFalse);
|
||||
});
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue