feat(curator): Chinese from CC-CEDICT, with the HSK levels of 2025
Chinese comes from CC-CEDICT (CC BY-SA 4.0), made for learners of Chinese: `curate cedict` writes two dictionaries of its entries, `cc-cedict-hans-en` (zh-Hans) and `cc-cedict-hant-en` (zh-Hant), so a studyset finds its words as it writes them, each finding a word by either script, by pinyin with or without tones and by its meanings. Readings are pinyin with tone marks, a syllable to a character; measure words are a remark beside the meanings. The HSK levels are those of the 2025 examination syllabus, which HSK 3.0's examination follows since July 2026 (decided 2026-10-14: level tags, credited), as hsk30 extracted them (MIT): `hsk:1` … `hsk:6`, and `hsk:7` for 7 to 9, each given to the entry read as the syllabus reads the word (的 de, not 的 dì; 书 shū, not the Book of Documents, Shū), ranking the words, HSK 1 first. `curate update --languages de,el,en,zh` takes over from `--wiktionary`: Chinese from MDBG's file of the day as often as EDRDG's files, the rest from Wiktionary's dump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
6d57f50b16
commit
9d21f7f20e
GPG key ID:
07F6FCF405488B64
11 changed files with 11610 additions and 22 deletions
|
|
@ -75,6 +75,32 @@ Greek and English first, then Chinese, then Korean):
|
|||
whole dump takes a while; it is read once, streamed, keeping in memory only
|
||||
which forms belong to which word.
|
||||
|
||||
Chinese from CC-CEDICT (decided 2026-10-14: a dictionary of its own, made for
|
||||
learners of Chinese, where Wiktionary's Chinese entries sit at their traditional
|
||||
forms):
|
||||
|
||||
1. Download `cedict_1_0_ts_utf-8_mdbg.txt.gz` from
|
||||
[MDBG](https://www.mdbg.net/chinese/dictionary?page=cc-cedict), which makes it
|
||||
anew every day.
|
||||
2. Convert, in this package:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate cedict \
|
||||
--input cedict_1_0_ts_utf-8_mdbg.txt.gz \
|
||||
--output ../halkyon_learn_server/dictionaries
|
||||
```
|
||||
|
||||
It writes two dictionaries of the same entries, `cc-cedict-hans-en`
|
||||
(`zh-Hans`, headwords in simplified characters) and `cc-cedict-hant-en`
|
||||
(`zh-Hant`, in traditional ones), so a studyset finds its words as it writes
|
||||
them; each finds a word by either script, by its pinyin with or without tones
|
||||
(`xuéxí`, `xuexi`, `xue2xi2`) and by its meanings. Readings are pinyin with
|
||||
tone marks, a syllable to a character (`xué xí`). The HSK levels of the 2025
|
||||
examination syllabus (`data/hsk/`) label the words `hsk:1` … `hsk:6`, and
|
||||
`hsk:7` for the advanced levels 7 to 9, which the syllabus doesn't tell
|
||||
apart, each given to the entry read as the syllabus reads the word (的 *de*,
|
||||
not 的 *dì*), and rank them, HSK 1 first; wordfreq ranks the rest.
|
||||
|
||||
Put the files into the server's `dictionaries` folder. The server loads each on
|
||||
its next start, and again whenever it changes (see the server's README).
|
||||
|
||||
|
|
@ -103,7 +129,8 @@ so in a container of its own (the server's README). Beside the files,
|
|||
from the same download (a label, a field), and an instance converts again as
|
||||
soon as the new curator runs, not a week later.
|
||||
|
||||
With `--wiktionary de,el,en` it also keeps those languages current from
|
||||
With `--languages de,el,en,zh` it also keeps those languages current: Chinese
|
||||
from MDBG's CC-CEDICT of the day as often as EDRDG's files (below), the rest from
|
||||
Wiktionary: each time it wakes it asks kaikki.org when its dump last changed
|
||||
(`Last-Modified`, without downloading it), and converts it again only when it is
|
||||
newer than the files' (about monthly), when the languages named changed, or when
|
||||
|
|
@ -132,12 +159,12 @@ The **first line** says what the file holds:
|
|||
|---|---|
|
||||
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
|
||||
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
|
||||
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
|
||||
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary and CC-CEDICT the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1`, `hsk:1` … `hsk:7` (later e.g. `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
|
||||
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
|
||||
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
||||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
|
||||
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). |
|
||||
|
||||
## The app's stroke files
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import 'dart:convert';
|
|||
import 'dart:io';
|
||||
|
||||
import 'package:args/args.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/cedict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jlpt.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jmdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
|
||||
|
|
@ -33,13 +34,19 @@ import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
|||
/// --input raw-wiktextract-data.jsonl.gz --languages de,el,en \
|
||||
/// --output ../halkyon_learn_server/dictionaries
|
||||
///
|
||||
/// CC-CEDICT into its two dictionary files, simplified and traditional:
|
||||
///
|
||||
/// dart run halkyon_learn_curator:curate cedict \
|
||||
/// --input cedict_1_0_ts_utf-8_mdbg.txt.gz \
|
||||
/// --output ../halkyon_learn_server/dictionaries
|
||||
///
|
||||
/// Or brings a server's dictionaries up to date from EDRDG's files of the
|
||||
/// day and, for the languages named, Wiktionary's latest dump, once or
|
||||
/// every so many days:
|
||||
/// day and, for the languages named, Wiktionary's latest dump and MDBG's
|
||||
/// CC-CEDICT of the day (Chinese), once or every so many days:
|
||||
///
|
||||
/// dart run halkyon_learn_curator:curate update \
|
||||
/// --into ../halkyon_learn_server/dictionaries [--every 7] \
|
||||
/// [--wiktionary de,el,en]
|
||||
/// [--languages de,el,en,zh]
|
||||
Future<void> main(List<String> args) async {
|
||||
final parser = ArgParser()
|
||||
..addOption(
|
||||
|
|
@ -85,19 +92,15 @@ Future<void> main(List<String> args) async {
|
|||
'languages',
|
||||
help:
|
||||
'wiktionary: the languages to write dictionaries of, by their '
|
||||
'BCP 47 tags (de,el,en).',
|
||||
'BCP 47 tags (de,el,en); update: the languages besides Japanese '
|
||||
'to keep current (de,el,en,zh; Chinese from CC-CEDICT, the rest '
|
||||
"from Wiktionary's dump), none without.",
|
||||
)
|
||||
..addOption(
|
||||
'version',
|
||||
help:
|
||||
"wiktionary: the dump's date (2026-10-03); the input file's date "
|
||||
'without.',
|
||||
)
|
||||
..addOption(
|
||||
'wiktionary',
|
||||
help:
|
||||
"update: the languages to keep current from Wiktionary's dump "
|
||||
'(de,el,en); none without.',
|
||||
);
|
||||
final ArgResults options;
|
||||
try {
|
||||
|
|
@ -113,14 +116,57 @@ Future<void> main(List<String> args) async {
|
|||
if (into == null || (every != null && (days == null || days < 1))) {
|
||||
_fail(
|
||||
'Usage: curate update --into <folder> [--data <folder>] '
|
||||
'[--every <days>] [--wiktionary <languages>]\n${parser.usage}',
|
||||
'[--every <days>] [--languages <languages>]\n${parser.usage}',
|
||||
);
|
||||
}
|
||||
await _update(
|
||||
Directory(into),
|
||||
Directory(options.option('data')!),
|
||||
days,
|
||||
wiktionary: _languages(options.option('wiktionary')),
|
||||
languages: _languages(options.option('languages')),
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (source == 'cedict') {
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
if (input == null || output == null) {
|
||||
_fail(
|
||||
'Usage: curate cedict --input <file> --output <folder> '
|
||||
'[--data <folder>]\n${parser.usage}',
|
||||
);
|
||||
}
|
||||
final read = File(input).openRead();
|
||||
final text =
|
||||
await (input.endsWith('.gz') ? read.transform(gzip.decoder) : read)
|
||||
.transform(utf8.decoder)
|
||||
.join();
|
||||
final data = options.option('data')!;
|
||||
final hsk = await readHskLevels(hskFile(Directory(data)));
|
||||
final frequencies = File('$data/wordfreq/small_zh.msgpack.gz');
|
||||
final words = frequencies.existsSync()
|
||||
? await WordFrequencies.read(frequencies)
|
||||
: WordFrequencies.none;
|
||||
final folder = await Directory(output).create(recursive: true);
|
||||
for (final traditional in [false, true]) {
|
||||
await Stream.fromIterable(
|
||||
cedictDictionary(
|
||||
text,
|
||||
traditional: traditional,
|
||||
hsk: hsk,
|
||||
frequencies: words,
|
||||
),
|
||||
)
|
||||
.transform(utf8.encoder)
|
||||
.transform(gzip.encoder)
|
||||
.pipe(
|
||||
File('${folder.path}/${cedictFileName(traditional: traditional)}')
|
||||
.openWrite(),
|
||||
);
|
||||
}
|
||||
stdout.writeln(
|
||||
'Wrote ${cedictFileName(traditional: false)} and '
|
||||
'${cedictFileName(traditional: true)} to $output.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
|
@ -234,15 +280,16 @@ Future<void> main(List<String> args) async {
|
|||
/// Updates the dictionaries in [into]; with [every], keeps running and
|
||||
/// updates them whenever they are that many days old (as the files' dates
|
||||
/// say, so a restart doesn't download them again), trying again after six
|
||||
/// hours when an update fails. The languages of [wiktionary] are looked at
|
||||
/// each time it wakes, and converted again only when Wiktionary's dump is
|
||||
/// newer than theirs.
|
||||
/// hours when an update fails. Of [languages], Chinese comes from
|
||||
/// CC-CEDICT as often as EDRDG's files, the rest from Wiktionary, looked at
|
||||
/// each time it wakes and converted again only when the dump is newer.
|
||||
Future<void> _update(
|
||||
Directory into,
|
||||
Directory data,
|
||||
int? every, {
|
||||
Set<String> wiktionary = const {},
|
||||
Set<String> languages = const {},
|
||||
}) async {
|
||||
final (:wiktionary, :chinese) = dictionarySources(languages);
|
||||
void say(String message) =>
|
||||
stdout.writeln('${DateTime.now().toUtc().toIso8601String()} $message');
|
||||
void written(List<File> files) => say(
|
||||
|
|
@ -275,8 +322,25 @@ Future<void> _update(
|
|||
}
|
||||
}
|
||||
|
||||
// CC-CEDICT once its files are as old as EDRDG's may be.
|
||||
Future<bool> chineseOnce() async {
|
||||
final updated = cedictUpdated(into);
|
||||
if (!chinese ||
|
||||
updated != null &&
|
||||
DateTime.now().difference(updated) < Duration(days: every ?? 1)) {
|
||||
return true;
|
||||
}
|
||||
try {
|
||||
written(await updateCedict(into: into, data: data));
|
||||
return true;
|
||||
} on Object catch (error) {
|
||||
say('Updating CC-CEDICT failed: $error');
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (every == null) {
|
||||
if (!await once() || !await wiktionaryOnce()) {
|
||||
if (!await once() || !await chineseOnce() || !await wiktionaryOnce()) {
|
||||
exit(1);
|
||||
}
|
||||
return;
|
||||
|
|
@ -290,7 +354,7 @@ Future<void> _update(
|
|||
}
|
||||
continue;
|
||||
}
|
||||
final retry = !await wiktionaryOnce();
|
||||
final retry = !await chineseOnce() | !await wiktionaryOnce();
|
||||
final next = retry && due.difference(now) > const Duration(hours: 6)
|
||||
? DateTime.now().add(const Duration(hours: 6))
|
||||
: due;
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
# Data the curator brings along
|
||||
|
||||
What the dictionaries draw on besides their sources, and never changes, so
|
||||
`curate update` needs only EDRDG's files of the day and Wiktionary's dump.
|
||||
`curate update` needs only EDRDG's files of the day, MDBG's CC-CEDICT and
|
||||
Wiktionary's dump.
|
||||
|
||||
`jlpt/n1.csv` … `n5.csv` are the JLPT vocabulary lists of
|
||||
[yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab)
|
||||
|
|
@ -26,3 +27,15 @@ credited, and who made SUBTLEX freely available), OpenSubtitles, Wikipedia,
|
|||
Google Books Ngrams and others, as wordfreq's README lists them. The Wiktionary
|
||||
dictionaries take their ranks and their `common` label from them and credit
|
||||
both. wordfreq's data won't change: it was sunset in 2024.
|
||||
|
||||
`hsk/syllabus2025_words.tsv` lists the 10,896 words of the HSK examination
|
||||
syllabus published in November 2025 by the Center for Language Education and
|
||||
Cooperation (中外语言交流合作中心), the one HSK 3.0's examination follows since
|
||||
July 2026, each with its level (1 to 6, 7 for 7 to 9), the level of a homograph
|
||||
graded twice, its pinyin and its part of speech: as extracted from the
|
||||
syllabus's PDF by Alvaro Serrano's [hsk30](https://github.com/harukicoder/hsk30)
|
||||
(`corpus/syllabus2025/`, as of 2026-09-06), MIT (`hsk/LICENSE.txt`), unchanged.
|
||||
Only each word's level reaches the dictionaries (`hsk:N`), with the pinyin to
|
||||
tell CC-CEDICT's readings of one word apart; the curator credits both. The
|
||||
Ministry claims copyright on the standard's documents; a word's level is a fact
|
||||
about it, which the dictionaries keep with attribution (decided 2026-10-14).
|
||||
|
|
|
|||
21
halkyon_learn_curator/data/hsk/LICENSE.txt
Normal file
21
halkyon_learn_curator/data/hsk/LICENSE.txt
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2026 Alvaro Serrano
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
10897
halkyon_learn_curator/data/hsk/syllabus2025_words.tsv
Normal file
10897
halkyon_learn_curator/data/hsk/syllabus2025_words.tsv
Normal file
File diff suppressed because it is too large
Load diff
266
halkyon_learn_curator/lib/src/sources/cedict.dart
Normal file
266
halkyon_learn_curator/lib/src/sources/cedict.dart
Normal file
|
|
@ -0,0 +1,266 @@
|
|||
import 'dart:io';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/dictionary_file.dart';
|
||||
import 'package:halkyon_learn_curator/src/gloss_keys.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
|
||||
/// Who makes CC-CEDICT.
|
||||
const cedictCredit = Credit(
|
||||
name: 'CC-CEDICT',
|
||||
by: 'MDBG and the CC-CEDICT contributors',
|
||||
license: 'CC-BY-SA-4.0',
|
||||
url: 'https://www.mdbg.net/chinese/dictionary?page=cc-cedict',
|
||||
);
|
||||
|
||||
/// Who set the HSK levels, and who made them readable.
|
||||
const hskCredit = Credit(
|
||||
name: 'HSK 3.0 vocabulary (examination syllabus, 2025)',
|
||||
by:
|
||||
'Center for Language Education and Cooperation; extracted by '
|
||||
'Alvaro Serrano (hsk30)',
|
||||
license: 'MIT',
|
||||
url: 'https://github.com/harukicoder/hsk30',
|
||||
);
|
||||
|
||||
/// The name of CC-CEDICT's dictionary file in simplified or [traditional]
|
||||
/// characters.
|
||||
String cedictFileName({required bool traditional}) =>
|
||||
'cc-cedict-${traditional ? 'hant' : 'hans'}-en.jsonl.gz';
|
||||
|
||||
/// HSK levels by word (simplified) and pinyin without tones: 1 to 6, and 7
|
||||
/// for the advanced levels 7 to 9, which the syllabus doesn't tell apart.
|
||||
typedef HskLevels = Map<(String, String), int>;
|
||||
|
||||
/// The HSK levels of the 2025 examination syllabus, as `hsk30` extracted
|
||||
/// them ([file]: word, level, another level of a homograph, pinyin, part of
|
||||
/// speech). A word graded twice keeps its lower level.
|
||||
Future<HskLevels> readHskLevels(File file) async {
|
||||
final levels = <(String, String), int>{};
|
||||
for (final line in await file.readAsLines()) {
|
||||
final [word, level, ..., pinyin, _] = switch (line.split('\t')) {
|
||||
final fields when fields.length >= 5 => fields,
|
||||
_ => const ['', '', '', '', ''],
|
||||
};
|
||||
final number = int.tryParse(level);
|
||||
if (number == null || word.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
final key = (word, tonelessPinyin(pinyin));
|
||||
final known = levels[key];
|
||||
levels[key] = known == null || number < known ? number : known;
|
||||
}
|
||||
return levels;
|
||||
}
|
||||
|
||||
/// CC-CEDICT ([text], its file as MDBG publishes it) as the lines of a
|
||||
/// dictionary file of Chinese words with English meanings: in simplified
|
||||
/// characters (`zh-Hans`, the headword simplified, the traditional among
|
||||
/// its forms) or [traditional] ones (`zh-Hant`), each a dictionary of its
|
||||
/// own so a studyset in either finds its words as it writes them. Readings
|
||||
/// are pinyin with tone marks, a syllable to a character. [hsk] gives the
|
||||
/// words' HSK levels (`hsk:1` … `hsk:7`, the last for 7 to 9), which also
|
||||
/// rank them (HSK 1 first), then [frequencies] the rest.
|
||||
Iterable<String> cedictDictionary(
|
||||
String text, {
|
||||
required bool traditional,
|
||||
HskLevels hsk = const {},
|
||||
WordFrequencies? frequencies,
|
||||
}) sync* {
|
||||
final version = RegExp(r'^#! date=(\S+)', multiLine: true).firstMatch(text);
|
||||
yield headerLine(
|
||||
DictionaryHeader(
|
||||
source: 'cc-cedict-${traditional ? 'hant' : 'hans'}',
|
||||
sourceVersion: version?[1]?.substring(0, 10) ?? '',
|
||||
language: traditional ? 'zh-Hant' : 'zh-Hans',
|
||||
glossLanguage: 'en',
|
||||
license: 'CC-BY-SA-4.0',
|
||||
credits: [
|
||||
cedictCredit,
|
||||
if (hsk.isNotEmpty) hskCredit,
|
||||
if (frequencies != null && frequencies.length > 0) ...wordfreqCredits,
|
||||
],
|
||||
),
|
||||
);
|
||||
final line = RegExp(r'^(\S+) (\S+) \[([^\]]*)\] /(.*)/\s*$');
|
||||
final matches = [
|
||||
for (final row in text.split('\n'))
|
||||
if (!row.startsWith('#')) ?line.firstMatch(row),
|
||||
];
|
||||
// Words with an entry in lower-case pinyin: their level is that one's,
|
||||
// not a name's written alike (书 shū, a book, not 書 Shū, the Book of
|
||||
// Documents).
|
||||
final common = {
|
||||
for (final match in matches)
|
||||
if (match[3]!.toLowerCase() == match[3])
|
||||
(match[2]!, tonelessPinyin(match[3]!)),
|
||||
};
|
||||
for (final match in matches) {
|
||||
final named = match[3]!.toLowerCase() != match[3];
|
||||
final key = (match[2]!, tonelessPinyin(match[3]!));
|
||||
yield entryLine(
|
||||
cedictEntry(
|
||||
traditionalForm: match[1]!,
|
||||
simplifiedForm: match[2]!,
|
||||
pinyin: match[3]!,
|
||||
definitions: match[4]!.split('/'),
|
||||
traditional: traditional,
|
||||
hsk: named && common.contains(key) ? const {} : hsk,
|
||||
frequencies: frequencies,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// One line of CC-CEDICT as a dictionary entry.
|
||||
DictionaryEntry cedictEntry({
|
||||
required String traditionalForm,
|
||||
required String simplifiedForm,
|
||||
required String pinyin,
|
||||
required List<String> definitions,
|
||||
required bool traditional,
|
||||
HskLevels hsk = const {},
|
||||
WordFrequencies? frequencies,
|
||||
}) {
|
||||
final headword = traditional ? traditionalForm : simplifiedForm;
|
||||
final other = traditional ? simplifiedForm : traditionalForm;
|
||||
final marked = pinyinWithToneMarks(pinyin);
|
||||
final toneless = tonelessPinyin(pinyin);
|
||||
final level = hsk[(simplifiedForm, toneless)];
|
||||
final place = frequencies?.placeOf(simplifiedForm);
|
||||
// Measure words are a remark on the word, not a meaning.
|
||||
final classifiers = [
|
||||
for (final definition in definitions)
|
||||
if (definition.startsWith('CL:'))
|
||||
for (final classifier in definition.substring(3).split(','))
|
||||
_readableCrossReference(classifier.trim()),
|
||||
];
|
||||
final senses = [
|
||||
for (final definition in definitions)
|
||||
if (definition.trim().isNotEmpty && !definition.startsWith('CL:'))
|
||||
_readableCrossReference(definition.trim()),
|
||||
];
|
||||
return DictionaryEntry(
|
||||
// Its line's own key, as CC-CEDICT's editor names an entry.
|
||||
id: '$traditionalForm $simplifiedForm [$pinyin]',
|
||||
headword: headword,
|
||||
reading: marked,
|
||||
labels: [
|
||||
if (place != null && place < 15000) 'common',
|
||||
if (level != null) 'hsk:$level',
|
||||
],
|
||||
rank: switch (level) {
|
||||
final level? => level * 100000 + (place ?? 99999).clamp(0, 99999),
|
||||
null => 1000000 + (place ?? 900000),
|
||||
},
|
||||
keys: {
|
||||
headword,
|
||||
other,
|
||||
marked.replaceAll(' ', '').toLowerCase(),
|
||||
toneless,
|
||||
pinyin.replaceAll(' ', '').toLowerCase(),
|
||||
for (final sense in senses)
|
||||
for (final gloss in sense.split(';')) ...glossKeys(gloss),
|
||||
}.toList(),
|
||||
looseKeys: [marked.toLowerCase(), pinyin.toLowerCase()],
|
||||
textKeys: {headword, other}.toList(),
|
||||
entry: {
|
||||
if (other != headword)
|
||||
'forms': [
|
||||
{'text': other},
|
||||
],
|
||||
'readings': [
|
||||
{'text': marked},
|
||||
],
|
||||
'senses': [
|
||||
for (final sense in senses)
|
||||
{
|
||||
'glosses': [
|
||||
for (final gloss in sense.split(';'))
|
||||
if (gloss.trim().isNotEmpty) gloss.trim(),
|
||||
],
|
||||
},
|
||||
],
|
||||
if (classifiers.isNotEmpty) 'classifiers': classifiers,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
/// CC-CEDICT's cross-references (`個|个[ge4]`) as people read them: the
|
||||
/// characters once where both scripts write them alike, the pinyin with
|
||||
/// tone marks (`個|个 gè`).
|
||||
String _readableCrossReference(String text) => text.replaceAllMapped(
|
||||
RegExp(r'\[([^\]]+)\]'),
|
||||
(match) => ' ${pinyinWithToneMarks(match[1]!)}',
|
||||
);
|
||||
|
||||
/// CC-CEDICT's numbered pinyin ([numbered]: `xue2 xi2`, `nu:3 er2`) with
|
||||
/// tone marks (`xué xí`, `nǚ ér`): the mark on a or e, on the o of ou,
|
||||
/// else on the last vowel; the fifth (neutral) tone unmarked. What is no
|
||||
/// syllable (letters, digits, `·`) stays as it is.
|
||||
String pinyinWithToneMarks(String numbered) =>
|
||||
[for (final syllable in numbered.split(' ')) _marked(syllable)].join(' ');
|
||||
|
||||
String _marked(String syllable) {
|
||||
final match = RegExp(r'^([A-Za-z:]+)([1-5])$').firstMatch(syllable);
|
||||
if (match == null) {
|
||||
return syllable.replaceAll('u:', 'ü').replaceAll('U:', 'Ü');
|
||||
}
|
||||
final letters = match[1]!
|
||||
.replaceAll('u:', 'ü')
|
||||
.replaceAll('U:', 'Ü')
|
||||
.replaceAll('v', 'ü')
|
||||
.replaceAll('V', 'Ü');
|
||||
final tone = int.parse(match[2]!);
|
||||
if (tone == 5) {
|
||||
return letters;
|
||||
}
|
||||
final lower = letters.toLowerCase();
|
||||
final at = switch (lower) {
|
||||
_ when lower.contains('a') => lower.indexOf('a'),
|
||||
_ when lower.contains('e') => lower.indexOf('e'),
|
||||
_ when lower.contains('ou') => lower.indexOf('o'),
|
||||
_ => lower.lastIndexOf(RegExp('[iouü]')),
|
||||
};
|
||||
if (at < 0) {
|
||||
return letters;
|
||||
}
|
||||
final vowel = letters[at];
|
||||
final marks = _toneMarks[vowel.toLowerCase()]!;
|
||||
final markedVowel = marks[tone - 1];
|
||||
return letters.replaceRange(
|
||||
at,
|
||||
at + 1,
|
||||
vowel == vowel.toUpperCase() ? markedVowel.toUpperCase() : markedVowel,
|
||||
);
|
||||
}
|
||||
|
||||
const _toneMarks = {
|
||||
'a': ['ā', 'á', 'ǎ', 'à'],
|
||||
'e': ['ē', 'é', 'ě', 'è'],
|
||||
'i': ['ī', 'í', 'ǐ', 'ì'],
|
||||
'o': ['ō', 'ó', 'ǒ', 'ò'],
|
||||
'u': ['ū', 'ú', 'ǔ', 'ù'],
|
||||
'ü': ['ǖ', 'ǘ', 'ǚ', 'ǜ'],
|
||||
};
|
||||
|
||||
/// [pinyin], numbered or with tone marks, as it is typed without tones:
|
||||
/// letters alone, lower case, ü as u (`xué xí`, `xue2 xi2` → `xuexi`).
|
||||
String tonelessPinyin(String pinyin) {
|
||||
final out = StringBuffer();
|
||||
for (final char in pinyin.toLowerCase().split('')) {
|
||||
final bare = _toneless[char] ?? char;
|
||||
if (RegExp('[a-z]').hasMatch(bare)) {
|
||||
out.write(bare);
|
||||
}
|
||||
}
|
||||
return out.toString();
|
||||
}
|
||||
|
||||
/// Every vowel with a tone mark (and ü) by the letter it is typed as.
|
||||
final _toneless = {
|
||||
for (final MapEntry(key: vowel, value: marks) in _toneMarks.entries)
|
||||
for (final marked in [...marks, vowel]) marked: vowel == 'ü' ? 'u' : vowel,
|
||||
};
|
||||
|
||||
/// The HSK levels file in the curator's [data] folder.
|
||||
File hskFile(Directory data) => File('${data.path}/hsk/syllabus2025_words.tsv');
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/sources/cedict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jlpt.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
|
||||
|
|
@ -267,3 +268,83 @@ String _wiktionaryState(DateTime dumped, Set<String> languages) =>
|
|||
'dump': dumped.toUtc().toIso8601String(),
|
||||
'languages': [...languages]..sort(),
|
||||
})}\n';
|
||||
|
||||
/// CC-CEDICT, as MDBG publishes it every day.
|
||||
final cedictSource = Uri.parse(
|
||||
'https://www.mdbg.net/chinese/export/cedict/cedict_1_0_ts_utf-8_mdbg.txt.gz',
|
||||
);
|
||||
|
||||
/// Brings CC-CEDICT's two files in [into] (simplified and traditional
|
||||
/// characters) up to date from MDBG's file of the day, with the HSK levels
|
||||
/// and word frequencies in the curator's [data] folder; each written
|
||||
/// beside its place and renamed over it. Returns the files written.
|
||||
Future<List<File>> updateCedict({
|
||||
required Directory into,
|
||||
required Directory data,
|
||||
Fetch fetch = httpFetch,
|
||||
}) async {
|
||||
final downloads = await Directory.systemTemp.createTemp('cedict');
|
||||
try {
|
||||
final download = File('${downloads.path}/cedict.txt.gz');
|
||||
await fetch(cedictSource, download);
|
||||
final text = utf8.decode(gzip.decode(await download.readAsBytes()));
|
||||
final hsk = await readHskLevels(hskFile(data));
|
||||
final frequencies = File('${data.path}/wordfreq/small_zh.msgpack.gz');
|
||||
final words = frequencies.existsSync()
|
||||
? await WordFrequencies.read(frequencies)
|
||||
: WordFrequencies.none;
|
||||
final written = [
|
||||
for (final traditional in [false, true])
|
||||
await _write(
|
||||
File('${downloads.path}/${cedictFileName(traditional: traditional)}'),
|
||||
Stream.fromIterable(
|
||||
cedictDictionary(
|
||||
text,
|
||||
traditional: traditional,
|
||||
hsk: hsk,
|
||||
frequencies: words,
|
||||
),
|
||||
),
|
||||
),
|
||||
];
|
||||
await into.create(recursive: true);
|
||||
return [for (final file in written) await _putInPlace(file, into)];
|
||||
} finally {
|
||||
await downloads.delete(recursive: true);
|
||||
}
|
||||
}
|
||||
|
||||
/// When CC-CEDICT's files in [folder] were last brought up to date: the
|
||||
/// older of them; null when one is missing.
|
||||
DateTime? cedictUpdated(Directory folder) {
|
||||
DateTime? oldest;
|
||||
for (final traditional in [false, true]) {
|
||||
final file = File(
|
||||
'${folder.path}/${cedictFileName(traditional: traditional)}',
|
||||
);
|
||||
if (!file.existsSync()) {
|
||||
return null;
|
||||
}
|
||||
final modified = file.lastModifiedSync();
|
||||
if (oldest == null || modified.isBefore(oldest)) {
|
||||
oldest = modified;
|
||||
}
|
||||
}
|
||||
return oldest;
|
||||
}
|
||||
|
||||
/// Of the languages an instance keeps dictionaries of besides Japanese
|
||||
/// ([languages], BCP 47 tags), those Wiktionary gives, and whether Chinese
|
||||
/// (CC-CEDICT, in both scripts) is among them.
|
||||
({Set<String> wiktionary, bool chinese}) dictionarySources(
|
||||
Set<String> languages,
|
||||
) {
|
||||
bool isChinese(String tag) => tag.split('-').first.toLowerCase() == 'zh';
|
||||
return (
|
||||
wiktionary: {
|
||||
for (final language in languages)
|
||||
if (!isChinese(language)) language,
|
||||
},
|
||||
chinese: languages.any(isChinese),
|
||||
);
|
||||
}
|
||||
|
|
|
|||
133
halkyon_learn_curator/test/cedict_test.dart
Normal file
133
halkyon_learn_curator/test/cedict_test.dart
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/sources/cedict.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
void main() {
|
||||
late String sample;
|
||||
late HskLevels hsk;
|
||||
|
||||
setUpAll(() async {
|
||||
sample = await File('test/fixtures/cedict-sample.txt').readAsString();
|
||||
hsk = await readHskLevels(File('data/hsk/syllabus2025_words.tsv'));
|
||||
});
|
||||
|
||||
List<Map<String, Object?>> linesOf({required bool traditional}) => [
|
||||
for (final line in cedictDictionary(
|
||||
sample,
|
||||
traditional: traditional,
|
||||
hsk: hsk,
|
||||
))
|
||||
jsonDecode(line) as Map<String, Object?>,
|
||||
];
|
||||
|
||||
Map<String, Object?> entryOf(String id, {bool traditional = false}) =>
|
||||
linesOf(traditional: traditional)
|
||||
.skip(1)
|
||||
.singleWhere((line) => line['id'] == id);
|
||||
|
||||
test('a file in simplified characters and one in traditional ones, each '
|
||||
'a dictionary of its own', () {
|
||||
final simplified = linesOf(traditional: false).first;
|
||||
expect(simplified['source'], 'cc-cedict-hans');
|
||||
expect(simplified['language'], 'zh-Hans');
|
||||
expect(simplified['glossLanguage'], 'en');
|
||||
expect(simplified['sourceVersion'], '2026-10-13');
|
||||
expect(
|
||||
[
|
||||
for (final credit in simplified['credits']! as List)
|
||||
(credit as Map)['name'],
|
||||
],
|
||||
['CC-CEDICT', 'HSK 3.0 vocabulary (examination syllabus, 2025)'],
|
||||
);
|
||||
final traditional = linesOf(traditional: true).first;
|
||||
expect(traditional['source'], 'cc-cedict-hant');
|
||||
expect(traditional['language'], 'zh-Hant');
|
||||
expect(cedictFileName(traditional: true), 'cc-cedict-hant-en.jsonl.gz');
|
||||
});
|
||||
|
||||
test('a word: its characters in either script, its pinyin with tone '
|
||||
'marks, found typed without tones, in text by its characters', () {
|
||||
final xuexi = entryOf('學習 学习 [xue2 xi2]');
|
||||
expect(xuexi['headword'], '学习');
|
||||
expect(xuexi['reading'], 'xué xí');
|
||||
expect(
|
||||
(xuexi['keys']! as List).cast<String>(),
|
||||
containsAll([
|
||||
'学习',
|
||||
'學習',
|
||||
'xuexi',
|
||||
'xuéxí',
|
||||
'xue2xi2',
|
||||
'to learn',
|
||||
'learn',
|
||||
'study',
|
||||
]),
|
||||
);
|
||||
expect(xuexi['textKeys'], ['学习', '學習']);
|
||||
final entry = xuexi['entry']! as Map;
|
||||
expect(entry['forms'], [
|
||||
{'text': '學習'},
|
||||
]);
|
||||
expect(entry['senses'], [
|
||||
{
|
||||
'glosses': ['to learn', 'to study'],
|
||||
},
|
||||
]);
|
||||
expect(entryOf('學習 学习 [xue2 xi2]', traditional: true)['headword'], '學習');
|
||||
});
|
||||
|
||||
test('HSK levels go to the entry read as the syllabus reads the word: 的 '
|
||||
'de, not dì; 书 shū, not the Book of Documents, Shū', () {
|
||||
expect(entryOf('的 的 [de5]')['labels'], contains('hsk:1'));
|
||||
expect(entryOf('的 的 [di4]')['labels'], isNot(contains('hsk:1')));
|
||||
expect(entryOf('書 书 [shu1]')['labels'], contains('hsk:1'));
|
||||
expect(entryOf('書 书 [Shu1]')['labels'], isEmpty);
|
||||
expect(entryOf('中國 中国 [Zhong1 guo2]')['labels'], contains('hsk:1'));
|
||||
expect(entryOf('銀行 银行 [yin2 hang2]')['labels'], contains('hsk:3'));
|
||||
// HSK 1 before HSK 3, both before words of no level.
|
||||
expect(
|
||||
entryOf('的 的 [de5]')['rank']! as int,
|
||||
lessThan(entryOf('銀行 银行 [yin2 hang2]')['rank']! as int),
|
||||
);
|
||||
expect(
|
||||
entryOf('銀行 银行 [yin2 hang2]')['rank']! as int,
|
||||
lessThan(entryOf('的 的 [di4]')['rank']! as int),
|
||||
);
|
||||
});
|
||||
|
||||
test('measure words are a remark beside the meanings, pinyin marked', () {
|
||||
final book = entryOf('書 书 [shu1]')['entry']! as Map;
|
||||
expect(book['classifiers'], ['本 běn', '冊|册 cè', '部 bù']);
|
||||
expect(
|
||||
[
|
||||
for (final sense in book['senses']! as List)
|
||||
...(sense as Map)['glosses']! as List,
|
||||
],
|
||||
['book', 'letter', 'document', 'to write'],
|
||||
);
|
||||
});
|
||||
|
||||
group('pinyin', () {
|
||||
test('takes its tone marks where they belong', () {
|
||||
expect(pinyinWithToneMarks('xue2 xi2'), 'xué xí');
|
||||
expect(pinyinWithToneMarks('Zhong1 guo2'), 'Zhōng guó');
|
||||
expect(pinyinWithToneMarks('nu:3 er2'), 'nǚ ér');
|
||||
expect(pinyinWithToneMarks('lu:e4'), 'lüè');
|
||||
expect(
|
||||
pinyinWithToneMarks('shuo1 gui4 liu2 xiong2'),
|
||||
'shuō guì liú xióng',
|
||||
);
|
||||
expect(pinyinWithToneMarks('dou1 le5 r5'), 'dōu le r');
|
||||
expect(pinyinWithToneMarks('A A zhi4'), 'A A zhì');
|
||||
});
|
||||
|
||||
test('is typed without its tones', () {
|
||||
expect(tonelessPinyin('xué xí'), 'xuexi');
|
||||
expect(tonelessPinyin('nǚ’ér'), 'nuer');
|
||||
expect(tonelessPinyin('yìsī-bùgǒu'), 'yisibugou');
|
||||
expect(tonelessPinyin('nu:3 er2'), 'nuer');
|
||||
});
|
||||
});
|
||||
}
|
||||
|
|
@ -39,3 +39,8 @@ etymologies or related words; at most six senses, two examples, 40 forms, two
|
|||
sounds, and 14 translations, those into German and Greek). Wiktionary's text is
|
||||
by its contributors, CC BY-SA 4.0 (<https://en.wiktionary.org/wiki/Wiktionary:Copyrights>);
|
||||
the extraction is Tatu Ylonen's Wiktextract (<https://kaikki.org>).
|
||||
|
||||
`cedict-sample.txt` holds CC-CEDICT's header and 16 of its lines (MDBG's file of
|
||||
2026-10-13): 中国, 了, 女儿, 学习, 书, 熊猫, 略, 的, 说 and 银行, with their
|
||||
readings. CC-CEDICT is by MDBG and its contributors, CC BY-SA 4.0
|
||||
(<https://www.mdbg.net/chinese/dictionary?page=cc-cedict>).
|
||||
|
|
|
|||
46
halkyon_learn_curator/test/fixtures/cedict-sample.txt
vendored
Normal file
46
halkyon_learn_curator/test/fixtures/cedict-sample.txt
vendored
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
# CC-CEDICT
|
||||
# Community maintained free Chinese-English dictionary.
|
||||
#
|
||||
# Published by MDBG
|
||||
#
|
||||
# License:
|
||||
# Creative Commons Attribution-ShareAlike 4.0 International License
|
||||
# https://creativecommons.org/licenses/by-sa/4.0/
|
||||
#
|
||||
# Referenced works:
|
||||
# CEDICT - Copyright (C) 1997, 1998 Paul Andrew Denisowski
|
||||
#
|
||||
# CC-CEDICT can be downloaded from:
|
||||
# https://www.mdbg.net/chinese/dictionary?page=cc-cedict
|
||||
#
|
||||
# Additions and corrections can be sent through:
|
||||
# https://cc-cedict.org/editor/editor.php
|
||||
#
|
||||
# For more information about CC-CEDICT see:
|
||||
# https://cc-cedict.org/wiki/
|
||||
#
|
||||
#! version=1
|
||||
#! subversion=0
|
||||
#! format=ts
|
||||
#! charset=UTF-8
|
||||
#! entries=125264
|
||||
#! publisher=MDBG
|
||||
#! license=https://creativecommons.org/licenses/by-sa/4.0/
|
||||
#! date=2026-10-13T08:59:07Z
|
||||
#! time=1791881947
|
||||
中國 中国 [Zhong1 guo2] /China/
|
||||
了 了 [le5] /(completed action marker)/(modal particle indicating change of state, situation now)/(modal particle intensifying preceding clause)/
|
||||
了 了 [liao3] /to finish/(used with 得[de2] or 不[bu4] after a verb to express (im)possibility, as in 忘不了[wang4 bu5 liao3] "cannot forget")/(literary) (usually followed by a negative such as 無|无[wu2] or 不[bu4]) completely (not); entirely (not); (not) in the least/variant of 瞭|了[liao3]/
|
||||
女兒 女儿 [nu:3 er2] /daughter/
|
||||
學習 学习 [xue2 xi2] /to learn; to study/
|
||||
書 书 [Shu1] /abbr. for 書經|书经[Shu1 jing1]/
|
||||
書 书 [shu1] /book/letter/document/CL:本[ben3],冊|册[ce4],部[bu4]/to write/
|
||||
熊貓 熊猫 [xiong2 mao1] /panda/CL:隻|只[zhi1]/
|
||||
略 略 [lu:e4] /brief; sketchy/outline; summary/to omit/(bound form before a single-character verb) a bit; somewhat; slightly/plan; strategy/to capture (territory)/
|
||||
的 的 [de5] /of; ~'s (possessive particle)/(used after an attribute when it modifies a noun)/(used at the end of a declarative sentence for emphasis)/(used after a noun, verb or adjective to form a nominal expression, as in 皮革的[pi2 ge2 de5] "one made of leather" or 跑堂兒的|跑堂儿的[pao3 tang2 r5 de5] "a waiter (literally, one who runs back and forth in a restaurant)" or 新的[xin1 de5] "new one")/also pr. [di4] or [di5] in poetry and songs/
|
||||
的 的 [di1] /(bound form) a taxi; a cab (abbr. for 的士[di1 shi4])/
|
||||
的 的 [di2] /really and truly/
|
||||
的 的 [di4] /(bound form) bull's-eye; target/
|
||||
說 说 [shui4] /to persuade/
|
||||
說 说 [shuo1] /to speak; to talk; to say/to explain; to comment/to scold; to tell off/(bound form) theory; doctrine/
|
||||
銀行 银行 [yin2 hang2] /bank/CL:家[jia1],個|个[ge4]/
|
||||
|
|
@ -178,4 +178,39 @@ void main() {
|
|||
expect(opened, 0);
|
||||
});
|
||||
});
|
||||
|
||||
test('CC-CEDICT is converted in both scripts with the HSK levels, and '
|
||||
'said to be up to date by its files', () async {
|
||||
expect(cedictUpdated(into), isNull);
|
||||
|
||||
final written = await updateCedict(
|
||||
into: into,
|
||||
data: Directory('data'),
|
||||
fetch: (uri, file) async {
|
||||
expect(uri, cedictSource);
|
||||
await file.writeAsBytes(
|
||||
gzip.encode(
|
||||
File('test/fixtures/cedict-sample.txt').readAsBytesSync(),
|
||||
),
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
expect(
|
||||
[for (final file in written) file.uri.pathSegments.last],
|
||||
['cc-cedict-hans-en.jsonl.gz', 'cc-cedict-hant-en.jsonl.gz'],
|
||||
);
|
||||
expect(header('cc-cedict-hant-en.jsonl.gz')['language'], 'zh-Hant');
|
||||
expect(cedictUpdated(into), isNotNull);
|
||||
});
|
||||
|
||||
test('Chinese comes from CC-CEDICT, the other languages from Wiktionary', () {
|
||||
final all = dictionarySources({'de', 'el', 'en', 'zh'});
|
||||
expect(all.wiktionary, {'de', 'el', 'en'});
|
||||
expect(all.chinese, isTrue);
|
||||
final traditional = dictionarySources({'zh-Hant'});
|
||||
expect(traditional.wiktionary, isEmpty);
|
||||
expect(traditional.chinese, isTrue);
|
||||
expect(dictionarySources({}).chinese, isFalse);
|
||||
});
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue