feat(curator): characters of Chinese and Korean, the Greek alphabet, Korean words from KRDict
For parity with Japanese's KANJIDIC2 and kana: - Unihan (Unicode License v3) makes characters' dictionaries: the 8,105 standard simplified characters (unihan-hans) and the traditional ones in use (unihan-hant), told apart and ranked by CC-CEDICT's words (Unihan calls 这 its own traditional variant; 後 ranks as the 后 of 以后), with pinyin, meanings, strokes, Kangxi radical, HSK 2025 character levels, the 3,500 common ones and Hong Kong's grades; and Korean Hanja (unihan-ko) with their sounds, the 1,800 of education, and their native readings from Wiktionary (물 수), with the 40 Hangul letters as kana are. - The Greek alphabet, the curator's own (CC0): letters with names, transliteration and sound. - KRDict, the Basic Korean Dictionary for learners (CC BY-SA 2.0 KR, downloaded without an account), replaces Wiktionary for Korean's words: learners' levels, pronunciation where it differs from the spelling (같이 [가치]), Hanja, English senses; Wiktionary's update keeps the words' forms (가다's 99) and the Hanja's readings beside it for both. - The update converts KRDict when its page offers another export and the characters weekly, and removes the files of languages no longer kept or of sources replaced. Measured on the real data: Unihan 3 s; KRDict 54,870 entries in 24 s and under 600 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
4e6581b9a5
commit
8820cefde3
GPG key ID:
07F6FCF405488B64
22 changed files with 6997 additions and 85 deletions
|
|
@ -110,7 +110,7 @@ services:
|
|||
# Docker's stop signal reaches the curator through a small init.
|
||||
init: true
|
||||
# JMdict's conversion takes some 400 MB for a minute a week, Wiktionary's some
|
||||
# 300 MB for ten minutes a month.
|
||||
# 300 MB for ten minutes a month, KRDict's some 600 MB for half a minute.
|
||||
mem_limit: 1g
|
||||
# Unset: German, Greek, English, Korean and Chinese; empty: none.
|
||||
command: ["--every", "7", "--languages", "${DICTIONARY_LANGUAGES-de,el,en,ko,zh}"]
|
||||
|
|
|
|||
|
|
@ -49,8 +49,8 @@ KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:
|
|||
|
||||
About 12,800 characters, in some 5 seconds.
|
||||
|
||||
Wiktionary, for the languages other than Japanese and Chinese (decided
|
||||
2026-10-14: German, Greek and English first, then Chinese, then Korean):
|
||||
Wiktionary, for the words of German, Greek and English (decided 2026-10-14),
|
||||
and for Korean the forms of its words and the native readings of its Hanja:
|
||||
|
||||
1. Download `raw-wiktextract-data.jsonl.gz` from
|
||||
[kaikki.org](https://kaikki.org/dictionary/rawdata.html): the English
|
||||
|
|
@ -65,8 +65,7 @@ Wiktionary, for the languages other than Japanese and Chinese (decided
|
|||
```
|
||||
|
||||
Each language but English becomes its words with Wiktionary's English
|
||||
meanings (`wiktionary-de-en.jsonl.gz`, source `wiktionary-de`; a Korean word
|
||||
with its Hanja and romanisation); English
|
||||
meanings (`wiktionary-de-en.jsonl.gz`, source `wiktionary-de`); English
|
||||
becomes its words with what their translation tables give in German
|
||||
(`wiktionary-en-de.jsonl.gz`), German being the app's other language. An
|
||||
inflected form, and an entry Wiktionary has for one alone ("Häusern: dative
|
||||
|
|
@ -102,6 +101,76 @@ forms):
|
|||
apart, each given to the entry read as the syllabus reads the word (的 *de*,
|
||||
not 的 *dì*), and rank them, HSK 1 first; wordfreq ranks the rest.
|
||||
|
||||
Chinese characters and Korean Hanja from Unihan, the Unicode Consortium's
|
||||
database of Han characters (decided 2026-10-14, for parity with KANJIDIC2;
|
||||
Unicode License v3):
|
||||
|
||||
1. Download `Unihan.zip` from
|
||||
[unicode.org](https://www.unicode.org/Public/UCD/latest/ucd/Unihan.zip)
|
||||
(8 MB, a new one with each version of Unicode), and CC-CEDICT as above.
|
||||
2. Convert, naming the languages:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate unihan --input Unihan.zip \
|
||||
--languages ko,zh --cedict cedict_1_0_ts_utf-8_mdbg.txt.gz \
|
||||
[--korean .wiktionary-ko.json.gz] \
|
||||
--output ../halkyon_learn_server/dictionaries
|
||||
```
|
||||
|
||||
For Chinese, two dictionaries of characters as for the words: the 8,105
|
||||
General Standard Chinese Characters of 2013 (`unihan-hans-en`, `zh-Hans`),
|
||||
and the traditional characters in use (`unihan-hant-en`, `zh-Hant`): each of
|
||||
them as Taiwan and Hong Kong write it, where it is a form of its own or a
|
||||
word of CC-CEDICT writes it so (Unihan calls 这 its own traditional variant,
|
||||
but no word is written with it), with those Hong Kong's primary schools
|
||||
teach. Each with its pinyin (the usual reading first: the mainland's in
|
||||
simplified, Taiwan's in traditional characters), meanings, stroke count and
|
||||
other script's forms; labelled by its HSK level (`hsk:1` … `hsk:7`, from the
|
||||
syllabus's characters in `data/hsk/`), the 3,500 standard characters in
|
||||
common use (`common`), a Hong Kong grade (`grade:1` … `grade:6`) and its
|
||||
Kangxi radical (`radical:N`), and ranked by HSK level, then by the most used
|
||||
word of CC-CEDICT written with it (wordfreq), so 後 ranks as the 后 of 以后
|
||||
and 后, the queen, by 皇后. For Korean, the 7,800 Hanja with a Korean reading
|
||||
(`unihan-ko-en`): their sounds (음, the one taught first), their native
|
||||
meanings with their sounds as Wiktionary gives them (훈, `물 수` for 水; from
|
||||
the Wiktionary update's `.wiktionary-ko.json.gz`), meanings and strokes,
|
||||
labelled `education` for the 1,800 taught in schools; and the 40 Hangul
|
||||
letters (jamo) as the kana are in KANJIDIC2's, with their sound and name,
|
||||
labelled `hangul`. Some 3 seconds.
|
||||
|
||||
The Greek alphabet, written by the curator itself (`greek-alphabet-en`, `el`,
|
||||
CC0): each letter with its capital, its Greek and English names, its
|
||||
transliteration (ELOT 743) and its sound, labelled `greek`, for a characters
|
||||
tab as kana have.
|
||||
|
||||
Korean words from KRDict, the National Institute of Korean Language's Basic
|
||||
Korean Dictionary for learners (decided 2026-10-14: levels, pronunciation and
|
||||
Hanja, which Wiktionary lacks; CC BY-SA 2.0 KR, later versions allowed):
|
||||
|
||||
1. Download its JSON export from
|
||||
[its page](https://krdict.korean.go.kr/download/downloadPopup) (Json 전체
|
||||
내려받기; no account needed; some 84 MB, a gigabyte unpacked, renewed about
|
||||
monthly).
|
||||
2. Convert:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate krdict \
|
||||
--input krdict_json_20260919.zip [--korean .wiktionary-ko.json.gz] \
|
||||
--output ../halkyon_learn_server/dictionaries
|
||||
```
|
||||
|
||||
`krdict-ko-en.jsonl.gz` holds its 54,870 words, idioms and proverbs, each
|
||||
with its part of speech, its pronunciation as its reading where it differs
|
||||
from the spelling (같이 [가치]), its Hanja, the few forms KRDict gives and
|
||||
its senses (KRDict's English words, its English definition as `info`);
|
||||
labelled by its learners' level (`level:beginner`, `level:intermediate`,
|
||||
`level:advanced`) and ranked by it, then by wordfreq (a verb, rarely written
|
||||
as the dictionary writes it, by its most used form). With the Wiktionary
|
||||
update's file, every form Wiktionary's tables give (가다's 99) finds a word in
|
||||
running text too. An idiom or proverb carries the id of the word it is filed
|
||||
under, so its own id adds itself (`27500:가는 날이 장날`). Read an entry at a
|
||||
time, some 25 seconds and 600 MB.
|
||||
|
||||
Example sentences from Tatoeba for those dictionaries (JMdict brings its own),
|
||||
as hints on the cards made from them:
|
||||
|
||||
|
|
@ -149,15 +218,23 @@ from the same download (a label, a field), and an instance converts again as
|
|||
soon as the new curator runs, not a week later.
|
||||
|
||||
With `--languages de,el,en,ko,zh` it also keeps those languages current, each
|
||||
file with Tatoeba's sentences of the week as examples (a dictionary whose
|
||||
sentences can't be had keeps its words without them): Chinese from MDBG's
|
||||
CC-CEDICT of the day as often as EDRDG's files (below), the rest from
|
||||
Wiktionary: each time it wakes it asks kaikki.org when its dump last changed
|
||||
(`Last-Modified`, without downloading it), and converts it again only when it is
|
||||
newer than the files' (about monthly), when the languages named changed, or when
|
||||
`wiktionaryConversion` in `lib/src/update.dart` was raised; `.wiktionary` beside
|
||||
the files says which. The dump is converted as it downloads, never stored.
|
||||
Files of a language no longer named stay until removed by hand.
|
||||
file of words with Tatoeba's sentences of the week as examples (a dictionary
|
||||
whose sentences can't be had keeps its words without them), in this order:
|
||||
Chinese from MDBG's CC-CEDICT of the day as often as EDRDG's files; the words of
|
||||
the rest from Wiktionary: each time it wakes it asks kaikki.org when its dump
|
||||
last changed (`Last-Modified`, without downloading it), and converts it again
|
||||
only when it is newer than the files' (about monthly), when the languages named
|
||||
changed, or when `wiktionaryConversion` in `lib/src/update.dart` was raised;
|
||||
`.wiktionary` beside the files says which (for Korean it keeps
|
||||
`.wiktionary-ko.json.gz`, the forms and Hanja readings). The dump is converted
|
||||
as it downloads, never stored. Korean's words from KRDict: each time it wakes
|
||||
it reads KRDict's page, and converts again only when it offers another export,
|
||||
when Wiktionary's Korean file changed, or when `krdictConversion` was raised
|
||||
(`.krdict`). Then the characters (Unihan for Chinese and Korean, with
|
||||
CC-CEDICT's words; the Greek alphabet) as often as EDRDG's files, and at once
|
||||
when Wiktionary's Korean file is newer than the Hanja's. Files the curator wrote
|
||||
for a language no longer named, or for a source replaced (Korean's words from
|
||||
Wiktionary, before KRDict), are removed, and the server unloads them.
|
||||
|
||||
## The format (`halkyon-dictionary`, version 1)
|
||||
|
||||
|
|
@ -180,12 +257,12 @@ The **first line** says what the file holds:
|
|||
|---|---|
|
||||
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
|
||||
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
|
||||
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary and CC-CEDICT the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1`, `hsk:1` … `hsk:7` (later e.g. `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
|
||||
| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary, CC-CEDICT and KRDict the 15,000 words wordfreq finds most used; for Chinese characters the 3,500 standard ones in common use), `jlpt:n5` … `jlpt:n1`, `hsk:1` … `hsk:7`, KRDict's learners' levels (`level:beginner`, `level:intermediate`, `level:advanced`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji; Hong Kong's for traditional Chinese), Korea's 1,800 Hanja of education (`education`), a character's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one, Unihan's `kRSUnicode`) and the script (`hiragana`, `katakana`, `hangul`, `greek`). |
|
||||
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
|
||||
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
||||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
|
||||
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |
|
||||
|
||||
## The app's stroke files
|
||||
|
||||
|
|
|
|||
|
|
@ -9,7 +9,9 @@ import 'package:halkyon_learn_curator/src/sources/jmdict.dart';
|
|||
import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/krdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/update.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
|
|
@ -42,6 +44,22 @@ import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
|||
/// --input cedict_1_0_ts_utf-8_mdbg.txt.gz \
|
||||
/// --output ../halkyon_learn_server/dictionaries
|
||||
///
|
||||
/// Unihan into dictionary files of Chinese characters (simplified and
|
||||
/// traditional) and of Korean Hanja, for the languages named, the Hanja's
|
||||
/// native readings from a Wiktionary update's `.wiktionary-ko.json.gz`:
|
||||
///
|
||||
/// dart run halkyon_learn_curator:curate unihan \
|
||||
/// --input Unihan.zip --languages ko,zh [--cedict <file>] \
|
||||
/// [--korean <file>] \
|
||||
/// --output ../halkyon_learn_server/dictionaries
|
||||
///
|
||||
/// KRDict's JSON export into the dictionary file of Korean words, with
|
||||
/// the forms of each a Wiktionary update kept (`.wiktionary-ko.json.gz`):
|
||||
///
|
||||
/// dart run halkyon_learn_curator:curate krdict \
|
||||
/// --input krdict_json_20260919.zip [--korean <file>] \
|
||||
/// --output ../halkyon_learn_server/dictionaries
|
||||
///
|
||||
/// Tatoeba's sentences as examples into a dictionary file of words (its
|
||||
/// exports of the dictionary's language, of its meaning language, and of
|
||||
/// the links between them, bzip2 or not):
|
||||
|
|
@ -119,6 +137,19 @@ Future<void> main(List<String> args) async {
|
|||
'links',
|
||||
help: "tatoeba: Tatoeba's links between the two languages.",
|
||||
)
|
||||
..addOption(
|
||||
'cedict',
|
||||
help:
|
||||
'unihan: CC-CEDICT, whose words tell traditional characters in use '
|
||||
'and rank the characters.',
|
||||
)
|
||||
..addOption(
|
||||
'korean',
|
||||
help:
|
||||
'unihan, krdict: what a Wiktionary update kept for Korean '
|
||||
"(.wiktionary-ko.json.gz: the Hanja's native readings, the "
|
||||
"words' forms).",
|
||||
)
|
||||
..addOption(
|
||||
'version',
|
||||
help:
|
||||
|
|
@ -243,6 +274,77 @@ Future<void> main(List<String> args) async {
|
|||
);
|
||||
return;
|
||||
}
|
||||
if (source == 'krdict') {
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
if (input == null || output == null) {
|
||||
_fail(
|
||||
'Usage: curate krdict --input <zip> --output <folder> '
|
||||
'[--korean <file>] [--data <folder>]\n${parser.usage}',
|
||||
);
|
||||
}
|
||||
final korean = options.option('korean');
|
||||
final frequencies = File(
|
||||
'${options.option('data')}/wordfreq/small_ko.msgpack.gz',
|
||||
);
|
||||
final folder = await Directory(output).create(recursive: true);
|
||||
final (:version, :entries) = krdictExport(await File(input).readAsBytes());
|
||||
await krdictDictionary(
|
||||
entries,
|
||||
sourceVersion: version,
|
||||
frequencies: frequencies.existsSync()
|
||||
? await WordFrequencies.read(frequencies)
|
||||
: null,
|
||||
korean: korean == null
|
||||
? null
|
||||
: WiktionaryKorean.fromJson(
|
||||
utf8.decode(gzip.decode(await File(korean).readAsBytes())),
|
||||
),
|
||||
)
|
||||
.transform(utf8.encoder)
|
||||
.transform(gzip.encoder)
|
||||
.pipe(File('${folder.path}/$krdictFileName').openWrite());
|
||||
stdout.writeln('Wrote $krdictFileName to $output.');
|
||||
return;
|
||||
}
|
||||
if (source == 'unihan') {
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
final languages = _languages(options.option('languages'));
|
||||
if (input == null || output == null || languages.isEmpty) {
|
||||
_fail(
|
||||
'Usage: curate unihan --input <Unihan.zip> --languages <languages> '
|
||||
'--output <folder> [--cedict <file>] [--korean <file>] '
|
||||
'[--data <folder>]'
|
||||
'\n${parser.usage}',
|
||||
);
|
||||
}
|
||||
final korean = options.option('korean');
|
||||
final cedict = options.option('cedict');
|
||||
final written = await unihanDictionaries(
|
||||
await File(input).readAsBytes(),
|
||||
into: await Directory(output).create(recursive: true),
|
||||
languages: languages,
|
||||
data: Directory(options.option('data')!),
|
||||
cedict: cedict == null
|
||||
? null
|
||||
: await (cedict.endsWith('.gz')
|
||||
? File(cedict).openRead().transform(gzip.decoder)
|
||||
: File(cedict).openRead())
|
||||
.transform(utf8.decoder)
|
||||
.join(),
|
||||
eumhun: korean == null
|
||||
? const {}
|
||||
: WiktionaryKorean.fromJson(
|
||||
utf8.decode(gzip.decode(await File(korean).readAsBytes())),
|
||||
).hanja,
|
||||
);
|
||||
stdout.writeln(
|
||||
'Wrote ${[for (final file in written) file.uri.pathSegments.last].join(', ')} '
|
||||
'to $output.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (source == 'wiktionary') {
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
|
|
@ -354,66 +456,96 @@ Future<void> main(List<String> args) async {
|
|||
/// updates them whenever they are that many days old (as the files' dates
|
||||
/// say, so a restart doesn't download them again), trying again after six
|
||||
/// hours when an update fails. Of [languages], Chinese comes from
|
||||
/// CC-CEDICT as often as EDRDG's files, the rest from Wiktionary, looked at
|
||||
/// each time it wakes and converted again only when the dump is newer.
|
||||
/// CC-CEDICT as often as EDRDG's files, and its characters from Unihan;
|
||||
/// Korean from KRDict, looked at each time it wakes and converted again
|
||||
/// only when it offers another export, and its Hanja from Unihan; the rest
|
||||
/// from Wiktionary, looked at each time it wakes and converted again only
|
||||
/// when the dump is newer (Korean's forms and the Hanja's readings come
|
||||
/// from it too); Greek's alphabet with the characters. Files of languages
|
||||
/// no longer kept go.
|
||||
Future<void> _update(
|
||||
Directory into,
|
||||
Directory data,
|
||||
int? every, {
|
||||
Set<String> languages = const {},
|
||||
}) async {
|
||||
final (:wiktionary, :chinese) = dictionarySources(languages);
|
||||
final (:wiktionary, :chinese, :korean, :greek) = dictionarySources(languages);
|
||||
void say(String message) =>
|
||||
stdout.writeln('${DateTime.now().toUtc().toIso8601String()} $message');
|
||||
void written(List<File> files) => say(
|
||||
'Updated ${[for (final file in files) file.uri.pathSegments.last].join(', ')}.',
|
||||
);
|
||||
Future<bool> once() async {
|
||||
try {
|
||||
written(await updateDictionaries(into: into, data: data));
|
||||
return true;
|
||||
} on Object catch (error) {
|
||||
say('Updating the dictionaries failed: $error');
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
Future<bool> wiktionaryOnce() async {
|
||||
try {
|
||||
final files = await updateWiktionary(
|
||||
into: into,
|
||||
data: data,
|
||||
languages: wiktionary,
|
||||
void written(List<File> files) {
|
||||
if (files.isNotEmpty) {
|
||||
say(
|
||||
'Updated ${[for (final file in files) file.uri.pathSegments.last].join(', ')}.',
|
||||
);
|
||||
if (files.isNotEmpty) {
|
||||
written(files);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs [step], saying what it wrote; false when it failed.
|
||||
Future<bool> attempt(String name, Future<List<File>> Function() step) async {
|
||||
try {
|
||||
written(await step());
|
||||
return true;
|
||||
} on Object catch (error) {
|
||||
say('Updating Wiktionary failed: $error');
|
||||
say('Updating $name failed: $error');
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// CC-CEDICT once its files are as old as EDRDG's may be.
|
||||
Future<bool> chineseOnce() async {
|
||||
final updated = cedictUpdated(into);
|
||||
if (!chinese ||
|
||||
updated != null &&
|
||||
DateTime.now().difference(updated) < Duration(days: every ?? 1)) {
|
||||
return true;
|
||||
}
|
||||
try {
|
||||
written(await updateCedict(into: into, data: data));
|
||||
return true;
|
||||
} on Object catch (error) {
|
||||
say('Updating CC-CEDICT failed: $error');
|
||||
return false;
|
||||
Future<bool> once() => attempt(
|
||||
'the dictionaries',
|
||||
() => updateDictionaries(into: into, data: data),
|
||||
);
|
||||
|
||||
// Files as old as EDRDG's may be: CC-CEDICT's, and the characters'.
|
||||
bool due(DateTime? updated) =>
|
||||
updated == null ||
|
||||
DateTime.now().difference(updated) >= Duration(days: every ?? 1);
|
||||
Future<bool> chineseOnce() async =>
|
||||
!chinese ||
|
||||
!due(cedictUpdated(into)) ||
|
||||
await attempt('CC-CEDICT', () => updateCedict(into: into, data: data));
|
||||
Future<bool> wiktionaryOnce() => attempt(
|
||||
'Wiktionary',
|
||||
() => updateWiktionary(
|
||||
into: into,
|
||||
data: data,
|
||||
languages: wiktionary,
|
||||
korean: korean,
|
||||
),
|
||||
);
|
||||
Future<bool> koreanOnce() async =>
|
||||
!korean ||
|
||||
await attempt('KRDict', () => updateKrdict(into: into, data: data));
|
||||
Future<bool> charactersOnce() async =>
|
||||
!due(
|
||||
charactersUpdated(into, chinese: chinese, korean: korean, greek: greek),
|
||||
) ||
|
||||
await attempt(
|
||||
'the characters',
|
||||
() => updateCharacters(
|
||||
into: into,
|
||||
data: data,
|
||||
chinese: chinese,
|
||||
korean: korean,
|
||||
greek: greek,
|
||||
),
|
||||
);
|
||||
// In this order: Korean's words and Hanja take what Wiktionary kept.
|
||||
Future<bool> languagesOnce() async {
|
||||
final removed = removeUnwanted(into, languages);
|
||||
if (removed.isNotEmpty) {
|
||||
say(
|
||||
'Removed ${[for (final file in removed) file.uri.pathSegments.last].join(', ')}.',
|
||||
);
|
||||
}
|
||||
return !(!await chineseOnce() |
|
||||
!await wiktionaryOnce() |
|
||||
!await koreanOnce() |
|
||||
!await charactersOnce());
|
||||
}
|
||||
|
||||
if (every == null) {
|
||||
if (!await once() || !await chineseOnce() || !await wiktionaryOnce()) {
|
||||
if (!await once() || !await languagesOnce()) {
|
||||
exit(1);
|
||||
}
|
||||
return;
|
||||
|
|
@ -427,7 +559,7 @@ Future<void> _update(
|
|||
}
|
||||
continue;
|
||||
}
|
||||
final retry = !await chineseOnce() | !await wiktionaryOnce();
|
||||
final retry = !await languagesOnce();
|
||||
final next = retry && due.difference(now) > const Duration(hours: 6)
|
||||
? DateTime.now().add(const Duration(hours: 6))
|
||||
: due;
|
||||
|
|
|
|||
|
|
@ -39,3 +39,8 @@ Only each word's level reaches the dictionaries (`hsk:N`), with the pinyin to
|
|||
tell CC-CEDICT's readings of one word apart; the curator credits both. The
|
||||
Ministry claims copyright on the standard's documents; a word's level is a fact
|
||||
about it, which the dictionaries keep with attribution (decided 2026-10-14).
|
||||
`hsk/syllabus2025_chars.tsv` lists the syllabus's 3,088 characters to recognise
|
||||
(认读字) with their levels (246, 125, 284, 441, 431 and 413 for levels 1 to 6,
|
||||
1,148 for 7 to 9), from the same repository and commit (`corpus/syllabus2025/`),
|
||||
MIT, unchanged; they label the Chinese characters (`hsk:N`) as the words' label
|
||||
the words.
|
||||
|
|
|
|||
3089
halkyon_learn_curator/data/hsk/syllabus2025_chars.tsv
Normal file
3089
halkyon_learn_curator/data/hsk/syllabus2025_chars.tsv
Normal file
File diff suppressed because it is too large
Load diff
154
halkyon_learn_curator/lib/src/sources/alphabets.dart
Normal file
154
halkyon_learn_curator/lib/src/sources/alphabets.dart
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
import 'package:halkyon_learn_curator/src/dictionary_file.dart';
|
||||
import 'package:halkyon_learn_curator/src/fold.dart';
|
||||
|
||||
/// The Hangul letters (jamo) in the order Korean schools teach them: the
|
||||
/// fourteen consonants, the five doubled, the ten vowels and the eleven
|
||||
/// made of them; each with its sound in the Revised Romanization (where it
|
||||
/// opens a syllable, then where it closes one), and a consonant's name.
|
||||
const _jamo = [
|
||||
('ㄱ', 'g, k', '기역'),
|
||||
('ㄴ', 'n', '니은'),
|
||||
('ㄷ', 'd, t', '디귿'),
|
||||
('ㄹ', 'r, l', '리을'),
|
||||
('ㅁ', 'm', '미음'),
|
||||
('ㅂ', 'b, p', '비읍'),
|
||||
('ㅅ', 's, t', '시옷'),
|
||||
('ㅇ', '–, ng', '이응'),
|
||||
('ㅈ', 'j, t', '지읒'),
|
||||
('ㅊ', 'ch, t', '치읓'),
|
||||
('ㅋ', 'k', '키읔'),
|
||||
('ㅌ', 't', '티읕'),
|
||||
('ㅍ', 'p', '피읖'),
|
||||
('ㅎ', 'h, t', '히읗'),
|
||||
('ㄲ', 'kk, k', '쌍기역'),
|
||||
('ㄸ', 'tt', '쌍디귿'),
|
||||
('ㅃ', 'pp', '쌍비읍'),
|
||||
('ㅆ', 'ss, t', '쌍시옷'),
|
||||
('ㅉ', 'jj', '쌍지읒'),
|
||||
('ㅏ', 'a', '아'),
|
||||
('ㅑ', 'ya', '야'),
|
||||
('ㅓ', 'eo', '어'),
|
||||
('ㅕ', 'yeo', '여'),
|
||||
('ㅗ', 'o', '오'),
|
||||
('ㅛ', 'yo', '요'),
|
||||
('ㅜ', 'u', '우'),
|
||||
('ㅠ', 'yu', '유'),
|
||||
('ㅡ', 'eu', '으'),
|
||||
('ㅣ', 'i', '이'),
|
||||
('ㅐ', 'ae', '애'),
|
||||
('ㅒ', 'yae', '얘'),
|
||||
('ㅔ', 'e', '에'),
|
||||
('ㅖ', 'ye', '예'),
|
||||
('ㅘ', 'wa', '와'),
|
||||
('ㅙ', 'wae', '왜'),
|
||||
('ㅚ', 'oe', '외'),
|
||||
('ㅝ', 'wo', '워'),
|
||||
('ㅞ', 'we', '웨'),
|
||||
('ㅟ', 'wi', '위'),
|
||||
('ㅢ', 'ui', '의'),
|
||||
];
|
||||
|
||||
/// The Hangul letters as entries of the Korean characters' dictionary, as
|
||||
/// the kana are of the Japanese one: each with its sound (`romaji`, so a
|
||||
/// card made of it is studied as a kana's is) and its name, found by
|
||||
/// either, labelled `hangul` and ranked before every Hanja, in the order
|
||||
/// schools teach them.
|
||||
List<DictionaryEntry> hangulEntries() => [
|
||||
for (final (place, (letter, sound, name)) in _jamo.indexed)
|
||||
DictionaryEntry(
|
||||
id: letter,
|
||||
headword: letter,
|
||||
reading: null,
|
||||
labels: const ['hangul'],
|
||||
rank: place,
|
||||
keys: {
|
||||
letter,
|
||||
name,
|
||||
for (final part in sound.split(', '))
|
||||
if (part != '–') part,
|
||||
}.toList(),
|
||||
entry: {'romaji': sound, 'script': 'hangul', 'name': name},
|
||||
),
|
||||
];
|
||||
|
||||
/// The Greek alphabet in its order: each small letter with its capital
|
||||
/// (and final sigma), its Greek and English names, its transliteration
|
||||
/// (ELOT 743, as Greece's passports write names) and its sound in Modern
|
||||
/// Greek (IPA; γ and χ are softer before e and i sounds).
|
||||
const _greek = [
|
||||
('α', 'Α', 'άλφα', 'alpha', 'a', 'a'),
|
||||
('β', 'Β', 'βήτα', 'beta', 'v', 'v'),
|
||||
('γ', 'Γ', 'γάμμα', 'gamma', 'g', 'ɣ, ʝ'),
|
||||
('δ', 'Δ', 'δέλτα', 'delta', 'd', 'ð'),
|
||||
('ε', 'Ε', 'έψιλον', 'epsilon', 'e', 'e'),
|
||||
('ζ', 'Ζ', 'ζήτα', 'zeta', 'z', 'z'),
|
||||
('η', 'Η', 'ήτα', 'eta', 'i', 'i'),
|
||||
('θ', 'Θ', 'θήτα', 'theta', 'th', 'θ'),
|
||||
('ι', 'Ι', 'γιώτα', 'iota', 'i', 'i'),
|
||||
('κ', 'Κ', 'κάπα', 'kappa', 'k', 'k, c'),
|
||||
('λ', 'Λ', 'λάμδα', 'lambda', 'l', 'l'),
|
||||
('μ', 'Μ', 'μι', 'mu', 'm', 'm'),
|
||||
('ν', 'Ν', 'νι', 'nu', 'n', 'n'),
|
||||
('ξ', 'Ξ', 'ξι', 'xi', 'x', 'ks'),
|
||||
('ο', 'Ο', 'όμικρον', 'omicron', 'o', 'o'),
|
||||
('π', 'Π', 'πι', 'pi', 'p', 'p'),
|
||||
('ρ', 'Ρ', 'ρο', 'rho', 'r', 'r'),
|
||||
('σ', 'Σ', 'σίγμα', 'sigma', 's', 's'),
|
||||
('τ', 'Τ', 'ταυ', 'tau', 't', 't'),
|
||||
('υ', 'Υ', 'ύψιλον', 'upsilon', 'y', 'i'),
|
||||
('φ', 'Φ', 'φι', 'phi', 'f', 'f'),
|
||||
('χ', 'Χ', 'χι', 'chi', 'ch', 'x, ç'),
|
||||
('ψ', 'Ψ', 'ψι', 'psi', 'ps', 'ps'),
|
||||
('ω', 'Ω', 'ωμέγα', 'omega', 'o', 'o'),
|
||||
];
|
||||
|
||||
/// Who wrote the alphabets down: facts of the scripts, the curator's own.
|
||||
const alphabetCredit = Credit(
|
||||
name: 'Alphabets (Hangul, Greek)',
|
||||
by: 'Halkyon Learn contributors',
|
||||
license: 'CC0-1.0',
|
||||
url: 'https://code.halkyon.app/project-halkyon/halkyon_learn',
|
||||
);
|
||||
|
||||
/// The name of the dictionary file of the Greek alphabet.
|
||||
const greekAlphabetFileName = 'greek-alphabet-en.jsonl.gz';
|
||||
|
||||
/// The Greek alphabet as the lines of a dictionary file of Greek
|
||||
/// characters: each letter with its capital (`Α α`; sigma with its final
|
||||
/// form, `Σ σ ς`), its transliteration (`romaji`, so a card made of it is
|
||||
/// studied as a kana's is), its names and its sound, found by any of them
|
||||
/// and labelled `greek`, in the alphabet's order.
|
||||
Iterable<String> greekAlphabet() sync* {
|
||||
yield headerLine(
|
||||
const DictionaryHeader(
|
||||
source: 'greek-alphabet',
|
||||
kind: DictionaryKind.characters,
|
||||
sourceVersion: '1',
|
||||
language: 'el',
|
||||
glossLanguage: 'en',
|
||||
license: 'CC0-1.0',
|
||||
credits: [alphabetCredit],
|
||||
),
|
||||
);
|
||||
for (final (place, (small, capital, name, english, latin, sound))
|
||||
in _greek.indexed) {
|
||||
final forms = [capital, small, if (small == 'σ') 'ς'];
|
||||
yield entryLine(
|
||||
DictionaryEntry(
|
||||
id: small,
|
||||
headword: forms.join(' '),
|
||||
reading: null,
|
||||
labels: const ['greek'],
|
||||
rank: place,
|
||||
keys: {...forms, name, withoutAccents(name), english, latin}.toList(),
|
||||
entry: {
|
||||
'romaji': latin,
|
||||
'script': 'greek',
|
||||
'name': name,
|
||||
'englishName': english,
|
||||
'sound': sound,
|
||||
},
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -338,5 +338,83 @@ final _toneless = {
|
|||
for (final marked in [...marks, vowel]) marked: vowel == 'ü' ? 'u' : vowel,
|
||||
};
|
||||
|
||||
/// Each character's place among those the words of CC-CEDICT ([text]) are
|
||||
/// written with, as each script writes them: the place of the most used
|
||||
/// word with it ([frequencies], by the simplified word), 的 first. A
|
||||
/// character only in words the frequencies lack, or only in entries that
|
||||
/// are variants of others, has none; traditional characters are so told
|
||||
/// from simplified ones written alike in Unihan (这 is its own variant
|
||||
/// there, but no word in traditional characters is written with it). A
|
||||
/// simplified character standing alone for several traditional ones (后
|
||||
/// for 後 and 后) gives its own place to the one the longer words use most
|
||||
/// (後, of 以後 and 最後), as the frequencies can't tell them apart.
|
||||
({Map<String, int> simplified, Map<String, int> traditional})
|
||||
cedictCharacterPlaces(String text, WordFrequencies frequencies) {
|
||||
final simplified = <String, int>{};
|
||||
final traditional = <String, int>{};
|
||||
void keep(Map<String, int> places, String character, int place) {
|
||||
final known = places[character];
|
||||
if (known == null || place < known) {
|
||||
places[character] = place;
|
||||
}
|
||||
}
|
||||
|
||||
// Single characters' places by their traditional forms, until the
|
||||
// longer words have shown which of them is used.
|
||||
final alone = <String, Map<String, int>>{};
|
||||
final line = RegExp(r'^(\S+) (\S+) \[[^\]]*\] /(.*)/\s*$');
|
||||
for (final row in text.split('\n')) {
|
||||
if (row.startsWith('#')) {
|
||||
continue;
|
||||
}
|
||||
final match = line.firstMatch(row);
|
||||
if (match == null ||
|
||||
match[3]!
|
||||
.split('/')
|
||||
.map((definition) => definition.trim())
|
||||
.where((definition) => definition.isNotEmpty)
|
||||
.every(_variantTarget.hasMatch)) {
|
||||
continue;
|
||||
}
|
||||
final place = frequencies.placeOf(match[2]!);
|
||||
if (place == null) {
|
||||
continue;
|
||||
}
|
||||
for (final character in match[2]!.split('')) {
|
||||
keep(simplified, character, place);
|
||||
}
|
||||
if (match[2]!.length == 1) {
|
||||
(alone[match[2]!] ??= {})[match[1]!] = place;
|
||||
continue;
|
||||
}
|
||||
for (final character in match[1]!.split('')) {
|
||||
keep(traditional, character, place);
|
||||
}
|
||||
}
|
||||
for (final forms in alone.values) {
|
||||
final used = forms.keys.toList()
|
||||
..sort(
|
||||
(a, b) =>
|
||||
(traditional[a] ?? 1 << 30).compareTo(traditional[b] ?? 1 << 30),
|
||||
);
|
||||
keep(traditional, used.first, forms[used.first]!);
|
||||
}
|
||||
return (simplified: simplified, traditional: traditional);
|
||||
}
|
||||
|
||||
/// The HSK levels of characters to recognise in the 2025 examination
|
||||
/// syllabus, as `hsk30` extracted them ([file]: character, level; 7 for
|
||||
/// 7 to 9).
|
||||
Future<Map<String, int>> readHskCharacters(File file) async => {
|
||||
for (final line in await file.readAsLines())
|
||||
if (line.split('\t') case [final character, final level]
|
||||
when character.runes.length == 1)
|
||||
character: ?int.tryParse(level),
|
||||
};
|
||||
|
||||
/// The HSK character levels file in the curator's [data] folder.
|
||||
File hskCharactersFile(Directory data) =>
|
||||
File('${data.path}/hsk/syllabus2025_chars.tsv');
|
||||
|
||||
/// The HSK levels file in the curator's [data] folder.
|
||||
File hskFile(Directory data) => File('${data.path}/hsk/syllabus2025_words.tsv');
|
||||
|
|
|
|||
324
halkyon_learn_curator/lib/src/sources/krdict.dart
Normal file
324
halkyon_learn_curator/lib/src/sources/krdict.dart
Normal file
|
|
@ -0,0 +1,324 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:math';
|
||||
import 'dart:typed_data';
|
||||
|
||||
import 'package:archive/archive.dart';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/dictionary_file.dart';
|
||||
import 'package:halkyon_learn_curator/src/gloss_keys.dart';
|
||||
import 'package:halkyon_learn_curator/src/json_array_stream.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
|
||||
/// Who makes KRDict, the Basic Korean Dictionary for learners.
|
||||
const krdictCredit = Credit(
|
||||
name: '한국어기초사전 (Basic Korean Dictionary)',
|
||||
by: 'National Institute of Korean Language (국립국어원)',
|
||||
license: 'CC-BY-SA-2.0-KR',
|
||||
url: 'https://krdict.korean.go.kr/kboardPolicy/copyRightTermsInfo',
|
||||
);
|
||||
|
||||
/// The name of KRDict's dictionary file.
|
||||
const krdictFileName = 'krdict-ko-en.jsonl.gz';
|
||||
|
||||
/// KRDict's parts of speech as Wiktionary names them, which the app spells
|
||||
/// out; a bound noun (의존 명사) is a noun, an auxiliary verb a verb.
|
||||
const _partsOfSpeech = {
|
||||
'명사': 'noun',
|
||||
'의존 명사': 'noun',
|
||||
'대명사': 'pron',
|
||||
'수사': 'num',
|
||||
'동사': 'verb',
|
||||
'보조 동사': 'verb',
|
||||
'형용사': 'adj',
|
||||
'보조 형용사': 'adj',
|
||||
'관형사': 'det',
|
||||
'부사': 'adv',
|
||||
'감탄사': 'intj',
|
||||
'조사': 'particle',
|
||||
'어미': 'ending',
|
||||
'접사': 'affix',
|
||||
};
|
||||
|
||||
/// KRDict's learners' levels by their labels: beginner (초급),
|
||||
/// intermediate (중급), advanced (고급).
|
||||
const _levels = {'초급': 1, '중급': 2, '고급': 3};
|
||||
|
||||
/// The label of a learners' level, 1 to 3.
|
||||
String koreanLevelLabel(int level) =>
|
||||
'level:${const ['beginner', 'intermediate', 'advanced'][level - 1]}';
|
||||
|
||||
/// How many senses an entry keeps, and how many forms it shows.
|
||||
const _maxSenses = 16;
|
||||
const _maxInflections = 6;
|
||||
|
||||
/// KRDict's entries ([entries], each a `LexicalEntry` of its JSON export as
|
||||
/// decoded) as the lines of a dictionary file of Korean words with English
|
||||
/// meanings: each with its part of speech, its pronunciation as its
|
||||
/// reading where it differs from the spelling (같이 [가치]; length marks
|
||||
/// left out), its Hanja, the forms KRDict gives beside it, and its senses,
|
||||
/// each with KRDict's English words and definition. Its learners' level
|
||||
/// labels it (`level:beginner`, `level:intermediate`, `level:advanced`)
|
||||
/// and ranks it, then [frequencies]; [korean] adds the forms Wiktionary's
|
||||
/// tables give (가다's 99), which find it in running text with KRDict's.
|
||||
Stream<String> krdictDictionary(
|
||||
Stream<Object?> entries, {
|
||||
required String sourceVersion,
|
||||
WordFrequencies? frequencies,
|
||||
WiktionaryKorean? korean,
|
||||
}) async* {
|
||||
yield headerLine(
|
||||
DictionaryHeader(
|
||||
source: 'krdict',
|
||||
sourceVersion: sourceVersion,
|
||||
language: 'ko',
|
||||
glossLanguage: 'en',
|
||||
license: 'CC-BY-SA-4.0',
|
||||
credits: [
|
||||
krdictCredit,
|
||||
if (korean != null) ...wiktionaryCredits,
|
||||
if (frequencies != null && frequencies.length > 0) ...wordfreqCredits,
|
||||
],
|
||||
),
|
||||
);
|
||||
// A word's frequency goes to its first homographs only, as for
|
||||
// Wiktionary's words, the most basic first.
|
||||
final seen = <String, int>{};
|
||||
await for (final json in entries) {
|
||||
if (json is! Map<String, Object?>) {
|
||||
continue;
|
||||
}
|
||||
final entry = krdictEntry(json, frequencies: frequencies, korean: korean);
|
||||
if (entry == null) {
|
||||
continue;
|
||||
}
|
||||
final homograph = seen[entry.headword] = (seen[entry.headword] ?? 0) + 1;
|
||||
yield entryLine(
|
||||
homograph <= commonHomographs
|
||||
? entry
|
||||
: DictionaryEntry(
|
||||
id: entry.id,
|
||||
headword: entry.headword,
|
||||
reading: entry.reading,
|
||||
labels: [
|
||||
for (final label in entry.labels)
|
||||
if (label != 'common') label,
|
||||
],
|
||||
rank: entry.rank + 900000,
|
||||
keys: entry.keys,
|
||||
looseKeys: entry.looseKeys,
|
||||
textKeys: entry.textKeys,
|
||||
entry: entry.entry,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// One of KRDict's entries ([json]) as an entry; null for one without an
|
||||
/// English meaning.
|
||||
DictionaryEntry? krdictEntry(
|
||||
Map<String, Object?> json, {
|
||||
WordFrequencies? frequencies,
|
||||
WiktionaryKorean? korean,
|
||||
}) {
|
||||
final id = json['att'] == 'id' ? json['val']?.toString() : null;
|
||||
final features = _features(json);
|
||||
final written = [
|
||||
for (final lemma in _list(json['Lemma']))
|
||||
if (lemma is Map<String, Object?>) ?_features(lemma)['writtenForm'],
|
||||
];
|
||||
if (id == null || written.isEmpty) {
|
||||
return null;
|
||||
}
|
||||
final headword = written.first;
|
||||
final senses = [
|
||||
for (final sense in _list(json['Sense']))
|
||||
if (sense is Map<String, Object?>) ?_sense(sense),
|
||||
];
|
||||
if (senses.isEmpty) {
|
||||
return null;
|
||||
}
|
||||
final unit = features['lexicalUnit'];
|
||||
final pos = switch (unit) {
|
||||
'관용구' || '구' || '문법‧표현' => 'phrase',
|
||||
'속담' => 'proverb',
|
||||
_ => _partsOfSpeech[features['partOfSpeech']],
|
||||
};
|
||||
final wordForms = [
|
||||
for (final form in _list(json['WordForm']))
|
||||
if (form is Map<String, Object?>) form,
|
||||
];
|
||||
// Its own pronunciation, then each form as written (가까워지니).
|
||||
final pronunciation = [
|
||||
for (final form in wordForms)
|
||||
if (_features(form)['type'] == '발음') ?_features(form)['pronunciation'],
|
||||
].firstOrNull?.replaceAll('ː', '');
|
||||
final inflections = {
|
||||
for (final form in wordForms)
|
||||
if (_features(form) case {'type': '활용', 'writtenForm': final text}) text,
|
||||
}.toList();
|
||||
final level = _levels[features['vocabularyLevel']];
|
||||
final origin = features['origin'];
|
||||
final hanja =
|
||||
origin != null &&
|
||||
RegExp(r'\p{Script=Han}', unicode: true).hasMatch(origin)
|
||||
? origin
|
||||
: null;
|
||||
final isWord = unit == null || unit == '단어';
|
||||
// An idiom or a proverb carries the id of the word it is filed under
|
||||
// (가는 날이 장날 that of 가다): itself as written tells them apart.
|
||||
final textKeys = isWord
|
||||
? {
|
||||
for (final text in written) text,
|
||||
...inflections,
|
||||
...?korean?.forms[headword],
|
||||
}.where((text) => !text.contains(' ')).toList()
|
||||
: const <String>[];
|
||||
// A verb is used in its forms, rarely as the dictionary writes it
|
||||
// (먹어, 먹었다, not 먹다): its own place, else its best form's, behind
|
||||
// words used as they stand.
|
||||
final formPlaces = [
|
||||
for (final text in textKeys)
|
||||
if (text != headword) ?frequencies?.placeOf(text),
|
||||
];
|
||||
final place =
|
||||
frequencies?.placeOf(headword) ??
|
||||
(formPlaces.isEmpty ? null : formPlaces.reduce(min) * 2);
|
||||
return DictionaryEntry(
|
||||
id: isWord ? id : '$id:$headword',
|
||||
headword: headword,
|
||||
reading:
|
||||
pronunciation != null &&
|
||||
pronunciation.isNotEmpty &&
|
||||
pronunciation.replaceAll(' ', '') != headword.replaceAll(' ', '')
|
||||
? pronunciation
|
||||
: null,
|
||||
labels: [
|
||||
if (level != null) koreanLevelLabel(level),
|
||||
if (place != null && place < commonPlaces) 'common',
|
||||
],
|
||||
rank: (level ?? 4) * 100000 + (place ?? 99999).clamp(0, 99999),
|
||||
keys: {
|
||||
...written,
|
||||
?hanja,
|
||||
for (final sense in senses)
|
||||
for (final gloss in sense.glosses) ...glossKeys(gloss),
|
||||
}.toList(),
|
||||
textKeys: textKeys,
|
||||
entry: {
|
||||
'pos': ?pos,
|
||||
if (hanja != null) 'hanja': [hanja],
|
||||
if (inflections.isNotEmpty)
|
||||
'inflections': [
|
||||
for (final text in inflections.take(_maxInflections)) {'text': text},
|
||||
],
|
||||
'senses': [
|
||||
for (final sense in senses.take(_maxSenses))
|
||||
{
|
||||
'glosses': sense.glosses,
|
||||
if (sense.definition != null) 'info': [sense.definition],
|
||||
},
|
||||
],
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
/// A sense with its English words and definition; null without them.
|
||||
({List<String> glosses, String? definition})? _sense(
|
||||
Map<String, Object?> sense,
|
||||
) {
|
||||
for (final equivalent in _list(sense['Equivalent'])) {
|
||||
if (equivalent is! Map<String, Object?>) {
|
||||
continue;
|
||||
}
|
||||
final features = _features(equivalent);
|
||||
if (features['language'] != '영어') {
|
||||
continue;
|
||||
}
|
||||
final glosses = [
|
||||
for (final gloss in (features['lemma'] ?? '').split(RegExp('[;,]')))
|
||||
if (gloss.trim().isNotEmpty) gloss.trim(),
|
||||
];
|
||||
if (glosses.isEmpty) {
|
||||
return null;
|
||||
}
|
||||
final definition = features['definition']?.trim();
|
||||
return (
|
||||
glosses: glosses,
|
||||
definition: definition == null || definition.isEmpty ? null : definition,
|
||||
);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/// The features (`{"att": …, "val": …}`) of a part of an entry, by name;
|
||||
/// the first of a name, and none without a value.
|
||||
Map<String, String> _features(Map<String, Object?> part) => {
|
||||
for (final feature in _list(part['feat']).reversed)
|
||||
if (feature case {'att': final String name, 'val': final Object value})
|
||||
name: value.toString(),
|
||||
};
|
||||
|
||||
/// A part that is one item or a list of them, as a list.
|
||||
List<Object?> _list(Object? value) => switch (value) {
|
||||
null => const [],
|
||||
final List<Object?> list => list,
|
||||
final item => [item],
|
||||
};
|
||||
|
||||
/// The version KRDict's export names ([fileName], `…_json_20260919.zip`, or
|
||||
/// one of its files, `1_5000_20260919.json`): its date, `2026-09-19`.
|
||||
String krdictVersionOf(String fileName) {
|
||||
final match = RegExp(r'(\d{4})(\d{2})(\d{2})').firstMatch(fileName);
|
||||
return match == null ? '' : '${match[1]}-${match[2]}-${match[3]}';
|
||||
}
|
||||
|
||||
/// The download KRDict's page of exports offers as JSON ([page], its HTML):
|
||||
/// the number of `batchDownload('217')`, the third of Excel, XML and JSON;
|
||||
/// null when the page offers none.
|
||||
String? krdictJsonDownloadOf(String page) {
|
||||
final numbers = [
|
||||
for (final match in RegExp(r"batchDownload\('(\d+)'\)").allMatches(page))
|
||||
match[1]!,
|
||||
];
|
||||
return numbers.length >= 3 && numbers[2] != '0' ? numbers[2] : null;
|
||||
}
|
||||
|
||||
/// KRDict's JSON export ([zip], as `dicBatchDownload` gives it: its files
|
||||
/// `1_5000_….json` to `11_3671_….json`, each a `LexicalResource`): the
|
||||
/// date it names as its version, and its entries in its files' order,
|
||||
/// each decoded on its own (a file unpacks to some 95 MB, the export to a
|
||||
/// gigabyte).
|
||||
({String version, Stream<Object?> entries}) krdictExport(List<int> zip) {
|
||||
int numberOf(ArchiveFile file) =>
|
||||
int.tryParse(file.name.split('/').last.split('_').first) ?? 0;
|
||||
final files = [
|
||||
for (final file in ZipDecoder().decodeBytes(zip).files)
|
||||
if (file.isFile && file.name.endsWith('.json')) file,
|
||||
]..sort((a, b) => numberOf(a).compareTo(numberOf(b)));
|
||||
Stream<Object?> entries() async* {
|
||||
for (final file in files) {
|
||||
final bytes = file.content;
|
||||
Stream<List<int>> chunks() async* {
|
||||
for (var start = 0; start < bytes.length; start += 1 << 20) {
|
||||
yield Uint8List.sublistView(
|
||||
bytes,
|
||||
start,
|
||||
min(start + (1 << 20), bytes.length),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
yield* jsonArrayElements(
|
||||
utf8.decoder.bind(chunks()),
|
||||
key: 'LexicalEntry',
|
||||
);
|
||||
file.clear();
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
version: files.isEmpty ? '' : krdictVersionOf(files.first.name),
|
||||
entries: entries(),
|
||||
);
|
||||
}
|
||||
592
halkyon_learn_curator/lib/src/sources/unihan.dart
Normal file
592
halkyon_learn_curator/lib/src/sources/unihan.dart
Normal file
|
|
@ -0,0 +1,592 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:archive/archive.dart';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/dictionary_file.dart';
|
||||
import 'package:halkyon_learn_curator/src/gloss_keys.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/alphabets.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/cedict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
|
||||
/// Who makes the Unihan database.
|
||||
const unihanCredit = Credit(
|
||||
name: 'Unihan Database',
|
||||
by: 'Unicode, Inc.',
|
||||
license: 'Unicode-3.0',
|
||||
url: 'https://www.unicode.org/reports/tr38/',
|
||||
);
|
||||
|
||||
/// The name of the dictionary file of Chinese characters, simplified or
|
||||
/// [traditional], or of Korean Hanja ([korean]).
|
||||
String unihanFileName({bool traditional = false, bool korean = false}) =>
|
||||
'unihan-${korean
|
||||
? 'ko'
|
||||
: traditional
|
||||
? 'hant'
|
||||
: 'hans'}-en.jsonl.gz';
|
||||
|
||||
/// The fields of Unihan's files the dictionaries are made of.
|
||||
const _wanted = {
|
||||
'kDefinition',
|
||||
'kMandarin',
|
||||
'kTGHZ2013',
|
||||
'kXHC1983',
|
||||
'kHanyuPinyin',
|
||||
'kHangul',
|
||||
'kTotalStrokes',
|
||||
'kRSUnicode',
|
||||
'kSimplifiedVariant',
|
||||
'kTraditionalVariant',
|
||||
'kTGH',
|
||||
'kGradeLevel',
|
||||
'kKoreanEducationHanja',
|
||||
};
|
||||
|
||||
/// Unihan's fields of each character ([_wanted] of them), as its files
|
||||
/// (`Unihan_*.txt`) give them: a code point, a field and its value to a
|
||||
/// line, tab-separated.
|
||||
final class Unihan(final Map<String, Map<String, String>> _fields) {
|
||||
/// The fields in [lines], all of Unihan's files' lines.
|
||||
factory parse(Iterable<String> lines) {
|
||||
final fields = <String, Map<String, String>>{};
|
||||
for (final line in lines) {
|
||||
if (line.startsWith('#') || !line.startsWith('U+')) {
|
||||
continue;
|
||||
}
|
||||
final [code, field, value] = switch (line.split('\t')) {
|
||||
final parts when parts.length == 3 => parts,
|
||||
_ => const ['', '', ''],
|
||||
};
|
||||
if (!_wanted.contains(field)) {
|
||||
continue;
|
||||
}
|
||||
final character = _character(code);
|
||||
if (character != null) {
|
||||
(fields[character] ??= {})[field] = value;
|
||||
}
|
||||
}
|
||||
return Unihan(fields);
|
||||
}
|
||||
|
||||
/// The version of Unicode a file of Unihan's ([lines]) is of
|
||||
/// (`# Unicode Version 18.0.0` → `18.0.0`), or ''.
|
||||
static String versionOf(Iterable<String> lines) {
|
||||
for (final line in lines.take(20)) {
|
||||
if (RegExp(r'^# Unicode Version (\S+)').firstMatch(line)
|
||||
case final match?) {
|
||||
return match[1]!;
|
||||
}
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
/// Unihan's files in [zip] (`Unihan.zip`, as unicode.org publishes it),
|
||||
/// and the version of Unicode they are of.
|
||||
static ({Unihan unihan, String version}) ofZip(List<int> zip) {
|
||||
final files = [
|
||||
for (final file in ZipDecoder().decodeBytes(zip).files)
|
||||
if (file.isFile &&
|
||||
file.name.startsWith('Unihan_') &&
|
||||
file.name.endsWith('.txt'))
|
||||
file,
|
||||
];
|
||||
Iterable<String> linesOf(ArchiveFile file) =>
|
||||
const LineSplitter().convert(utf8.decode(file.content));
|
||||
Iterable<String> lines() sync* {
|
||||
for (final file in files) {
|
||||
yield* linesOf(file);
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
unihan: Unihan.parse(lines()),
|
||||
version: files.isEmpty ? '' : versionOf(linesOf(files.first)),
|
||||
);
|
||||
}
|
||||
|
||||
/// Every character Unihan says something of here.
|
||||
Iterable<String> get characters => _fields.keys;
|
||||
|
||||
/// Whether [character] has a value of [field].
|
||||
bool has(String character, String field) =>
|
||||
_fields[character]?.containsKey(field) ?? false;
|
||||
|
||||
String? _of(String character, String field) => _fields[character]?[field];
|
||||
}
|
||||
|
||||
/// `U+6C34` as the character 水; null for no code point.
|
||||
String? _character(String code) {
|
||||
final value = int.tryParse(code.substring(2), radix: 16);
|
||||
return value == null ? null : String.fromCharCode(value);
|
||||
}
|
||||
|
||||
/// The characters a variant field names (`U+8AAA U+8AAC`, in
|
||||
/// `kTraditionalVariant`), each as written.
|
||||
List<String> _variants(String? value) => [
|
||||
if (value != null)
|
||||
for (final code in value.split(' '))
|
||||
if (code.startsWith('U+')) ?_character(code.split('<').first),
|
||||
];
|
||||
|
||||
/// Unihan's English definition ([definition]) as meanings: one for each
|
||||
/// part between semicolons and commas outside brackets (`water, liquid`;
|
||||
/// `(same as 丘) hillock or mound` stays one).
|
||||
List<String> _meanings(String? definition) {
|
||||
if (definition == null) {
|
||||
return const [];
|
||||
}
|
||||
final meanings = <String>[];
|
||||
final current = StringBuffer();
|
||||
var depth = 0;
|
||||
for (final char in definition.split('')) {
|
||||
if (char == '(' || char == '[') {
|
||||
depth++;
|
||||
} else if ((char == ')' || char == ']') && depth > 0) {
|
||||
depth--;
|
||||
}
|
||||
if (depth == 0 && (char == ',' || char == ';')) {
|
||||
meanings.add(current.toString().trim());
|
||||
current.clear();
|
||||
} else {
|
||||
current.write(char);
|
||||
}
|
||||
}
|
||||
meanings.add(current.toString().trim());
|
||||
return [
|
||||
for (final meaning in meanings)
|
||||
if (meaning.isNotEmpty) meaning,
|
||||
];
|
||||
}
|
||||
|
||||
/// The readings a reading field's value gives (`131.140:háng 408.120:xíng`,
|
||||
/// `20811.060:háng,xìng`, `1424.050*:háng`): every reading after a colon.
|
||||
Iterable<String> _readingsOf(String? value) sync* {
|
||||
if (value == null) {
|
||||
return;
|
||||
}
|
||||
for (final part in value.split(' ')) {
|
||||
final readings = part.contains(':') ? part.split(':').last : part;
|
||||
yield* readings.split(',').where((reading) => reading.isNotEmpty);
|
||||
}
|
||||
}
|
||||
|
||||
/// [character]'s readings in Mandarin, the usual one first: in simplified
|
||||
/// characters the mainland's (`kMandarin`'s first), in [traditional] ones
|
||||
/// Taiwan's (its last), then those the dictionaries of the standard
|
||||
/// characters (2013), of modern Chinese (1983) and the Hanyu Da Zidian
|
||||
/// give.
|
||||
List<String> _pinyin(
|
||||
Unihan unihan,
|
||||
String character, {
|
||||
required bool traditional,
|
||||
}) {
|
||||
final usual = unihan._of(character, 'kMandarin')?.split(' ') ?? const [];
|
||||
return {
|
||||
...traditional ? usual.reversed : usual,
|
||||
..._readingsOf(unihan._of(character, 'kTGHZ2013')),
|
||||
..._readingsOf(unihan._of(character, 'kXHC1983')),
|
||||
..._readingsOf(unihan._of(character, 'kHanyuPinyin')),
|
||||
}.toList();
|
||||
}
|
||||
|
||||
/// [character]'s stroke count in its simplified or [traditional] form,
|
||||
/// which `kTotalStrokes` gives first and last where they differ.
|
||||
int? _strokes(Unihan unihan, String character, {bool traditional = false}) {
|
||||
final counts = unihan._of(character, 'kTotalStrokes')?.split(' ') ?? const [];
|
||||
return counts.isEmpty
|
||||
? null
|
||||
: int.tryParse(traditional ? counts.last : counts.first);
|
||||
}
|
||||
|
||||
/// The Kangxi radical (1 to 214) [character] is filed under:
|
||||
/// `kRSUnicode`'s first, `149'.7` (the simplified form of radical 149) as
|
||||
/// 149.
|
||||
int? _radical(Unihan unihan, String character) {
|
||||
final value = unihan._of(character, 'kRSUnicode')?.split(' ').first;
|
||||
final radical = int.tryParse(
|
||||
value?.split('.').first.replaceAll("'", '') ?? '',
|
||||
);
|
||||
return radical != null && radical >= 1 && radical <= 214 ? radical : null;
|
||||
}
|
||||
|
||||
/// A syllable of pinyin with tone marks (`shuǐ`) numbered as typed
|
||||
/// without them (`shui3`; the neutral tone `5`).
|
||||
String _numbered(String syllable) {
|
||||
for (final (tone, marks) in const [
|
||||
(1, 'āēīōūǖ'),
|
||||
(2, 'áéíóúǘ'),
|
||||
(3, 'ǎěǐǒǔǚ'),
|
||||
(4, 'àèìòùǜ'),
|
||||
]) {
|
||||
if (syllable.split('').any(marks.contains)) {
|
||||
return '${tonelessPinyin(syllable)}$tone';
|
||||
}
|
||||
}
|
||||
return '${tonelessPinyin(syllable)}5';
|
||||
}
|
||||
|
||||
/// The tier of the General Standard Chinese Characters (2013) [character]
|
||||
/// is in, by its number in the table (`kTGH`, `2013:483`): 1 for the 3,500
|
||||
/// in common use, 2 for the next 3,000, 3 for the rest.
|
||||
int? _standardTier(Unihan unihan, String character) {
|
||||
final number = int.tryParse(
|
||||
unihan._of(character, 'kTGH')?.split(':').last ?? '',
|
||||
);
|
||||
return switch (number) {
|
||||
null => null,
|
||||
<= 3500 => 1,
|
||||
<= 6500 => 2,
|
||||
_ => 3,
|
||||
};
|
||||
}
|
||||
|
||||
/// Each Chinese character's place among those the words in use are
|
||||
/// written with in each script, 的 first (`cedictCharacterPlaces`).
|
||||
typedef CharacterPlaces = ({
|
||||
Map<String, int> simplified,
|
||||
Map<String, int> traditional,
|
||||
});
|
||||
|
||||
/// No places known.
|
||||
const noCharacterPlaces = (
|
||||
simplified: <String, int>{},
|
||||
traditional: <String, int>{},
|
||||
);
|
||||
|
||||
/// The Chinese characters of simplified writing: the General Standard
|
||||
/// Chinese Characters (2013, `kTGH`); or those of [traditional] writing:
|
||||
/// each of them as Taiwan and Hong Kong write it (`kTraditionalVariant`,
|
||||
/// where it is a form of its own or a word in use writes it so: [places]),
|
||||
/// the characters of the words in use, and those taught in Hong Kong's
|
||||
/// primary schools (`kGradeLevel`).
|
||||
Set<String> chineseCharactersOf(
|
||||
Unihan unihan, {
|
||||
required bool traditional,
|
||||
CharacterPlaces places = noCharacterPlaces,
|
||||
}) {
|
||||
final standard = {
|
||||
for (final character in unihan.characters)
|
||||
if (unihan.has(character, 'kTGH')) character,
|
||||
};
|
||||
if (!traditional) {
|
||||
return standard;
|
||||
}
|
||||
return {
|
||||
for (final character in standard)
|
||||
...switch (_variants(unihan._of(character, 'kTraditionalVariant'))) {
|
||||
final forms when forms.isEmpty => [character],
|
||||
final forms => [
|
||||
for (final form in forms)
|
||||
if (form != character || places.traditional.containsKey(character))
|
||||
form,
|
||||
],
|
||||
},
|
||||
...places.traditional.keys,
|
||||
for (final character in unihan.characters)
|
||||
if (unihan.has(character, 'kGradeLevel')) character,
|
||||
}..removeWhere(
|
||||
(character) =>
|
||||
!unihan.has(character, 'kMandarin') &&
|
||||
!unihan.has(character, 'kDefinition'),
|
||||
);
|
||||
}
|
||||
|
||||
/// Each character's place among those [frequencies] (Chinese words, the
|
||||
/// most used first) are written with: the place of the most used word
|
||||
/// with it, 的 first. Simplified characters alone, as wordfreq writes
|
||||
/// Chinese.
|
||||
Map<String, int> characterPlaces(WordFrequencies frequencies) {
|
||||
final places = <String, int>{};
|
||||
for (final (place, word) in frequencies.words.indexed) {
|
||||
for (final character in word.split('')) {
|
||||
places.putIfAbsent(character, () => place);
|
||||
}
|
||||
}
|
||||
return places;
|
||||
}
|
||||
|
||||
/// Unihan as the lines of a dictionary file of Chinese characters with
|
||||
/// English meanings, in simplified characters (`zh-Hans`) or [traditional]
|
||||
/// ones (`zh-Hant`), each a dictionary of its own, as CC-CEDICT's words
|
||||
/// are: every character's readings in pinyin (the usual first), meanings,
|
||||
/// stroke count and its other script's forms. Labels pick its HSK level
|
||||
/// ([hsk], by the simplified character: `hsk:1` … `hsk:7`), the 3,500
|
||||
/// standard characters in common use (`common`), a grade of Hong Kong's
|
||||
/// primary schools (`grade:1` … `grade:6`) and its Kangxi radical
|
||||
/// (`radical:1` … `radical:214`); a traditional character takes the level
|
||||
/// and `common` of its simplified one only as its usual form (後, not 后,
|
||||
/// for 后; 了, not 瞭, for 了). Characters rank by HSK level, then by how
|
||||
/// often the words written with them are used ([places]).
|
||||
Iterable<String> chineseCharacters(
|
||||
Unihan unihan, {
|
||||
required bool traditional,
|
||||
String sourceVersion = '',
|
||||
Map<String, int> hsk = const {},
|
||||
CharacterPlaces places = noCharacterPlaces,
|
||||
List<Credit> placesCredits = const [],
|
||||
}) sync* {
|
||||
yield headerLine(
|
||||
DictionaryHeader(
|
||||
source: 'unihan-${traditional ? 'hant' : 'hans'}',
|
||||
kind: DictionaryKind.characters,
|
||||
sourceVersion: sourceVersion,
|
||||
language: traditional ? 'zh-Hant' : 'zh-Hans',
|
||||
glossLanguage: 'en',
|
||||
license: placesCredits.isEmpty ? 'Unicode-3.0' : 'CC-BY-SA-4.0',
|
||||
credits: [unihanCredit, if (hsk.isNotEmpty) hskCredit, ...placesCredits],
|
||||
),
|
||||
);
|
||||
final scriptPlaces = traditional ? places.traditional : places.simplified;
|
||||
// A simplified character's usual traditional form: the one the most used
|
||||
// word writes, else one of its own, else itself.
|
||||
String usualForm(String simplified) {
|
||||
final forms = _variants(unihan._of(simplified, 'kTraditionalVariant'));
|
||||
if (forms.isEmpty) {
|
||||
return simplified;
|
||||
}
|
||||
final used = [
|
||||
for (final form in forms)
|
||||
if (places.traditional[form] case final place?) (form, place),
|
||||
]..sort((a, b) => a.$2.compareTo(b.$2));
|
||||
return used.firstOrNull?.$1 ??
|
||||
forms.firstWhere(
|
||||
(form) => form != simplified,
|
||||
orElse: () => forms.first,
|
||||
);
|
||||
}
|
||||
|
||||
for (final character in chineseCharactersOf(
|
||||
unihan,
|
||||
traditional: traditional,
|
||||
places: places,
|
||||
)) {
|
||||
// A traditional character's level and tier are its simplified form's,
|
||||
// which the syllabus and the standard write, when it is that one's
|
||||
// usual form.
|
||||
final simplified = traditional
|
||||
? _variants(unihan._of(character, 'kSimplifiedVariant')).firstOrNull ??
|
||||
character
|
||||
: character;
|
||||
final usual = !traditional || usualForm(simplified) == character;
|
||||
final otherScript = _variants(
|
||||
unihan._of(
|
||||
character,
|
||||
traditional ? 'kSimplifiedVariant' : 'kTraditionalVariant',
|
||||
),
|
||||
).where((form) => form != character).toList();
|
||||
final pinyin = _pinyin(unihan, character, traditional: traditional);
|
||||
final meanings = _meanings(unihan._of(character, 'kDefinition'));
|
||||
if (pinyin.isEmpty && meanings.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
final level = usual ? hsk[simplified] : null;
|
||||
final place = scriptPlaces[character];
|
||||
final tier = usual ? _standardTier(unihan, simplified) : null;
|
||||
final grade = traditional
|
||||
? int.tryParse(unihan._of(character, 'kGradeLevel') ?? '')
|
||||
: null;
|
||||
final radical = _radical(unihan, character);
|
||||
final strokes = _strokes(unihan, character, traditional: traditional);
|
||||
yield entryLine(
|
||||
DictionaryEntry(
|
||||
id: character,
|
||||
headword: character,
|
||||
reading: null,
|
||||
labels: [
|
||||
if (level != null) 'hsk:$level',
|
||||
if (tier == 1) 'common',
|
||||
if (grade != null && grade >= 1 && grade <= 6) 'grade:$grade',
|
||||
if (radical != null) 'radical:$radical',
|
||||
],
|
||||
rank: switch (level) {
|
||||
final level? => level * 10000 + (place ?? 9999).clamp(0, 9999),
|
||||
null =>
|
||||
100000 + (tier ?? 4) * 100000 + (place ?? 99999).clamp(0, 99999),
|
||||
},
|
||||
keys: {
|
||||
character,
|
||||
for (final reading in pinyin) ...[
|
||||
reading.toLowerCase(),
|
||||
tonelessPinyin(reading),
|
||||
_numbered(reading.toLowerCase()),
|
||||
],
|
||||
for (final meaning in meanings) ...glossKeys(meaning),
|
||||
}.toList(),
|
||||
// Written in the other script, it is still this character.
|
||||
looseKeys: otherScript,
|
||||
entry: {
|
||||
'meanings': meanings,
|
||||
'pinyin': pinyin,
|
||||
'strokes': ?strokes,
|
||||
if (otherScript.isNotEmpty)
|
||||
traditional ? 'simplified' : 'traditional': otherScript,
|
||||
'hsk': ?level,
|
||||
'frequency': ?place,
|
||||
},
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A Hanja's readings as Korean dictionaries give them, its meaning in
|
||||
/// native Korean (훈, hun) and its sound (음, eum): `물 수` for 水.
|
||||
typedef Eumhun = ({String hun, String eum});
|
||||
|
||||
/// The Hanja Korean writes ([unihan]'s characters with a Korean reading,
|
||||
/// `kHangul`) as the lines of a dictionary file with English meanings:
|
||||
/// each with its sounds (음, the reading taught in schools first), its
|
||||
/// native meanings and sounds together as [eumhun] gives them (Wiktionary,
|
||||
/// `물 수`), meanings and stroke count. Labels pick the 1,800 Hanja taught
|
||||
/// in schools (`education`) and a character's Kangxi radical; those rank
|
||||
/// first, then the rest by stroke count.
|
||||
Iterable<String> koreanHanja(
|
||||
Unihan unihan, {
|
||||
String sourceVersion = '',
|
||||
Map<String, List<Eumhun>> eumhun = const {},
|
||||
}) sync* {
|
||||
yield headerLine(
|
||||
DictionaryHeader(
|
||||
source: 'unihan-ko',
|
||||
kind: DictionaryKind.characters,
|
||||
sourceVersion: sourceVersion,
|
||||
language: 'ko',
|
||||
glossLanguage: 'en',
|
||||
license: eumhun.isEmpty ? 'Unicode-3.0' : 'CC-BY-SA-4.0',
|
||||
credits: [
|
||||
unihanCredit,
|
||||
// The Hanja's native readings are Wiktionary's.
|
||||
if (eumhun.isNotEmpty) ...wiktionaryCredits,
|
||||
alphabetCredit,
|
||||
],
|
||||
),
|
||||
);
|
||||
for (final character in unihan.characters) {
|
||||
final value = unihan._of(character, 'kHangul');
|
||||
if (value == null) {
|
||||
continue;
|
||||
}
|
||||
// `항:0N 행:0E`: the reading, then where it comes from; `E` marks the
|
||||
// reading taught with the 1,800 Hanja of education, first.
|
||||
final readings = [
|
||||
for (final reading in value.split(' '))
|
||||
(reading.split(':').first, reading.contains('E')),
|
||||
];
|
||||
final eum = {
|
||||
for (final (reading, taught) in readings)
|
||||
if (taught) reading,
|
||||
for (final (reading, _) in readings) reading,
|
||||
}.toList();
|
||||
final native = eumhun[character] ?? const [];
|
||||
final meanings = _meanings(unihan._of(character, 'kDefinition'));
|
||||
if (meanings.isEmpty && native.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
final taught = unihan.has(character, 'kKoreanEducationHanja');
|
||||
final radical = _radical(unihan, character);
|
||||
final strokes = _strokes(unihan, character, traditional: true);
|
||||
yield entryLine(
|
||||
DictionaryEntry(
|
||||
id: character,
|
||||
headword: character,
|
||||
reading: null,
|
||||
labels: [
|
||||
if (taught) 'education',
|
||||
if (radical != null) 'radical:$radical',
|
||||
],
|
||||
rank: (taught ? 0 : 100000) + (strokes ?? 99) * 1000,
|
||||
keys: {
|
||||
character,
|
||||
...eum,
|
||||
for (final (:hun, :eum) in native) ...[hun, '$hun $eum'],
|
||||
for (final meaning in meanings) ...glossKeys(meaning),
|
||||
}.toList(),
|
||||
entry: {
|
||||
'meanings': meanings,
|
||||
'eum': eum,
|
||||
if (native.isNotEmpty)
|
||||
'hun': [for (final (:hun, eum: _) in native) hun],
|
||||
if (native.isNotEmpty)
|
||||
'eumhun': [for (final (:hun, :eum) in native) '$hun $eum'],
|
||||
'strokes': ?strokes,
|
||||
},
|
||||
),
|
||||
);
|
||||
}
|
||||
for (final letter in hangulEntries()) {
|
||||
yield entryLine(letter);
|
||||
}
|
||||
}
|
||||
|
||||
/// Unihan's files in [zip] (`Unihan.zip`) as gzipped dictionary files in
|
||||
/// [into], for those of [languages] it serves: Chinese characters in both
|
||||
/// scripts for Chinese (`zh`, `zh-Hans`, …), with the HSK levels of
|
||||
/// characters in the curator's [data] folder and their places in the
|
||||
/// words of CC-CEDICT ([cedict], its text) as the word frequencies there
|
||||
/// rank them; Korean Hanja for `ko`, with [eumhun]. Returns the files
|
||||
/// written.
|
||||
Future<List<File>> unihanDictionaries(
|
||||
List<int> zip, {
|
||||
required Directory into,
|
||||
required Set<String> languages,
|
||||
required Directory data,
|
||||
String? cedict,
|
||||
Map<String, List<Eumhun>> eumhun = const {},
|
||||
}) async {
|
||||
bool primary(String language) =>
|
||||
languages.any((tag) => tag.split('-').first.toLowerCase() == language);
|
||||
if (!primary('zh') && !primary('ko')) {
|
||||
return const [];
|
||||
}
|
||||
final (:unihan, :version) = Unihan.ofZip(zip);
|
||||
final hsk = primary('zh') && hskCharactersFile(data).existsSync()
|
||||
? await readHskCharacters(hskCharactersFile(data))
|
||||
: const <String, int>{};
|
||||
final frequencies = File('${data.path}/wordfreq/small_zh.msgpack.gz');
|
||||
final words = primary('zh') && frequencies.existsSync()
|
||||
? await WordFrequencies.read(frequencies)
|
||||
: null;
|
||||
// Without CC-CEDICT, the simplified characters' places alone.
|
||||
final CharacterPlaces places = switch ((words, cedict)) {
|
||||
(final words?, final cedict?) => cedictCharacterPlaces(cedict, words),
|
||||
(final words?, null) => (
|
||||
simplified: characterPlaces(words),
|
||||
traditional: const {},
|
||||
),
|
||||
_ => noCharacterPlaces,
|
||||
};
|
||||
final placesCredits = [
|
||||
if (cedict != null && words != null) cedictCredit,
|
||||
if (words != null) ...wordfreqCredits,
|
||||
];
|
||||
Future<File> write(String name, Iterable<String> lines) async {
|
||||
final file = File('${into.path}/$name');
|
||||
await Stream.fromIterable(lines)
|
||||
.transform(utf8.encoder)
|
||||
.transform(gzip.encoder)
|
||||
.pipe(file.openWrite());
|
||||
return file;
|
||||
}
|
||||
|
||||
return [
|
||||
if (primary('zh'))
|
||||
for (final traditional in [false, true])
|
||||
await write(
|
||||
unihanFileName(traditional: traditional),
|
||||
chineseCharacters(
|
||||
unihan,
|
||||
traditional: traditional,
|
||||
sourceVersion: version,
|
||||
hsk: hsk,
|
||||
places: places,
|
||||
placesCredits: placesCredits,
|
||||
),
|
||||
),
|
||||
if (primary('ko'))
|
||||
await write(
|
||||
unihanFileName(korean: true),
|
||||
koreanHanja(unihan, sourceVersion: version, eumhun: eumhun),
|
||||
),
|
||||
];
|
||||
}
|
||||
|
|
@ -4,6 +4,7 @@ import 'dart:io';
|
|||
import 'package:halkyon_learn_curator/src/dictionary_file.dart';
|
||||
import 'package:halkyon_learn_curator/src/fold.dart';
|
||||
import 'package:halkyon_learn_curator/src/gloss_keys.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
|
||||
/// Who wrote Wiktionary, and who made it readable here.
|
||||
|
|
@ -141,6 +142,10 @@ const commonHomographs = 3;
|
|||
/// own; names and affixes are left out. [frequencies] by language rank the
|
||||
/// words, the most used first, and say which are `common`.
|
||||
///
|
||||
/// With [korean], it also gathers there what Korean's dictionaries take
|
||||
/// from Wiktionary beside KRDict's words: each Hanja's readings and each
|
||||
/// Korean word's forms ([WiktionaryKorean]).
|
||||
///
|
||||
/// Reads the dump once, keeping in memory only which forms belong to which
|
||||
/// word; the entries wait in temporary files until it is read. Returns
|
||||
/// the files written.
|
||||
|
|
@ -151,6 +156,7 @@ Future<List<File>> wiktionaryDictionaries(
|
|||
Set<String> languages = const {},
|
||||
Set<String> translatedInto = const {},
|
||||
Map<String, WordFrequencies> frequencies = const {},
|
||||
WiktionaryKorean? korean,
|
||||
}) async {
|
||||
final work = await Directory.systemTemp.createTemp('wiktionary');
|
||||
try {
|
||||
|
|
@ -162,7 +168,11 @@ Future<List<File>> wiktionaryDictionaries(
|
|||
};
|
||||
// Each language's forms, by the word they are forms of.
|
||||
final formsOf = <String, Map<String, Set<String>>>{};
|
||||
final wanted = {...languages, if (translatedInto.isNotEmpty) 'en'};
|
||||
final wanted = {
|
||||
...languages,
|
||||
if (translatedInto.isNotEmpty) 'en',
|
||||
if (korean != null) 'ko',
|
||||
};
|
||||
// A quick look before reading a line as JSON: written with a space
|
||||
// after the colon, as kaikki.org writes it, or without.
|
||||
final marks = [
|
||||
|
|
@ -185,6 +195,15 @@ Future<List<File>> wiktionaryDictionaries(
|
|||
}
|
||||
final word = json['word'];
|
||||
final pos = json['pos'];
|
||||
if (korean != null &&
|
||||
language == 'ko' &&
|
||||
word is String &&
|
||||
pos is String) {
|
||||
korean.add(json, word, pos);
|
||||
}
|
||||
if (language == 'ko' && pos == 'character') {
|
||||
continue;
|
||||
}
|
||||
if (word is! String || pos is! String || _notWords.contains(pos)) {
|
||||
continue;
|
||||
}
|
||||
|
|
@ -243,6 +262,80 @@ Future<List<File>> wiktionaryDictionaries(
|
|||
}
|
||||
}
|
||||
|
||||
/// What Korean's dictionaries take from Wiktionary beside KRDict's words,
|
||||
/// which lists only a few of a word's forms and no native readings of
|
||||
/// Hanja: each Hanja's readings as Korean dictionaries give them, its
|
||||
/// native meaning and its sound ([hanja]: `물 수` for 水, from Wiktionary's
|
||||
/// Korean entries of Han characters), and each word's forms ([forms]: the
|
||||
/// 99 of 가다 its conjugation table gives, to find it in running text).
|
||||
final class WiktionaryKorean() {
|
||||
/// What [json] ([toJson]) keeps.
|
||||
factory fromJson(String json) {
|
||||
final korean = WiktionaryKorean();
|
||||
final map = jsonDecode(json) as Map<String, Object?>;
|
||||
for (final MapEntry(key: character, value: readings)
|
||||
in (map['hanja']! as Map<String, Object?>).entries) {
|
||||
korean.hanja[character] = [
|
||||
for (final reading in (readings! as List).cast<String>())
|
||||
if (reading.lastIndexOf(' ') case final at when at > 0)
|
||||
(hun: reading.substring(0, at), eum: reading.substring(at + 1)),
|
||||
];
|
||||
}
|
||||
for (final MapEntry(key: word, value: forms)
|
||||
in (map['forms']! as Map<String, Object?>).entries) {
|
||||
korean.forms[word] = {...(forms! as List).cast<String>()};
|
||||
}
|
||||
return korean;
|
||||
}
|
||||
|
||||
final hanja = <String, List<Eumhun>>{};
|
||||
final forms = <String, Set<String>>{};
|
||||
|
||||
/// Takes what [json], a Korean entry of the dump for [word] as [pos],
|
||||
/// gives.
|
||||
void add(Map<String, Object?> json, String word, String pos) {
|
||||
if (pos == 'character') {
|
||||
if (_eumhun(json) case final readings when readings.isNotEmpty) {
|
||||
(hanja[word] ??= []).addAll(readings);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (_notWords.contains(pos)) {
|
||||
return;
|
||||
}
|
||||
final own = _textKeys(word, pos, 'ko', _maps(json['forms']))
|
||||
..remove(word.toLowerCase());
|
||||
if (own.isNotEmpty) {
|
||||
(forms[word] ??= {}).addAll(own);
|
||||
}
|
||||
}
|
||||
|
||||
/// What it holds, as a JSON file keeps it between updates.
|
||||
String toJson() => jsonEncode({
|
||||
'hanja': {
|
||||
for (final MapEntry(key: character, value: readings) in hanja.entries)
|
||||
character: [for (final (:hun, :eum) in readings) '$hun $eum'],
|
||||
},
|
||||
'forms': {
|
||||
for (final MapEntry(key: word, value: forms) in forms.entries)
|
||||
word: forms.toList(),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
/// A Korean entry of a Han character's readings, each its native meaning
|
||||
/// and its sound (`{"form": "물 수", "tags": ["eumhun"]}`).
|
||||
List<Eumhun> _eumhun(Map<String, Object?> json) => [
|
||||
for (final form in _maps(json['forms']))
|
||||
if (_strings(form['tags']).contains('eumhun'))
|
||||
if (form['form'] case final String reading)
|
||||
if (reading.trim().lastIndexOf(' ') case final at when at > 0)
|
||||
(
|
||||
hun: reading.trim().substring(0, at).trim(),
|
||||
eum: reading.trim().substring(at + 1),
|
||||
),
|
||||
];
|
||||
|
||||
/// The entries of one dictionary file, waiting in a temporary file until
|
||||
/// the dump is read.
|
||||
final class _Output(
|
||||
|
|
|
|||
|
|
@ -3,11 +3,14 @@ import 'dart:io';
|
|||
|
||||
import 'package:archive/archive_io.dart';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/sources/alphabets.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/cedict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jlpt.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/krdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
|
||||
|
|
@ -151,7 +154,14 @@ final wiktionarySource = Uri.parse(
|
|||
);
|
||||
|
||||
/// How the curator converts Wiktionary now; raise it as [conversion].
|
||||
const wiktionaryConversion = 1;
|
||||
/// 2: Korean's words from KRDict, Wiktionary keeping their forms and the
|
||||
/// Hanja's readings beside them ([wiktionaryKoreanFile]).
|
||||
const wiktionaryConversion = 2;
|
||||
|
||||
/// The file in the dictionaries' folder where a Wiktionary update keeps
|
||||
/// what Korean's dictionaries take from it ([WiktionaryKorean], gzipped
|
||||
/// JSON; the server reads only dictionary files).
|
||||
const wiktionaryKoreanFile = '.wiktionary-ko.json.gz';
|
||||
|
||||
/// The file in the dictionaries' folder that says which dump, languages
|
||||
/// and [wiktionaryConversion] made the Wiktionary files.
|
||||
|
|
@ -223,22 +233,24 @@ DateTime _modifiedOf(Uri uri, HttpClientResponse response) {
|
|||
/// as it downloads, never kept (some 3 GB); each file is written beside
|
||||
/// its place and renamed over it. The word frequencies are in the
|
||||
/// curator's [data] folder (`wordfreq/`). Each file takes Tatoeba's
|
||||
/// sentences as examples before it is put in place. Returns the files
|
||||
/// written, none when they were up to date.
|
||||
/// sentences as examples before it is put in place. For [korean], it keeps
|
||||
/// what Korean's dictionaries take from the dump ([wiktionaryKoreanFile]).
|
||||
/// Returns the files written, none when they were up to date.
|
||||
Future<List<File>> updateWiktionary({
|
||||
required Directory into,
|
||||
required Directory data,
|
||||
required Set<String> languages,
|
||||
bool korean = false,
|
||||
Modified modified = httpModified,
|
||||
Open open = httpOpen,
|
||||
Fetch fetch = httpFetch,
|
||||
}) async {
|
||||
if (languages.isEmpty) {
|
||||
if (languages.isEmpty && !korean) {
|
||||
return const [];
|
||||
}
|
||||
final state = File('${into.path}/$wiktionaryStateFile');
|
||||
final dumped = await modified(wiktionarySource);
|
||||
final wanted = _wiktionaryState(dumped, languages);
|
||||
final wanted = _wiktionaryState(dumped, languages, korean: korean);
|
||||
if (state.existsSync() && state.readAsStringSync() == wanted) {
|
||||
return const [];
|
||||
}
|
||||
|
|
@ -251,6 +263,7 @@ Future<List<File>> updateWiktionary({
|
|||
language: await WordFrequencies.read(file),
|
||||
};
|
||||
final work = await Directory.systemTemp.createTemp('wiktionary-update');
|
||||
final koreanFromDump = korean ? WiktionaryKorean() : null;
|
||||
try {
|
||||
final converted = await wiktionaryDictionaries(
|
||||
download.bytes
|
||||
|
|
@ -262,8 +275,16 @@ Future<List<File>> updateWiktionary({
|
|||
languages: outputs.languages,
|
||||
translatedInto: outputs.translatedInto,
|
||||
frequencies: frequencies,
|
||||
korean: koreanFromDump,
|
||||
);
|
||||
await into.create(recursive: true);
|
||||
if (koreanFromDump != null) {
|
||||
final kept = File('${work.path}/$wiktionaryKoreanFile');
|
||||
await kept.writeAsBytes(
|
||||
gzip.encode(utf8.encode(koreanFromDump.toJson())),
|
||||
);
|
||||
await _putInPlace(kept, into);
|
||||
}
|
||||
final written = [
|
||||
for (final file in converted)
|
||||
await _putInPlace(
|
||||
|
|
@ -272,7 +293,9 @@ Future<List<File>> updateWiktionary({
|
|||
name: file.uri.pathSegments.last,
|
||||
),
|
||||
];
|
||||
await state.writeAsString(_wiktionaryState(download.modified, languages));
|
||||
await state.writeAsString(
|
||||
_wiktionaryState(download.modified, languages, korean: korean),
|
||||
);
|
||||
return written;
|
||||
} finally {
|
||||
await work.delete(recursive: true);
|
||||
|
|
@ -296,11 +319,16 @@ Future<File> _withExamples(
|
|||
|
||||
/// What [wiktionaryStateFile] says of files made from the dump of
|
||||
/// [dumped] for [languages].
|
||||
String _wiktionaryState(DateTime dumped, Set<String> languages) =>
|
||||
String _wiktionaryState(
|
||||
DateTime dumped,
|
||||
Set<String> languages, {
|
||||
required bool korean,
|
||||
}) =>
|
||||
'${jsonEncode({
|
||||
'conversion': wiktionaryConversion,
|
||||
'dump': dumped.toUtc().toIso8601String(),
|
||||
'languages': [...languages]..sort(),
|
||||
'korean': korean,
|
||||
})}\n';
|
||||
|
||||
/// CC-CEDICT, as MDBG publishes it every day.
|
||||
|
|
@ -376,21 +404,75 @@ DateTime? cedictUpdated(Directory folder) {
|
|||
}
|
||||
|
||||
/// Of the languages an instance keeps dictionaries of besides Japanese
|
||||
/// ([languages], BCP 47 tags), those Wiktionary gives, and whether Chinese
|
||||
/// (CC-CEDICT, in both scripts) is among them.
|
||||
({Set<String> wiktionary, bool chinese}) dictionarySources(
|
||||
Set<String> languages,
|
||||
) {
|
||||
bool isChinese(String tag) => tag.split('-').first.toLowerCase() == 'zh';
|
||||
/// ([languages], BCP 47 tags), those whose words Wiktionary gives; whether
|
||||
/// Chinese is among them (CC-CEDICT's words, Unihan's characters, in both
|
||||
/// scripts), Korean (KRDict's words, Unihan's Hanja, the Hangul letters)
|
||||
/// and Greek (its alphabet besides Wiktionary's words).
|
||||
({Set<String> wiktionary, bool chinese, bool korean, bool greek})
|
||||
dictionarySources(Set<String> languages) {
|
||||
String primary(String tag) => tag.split('-').first.toLowerCase();
|
||||
return (
|
||||
wiktionary: {
|
||||
for (final language in languages)
|
||||
if (!isChinese(language)) language,
|
||||
if (!{'zh', 'ko'}.contains(primary(language))) language,
|
||||
},
|
||||
chinese: languages.any(isChinese),
|
||||
chinese: languages.any((tag) => primary(tag) == 'zh'),
|
||||
korean: languages.any((tag) => primary(tag) == 'ko'),
|
||||
greek: languages.any((tag) => primary(tag) == 'el'),
|
||||
);
|
||||
}
|
||||
|
||||
/// The dictionary files an update keeps in its folder for [languages]
|
||||
/// besides Japanese; the curator's other files there are of languages no
|
||||
/// longer kept, or of sources replaced (Korean's words from Wiktionary
|
||||
/// before KRDict), and go ([removeUnwanted]).
|
||||
Set<String> wantedFiles(Set<String> languages) {
|
||||
final (:wiktionary, :chinese, :korean, :greek) = dictionarySources(languages);
|
||||
final outputs = wiktionaryOutputs(wiktionary);
|
||||
return {
|
||||
...updatedFiles,
|
||||
for (final language in outputs.languages)
|
||||
wiktionaryFileName(language, 'en'),
|
||||
for (final gloss in outputs.translatedInto) wiktionaryFileName('en', gloss),
|
||||
if (chinese) ...[
|
||||
cedictFileName(traditional: false),
|
||||
cedictFileName(traditional: true),
|
||||
unihanFileName(),
|
||||
unihanFileName(traditional: true),
|
||||
],
|
||||
if (korean) ...[krdictFileName, unihanFileName(korean: true)],
|
||||
if (greek) greekAlphabetFileName,
|
||||
};
|
||||
}
|
||||
|
||||
/// Removes from [into] the dictionary files the curator writes that
|
||||
/// [languages] no longer want ([wantedFiles]), so the server unloads them;
|
||||
/// files of others (an operator's own) stay. Returns the files removed.
|
||||
List<File> removeUnwanted(Directory into, Set<String> languages) {
|
||||
if (!into.existsSync()) {
|
||||
return const [];
|
||||
}
|
||||
final wanted = wantedFiles(languages);
|
||||
final removed = <File>[];
|
||||
for (final entity in into.listSync()) {
|
||||
final name = entity.uri.pathSegments.last;
|
||||
if (entity is File &&
|
||||
name.endsWith('.jsonl.gz') &&
|
||||
const [
|
||||
'wiktionary-',
|
||||
'cc-cedict-',
|
||||
'unihan-',
|
||||
'krdict-',
|
||||
'greek-alphabet-',
|
||||
].any(name.startsWith) &&
|
||||
!wanted.contains(name)) {
|
||||
entity.deleteSync();
|
||||
removed.add(entity);
|
||||
}
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
|
||||
/// Tatoeba's weekly export of a language's sentences ([code], ISO 639-3).
|
||||
Uri tatoebaSentencesOf(String code) => Uri.parse(
|
||||
'https://downloads.tatoeba.org/exports/per_language/$code/'
|
||||
|
|
@ -470,3 +552,186 @@ Future<File> addTatoebaExamples(
|
|||
),
|
||||
);
|
||||
}
|
||||
|
||||
/// KRDict's page of exports, which names the current one's download.
|
||||
final krdictPage = Uri.parse(
|
||||
'https://krdict.korean.go.kr/download/downloadPopup',
|
||||
);
|
||||
|
||||
/// The download of KRDict's export numbered [number].
|
||||
Uri krdictDownloadOf(String number) =>
|
||||
Uri.parse('https://krdict.korean.go.kr/dicBatchDownload?seq=$number');
|
||||
|
||||
/// How the curator converts KRDict now; raise it as [conversion].
|
||||
const krdictConversion = 1;
|
||||
|
||||
/// The file in the dictionaries' folder that says which export, which of
|
||||
/// Wiktionary's Korean forms ([wiktionaryKoreanFile]) and which
|
||||
/// [krdictConversion] made KRDict's file.
|
||||
const krdictStateFile = '.krdict';
|
||||
|
||||
/// Brings KRDict's file in [into] up to date: only when its page offers
|
||||
/// another export (about monthly, some 84 MB zipped, a gigabyte of JSON
|
||||
/// read an entry at a time), Wiktionary's Korean forms changed, or how the
|
||||
/// curator converts it; with the word frequencies in the curator's [data]
|
||||
/// folder and Tatoeba's sentences as examples. Returns the files written,
|
||||
/// none when it was up to date.
|
||||
Future<List<File>> updateKrdict({
|
||||
required Directory into,
|
||||
required Directory data,
|
||||
Fetch fetch = httpFetch,
|
||||
}) async {
|
||||
final work = await Directory.systemTemp.createTemp('krdict');
|
||||
try {
|
||||
final page = File('${work.path}/page.html');
|
||||
await fetch(krdictPage, page);
|
||||
final number = krdictJsonDownloadOf(await page.readAsString());
|
||||
if (number == null) {
|
||||
throw StateError("KRDict's page offers no export as JSON.");
|
||||
}
|
||||
final kept = File('${into.path}/$wiktionaryKoreanFile');
|
||||
final wanted =
|
||||
'${jsonEncode({'conversion': krdictConversion, 'export': number, 'wiktionary': kept.existsSync() ? kept.lastModifiedSync().toUtc().toIso8601String() : null})}\n';
|
||||
final state = File('${into.path}/$krdictStateFile');
|
||||
if (File('${into.path}/$krdictFileName').existsSync() &&
|
||||
state.existsSync() &&
|
||||
state.readAsStringSync() == wanted) {
|
||||
return const [];
|
||||
}
|
||||
final zip = File('${work.path}/krdict.zip');
|
||||
await fetch(krdictDownloadOf(number), zip);
|
||||
final (:version, :entries) = krdictExport(await zip.readAsBytes());
|
||||
final frequencies = File('${data.path}/wordfreq/small_ko.msgpack.gz');
|
||||
final written = await _write(
|
||||
File('${work.path}/$krdictFileName'),
|
||||
krdictDictionary(
|
||||
entries,
|
||||
sourceVersion: version,
|
||||
frequencies: frequencies.existsSync()
|
||||
? await WordFrequencies.read(frequencies)
|
||||
: null,
|
||||
korean: kept.existsSync()
|
||||
? WiktionaryKorean.fromJson(
|
||||
utf8.decode(gzip.decode(await kept.readAsBytes())),
|
||||
)
|
||||
: null,
|
||||
),
|
||||
);
|
||||
await zip.delete();
|
||||
await into.create(recursive: true);
|
||||
final placed = await _putInPlace(
|
||||
await _withExamples(written, work: work, fetch: fetch),
|
||||
into,
|
||||
name: krdictFileName,
|
||||
);
|
||||
await state.writeAsString(wanted);
|
||||
return [placed];
|
||||
} finally {
|
||||
await work.delete(recursive: true);
|
||||
}
|
||||
}
|
||||
|
||||
/// The Unihan database, as unicode.org publishes it with every version of
|
||||
/// Unicode.
|
||||
final unihanSource = Uri.parse(
|
||||
'https://www.unicode.org/Public/UCD/latest/ucd/Unihan.zip',
|
||||
);
|
||||
|
||||
/// Brings the dictionary files of characters in [into] up to date: Chinese
|
||||
/// characters in both scripts for [chinese] (Unihan, ranked by CC-CEDICT's
|
||||
/// words, which also tell the traditional ones in use; HSK levels from the
|
||||
/// curator's [data] folder), Korean Hanja with the Hangul letters for
|
||||
/// [korean] (with the Hanja's native readings a Wiktionary update kept),
|
||||
/// and the Greek alphabet for [greek]; each written beside its place and
|
||||
/// renamed over it. Returns the files written.
|
||||
Future<List<File>> updateCharacters({
|
||||
required Directory into,
|
||||
required Directory data,
|
||||
bool chinese = false,
|
||||
bool korean = false,
|
||||
bool greek = false,
|
||||
Fetch fetch = httpFetch,
|
||||
}) async {
|
||||
if (!chinese && !korean && !greek) {
|
||||
return const [];
|
||||
}
|
||||
final work = await Directory.systemTemp.createTemp('characters');
|
||||
try {
|
||||
final made = <File>[];
|
||||
if (chinese || korean) {
|
||||
final zip = File('${work.path}/Unihan.zip');
|
||||
await fetch(unihanSource, zip);
|
||||
String? cedict;
|
||||
if (chinese) {
|
||||
final download = File('${work.path}/cedict.txt.gz');
|
||||
await fetch(cedictSource, download);
|
||||
cedict = utf8.decode(gzip.decode(await download.readAsBytes()));
|
||||
}
|
||||
final kept = File('${into.path}/$wiktionaryKoreanFile');
|
||||
made.addAll(
|
||||
await unihanDictionaries(
|
||||
await zip.readAsBytes(),
|
||||
into: work,
|
||||
languages: {if (chinese) 'zh', if (korean) 'ko'},
|
||||
data: data,
|
||||
cedict: cedict,
|
||||
eumhun: korean && kept.existsSync()
|
||||
? WiktionaryKorean.fromJson(
|
||||
utf8.decode(gzip.decode(await kept.readAsBytes())),
|
||||
).hanja
|
||||
: const {},
|
||||
),
|
||||
);
|
||||
}
|
||||
if (greek) {
|
||||
made.add(
|
||||
await _write(
|
||||
File('${work.path}/$greekAlphabetFileName'),
|
||||
Stream.fromIterable(greekAlphabet()),
|
||||
),
|
||||
);
|
||||
}
|
||||
await into.create(recursive: true);
|
||||
return [for (final file in made) await _putInPlace(file, into)];
|
||||
} finally {
|
||||
await work.delete(recursive: true);
|
||||
}
|
||||
}
|
||||
|
||||
/// When the dictionary files of characters in [folder] for [chinese],
|
||||
/// [korean] and [greek] were last brought up to date: the oldest of them;
|
||||
/// null when one is missing, or when the Hanja's readings a Wiktionary
|
||||
/// update kept are newer than the Hanja's file.
|
||||
DateTime? charactersUpdated(
|
||||
Directory folder, {
|
||||
bool chinese = false,
|
||||
bool korean = false,
|
||||
bool greek = false,
|
||||
}) {
|
||||
final names = [
|
||||
if (chinese) ...[unihanFileName(), unihanFileName(traditional: true)],
|
||||
if (korean) unihanFileName(korean: true),
|
||||
if (greek) greekAlphabetFileName,
|
||||
];
|
||||
DateTime? oldest;
|
||||
for (final name in names) {
|
||||
final file = File('${folder.path}/$name');
|
||||
if (!file.existsSync()) {
|
||||
return null;
|
||||
}
|
||||
final modified = file.lastModifiedSync();
|
||||
if (oldest == null || modified.isBefore(oldest)) {
|
||||
oldest = modified;
|
||||
}
|
||||
}
|
||||
final kept = File('${folder.path}/$wiktionaryKoreanFile');
|
||||
if (korean &&
|
||||
kept.existsSync() &&
|
||||
kept.lastModifiedSync().isAfter(
|
||||
File('${folder.path}/${unihanFileName(korean: true)}')
|
||||
.lastModifiedSync(),
|
||||
)) {
|
||||
return null;
|
||||
}
|
||||
return oldest;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -52,6 +52,9 @@ final class WordFrequencies(final Map<String, int> _places) {
|
|||
|
||||
/// How many words the list holds.
|
||||
int get length => _places.length;
|
||||
|
||||
/// The words, the most used first.
|
||||
Iterable<String> get words => _places.keys;
|
||||
}
|
||||
|
||||
/// Just enough of msgpack (https://msgpack.org) to read wordfreq's lists:
|
||||
|
|
|
|||
45
halkyon_learn_curator/test/alphabets_test.dart
Normal file
45
halkyon_learn_curator/test/alphabets_test.dart
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
import 'dart:convert';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/sources/alphabets.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
void main() {
|
||||
test('the Greek alphabet in its order, each letter with its capital, '
|
||||
'names, transliteration and sound', () {
|
||||
final [header, ...letters] = [
|
||||
for (final line in greekAlphabet())
|
||||
jsonDecode(line) as Map<String, Object?>,
|
||||
];
|
||||
|
||||
expect(header['source'], 'greek-alphabet');
|
||||
expect(header['kind'], 'characters');
|
||||
expect(header['language'], 'el');
|
||||
expect(letters, hasLength(24));
|
||||
expect(letters.first['headword'], 'Α α');
|
||||
final sigma = letters.singleWhere((letter) => letter['id'] == 'σ');
|
||||
expect(sigma['headword'], 'Σ σ ς');
|
||||
expect(sigma['keys'], containsAll(['ς', 'σίγμα', 'sigma', 's']));
|
||||
expect(letters[1]['entry'], {
|
||||
'romaji': 'v',
|
||||
'script': 'greek',
|
||||
'name': 'βήτα',
|
||||
'englishName': 'beta',
|
||||
'sound': 'v',
|
||||
});
|
||||
// Found by a name typed without its accent.
|
||||
expect(letters.last['keys'], contains('ωμεγα'));
|
||||
});
|
||||
|
||||
test('the Hangul letters in the order schools teach them, found by sound '
|
||||
'and name', () {
|
||||
final letters = hangulEntries();
|
||||
|
||||
expect(letters, hasLength(40));
|
||||
expect(letters.first.headword, 'ㄱ');
|
||||
expect(letters.first.keys, containsAll(['ㄱ', '기역', 'g', 'k']));
|
||||
// ㅇ is silent opening a syllable: no key for its sound there.
|
||||
final ieung = letters.singleWhere((letter) => letter.id == 'ㅇ');
|
||||
expect(ieung.keys, ['ㅇ', '이응', 'ng']);
|
||||
expect(letters.last.headword, 'ㅢ');
|
||||
});
|
||||
}
|
||||
23
halkyon_learn_curator/test/fixtures/README.md
vendored
23
halkyon_learn_curator/test/fixtures/README.md
vendored
|
|
@ -30,11 +30,12 @@ declarations of the entities they use: JMdict is © the Electronic Dictionary
|
|||
Research and Development Group, used under its licence, CC BY-SA 4.0
|
||||
(<https://www.edrdg.org/edrdg/licence.html>).
|
||||
|
||||
`wiktionary-sample.jsonl` holds 30 lines of kaikki.org's Wiktextract data of the
|
||||
`wiktionary-sample.jsonl` holds 31 lines of kaikki.org's Wiktextract data of the
|
||||
English Wiktionary (the dump of 2026-10-03, read per word from kaikki.org): the
|
||||
German Haus, Häuser, Häusern, essen, aß, gegessen, Straße and im, the Greek σπίτι,
|
||||
σπιτιού and γράφω, the English house, houses, go and went, the French maison,
|
||||
and the Korean 먹다, 집 and 학생, each shortened to the fields the curator reads (no categories,
|
||||
the Korean 먹다, 집 and 학생, and the Korean entry of the Han character 水 (its readings,
|
||||
물 수), each shortened to the fields the curator reads (no categories,
|
||||
etymologies or related words; at most six senses, two examples, 40 forms, two
|
||||
sounds, and 14 translations, those into German and Greek). Wiktionary's text is
|
||||
by its contributors, CC BY-SA 4.0 (<https://en.wiktionary.org/wiki/Wiktionary:Copyrights>);
|
||||
|
|
@ -51,3 +52,21 @@ Korean sentences using Haus, aß, Straße, 学习, 집에서 and 학생이 (and
|
|||
without an English translation), the links to their English translations, and
|
||||
those translations. The sentences are by Tatoeba's contributors, CC BY 2.0 FR
|
||||
(<https://tatoeba.org/en/terms_of_use>).
|
||||
|
||||
`unihan-sample.txt` holds the lines of Unihan 18.0.0 (2026-07-31,
|
||||
<https://www.unicode.org/Public/UCD/latest/ucd/Unihan.zip>) the curator reads for
|
||||
水, 说, 說, 説, 這, 这, 了, 瞭, 后, 後, 行, 们, 們, 学 and 學, with a header like its
|
||||
files'. The Unihan Database is © Unicode, Inc., under the Unicode License v3
|
||||
(<https://www.unicode.org/license.txt>).
|
||||
|
||||
`cedict-characters-sample.txt` holds 17 lines of CC-CEDICT (MDBG's file of
|
||||
2026-10-14) for the words those characters write (這, 了, 瞭, 瞭解, 為了, 後, 后,
|
||||
以後, 皇后, 們, 我們, 說, 水, 行, 銀行, 學 and 學習), CC BY-SA 4.0 as above.
|
||||
|
||||
`krdict-sample.json` holds six entries of KRDict's JSON export of 2026-09-19
|
||||
(<https://krdict.korean.go.kr/download/downloadPopup>): 같이 (the adverb and the
|
||||
particle), 가다 with its proverb 가는 날이 장날, 읽다 and 학교, each cut to its first
|
||||
three senses, their English and Japanese equivalents and two examples, without
|
||||
sound files or related words. 한국어기초사전 is by the National Institute of
|
||||
Korean Language (국립국어원), CC BY-SA 2.0 KR
|
||||
(<https://krdict.korean.go.kr/kboardPolicy/copyRightTermsInfo>).
|
||||
|
|
|
|||
19
halkyon_learn_curator/test/fixtures/cedict-characters-sample.txt
vendored
Normal file
19
halkyon_learn_curator/test/fixtures/cedict-characters-sample.txt
vendored
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# CC-CEDICT
|
||||
#! date=2026-10-14T07:20:00Z
|
||||
了 了 [le5] /(completed action marker)/(modal particle indicating change of state, situation now)/(modal particle intensifying preceding clause)/
|
||||
以後 以后 [yi3 hou4] /after/later/afterwards/following/later on/in the future/
|
||||
們 们 [men5] /plural marker for pronouns, and nouns referring to individuals/
|
||||
后 后 [hou4] /empress/queen/(archaic) monarch/ruler/
|
||||
學 学 [xue2] /to learn/to study/to imitate/science/-ology/
|
||||
學習 学习 [xue2 xi2] /to learn; to study/
|
||||
後 后 [hou4] /back/behind/rear/afterwards/after/later/post-/
|
||||
我們 我们 [wo3 men5] /we; us; ourselves/
|
||||
水 水 [shui3] /water/(after a name) ... River/to swim (used mostly in 會水|会水[hui4 shui3] and 水性[shui3 xing4])/(coll.) lacking in substance; shoddy/(bound form) additional cost; extra income/classifier for washings of a garment/
|
||||
為了 为了 [wei4 le5] /for; for the purpose of; in order to/
|
||||
皇后 皇后 [huang2 hou4] /empress/imperial consort/
|
||||
瞭 了 [liao3] /(bound form) clear in one's mind; to understand clearly/
|
||||
瞭解 了解 [liao3 jie3] /variant of 了解[liao3 jie3]/
|
||||
行 行 [xing2] /(bound form) to walk; to go; to travel/(literary) trip; journey; visit/(bound form) temporary; makeshift/(bound form) current; in circulation/(bound form) to do; to perform/capable; competent/all right; OK!; will do/behavior; conduct (Taiwan pr. [xing4])/(literary) about to; soon/
|
||||
說 说 [shuo1] /to speak; to talk; to say/to explain; to comment/to scold; to tell off/(bound form) theory; doctrine/
|
||||
這 这 [zhe4] /(pronoun) this; these/(bound form) this; the (followed by a noun)/(bound form) this; these (followed by a classifier) (in this sense, commonly pr. [zhei4], esp. in Beijing)/
|
||||
銀行 银行 [yin2 hang2] /bank/CL:家[jia1],個|个[ge4]/
|
||||
1354
halkyon_learn_curator/test/fixtures/krdict-sample.json
vendored
Normal file
1354
halkyon_learn_curator/test/fixtures/krdict-sample.json
vendored
Normal file
File diff suppressed because it is too large
Load diff
147
halkyon_learn_curator/test/fixtures/unihan-sample.txt
vendored
Normal file
147
halkyon_learn_curator/test/fixtures/unihan-sample.txt
vendored
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# Unihan_Readings.txt
|
||||
# Date: 2026-07-31 00:00:00 GMT [KL]
|
||||
# Unicode Version 18.0.0
|
||||
U+4E86 kGradeLevel 1
|
||||
U+5011 kGradeLevel 1
|
||||
U+540E kGradeLevel 3
|
||||
U+5B78 kGradeLevel 1
|
||||
U+5F8C kGradeLevel 1
|
||||
U+6C34 kGradeLevel 1
|
||||
U+77AD kGradeLevel 4
|
||||
U+884C kGradeLevel 1
|
||||
U+8AAA kGradeLevel 1
|
||||
U+9019 kGradeLevel 1
|
||||
U+4E86 kRSUnicode 6.1
|
||||
U+4E86 kTotalStrokes 2
|
||||
U+4EEC kRSUnicode 9.3
|
||||
U+4EEC kTotalStrokes 5
|
||||
U+5011 kRSUnicode 9.8
|
||||
U+5011 kTotalStrokes 10
|
||||
U+540E kRSUnicode 30.3
|
||||
U+540E kTotalStrokes 6
|
||||
U+5B66 kRSUnicode 39.5
|
||||
U+5B66 kTotalStrokes 8
|
||||
U+5B78 kRSUnicode 39.13
|
||||
U+5B78 kTotalStrokes 16
|
||||
U+5F8C kRSUnicode 60.6
|
||||
U+5F8C kTotalStrokes 9
|
||||
U+6C34 kRSUnicode 85.0
|
||||
U+6C34 kTotalStrokes 4
|
||||
U+77AD kRSUnicode 109.12
|
||||
U+77AD kTotalStrokes 17
|
||||
U+884C kRSUnicode 144.0
|
||||
U+884C kTotalStrokes 6
|
||||
U+8AAA kRSUnicode 149.7
|
||||
U+8AAA kTotalStrokes 14
|
||||
U+8AAC kRSUnicode 149.7
|
||||
U+8AAC kTotalStrokes 14
|
||||
U+8BF4 kRSUnicode 149'.7
|
||||
U+8BF4 kTotalStrokes 9
|
||||
U+8FD9 kRSUnicode 162.4
|
||||
U+8FD9 kTotalStrokes 8
|
||||
U+9019 kRSUnicode 162.7
|
||||
U+9019 kTotalStrokes 11
|
||||
U+4E86 kKoreanEducationHanja 2007
|
||||
U+4E86 kTGH 2013:17
|
||||
U+4EEC kTGH 2013:273
|
||||
U+540E kTGH 2013:482
|
||||
U+5B66 kTGH 2013:1273
|
||||
U+5B78 kKoreanEducationHanja 2007
|
||||
U+5F8C kKoreanEducationHanja 2007
|
||||
U+6C34 kKoreanEducationHanja 2007
|
||||
U+6C34 kTGH 2013:117
|
||||
U+77AD kTGH 2013:3401
|
||||
U+884C kKoreanEducationHanja 2007
|
||||
U+884C kTGH 2013:483
|
||||
U+8AAA kKoreanEducationHanja 2007
|
||||
U+8BF4 kTGH 2013:1717
|
||||
U+8FD9 kTGH 2013:837
|
||||
U+4E86 kDefinition to finish; particle of completed action
|
||||
U+4E86 kHangul 료:0E 요:0
|
||||
U+4E86 kHanyuPinyin 10048.060:liǎo,le,liào
|
||||
U+4E86 kMandarin le
|
||||
U+4E86 kTGHZ2013 212.080:le 223.010,223.020:liǎo
|
||||
U+4E86 kXHC1983 0684.060:le 0711.150:liǎo 0712.070:liào
|
||||
U+4EEC kDefinition adjunct pronoun indicate plural
|
||||
U+4EEC kMandarin men
|
||||
U+4EEC kTGHZ2013 247.070:mén 247.150:men
|
||||
U+4EEC kXHC1983 0777.030:men
|
||||
U+5011 kDefinition adjunct pronoun indicate plural
|
||||
U+5011 kHangul 문:0N
|
||||
U+5011 kHanyuPinyin 10185.020:mèn,mén
|
||||
U+5011 kMandarin mén
|
||||
U+540E kDefinition queen, empress, sovereign; (simp. for 後) behind, rear, after
|
||||
U+540E kHangul 후:0N
|
||||
U+540E kHanyuPinyin 10580.010:hòu
|
||||
U+540E kMandarin hòu
|
||||
U+540E kTGHZ2013 139.070,139.080:hòu
|
||||
U+540E kXHC1983 0468.010:hòu
|
||||
U+5B66 kDefinition learning, knowledge; school
|
||||
U+5B66 kHangul 학:N
|
||||
U+5B66 kMandarin xué
|
||||
U+5B66 kTGHZ2013 416.090:xué
|
||||
U+5B66 kXHC1983 1308.120:xué
|
||||
U+5B78 kDefinition learning, knowledge; school
|
||||
U+5B78 kHangul 학:0E
|
||||
U+5B78 kHanyuPinyin 21019.110:xué,huá,jiào
|
||||
U+5B78 kMandarin xué
|
||||
U+5B78 kXHC1983 1308.121:xué
|
||||
U+5F8C kDefinition behind, rear, after; descendents
|
||||
U+5F8C kHangul 후:0E
|
||||
U+5F8C kHanyuPinyin 20822.030:hòu
|
||||
U+5F8C kMandarin hòu
|
||||
U+5F8C kXHC1983 0468.011:hòu
|
||||
U+6C34 kDefinition water, liquid, lotion, juice
|
||||
U+6C34 kHangul 수:0E
|
||||
U+6C34 kHanyuPinyin 31545.010:shuǐ
|
||||
U+6C34 kMandarin shuǐ
|
||||
U+6C34 kTGHZ2013 344.130:shuǐ
|
||||
U+6C34 kXHC1983 1075.010:shuǐ
|
||||
U+77AD kDefinition bright, clear; clear-sighted
|
||||
U+77AD kHangul 료:0N
|
||||
U+77AD kHanyuPinyin 42512.170:liǎo,liào
|
||||
U+77AD kMandarin liào liǎo
|
||||
U+77AD kTGHZ2013 223.130:liào
|
||||
U+77AD kXHC1983 0711.151:liǎo 0712.071:liào
|
||||
U+884C kDefinition go; walk; move, travel; circulate; Kangxi radical 144
|
||||
U+884C kHangul 항:0N 행:0E
|
||||
U+884C kHanyuPinyin 20811.060:háng,xìng,xíng,hàng,héng
|
||||
U+884C kMandarin xíng
|
||||
U+884C kTGHZ2013 131.140:háng 136.100:héng 408.120:xíng
|
||||
U+884C kXHC1983 0442.080:háng 0443.050:hàng 0460.010:héng 1290.030:xíng
|
||||
U+8AAA kDefinition speak, say, talk; scold, upbraid
|
||||
U+8AAA kHangul 설:0E 세:0N 열:0N
|
||||
U+8AAA kMandarin shuō
|
||||
U+8AAC kDefinition speak
|
||||
U+8AAC kHanyuPinyin 63979.030:shuō,shuì,yuè,tuō
|
||||
U+8AAC kMandarin shuō
|
||||
U+8BF4 kDefinition speak, say, talk; scold, upbraid
|
||||
U+8BF4 kMandarin shuō
|
||||
U+8BF4 kTGHZ2013 344.140:shuì 345.070:shuō
|
||||
U+8BF4 kXHC1983 1080.010:shuì 1082.010:shuō 1428.010:yuè
|
||||
U+8FD9 kDefinition this, the, here
|
||||
U+8FD9 kMandarin zhè
|
||||
U+8FD9 kTGHZ2013 470.100:zhè
|
||||
U+8FD9 kXHC1983 1464.130:zhè 1466.010:zhèi
|
||||
U+9019 kDefinition this, the, here
|
||||
U+9019 kHangul 저:0N
|
||||
U+9019 kHanyuPinyin 63844.050:yàn,zhè,zhèi
|
||||
U+9019 kMandarin zhè
|
||||
U+9019 kXHC1983 1464.131:zhè 1466.011:zhèi
|
||||
U+4E86 kSimplifiedVariant U+4E86
|
||||
U+4E86 kTraditionalVariant U+4E86 U+77AD
|
||||
U+4EEC kTraditionalVariant U+5011
|
||||
U+5011 kSimplifiedVariant U+4EEC
|
||||
U+540E kSimplifiedVariant U+540E
|
||||
U+540E kTraditionalVariant U+540E U+5F8C
|
||||
U+5B66 kTraditionalVariant U+5B78
|
||||
U+5B78 kSimplifiedVariant U+5B66
|
||||
U+5F8C kSimplifiedVariant U+540E
|
||||
U+77AD kSimplifiedVariant U+4E86 U+77AD
|
||||
U+77AD kTraditionalVariant U+77AD
|
||||
U+8AAA kSimplifiedVariant U+8BF4
|
||||
U+8AAC kSimplifiedVariant U+8BF4
|
||||
U+8BF4 kTraditionalVariant U+8AAA U+8AAC
|
||||
U+8FD9 kSimplifiedVariant U+8FD9
|
||||
U+8FD9 kTraditionalVariant U+8FD9 U+9019
|
||||
U+9019 kSimplifiedVariant U+8FD9
|
||||
|
|
@ -28,3 +28,4 @@
|
|||
{"senses": [{"examples": [{"text": "작품(作品) (jakpum, “works”) + 집(集) (jip, “collection”) → 작품집(作品集) (jakpumjip, “collected works”)", "type": "example"}], "glosses": ["collection of bound works; anthology, collected works"], "tags": ["morpheme"]}], "pos": "suffix", "head_templates": [{"name": "ko-suffix", "args": {"hanja": "集"}, "expansion": "—집 • (-jip) (hanja 集)"}], "forms": [{"form": "—집", "tags": ["canonical"]}, {"form": "-jip", "tags": ["romanization"]}, {"form": "集", "tags": ["hanja"]}], "etymology_number": "2", "word": "집", "lang": "Korean", "lang_code": "ko"}
|
||||
{"senses": [{"examples": [{"text": "최근에 박사 과정을 시작했는데 다시 학생이 돼서 기분이 좋습니다.", "bold_text_offsets": [[20, 23]], "roman": "Choegeun-e baksa gwajeong-eul sijakhaenneunde dasi haksaeng-i dwaeseo gibun-i joseumnida.", "bold_roman_offsets": [[51, 61]], "translation": "I recently started a PhD program, and it feels good to be a student again.", "english": "I recently started a PhD program, and it feels good to be a student again.", "bold_translation_offsets": [[60, 67]], "type": "example"}], "glosses": ["student, scholar, school pupil"]}], "pos": "noun", "head_templates": [{"name": "ko-noun", "args": {"hanja": "學生", "count": "명(名)"}, "expansion": "학생 • (haksaeng) (counter 명(名), hanja 學生)"}], "forms": [{"form": "haksaeng", "tags": ["romanization"]}, {"form": "명", "tags": ["counter", "error-unknown-tag"]}, {"form": "學生", "tags": ["hanja"]}], "sounds": [{"tags": ["SK-Standard", "Seoul"], "ipa": "[ha̠ks͈ɛŋ]"}, {"tags": ["SK-Standard", "Seoul"], "ipa": "[ha̠ks͈e̞ŋ]"}], "word": "학생", "lang": "Korean", "lang_code": "ko"}
|
||||
{"senses": [{"glosses": ["contraction of in + dem; in the"], "tags": ["abbreviation", "alt-of", "contraction"], "alt_of": [{"word": "in", "extra": "+ dem; in the"}]}], "pos": "contraction", "head_templates": [{"name": "head", "args": {"1": "de", "2": "contraction"}, "expansion": "im"}], "forms": [{"form": "no-table-tags", "source": "declension", "tags": ["table-tags"]}, {"form": "l", "source": "declension", "tags": ["inflection-template"]}, {"form": "in den", "tags": ["accusative", "masculine"], "source": "declension"}, {"form": "ins", "tags": ["accusative", "neuter"], "source": "declension"}, {"form": "in die", "tags": ["accusative", "feminine"], "source": "declension"}, {"form": "in die", "tags": ["accusative", "plural"], "source": "declension"}, {"form": "im", "tags": ["dative", "masculine"], "source": "declension"}, {"form": "im", "tags": ["dative", "neuter"], "source": "declension"}, {"form": "in der", "tags": ["dative", "feminine"], "source": "declension"}, {"form": "in den", "tags": ["dative", "plural"], "source": "declension"}], "sounds": [{"ipa": "/ɪm/"}, {"audio": "De-im.ogg", "ogg_url": "https://upload.wikimedia.org/wikipedia/commons/7/71/De-im.ogg", "mp3_url": "https://upload.wikimedia.org/wikipedia/commons/transcoded/7/71/De-im.ogg/De-im.ogg.mp3"}, {"audio": "De-im2.ogg", "tags": ["Germany", "Berlin"], "ogg_url": "https://upload.wikimedia.org/wikipedia/commons/4/44/De-im2.ogg", "mp3_url": "https://upload.wikimedia.org/wikipedia/commons/transcoded/4/44/De-im2.ogg/De-im2.ogg.mp3"}], "word": "im", "lang": "German", "lang_code": "de"}
|
||||
{"word": "水", "pos": "character", "lang": "Korean", "lang_code": "ko", "forms": [{"form": "물 수", "roman": "mul su", "tags": ["eumhun"]}], "head_templates": [{"name": "ko-hanja", "args": {"1": "물", "2": "수"}, "expansion": "水 (eumhun 물 수 (mul su))"}], "senses": [{"glosses": ["hanja form of 수 (“water”)"], "tags": ["form-of", "hanja"]}, {"glosses": ["hanja form of 수 (“short for 수요일(水曜日) (suyoil, “Wednesday”)”)"], "tags": ["form-of", "hanja"]}]}
|
||||
|
|
|
|||
123
halkyon_learn_curator/test/krdict_test.dart
Normal file
123
halkyon_learn_curator/test/krdict_test.dart
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:archive/archive.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/krdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
void main() {
|
||||
/// The fixture packed as KRDict packs its export: its first file.
|
||||
List<int> export() => ZipEncoder().encodeBytes(
|
||||
Archive()..add(
|
||||
ArchiveFile.bytes(
|
||||
'1_5000_20260919.json',
|
||||
File('test/fixtures/krdict-sample.json').readAsBytesSync(),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
Future<List<Map<String, Object?>>> converted({
|
||||
WordFrequencies? frequencies,
|
||||
WiktionaryKorean? korean,
|
||||
}) async {
|
||||
final (:version, :entries) = krdictExport(export());
|
||||
return [
|
||||
for (final line in await krdictDictionary(
|
||||
entries,
|
||||
sourceVersion: version,
|
||||
frequencies: frequencies,
|
||||
korean: korean,
|
||||
).toList())
|
||||
jsonDecode(line) as Map<String, Object?>,
|
||||
];
|
||||
}
|
||||
|
||||
test("KRDict's words with their level, pronunciation where it differs from "
|
||||
'the spelling, Hanja, forms and English senses', () async {
|
||||
final [header, ...entries] = await converted();
|
||||
|
||||
expect(header['source'], 'krdict');
|
||||
expect(header['language'], 'ko');
|
||||
expect(header['sourceVersion'], '2026-09-19');
|
||||
expect(
|
||||
[
|
||||
for (final credit in header['credits']! as List)
|
||||
(credit as Map)['name'],
|
||||
],
|
||||
['한국어기초사전 (Basic Korean Dictionary)'],
|
||||
);
|
||||
final school = entries.singleWhere((entry) => entry['headword'] == '학교');
|
||||
expect(school['id'], '73276');
|
||||
expect(school['reading'], '학꾜');
|
||||
expect(school['labels'], ['level:beginner']);
|
||||
expect(school['keys'], containsAll(['학교', '學校', 'school']));
|
||||
const definition =
|
||||
'An institution that educates students according to a specific '
|
||||
'purpose, curriculum, system, etc.';
|
||||
expect(school['entry'], {
|
||||
'pos': 'noun',
|
||||
'hanja': ['學校'],
|
||||
'senses': [
|
||||
{
|
||||
'glosses': ['school'],
|
||||
'info': [definition],
|
||||
},
|
||||
],
|
||||
});
|
||||
final read = entries.singleWhere((entry) => entry['headword'] == '읽다');
|
||||
expect(read['reading'], '익따');
|
||||
expect(read['textKeys'], containsAll(['읽다', '읽어', '읽고']));
|
||||
expect(
|
||||
(read['entry']! as Map)['inflections'],
|
||||
contains(equals({'text': '읽어'})),
|
||||
);
|
||||
final together = entries.where((entry) => entry['headword'] == '같이');
|
||||
expect(
|
||||
[for (final entry in together) (entry['entry']! as Map)['pos']],
|
||||
['particle', 'adv'],
|
||||
);
|
||||
});
|
||||
|
||||
test('an idiom or a proverb is an entry of its own, never found in running '
|
||||
'text, with an id told apart from its word', () async {
|
||||
final [_, ...entries] = await converted();
|
||||
|
||||
final proverb = entries.singleWhere(
|
||||
(entry) => entry['headword'] == '가는 날이 장날',
|
||||
);
|
||||
expect(proverb['id'], '27500:가는 날이 장날');
|
||||
expect((proverb['entry']! as Map)['pos'], 'proverb');
|
||||
expect(proverb.containsKey('textKeys'), isFalse);
|
||||
expect(
|
||||
entries.singleWhere((entry) => entry['headword'] == '가다')['id'],
|
||||
'27500',
|
||||
);
|
||||
});
|
||||
|
||||
test("Wiktionary's forms find a word in text; a verb rarely written as "
|
||||
'the dictionary writes it ranks by its most used form', () async {
|
||||
final korean = WiktionaryKorean()..forms['가다'] = {'가요', '갔어요', '갑니다'};
|
||||
final [_, ...entries] = await converted(
|
||||
korean: korean,
|
||||
frequencies: WordFrequencies(const {'갔어요': 50}),
|
||||
);
|
||||
|
||||
final go = entries.singleWhere((entry) => entry['headword'] == '가다');
|
||||
expect(go['textKeys'], containsAll(['가다', '가요', '갔어요']));
|
||||
expect(go['labels'], contains('common'));
|
||||
expect(go['rank'], 100000 + 100);
|
||||
});
|
||||
|
||||
test("the export's JSON download is the third its page offers", () {
|
||||
expect(
|
||||
krdictJsonDownloadOf('''
|
||||
<button onclick="batchDownload('215')">엑셀</button>
|
||||
<button onclick="batchDownload('216')">XML</button>
|
||||
<button onclick="batchDownload('217')">Json</button>'''),
|
||||
'217',
|
||||
);
|
||||
expect(krdictJsonDownloadOf("batchDownload('0')"), isNull);
|
||||
});
|
||||
}
|
||||
173
halkyon_learn_curator/test/unihan_test.dart
Normal file
173
halkyon_learn_curator/test/unihan_test.dart
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:archive/archive.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/cedict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
void main() {
|
||||
final unihan = Unihan.parse(
|
||||
File('test/fixtures/unihan-sample.txt').readAsLinesSync(),
|
||||
);
|
||||
final cedict = File('test/fixtures/cedict-characters-sample.txt')
|
||||
.readAsStringSync();
|
||||
// The words' places as wordfreq's list has them (的 0, 了 3, 后 41, …).
|
||||
final frequencies = WordFrequencies(const {
|
||||
'了': 3,
|
||||
'我们': 14,
|
||||
'说': 24,
|
||||
'为了': 170,
|
||||
'以后': 160,
|
||||
'后': 41,
|
||||
'学习': 380,
|
||||
'了解': 520,
|
||||
'这': 15,
|
||||
'水': 600,
|
||||
'银行': 1900,
|
||||
'皇后': 9000,
|
||||
});
|
||||
final places = cedictCharacterPlaces(cedict, frequencies);
|
||||
|
||||
/// The entries of a dictionary file's [lines], by character.
|
||||
Map<String, Map<String, Object?>> entriesOf(Iterable<String> lines) => {
|
||||
for (final line in lines.skip(1))
|
||||
if (jsonDecode(line) case final Map<String, Object?> entry)
|
||||
entry['headword']! as String: entry,
|
||||
};
|
||||
|
||||
test("the simplified characters are the standard's, with pinyin, the "
|
||||
'usual reading first, meanings, strokes, radical and HSK level', () {
|
||||
final lines = chineseCharacters(
|
||||
unihan,
|
||||
traditional: false,
|
||||
sourceVersion: '18.0.0',
|
||||
hsk: const {'说': 1, '行': 1, '后': 1},
|
||||
places: places,
|
||||
).toList();
|
||||
|
||||
final header = jsonDecode(lines.first) as Map<String, Object?>;
|
||||
expect(header['source'], 'unihan-hans');
|
||||
expect(header['kind'], 'characters');
|
||||
expect(header['language'], 'zh-Hans');
|
||||
expect(header['sourceVersion'], '18.0.0');
|
||||
final entries = entriesOf(lines);
|
||||
// No traditional character among them.
|
||||
expect(entries.keys, isNot(contains('說')));
|
||||
expect(entries.keys, containsAll(['说', '这', '后', '了', '行', '水']));
|
||||
final xing = entries['行']!;
|
||||
expect(((xing['entry']! as Map)['pinyin'] as List).take(2), [
|
||||
'xíng',
|
||||
'háng',
|
||||
]);
|
||||
expect((xing['entry']! as Map)['strokes'], 6);
|
||||
expect(xing['labels'], ['hsk:1', 'common', 'radical:144']);
|
||||
final shuo = entries['说']!;
|
||||
expect((shuo['entry']! as Map)['meanings'], [
|
||||
'speak',
|
||||
'say',
|
||||
'talk',
|
||||
'scold',
|
||||
'upbraid',
|
||||
]);
|
||||
expect((shuo['entry']! as Map)['traditional'], ['說', '説']);
|
||||
// Found by its pinyin with marks, without, and numbered; written in
|
||||
// the other script, loosely.
|
||||
expect(shuo['keys'], containsAll(['shuō', 'shuo', 'shuo1', 'speak']));
|
||||
expect(shuo['looseKeys'], ['說', '説']);
|
||||
});
|
||||
|
||||
test('the traditional characters are those words in use write: 這 and 了, '
|
||||
'not 这; 後 the usual form of 后, ranked and levelled as it', () {
|
||||
final entries = entriesOf(
|
||||
chineseCharacters(
|
||||
unihan,
|
||||
traditional: true,
|
||||
hsk: const {'后': 1, '了': 1},
|
||||
places: places,
|
||||
),
|
||||
);
|
||||
|
||||
expect(entries.keys, containsAll(['這', '了', '瞭', '後', '后', '說']));
|
||||
expect(entries.keys, isNot(contains('这')));
|
||||
// 後 takes 后's place and level; 后, the queen, ranks by its own words.
|
||||
expect(entries['後']!['labels'], contains('hsk:1'));
|
||||
expect(entries['后']!['labels'], isNot(contains('hsk:1')));
|
||||
expect(
|
||||
entries['後']!['rank']! as int,
|
||||
lessThan(entries['后']!['rank']! as int),
|
||||
);
|
||||
// 了 stands for itself; 瞭 is a form of it of its own.
|
||||
expect(entries['了']!['labels'], contains('hsk:1'));
|
||||
expect(entries['瞭']!['labels'], isNot(contains('hsk:1')));
|
||||
expect((entries['說']!['entry']! as Map)['simplified'], ['说']);
|
||||
// Hong Kong's primary schools teach 說 in the first grade.
|
||||
expect(entries['說']!['labels'], contains('grade:1'));
|
||||
});
|
||||
|
||||
test("CC-CEDICT's words tell each script's characters their places: a "
|
||||
"lone simplified character's to the traditional form longer words use "
|
||||
'most', () {
|
||||
expect(places.simplified['后'], 41);
|
||||
expect(places.traditional['後'], 41);
|
||||
expect(places.traditional['后'], 9000);
|
||||
expect(places.traditional['了'], 3);
|
||||
// 瞭解 is only a variant of 了解 there, and 了 the form 為了 uses.
|
||||
expect(places.traditional, isNot(contains('瞭')));
|
||||
expect(places.traditional, isNot(contains('这')));
|
||||
});
|
||||
|
||||
test('Korean Hanja with their sounds, the taught one first, their native '
|
||||
'readings from Wiktionary, and the Hangul letters before them', () {
|
||||
final lines = koreanHanja(
|
||||
unihan,
|
||||
sourceVersion: '18.0.0',
|
||||
eumhun: const {
|
||||
'水': [(hun: '물', eum: '수')],
|
||||
},
|
||||
).toList();
|
||||
|
||||
final header = jsonDecode(lines.first) as Map<String, Object?>;
|
||||
expect(header['language'], 'ko');
|
||||
expect(header['license'], 'CC-BY-SA-4.0');
|
||||
expect([
|
||||
for (final credit in header['credits']! as List) (credit as Map)['name'],
|
||||
], containsAll(['Unihan Database', 'Wiktionary']));
|
||||
final entries = entriesOf(lines);
|
||||
final water = entries['水']!;
|
||||
expect(water['entry'], {
|
||||
'meanings': ['water', 'liquid', 'lotion', 'juice'],
|
||||
'eum': ['수'],
|
||||
'hun': ['물'],
|
||||
'eumhun': ['물 수'],
|
||||
'strokes': 4,
|
||||
});
|
||||
expect(water['labels'], ['education', 'radical:85']);
|
||||
expect(water['keys'], containsAll(['水', '수', '물', '물 수', 'water']));
|
||||
// 行 is taught as 행, also read 항.
|
||||
expect((entries['行']!['entry']! as Map)['eum'], ['행', '항']);
|
||||
final giyeok = entries['ㄱ']!;
|
||||
expect(giyeok['labels'], ['hangul']);
|
||||
expect(giyeok['entry'], {
|
||||
'romaji': 'g, k',
|
||||
'script': 'hangul',
|
||||
'name': '기역',
|
||||
});
|
||||
expect(giyeok['rank']! as int, lessThan(water['rank']! as int));
|
||||
});
|
||||
|
||||
test("Unihan's zip is read whole, with the version of Unicode it is of", () {
|
||||
final archive = Archive()
|
||||
..add(
|
||||
ArchiveFile.bytes(
|
||||
'Unihan_Readings.txt',
|
||||
File('test/fixtures/unihan-sample.txt').readAsBytesSync(),
|
||||
),
|
||||
);
|
||||
final (:unihan, :version) = Unihan.ofZip(ZipEncoder().encodeBytes(archive));
|
||||
|
||||
expect(version, '18.0.0');
|
||||
expect(unihan.characters, contains('水'));
|
||||
});
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue