feat(learn): furigana and pinyin over hints, from Tatoeba's transcriptions
The curator puts Tatoeba's readings on a dictionary's examples by their Tatoeba ID (`withTatoebaReadings`, `curate readings`, and in `update`): furigana over JMdict's Japanese from its jpn transcriptions, pinyin with tone marks over Chinese from its cmn ones, each word over as many Han characters as it has syllables. An example gets `ruby`, `[start, end, reading]` each, and `rubyUnreviewed` where no one wrote or checked them (MeCab's furigana, the automatic pinyin). JMdict's conversion goes to 3, so instances convert again at once. A word's example hints carry them as ruby marks, the word still in bold. The hints sheet shows them from the start with a switch above (Furigana, Pinyin), kept on the device (`DeviceSettings.hintReadings`), and below a hint whose readings no one reviewed says so. Whats-New: Example sentences in a word's hints now show furigana or pinyin over them, with a switch to hide them. Whats-New-De: Beispielsätze in den Hinweisen eines Worts zeigen jetzt Furigana oder Pinyin darüber, mit einem Schalter zum Ausblenden. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
bed39dc2a2
commit
3257905536
GPG key ID:
07F6FCF405488B64
8 changed files with 1083 additions and 78 deletions
|
|
@ -189,6 +189,26 @@ up to three for each: those using the word as its headword writes it first, then
|
|||
those nearest seven words long, the shortest then; they come first in an entry's
|
||||
first sense, the word as it stands there beside. Some ten seconds for German.
|
||||
|
||||
Readings over the examples, from Tatoeba's transcriptions of their sentences
|
||||
(decided 9 Nov 2026): furigana over JMdict's Japanese, pinyin over Chinese.
|
||||
`update` adds them itself; by hand:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate readings \
|
||||
--input jmdict-en.jsonl.gz --transcriptions jpn_transcriptions.tsv.bz2 \
|
||||
--output jmdict-en.jsonl.gz
|
||||
```
|
||||
|
||||
from Tatoeba's `jpn_transcriptions.tsv.bz2` or `cmn_transcriptions.tsv.bz2`
|
||||
(CC BY 2.0 FR, the sentences' own licence). Each example with a Tatoeba ID
|
||||
gets `ruby`, a list of `[start, end, reading]` over its sentence (UTF-16 code
|
||||
units), where the transcription is of that very sentence: Japanese's
|
||||
`[漢字|かん|じ]` one reading for each kanji where there are as many, else one
|
||||
over the word; Chinese's tone numbers as tone marks, each of Tatoeba's words
|
||||
over as many Han characters as it has syllables, none where they don't count
|
||||
up (about 1 % of them). `rubyUnreviewed` marks those no one wrote or checked,
|
||||
MeCab's furigana and Tatoeba's automatic pinyin, some half of the Japanese.
|
||||
|
||||
Put the files into the server's `dictionaries` folder. The server loads each on
|
||||
its next start, and again whenever it changes (see the server's README).
|
||||
|
||||
|
|
@ -277,7 +297,7 @@ The **first line** says what the file holds:
|
|||
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
||||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
|
||||
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, the source's id, e.g. `tatoeba`, and where Tatoeba transcribed the sentence `ruby` and `rubyUnreviewed`, the readings over it). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |
|
||||
|
||||
## The app's stroke files
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
|
|||
import 'package:halkyon_learn_curator/src/sources/krdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/makemeahanzi.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba_readings.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/update.dart';
|
||||
|
|
@ -92,6 +93,16 @@ import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
|||
/// --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
|
||||
/// --output wiktionary-de-en.jsonl.gz
|
||||
///
|
||||
/// Readings over a dictionary file's examples, from Tatoeba's
|
||||
/// transcriptions of their sentences (furigana over Japanese, pinyin over
|
||||
/// Chinese; its `jpn_transcriptions.tsv` or `cmn_transcriptions.tsv`, bzip2
|
||||
/// or not):
|
||||
///
|
||||
/// dart run halkyon_learn_curator:curate readings \
|
||||
/// --input jmdict-en.jsonl.gz \
|
||||
/// --transcriptions jpn_transcriptions.tsv.bz2 \
|
||||
/// --output jmdict-en.jsonl.gz
|
||||
///
|
||||
/// Or brings a server's dictionaries up to date from EDRDG's files of the
|
||||
/// day and, for the languages named, Wiktionary's latest dump and MDBG's
|
||||
/// CC-CEDICT of the day (Chinese), once or every so many days:
|
||||
|
|
@ -160,6 +171,10 @@ Future<void> main(List<String> args) async {
|
|||
'links',
|
||||
help: "tatoeba: Tatoeba's links between the two languages.",
|
||||
)
|
||||
..addOption(
|
||||
'transcriptions',
|
||||
help: "readings: Tatoeba's transcriptions of the dictionary's language.",
|
||||
)
|
||||
..addOption(
|
||||
'cedict',
|
||||
help:
|
||||
|
|
@ -281,6 +296,44 @@ Future<void> main(List<String> args) async {
|
|||
);
|
||||
return;
|
||||
}
|
||||
if (source == 'readings') {
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
final transcriptions = options.option('transcriptions');
|
||||
if (input == null || output == null || transcriptions == null) {
|
||||
_fail(
|
||||
'Usage: curate readings --input <dictionary> --transcriptions <file> '
|
||||
'--output <dictionary>\n${parser.usage}',
|
||||
);
|
||||
}
|
||||
final read = await readTatoebaTranscriptions(
|
||||
(transcriptions.endsWith('.bz2')
|
||||
? Stream.value(
|
||||
BZip2Decoder().decodeBytes(
|
||||
File(transcriptions).readAsBytesSync(),
|
||||
),
|
||||
)
|
||||
: File(transcriptions).openRead())
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter()),
|
||||
);
|
||||
final bytes = File(input).openRead();
|
||||
final written = File('$output.part');
|
||||
final lines = withTatoebaReadings(
|
||||
(input.endsWith('.gz') ? bytes.transform(gzip.decoder) : bytes)
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter()),
|
||||
read,
|
||||
).transform(utf8.encoder);
|
||||
await (output.endsWith('.gz') ? lines.transform(gzip.encoder) : lines).pipe(
|
||||
written.openWrite(),
|
||||
);
|
||||
await written.rename(output);
|
||||
stdout.writeln(
|
||||
'Wrote $output with ${read.length} transcriptions to read from.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (source == 'cedict') {
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
|
|
|
|||
|
|
@ -0,0 +1,231 @@
|
|||
import 'dart:convert';
|
||||
|
||||
/// A transcription of one of Tatoeba's sentences: its `text` in Tatoeba's
|
||||
/// way of writing it, and whether someone wrote or checked it (`reviewed`)
|
||||
/// rather than Tatoeba making it alone (MeCab's furigana, the automatic
|
||||
/// pinyin).
|
||||
typedef Transcription = ({String text, bool reviewed});
|
||||
|
||||
/// A reading over a part of a sentence: from its start to its end (UTF-16 code
|
||||
/// units, as Dart's strings count), and what it reads.
|
||||
typedef Ruby = (int start, int end, String reading);
|
||||
|
||||
/// The scripts Tatoeba transcribes a language into that give readings over
|
||||
/// it: furigana over Japanese (`Hrkt`), pinyin over Mandarin (`Latn`).
|
||||
const _rubyScripts = {'jpn': 'Hrkt', 'cmn': 'Latn'};
|
||||
|
||||
/// Tatoeba's transcriptions of one language by their sentence's ID
|
||||
/// (Tatoeba's `<code>_transcriptions.tsv`: id, language, script, the user
|
||||
/// who wrote or last checked it, empty for none, and the text), only those
|
||||
/// in the script that gives readings over it.
|
||||
Future<Map<int, Transcription>> readTatoebaTranscriptions(
|
||||
Stream<String> lines,
|
||||
) async {
|
||||
final transcriptions = <int, Transcription>{};
|
||||
await for (final line in lines) {
|
||||
final fields = line.split('\t');
|
||||
if (fields.length < 5 || _rubyScripts[fields[1]] != fields[2]) {
|
||||
continue;
|
||||
}
|
||||
if (int.tryParse(fields[0]) case final id?) {
|
||||
transcriptions[id] = (
|
||||
text: fields.sublist(4).join('\t'),
|
||||
reviewed: fields[3].isNotEmpty,
|
||||
);
|
||||
}
|
||||
}
|
||||
return transcriptions;
|
||||
}
|
||||
|
||||
/// The readings [transcription] gives over [sentence] in [language] (a BCP
|
||||
/// 47 tag): furigana for Japanese, from Tatoeba's `[漢字|かん|じ]` (one
|
||||
/// reading for each character where there are as many, else one for all);
|
||||
/// pinyin with tone marks for Chinese, each of Tatoeba's words over as
|
||||
/// many of the sentence's Han characters as it has syllables. Null for
|
||||
/// another language, a transcription of another sentence or one that
|
||||
/// doesn't count up, and one without any reading.
|
||||
List<Ruby>? rubyOf(String language, String sentence, String transcription) {
|
||||
final ruby = switch (language.split('-').first.toLowerCase()) {
|
||||
'ja' => _furigana(sentence, transcription),
|
||||
'zh' => _pinyin(sentence, transcription),
|
||||
_ => null,
|
||||
};
|
||||
return ruby == null || ruby.isEmpty ? null : ruby;
|
||||
}
|
||||
|
||||
final _bracket = RegExp(r'\[([^\[\]|]+)((?:\|[^\[\]|]*)+)\]');
|
||||
|
||||
List<Ruby>? _furigana(String sentence, String transcription) {
|
||||
final ruby = <Ruby>[];
|
||||
final plain = StringBuffer();
|
||||
var at = 0;
|
||||
for (final match in _bracket.allMatches(transcription)) {
|
||||
plain.write(transcription.substring(at, match.start));
|
||||
final base = match[1]!;
|
||||
final readings = match[2]!.substring(1).split('|');
|
||||
final characters = base.runes.map(String.fromCharCode).toList();
|
||||
if (readings.length == characters.length) {
|
||||
for (final (i, character) in characters.indexed) {
|
||||
if (readings[i].isNotEmpty) {
|
||||
ruby.add((
|
||||
plain.length,
|
||||
plain.length + character.length,
|
||||
readings[i],
|
||||
));
|
||||
}
|
||||
plain.write(character);
|
||||
}
|
||||
} else {
|
||||
ruby.add((plain.length, plain.length + base.length, readings.join()));
|
||||
plain.write(base);
|
||||
}
|
||||
at = match.end;
|
||||
}
|
||||
plain.write(transcription.substring(at));
|
||||
return plain.toString() == sentence ? ruby : null;
|
||||
}
|
||||
|
||||
final _han = RegExp(r'\p{Script=Han}', unicode: true);
|
||||
|
||||
final _syllable = RegExp('([a-zü]+(?::[a-z]*)?)([1-5])', caseSensitive: false);
|
||||
|
||||
List<Ruby>? _pinyin(String sentence, String transcription) {
|
||||
// Where each Han character of the sentence is.
|
||||
final characters = [
|
||||
for (final match in _han.allMatches(sentence)) (match.start, match.end),
|
||||
];
|
||||
final ruby = <Ruby>[];
|
||||
var next = 0;
|
||||
for (final word in transcription.split(RegExp(r'\s+'))) {
|
||||
final syllables = [
|
||||
for (final match in _syllable.allMatches(word))
|
||||
_marked(match[1]!.toLowerCase(), int.parse(match[2]!)),
|
||||
];
|
||||
if (syllables.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
if (next + syllables.length > characters.length) {
|
||||
return null;
|
||||
}
|
||||
ruby.add((
|
||||
characters[next].$1,
|
||||
characters[next + syllables.length - 1].$2,
|
||||
syllables.join(),
|
||||
));
|
||||
next += syllables.length;
|
||||
}
|
||||
return next == characters.length ? ruby : null;
|
||||
}
|
||||
|
||||
const _toneMarks = {
|
||||
'a': 'āáǎà',
|
||||
'e': 'ēéěè',
|
||||
'i': 'īíǐì',
|
||||
'o': 'ōóǒò',
|
||||
'u': 'ūúǔù',
|
||||
'ü': 'ǖǘǚǜ',
|
||||
};
|
||||
|
||||
/// [syllable] with [tone] marked: on its a or e, the o of ou, else its
|
||||
/// last vowel; none for the neutral tone (5). `u:` and `v` are ü.
|
||||
String _marked(String syllable, int tone) {
|
||||
final letters = syllable.replaceAll('u:', 'ü').replaceAll('v', 'ü');
|
||||
if (tone == 5) {
|
||||
return letters;
|
||||
}
|
||||
final vowel = letters.contains('a')
|
||||
? letters.indexOf('a')
|
||||
: letters.contains('e')
|
||||
? letters.indexOf('e')
|
||||
: letters.contains('ou')
|
||||
? letters.indexOf('o')
|
||||
: letters.lastIndexOf(RegExp('[aeiouü]'));
|
||||
if (vowel < 0) {
|
||||
return letters;
|
||||
}
|
||||
return letters.replaceRange(
|
||||
vowel,
|
||||
vowel + 1,
|
||||
_toneMarks[letters[vowel]]![tone - 1],
|
||||
);
|
||||
}
|
||||
|
||||
/// A dictionary file's [lines] (without their line ends) with readings over its examples: each
|
||||
/// example with a Tatoeba ID among [transcriptions] whose readings fit its
|
||||
/// sentence gets them as `ruby` (`[start, end, reading]` each), and
|
||||
/// `rubyUnreviewed` where no one wrote or checked them. The header and
|
||||
/// every other line pass as they were, each ended.
|
||||
Stream<String> withTatoebaReadings(
|
||||
Stream<String> lines,
|
||||
Map<int, Transcription> transcriptions,
|
||||
) async* {
|
||||
String? language;
|
||||
await for (final line in lines) {
|
||||
if (language == null) {
|
||||
final header = jsonDecode(line) as Map<String, Object?>;
|
||||
language = header['language']! as String;
|
||||
yield '$line\n';
|
||||
continue;
|
||||
}
|
||||
if (line.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
final entry = jsonDecode(line) as Map<String, Object?>;
|
||||
var changed = false;
|
||||
Map<String, Object?> withRuby(Map<String, Object?> example) {
|
||||
final id = switch (example['tatoeba']) {
|
||||
final int id => id,
|
||||
final String id => int.tryParse(id),
|
||||
_ => null,
|
||||
};
|
||||
final transcription = transcriptions[id];
|
||||
if (transcription == null) {
|
||||
return example;
|
||||
}
|
||||
final sentence = example[language];
|
||||
if (sentence is! String) {
|
||||
return example;
|
||||
}
|
||||
final ruby = rubyOf(language!, sentence, transcription.text);
|
||||
if (ruby == null) {
|
||||
return example;
|
||||
}
|
||||
changed = true;
|
||||
return {
|
||||
...example,
|
||||
'ruby': [
|
||||
for (final (start, end, reading) in ruby) [start, end, reading],
|
||||
],
|
||||
if (!transcription.reviewed) 'rubyUnreviewed': true,
|
||||
};
|
||||
}
|
||||
|
||||
final body = entry['entry'];
|
||||
if (body is! Map<String, Object?> || body['senses'] is! List) {
|
||||
yield '$line\n';
|
||||
continue;
|
||||
}
|
||||
final senses = [
|
||||
for (final sense in body['senses']! as List)
|
||||
if (sense case {'examples': final List<Object?> examples})
|
||||
{
|
||||
...sense as Map<String, Object?>,
|
||||
'examples': [
|
||||
for (final example in examples)
|
||||
if (example is Map<String, Object?>)
|
||||
withRuby(example)
|
||||
else
|
||||
example,
|
||||
],
|
||||
}
|
||||
else
|
||||
sense,
|
||||
];
|
||||
yield changed
|
||||
? '${jsonEncode({
|
||||
...entry,
|
||||
'entry': {...body, 'senses': senses},
|
||||
})}\n'
|
||||
: '$line\n';
|
||||
}
|
||||
}
|
||||
|
|
@ -10,6 +10,7 @@ import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
|
|||
import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/krdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba_readings.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
|
|
@ -31,8 +32,9 @@ const updatedFiles = ['jmdict-en.jsonl.gz', 'kanjidic-en.jsonl.gz'];
|
|||
/// curator writes something else from the same download (a label, a field),
|
||||
/// so that an instance converts again as soon as the new curator runs
|
||||
/// rather than at its next update, up to a week later. 2: kanji labelled
|
||||
/// by their radical.
|
||||
const conversion = 2;
|
||||
/// by their radical. 3: furigana over JMdict's examples, from Tatoeba's
|
||||
/// transcriptions.
|
||||
const conversion = 3;
|
||||
|
||||
/// The file in the dictionaries' folder that says which [conversion] made
|
||||
/// them. The server reads only dictionary files (`.jsonl`, `.jsonl.gz`).
|
||||
|
|
@ -59,7 +61,8 @@ Future<void> httpFetch(Uri uri, File file) async {
|
|||
}
|
||||
|
||||
/// Brings the dictionary files in [into] up to date from EDRDG's files of
|
||||
/// the day: JMdict with its sentences and KANJIDIC2 with the kana, the
|
||||
/// the day: JMdict with its sentences, furigana over them from Tatoeba's
|
||||
/// transcriptions (`addTatoebaReadings`), and KANJIDIC2 with the kana, the
|
||||
/// JLPT levels from the curator's [data] folder (`data/README.md`). Each
|
||||
/// file is written beside its place first and then put there at once, so
|
||||
/// the server never reads half of one; should anything fail, the files
|
||||
|
|
@ -87,6 +90,12 @@ Future<List<File>> updateDictionaries({
|
|||
examples: true,
|
||||
),
|
||||
);
|
||||
final jmdictRead = await addTatoebaReadings(
|
||||
jmdict,
|
||||
code: 'jpn',
|
||||
downloads: downloads,
|
||||
fetch: fetch,
|
||||
);
|
||||
final kanjidic = await _write(
|
||||
File('${downloads.path}/${updatedFiles[1]}'),
|
||||
Stream.fromIterable(
|
||||
|
|
@ -98,7 +107,8 @@ Future<List<File>> updateDictionaries({
|
|||
);
|
||||
await into.create(recursive: true);
|
||||
final written = [
|
||||
for (final file in [jmdict, kanjidic]) await _putInPlace(file, into),
|
||||
await _putInPlace(jmdictRead, into, name: updatedFiles[0]),
|
||||
await _putInPlace(kanjidic, into),
|
||||
];
|
||||
await File('${into.path}/$conversionFile').writeAsString('$conversion');
|
||||
return written;
|
||||
|
|
@ -495,9 +505,84 @@ Uri tatoebaLinksOf(String from, String to) => Uri.parse(
|
|||
'$from-${to}_links.tsv.bz2',
|
||||
);
|
||||
|
||||
/// Tatoeba's weekly export of the transcriptions of a language's sentences
|
||||
/// ([code], ISO 639-3).
|
||||
Uri tatoebaTranscriptionsOf(String code) => Uri.parse(
|
||||
'https://downloads.tatoeba.org/exports/per_language/$code/'
|
||||
'${code}_transcriptions.tsv.bz2',
|
||||
);
|
||||
|
||||
/// The lines of Tatoeba's export at [uri], fetched into [downloads] and
|
||||
/// unpacked there once.
|
||||
Future<Stream<String>> _tatoebaLines(
|
||||
Uri uri, {
|
||||
required Directory downloads,
|
||||
required Fetch fetch,
|
||||
}) async {
|
||||
final name = uri.pathSegments.last;
|
||||
final packed = File('${downloads.path}/$name');
|
||||
final unpacked = File('${downloads.path}/${name.replaceAll('.bz2', '')}');
|
||||
if (!unpacked.existsSync()) {
|
||||
await fetch(uri, packed);
|
||||
final input = InputFileStream(packed.path);
|
||||
final output = OutputFileStream(unpacked.path);
|
||||
try {
|
||||
BZip2Decoder().decodeStream(input, output);
|
||||
} finally {
|
||||
await input.close();
|
||||
await output.close();
|
||||
}
|
||||
await packed.delete();
|
||||
}
|
||||
return unpacked
|
||||
.openRead()
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter());
|
||||
}
|
||||
|
||||
/// [dictionary], a dictionary file just written (gzipped), with readings
|
||||
/// over its examples from Tatoeba's transcriptions of the sentences in
|
||||
/// [code] (`withTatoebaReadings`: furigana over Japanese, pinyin over
|
||||
/// Chinese), downloaded into [downloads]; returns the file it wrote beside
|
||||
/// it, or [dictionary] as it was should Tatoeba fail: readings are worth
|
||||
/// having, never worth an update.
|
||||
Future<File> addTatoebaReadings(
|
||||
File dictionary, {
|
||||
required String code,
|
||||
required Directory downloads,
|
||||
Fetch fetch = httpFetch,
|
||||
}) async {
|
||||
try {
|
||||
final transcriptions = await readTatoebaTranscriptions(
|
||||
await _tatoebaLines(
|
||||
tatoebaTranscriptionsOf(code),
|
||||
downloads: downloads,
|
||||
fetch: fetch,
|
||||
),
|
||||
);
|
||||
return await _write(
|
||||
File('${dictionary.path}.readings.gz'),
|
||||
withTatoebaReadings(
|
||||
dictionary
|
||||
.openRead()
|
||||
.transform(gzip.decoder)
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter()),
|
||||
transcriptions,
|
||||
),
|
||||
);
|
||||
} on Object catch (error) {
|
||||
stdout.writeln(
|
||||
'Readings left out of ${dictionary.uri.pathSegments.last}: $error',
|
||||
);
|
||||
return dictionary;
|
||||
}
|
||||
}
|
||||
|
||||
/// [dictionary], a dictionary file just written (gzipped), with Tatoeba's
|
||||
/// sentences in its language that have a translation into its meaning
|
||||
/// language as its entries' first examples (`withTatoebaExamples`),
|
||||
/// language as its entries' first examples (`withTatoebaExamples`), and
|
||||
/// for Chinese pinyin over them (`addTatoebaReadings`),
|
||||
/// downloaded into [downloads]; returns the file it wrote beside it. A
|
||||
/// language Tatoeba has no code for here is left as it is.
|
||||
Future<File> addTatoebaExamples(
|
||||
|
|
@ -518,27 +603,8 @@ Future<File> addTatoebaExamples(
|
|||
if (from == null || to == null) {
|
||||
return dictionary;
|
||||
}
|
||||
Future<Stream<String>> download(Uri uri) async {
|
||||
final name = uri.pathSegments.last;
|
||||
final packed = File('${downloads.path}/$name');
|
||||
final unpacked = File('${downloads.path}/${name.replaceAll('.bz2', '')}');
|
||||
if (!unpacked.existsSync()) {
|
||||
await fetch(uri, packed);
|
||||
final input = InputFileStream(packed.path);
|
||||
final output = OutputFileStream(unpacked.path);
|
||||
try {
|
||||
BZip2Decoder().decodeStream(input, output);
|
||||
} finally {
|
||||
await input.close();
|
||||
await output.close();
|
||||
}
|
||||
await packed.delete();
|
||||
}
|
||||
return unpacked
|
||||
.openRead()
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter());
|
||||
}
|
||||
Future<Stream<String>> download(Uri uri) =>
|
||||
_tatoebaLines(uri, downloads: downloads, fetch: fetch);
|
||||
|
||||
final links = await readTatoebaLinks(
|
||||
await download(tatoebaLinksOf(from, to)),
|
||||
|
|
@ -552,7 +618,7 @@ Future<File> addTatoebaExamples(
|
|||
await download(tatoebaSentencesOf(to)),
|
||||
wanted: linked.contains,
|
||||
);
|
||||
return await _write(
|
||||
final examples = await _write(
|
||||
File('${dictionary.path}.tatoeba.gz'),
|
||||
withTatoebaExamples(
|
||||
dictionary,
|
||||
|
|
@ -561,6 +627,16 @@ Future<File> addTatoebaExamples(
|
|||
links: links,
|
||||
),
|
||||
);
|
||||
// Pinyin over Chinese; Tatoeba transcribes no other language of these
|
||||
// into readings.
|
||||
return from == 'cmn'
|
||||
? await addTatoebaReadings(
|
||||
examples,
|
||||
code: from,
|
||||
downloads: downloads,
|
||||
fetch: fetch,
|
||||
)
|
||||
: examples;
|
||||
}
|
||||
|
||||
/// KRDict's page of exports, which names the current one's download.
|
||||
|
|
|
|||
8
apps/learn/halkyon_learn_curator/test/fixtures/tatoeba-cmn_transcriptions.tsv
vendored
Normal file
8
apps/learn/halkyon_learn_curator/test/fixtures/tatoeba-cmn_transcriptions.tsv
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
750344 cmn Hant 我學習。
|
||||
750344 cmn Latn Wo3 xue2xi2.
|
||||
332662 cmn Hant 您學習嗎？
|
||||
332662 cmn Latn Yorwba Nin2 xue2xi2 ma5?
|
||||
4456365 cmn Hant 她學習數學
|
||||
4456365 cmn Latn Ta1 xue2xi2 shu4xue2
|
||||
333376 cmn Hant 我妹妹目前不學習。
|
||||
333376 cmn Latn Yorwba Wo3 mei4mei5 mu4qian2 bu4 xue2xi2.
|
||||
|
3
apps/learn/halkyon_learn_curator/test/fixtures/tatoeba-jpn_transcriptions.tsv
vendored
Normal file
3
apps/learn/halkyon_learn_curator/test/fixtures/tatoeba-jpn_transcriptions.tsv
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
193344 jpn Hrkt Waldelfe もっと[果物|くだ|もの]を[食|た]べるべきです。
|
||||
81964 jpn Hrkt [僕|ぼく]は[脚本|きゃく|ほん][家|か]で[食|た]べていく[決心|けっ|しん]をした。
|
||||
232489 jpn Hrkt あなたはどこでテレビを[見|み]ますか。
|
||||
|
167
apps/learn/halkyon_learn_curator/test/tatoeba_readings_test.dart
Normal file
167
apps/learn/halkyon_learn_curator/test/tatoeba_readings_test.dart
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba_readings.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
void main() {
|
||||
Stream<String> lines(String fixture) =>
|
||||
File('test/fixtures/$fixture')
|
||||
.openRead()
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter());
|
||||
|
||||
group('Japanese', () {
|
||||
test('a reading over each kanji where Tatoeba gives one each, over the '
|
||||
'whole word where it gives one for all', () {
|
||||
expect(rubyOf('ja', '僕は脚本家で食べる。', '[僕|ぼく]は[脚本|きゃく|ほん][家|か]で[食|た]べる。'), [
|
||||
(0, 1, 'ぼく'),
|
||||
(2, 3, 'きゃく'),
|
||||
(3, 4, 'ほん'),
|
||||
(4, 5, 'か'),
|
||||
(6, 7, 'た'),
|
||||
]);
|
||||
expect(rubyOf('ja', '一人いくら？', '[一人|ひとり]いくら？'), [(0, 2, 'ひとり')]);
|
||||
});
|
||||
|
||||
test('none for a transcription of another sentence than the example, '
|
||||
'nor for one without any reading', () {
|
||||
expect(rubyOf('ja', '僕は行く。', '[俺|おれ]は[行|い]く。'), isNull);
|
||||
expect(rubyOf('ja', 'ごめん。', 'ごめん。'), isNull);
|
||||
expect(rubyOf('ja', '僕。', '[僕|ぼく。'), isNull);
|
||||
});
|
||||
});
|
||||
|
||||
group('Chinese', () {
|
||||
test('pinyin with tone marks over each word, its syllables counted '
|
||||
"against the sentence's characters, whichever script", () {
|
||||
expect(rubyOf('zh', '我学习。', 'Wo3 xue2xi2.'), [
|
||||
(0, 1, 'wǒ'),
|
||||
(1, 3, 'xuéxí'),
|
||||
]);
|
||||
expect(rubyOf('zh', '您學習嗎？', 'Nin2 xue2xi2 ma5?'), [
|
||||
(0, 1, 'nín'),
|
||||
(1, 3, 'xuéxí'),
|
||||
(3, 4, 'ma'),
|
||||
]);
|
||||
expect(rubyOf('zh', '我妹妹去了', 'wo3 mei4mei5 qu4 le5'), [
|
||||
(0, 1, 'wǒ'),
|
||||
(1, 3, 'mèimei'),
|
||||
(3, 4, 'qù'),
|
||||
(4, 5, 'le'),
|
||||
]);
|
||||
});
|
||||
|
||||
test('marks the vowel pinyin marks: a or e first, the o of ou, else the '
|
||||
'last; ü written u: or v', () {
|
||||
expect(rubyOf('zh', '好够对女绿', 'hao3 gou4 dui4 nu:3 lv4'), [
|
||||
(0, 1, 'hǎo'),
|
||||
(1, 2, 'gòu'),
|
||||
(2, 3, 'duì'),
|
||||
(3, 4, 'nǚ'),
|
||||
(4, 5, 'lǜ'),
|
||||
]);
|
||||
});
|
||||
|
||||
test("none where the syllables don't match the characters in number", () {
|
||||
expect(rubyOf('zh', '我有3个', 'wo3 you3 san1 ge4'), isNull);
|
||||
expect(rubyOf('zh', '我学习', 'wo3 xue2'), isNull);
|
||||
});
|
||||
});
|
||||
|
||||
test('another language has none', () {
|
||||
expect(rubyOf('ko', '집', 'jip'), isNull);
|
||||
});
|
||||
|
||||
test("reads Tatoeba's transcriptions of one script, those no one wrote or "
|
||||
'checked (MeCab, or the automatic pinyin) unreviewed', () async {
|
||||
final japanese = await readTatoebaTranscriptions(
|
||||
lines('tatoeba-jpn_transcriptions.tsv'),
|
||||
);
|
||||
expect(japanese[193344], (
|
||||
text: 'もっと[果物|くだ|もの]を[食|た]べるべきです。',
|
||||
reviewed: true,
|
||||
));
|
||||
expect(japanese[81964]?.reviewed, isFalse);
|
||||
|
||||
final chinese = await readTatoebaTranscriptions(
|
||||
lines('tatoeba-cmn_transcriptions.tsv'),
|
||||
);
|
||||
expect(chinese.keys, unorderedEquals([750344, 332662, 4456365, 333376]));
|
||||
expect(chinese[750344], (text: 'Wo3 xue2xi2.', reviewed: false));
|
||||
expect(chinese[332662]?.reviewed, isTrue);
|
||||
});
|
||||
|
||||
test("puts readings on a dictionary's examples by their Tatoeba ID, those "
|
||||
'unreviewed marked, other lines as they were', () async {
|
||||
final header = jsonEncode({
|
||||
'format': 'halkyon-dictionary',
|
||||
'language': 'zh-Hans',
|
||||
'credits': <Object?>[],
|
||||
});
|
||||
String entry(List<Map<String, Object?>> examples) => jsonEncode({
|
||||
'headword': '学习',
|
||||
'entry': {
|
||||
'senses': [
|
||||
{
|
||||
'gloss': ['to learn'],
|
||||
'examples': examples,
|
||||
},
|
||||
{
|
||||
'gloss': ['study'],
|
||||
'examples': [
|
||||
{'text': '学习', 'zh-Hans': '您学习吗？', 'tatoeba': '332662'},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
final written = [
|
||||
for (final line in await withTatoebaReadings(
|
||||
Stream.fromIterable([
|
||||
header,
|
||||
entry([
|
||||
{
|
||||
'text': '学习',
|
||||
'zh-Hans': '我学习。',
|
||||
'en': 'I study.',
|
||||
'tatoeba': 750344,
|
||||
},
|
||||
{'text': '学习', 'zh-Hans': '我学。', 'tatoeba': 4456365},
|
||||
{'text': '学习', 'zh-Hans': '我学习。'},
|
||||
]),
|
||||
]),
|
||||
{
|
||||
750344: (text: 'Wo3 xue2xi2.', reviewed: false),
|
||||
332662: (text: 'Nin2 xue2xi2 ma5?', reviewed: true),
|
||||
4456365: (text: 'Ta1 xue2xi2 shu4xue2', reviewed: false),
|
||||
},
|
||||
).toList())
|
||||
jsonDecode(line) as Map<String, Object?>,
|
||||
];
|
||||
|
||||
expect(written.first, jsonDecode(header));
|
||||
final senses = (written[1]['entry']! as Map)['senses']! as List;
|
||||
final [first, second, third] = (senses[0] as Map)['examples']! as List;
|
||||
expect(first, {
|
||||
'text': '学习',
|
||||
'zh-Hans': '我学习。',
|
||||
'en': 'I study.',
|
||||
'tatoeba': 750344,
|
||||
'ruby': [
|
||||
[0, 1, 'wǒ'],
|
||||
[1, 3, 'xuéxí'],
|
||||
],
|
||||
'rubyUnreviewed': true,
|
||||
});
|
||||
expect(second, isNot(contains('ruby')));
|
||||
expect(third, isNot(contains('ruby')));
|
||||
final [other] = (senses[1] as Map)['examples']! as List;
|
||||
expect((other as Map)['ruby'], [
|
||||
[0, 1, 'nín'],
|
||||
[1, 3, 'xuéxí'],
|
||||
[3, 4, 'ma'],
|
||||
]);
|
||||
expect(other, isNot(contains('rubyUnreviewed')));
|
||||
});
|
||||
}
|
||||
|
|
@ -14,14 +14,6 @@ void main() {
|
|||
setUp(() => into = Directory.systemTemp.createTempSync('dictionaries'));
|
||||
tearDown(() => into.deleteSync(recursive: true));
|
||||
|
||||
/// Fetches the samples in place of EDRDG's files of the day.
|
||||
Future<void> samples(Uri uri, File file) async {
|
||||
final sample = uri == jmdictSource
|
||||
? 'test/fixtures/jmdict-e-examp-sample.xml'
|
||||
: 'test/fixtures/kanjidic2-sample.xml';
|
||||
await file.writeAsBytes(gzip.encode(await File(sample).readAsBytes()));
|
||||
}
|
||||
|
||||
/// Tatoeba's exports, as the fixtures hold a few of their lines, packed
|
||||
/// as Tatoeba packs them; null for any other address.
|
||||
List<int>? tatoeba(Uri uri) {
|
||||
|
|
@ -32,6 +24,33 @@ void main() {
|
|||
: null;
|
||||
}
|
||||
|
||||
/// Fetches the samples in place of EDRDG's files of the day, and
|
||||
/// Tatoeba's transcriptions.
|
||||
Future<void> samples(Uri uri, File file) async {
|
||||
if (tatoeba(uri) case final packed?) {
|
||||
await file.writeAsBytes(packed);
|
||||
return;
|
||||
}
|
||||
final sample = uri == jmdictSource
|
||||
? 'test/fixtures/jmdict-e-examp-sample.xml'
|
||||
: 'test/fixtures/kanjidic2-sample.xml';
|
||||
await file.writeAsBytes(gzip.encode(await File(sample).readAsBytes()));
|
||||
}
|
||||
|
||||
/// The examples of every sense in the dictionary file [name] in [into].
|
||||
List<Map<String, Object?>> examplesIn(String name) => [
|
||||
for (final line
|
||||
in utf8
|
||||
.decode(gzip.decode(File('${into.path}/$name').readAsBytesSync()))
|
||||
.split('\n')
|
||||
.skip(1))
|
||||
if (line.isNotEmpty)
|
||||
for (final sense
|
||||
in ((jsonDecode(line) as Map)['entry'] as Map)['senses'] as List)
|
||||
for (final example in (sense as Map)['examples'] as List? ?? const [])
|
||||
example as Map<String, Object?>,
|
||||
];
|
||||
|
||||
/// The header of the dictionary file [name] in [into].
|
||||
Map<String, Object?> header(String name) => jsonDecode(
|
||||
utf8
|
||||
|
|
@ -61,6 +80,19 @@ void main() {
|
|||
['JMdict', 'JLPT vocabulary lists', 'Tatoeba'],
|
||||
);
|
||||
expect(header('kanjidic-en.jsonl.gz')['kind'], 'characters');
|
||||
// Furigana over the sentences Tatoeba transcribed, those MeCab's alone
|
||||
// marked.
|
||||
final examples = {
|
||||
for (final example in examplesIn('jmdict-en.jsonl.gz'))
|
||||
'${example['tatoeba']}': example,
|
||||
};
|
||||
expect(examples['193344']?['ruby'], [
|
||||
[3, 4, 'くだ'],
|
||||
[4, 5, 'もの'],
|
||||
[6, 7, 'た'],
|
||||
]);
|
||||
expect(examples['193344'], isNot(contains('rubyUnreviewed')));
|
||||
expect(examples['81964']?['rubyUnreviewed'], isTrue);
|
||||
// Nothing half-written left behind; which conversion made them.
|
||||
expect(
|
||||
[for (final entity in into.listSync()) entity.uri.pathSegments.last]
|
||||
|
|
@ -242,6 +274,17 @@ void main() {
|
|||
(credit as Map)['name'],
|
||||
], contains('Tatoeba'));
|
||||
expect(cedictUpdated(into), isNotNull);
|
||||
// Pinyin over the sentences Tatoeba transcribed.
|
||||
expect(
|
||||
[
|
||||
for (final example in examplesIn('cc-cedict-hans-en.jsonl.gz'))
|
||||
if (example['tatoeba'] == 750344) example['ruby'],
|
||||
].first,
|
||||
[
|
||||
[0, 1, 'wǒ'],
|
||||
[1, 3, 'xuéxí'],
|
||||
],
|
||||
);
|
||||
// Nothing of the downloads left beside the files.
|
||||
expect(
|
||||
[for (final entity in into.listSync()) entity.uri.pathSegments.last]
|
||||
|
|
|
|||
Loading…
Reference in a new issue