feat(learn): furigana and pinyin over hints, from Tatoeba's transcriptions
The curator puts Tatoeba's readings on a dictionary's examples by their Tatoeba ID (`withTatoebaReadings`, `curate readings`, and in `update`): furigana over JMdict's Japanese from its jpn transcriptions, pinyin with tone marks over Chinese from its cmn ones, each word over as many Han characters as it has syllables. An example gets `ruby`, `[start, end, reading]` each, and `rubyUnreviewed` where no one wrote or checked them (MeCab's furigana, the automatic pinyin). JMdict's conversion goes to 3, so instances convert again at once. A word's example hints carry them as ruby marks, the word still in bold. The hints sheet shows them from the start with a switch above (Furigana, Pinyin), kept on the device (`DeviceSettings.hintReadings`), and below a hint whose readings no one reviewed says so. Whats-New: Example sentences in a word's hints now show furigana or pinyin over them, with a switch to hide them. Whats-New-De: Beispielsätze in den Hinweisen eines Worts zeigen jetzt Furigana oder Pinyin darüber, mit einem Schalter zum Ausblenden. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
bed39dc2a2
commit
3257905536
GPG key ID:
07F6FCF405488B64
25 changed files with 1083 additions and 78 deletions
File diff suppressed because one or more lines are too long
|
|
@ -189,6 +189,26 @@ up to three for each: those using the word as its headword writes it first, then
|
|||
those nearest seven words long, the shortest then; they come first in an entry's
|
||||
first sense, the word as it stands there beside. Some ten seconds for German.
|
||||
|
||||
Readings over the examples, from Tatoeba's transcriptions of their sentences
|
||||
(decided 9 Nov 2026): furigana over JMdict's Japanese, pinyin over Chinese.
|
||||
`update` adds them itself; by hand:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate readings \
|
||||
--input jmdict-en.jsonl.gz --transcriptions jpn_transcriptions.tsv.bz2 \
|
||||
--output jmdict-en.jsonl.gz
|
||||
```
|
||||
|
||||
from Tatoeba's `jpn_transcriptions.tsv.bz2` or `cmn_transcriptions.tsv.bz2`
|
||||
(CC BY 2.0 FR, the sentences' own licence). Each example with a Tatoeba ID
|
||||
gets `ruby`, a list of `[start, end, reading]` over its sentence (UTF-16 code
|
||||
units), where the transcription is of that very sentence: Japanese's
|
||||
`[漢字|かん|じ]` one reading for each kanji where there are as many, else one
|
||||
over the word; Chinese's tone numbers as tone marks, each of Tatoeba's words
|
||||
over as many Han characters as it has syllables, none where they don't count
|
||||
up (about 1 % of them). `rubyUnreviewed` marks those no one wrote or checked,
|
||||
MeCab's furigana and Tatoeba's automatic pinyin, some half of the Japanese.
|
||||
|
||||
Put the files into the server's `dictionaries` folder. The server loads each on
|
||||
its next start, and again whenever it changes (see the server's README).
|
||||
|
||||
|
|
@ -277,7 +297,7 @@ The **first line** says what the file holds:
|
|||
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
||||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
|
||||
| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, the source's id, e.g. `tatoeba`, and where Tatoeba transcribed the sentence `ruby` and `rubyUnreviewed`, the readings over it). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). A CC-CEDICT word's: `forms` (the word in the other script), `readings` (pinyin with tone marks), `senses` (each of CC-CEDICT's meanings, its glosses split at `;`) and `classifiers` (its measure words, `个 gè`). A KRDict word's: `pos`, `hanja`, `inflections` (`text`) and `senses` (`glosses`, its English definition as `info`). A Chinese character's: `meanings`, `pinyin` (the usual reading first), `strokes`, `traditional` or `simplified` (its forms in the other script), `hsk` and `frequency` (the place of the most used word written with it). A Hanja's: `meanings`, `eum` (its sounds in Hangul, the one taught first), `hun` and `eumhun` (its native meanings, and each with its sound, `물 수`) and `strokes`. A Hangul letter's: `romaji` (its sound, `g, k`), `script` and `name` (`기역`). A Greek letter's: `romaji` (its transliteration), `script`, `name`, `englishName` and `sound` (IPA). |
|
||||
|
||||
## The app's stroke files
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
|
|||
import 'package:halkyon_learn_curator/src/sources/krdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/makemeahanzi.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba_readings.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/update.dart';
|
||||
|
|
@ -92,6 +93,16 @@ import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
|||
/// --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
|
||||
/// --output wiktionary-de-en.jsonl.gz
|
||||
///
|
||||
/// Readings over a dictionary file's examples, from Tatoeba's
|
||||
/// transcriptions of their sentences (furigana over Japanese, pinyin over
|
||||
/// Chinese; its `jpn_transcriptions.tsv` or `cmn_transcriptions.tsv`, bzip2
|
||||
/// or not):
|
||||
///
|
||||
/// dart run halkyon_learn_curator:curate readings \
|
||||
/// --input jmdict-en.jsonl.gz \
|
||||
/// --transcriptions jpn_transcriptions.tsv.bz2 \
|
||||
/// --output jmdict-en.jsonl.gz
|
||||
///
|
||||
/// Or brings a server's dictionaries up to date from EDRDG's files of the
|
||||
/// day and, for the languages named, Wiktionary's latest dump and MDBG's
|
||||
/// CC-CEDICT of the day (Chinese), once or every so many days:
|
||||
|
|
@ -160,6 +171,10 @@ Future<void> main(List<String> args) async {
|
|||
'links',
|
||||
help: "tatoeba: Tatoeba's links between the two languages.",
|
||||
)
|
||||
..addOption(
|
||||
'transcriptions',
|
||||
help: "readings: Tatoeba's transcriptions of the dictionary's language.",
|
||||
)
|
||||
..addOption(
|
||||
'cedict',
|
||||
help:
|
||||
|
|
@ -281,6 +296,44 @@ Future<void> main(List<String> args) async {
|
|||
);
|
||||
return;
|
||||
}
|
||||
if (source == 'readings') {
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
final transcriptions = options.option('transcriptions');
|
||||
if (input == null || output == null || transcriptions == null) {
|
||||
_fail(
|
||||
'Usage: curate readings --input <dictionary> --transcriptions <file> '
|
||||
'--output <dictionary>\n${parser.usage}',
|
||||
);
|
||||
}
|
||||
final read = await readTatoebaTranscriptions(
|
||||
(transcriptions.endsWith('.bz2')
|
||||
? Stream.value(
|
||||
BZip2Decoder().decodeBytes(
|
||||
File(transcriptions).readAsBytesSync(),
|
||||
),
|
||||
)
|
||||
: File(transcriptions).openRead())
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter()),
|
||||
);
|
||||
final bytes = File(input).openRead();
|
||||
final written = File('$output.part');
|
||||
final lines = withTatoebaReadings(
|
||||
(input.endsWith('.gz') ? bytes.transform(gzip.decoder) : bytes)
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter()),
|
||||
read,
|
||||
).transform(utf8.encoder);
|
||||
await (output.endsWith('.gz') ? lines.transform(gzip.encoder) : lines).pipe(
|
||||
written.openWrite(),
|
||||
);
|
||||
await written.rename(output);
|
||||
stdout.writeln(
|
||||
'Wrote $output with ${read.length} transcriptions to read from.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (source == 'cedict') {
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
|
|
|
|||
|
|
@ -0,0 +1,231 @@
|
|||
import 'dart:convert';
|
||||
|
||||
/// A transcription of one of Tatoeba's sentences: its `text` in Tatoeba's
|
||||
/// way of writing it, and whether someone wrote or checked it (`reviewed`)
|
||||
/// rather than Tatoeba making it alone (MeCab's furigana, the automatic
|
||||
/// pinyin).
|
||||
typedef Transcription = ({String text, bool reviewed});
|
||||
|
||||
/// A reading over a part of a sentence: from its start to its end (UTF-16 code
|
||||
/// units, as Dart's strings count), and what it reads.
|
||||
typedef Ruby = (int start, int end, String reading);
|
||||
|
||||
/// The scripts Tatoeba transcribes a language into that give readings over
|
||||
/// it: furigana over Japanese (`Hrkt`), pinyin over Mandarin (`Latn`).
|
||||
const _rubyScripts = {'jpn': 'Hrkt', 'cmn': 'Latn'};
|
||||
|
||||
/// Tatoeba's transcriptions of one language by their sentence's ID
|
||||
/// (Tatoeba's `<code>_transcriptions.tsv`: id, language, script, the user
|
||||
/// who wrote or last checked it, empty for none, and the text), only those
|
||||
/// in the script that gives readings over it.
|
||||
Future<Map<int, Transcription>> readTatoebaTranscriptions(
|
||||
Stream<String> lines,
|
||||
) async {
|
||||
final transcriptions = <int, Transcription>{};
|
||||
await for (final line in lines) {
|
||||
final fields = line.split('\t');
|
||||
if (fields.length < 5 || _rubyScripts[fields[1]] != fields[2]) {
|
||||
continue;
|
||||
}
|
||||
if (int.tryParse(fields[0]) case final id?) {
|
||||
transcriptions[id] = (
|
||||
text: fields.sublist(4).join('\t'),
|
||||
reviewed: fields[3].isNotEmpty,
|
||||
);
|
||||
}
|
||||
}
|
||||
return transcriptions;
|
||||
}
|
||||
|
||||
/// The readings [transcription] gives over [sentence] in [language] (a BCP
|
||||
/// 47 tag): furigana for Japanese, from Tatoeba's `[漢字|かん|じ]` (one
|
||||
/// reading for each character where there are as many, else one for all);
|
||||
/// pinyin with tone marks for Chinese, each of Tatoeba's words over as
|
||||
/// many of the sentence's Han characters as it has syllables. Null for
|
||||
/// another language, a transcription of another sentence or one that
|
||||
/// doesn't count up, and one without any reading.
|
||||
List<Ruby>? rubyOf(String language, String sentence, String transcription) {
|
||||
final ruby = switch (language.split('-').first.toLowerCase()) {
|
||||
'ja' => _furigana(sentence, transcription),
|
||||
'zh' => _pinyin(sentence, transcription),
|
||||
_ => null,
|
||||
};
|
||||
return ruby == null || ruby.isEmpty ? null : ruby;
|
||||
}
|
||||
|
||||
final _bracket = RegExp(r'\[([^\[\]|]+)((?:\|[^\[\]|]*)+)\]');
|
||||
|
||||
List<Ruby>? _furigana(String sentence, String transcription) {
|
||||
final ruby = <Ruby>[];
|
||||
final plain = StringBuffer();
|
||||
var at = 0;
|
||||
for (final match in _bracket.allMatches(transcription)) {
|
||||
plain.write(transcription.substring(at, match.start));
|
||||
final base = match[1]!;
|
||||
final readings = match[2]!.substring(1).split('|');
|
||||
final characters = base.runes.map(String.fromCharCode).toList();
|
||||
if (readings.length == characters.length) {
|
||||
for (final (i, character) in characters.indexed) {
|
||||
if (readings[i].isNotEmpty) {
|
||||
ruby.add((
|
||||
plain.length,
|
||||
plain.length + character.length,
|
||||
readings[i],
|
||||
));
|
||||
}
|
||||
plain.write(character);
|
||||
}
|
||||
} else {
|
||||
ruby.add((plain.length, plain.length + base.length, readings.join()));
|
||||
plain.write(base);
|
||||
}
|
||||
at = match.end;
|
||||
}
|
||||
plain.write(transcription.substring(at));
|
||||
return plain.toString() == sentence ? ruby : null;
|
||||
}
|
||||
|
||||
final _han = RegExp(r'\p{Script=Han}', unicode: true);
|
||||
|
||||
final _syllable = RegExp('([a-zü]+(?::[a-z]*)?)([1-5])', caseSensitive: false);
|
||||
|
||||
List<Ruby>? _pinyin(String sentence, String transcription) {
|
||||
// Where each Han character of the sentence is.
|
||||
final characters = [
|
||||
for (final match in _han.allMatches(sentence)) (match.start, match.end),
|
||||
];
|
||||
final ruby = <Ruby>[];
|
||||
var next = 0;
|
||||
for (final word in transcription.split(RegExp(r'\s+'))) {
|
||||
final syllables = [
|
||||
for (final match in _syllable.allMatches(word))
|
||||
_marked(match[1]!.toLowerCase(), int.parse(match[2]!)),
|
||||
];
|
||||
if (syllables.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
if (next + syllables.length > characters.length) {
|
||||
return null;
|
||||
}
|
||||
ruby.add((
|
||||
characters[next].$1,
|
||||
characters[next + syllables.length - 1].$2,
|
||||
syllables.join(),
|
||||
));
|
||||
next += syllables.length;
|
||||
}
|
||||
return next == characters.length ? ruby : null;
|
||||
}
|
||||
|
||||
const _toneMarks = {
|
||||
'a': 'āáǎà',
|
||||
'e': 'ēéěè',
|
||||
'i': 'īíǐì',
|
||||
'o': 'ōóǒò',
|
||||
'u': 'ūúǔù',
|
||||
'ü': 'ǖǘǚǜ',
|
||||
};
|
||||
|
||||
/// [syllable] with [tone] marked: on its a or e, the o of ou, else its
|
||||
/// last vowel; none for the neutral tone (5). `u:` and `v` are ü.
|
||||
String _marked(String syllable, int tone) {
|
||||
final letters = syllable.replaceAll('u:', 'ü').replaceAll('v', 'ü');
|
||||
if (tone == 5) {
|
||||
return letters;
|
||||
}
|
||||
final vowel = letters.contains('a')
|
||||
? letters.indexOf('a')
|
||||
: letters.contains('e')
|
||||
? letters.indexOf('e')
|
||||
: letters.contains('ou')
|
||||
? letters.indexOf('o')
|
||||
: letters.lastIndexOf(RegExp('[aeiouü]'));
|
||||
if (vowel < 0) {
|
||||
return letters;
|
||||
}
|
||||
return letters.replaceRange(
|
||||
vowel,
|
||||
vowel + 1,
|
||||
_toneMarks[letters[vowel]]![tone - 1],
|
||||
);
|
||||
}
|
||||
|
||||
/// A dictionary file's [lines] (without their line ends) with readings over its examples: each
|
||||
/// example with a Tatoeba ID among [transcriptions] whose readings fit its
|
||||
/// sentence gets them as `ruby` (`[start, end, reading]` each), and
|
||||
/// `rubyUnreviewed` where no one wrote or checked them. The header and
|
||||
/// every other line pass as they were, each ended.
|
||||
Stream<String> withTatoebaReadings(
|
||||
Stream<String> lines,
|
||||
Map<int, Transcription> transcriptions,
|
||||
) async* {
|
||||
String? language;
|
||||
await for (final line in lines) {
|
||||
if (language == null) {
|
||||
final header = jsonDecode(line) as Map<String, Object?>;
|
||||
language = header['language']! as String;
|
||||
yield '$line\n';
|
||||
continue;
|
||||
}
|
||||
if (line.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
final entry = jsonDecode(line) as Map<String, Object?>;
|
||||
var changed = false;
|
||||
Map<String, Object?> withRuby(Map<String, Object?> example) {
|
||||
final id = switch (example['tatoeba']) {
|
||||
final int id => id,
|
||||
final String id => int.tryParse(id),
|
||||
_ => null,
|
||||
};
|
||||
final transcription = transcriptions[id];
|
||||
if (transcription == null) {
|
||||
return example;
|
||||
}
|
||||
final sentence = example[language];
|
||||
if (sentence is! String) {
|
||||
return example;
|
||||
}
|
||||
final ruby = rubyOf(language!, sentence, transcription.text);
|
||||
if (ruby == null) {
|
||||
return example;
|
||||
}
|
||||
changed = true;
|
||||
return {
|
||||
...example,
|
||||
'ruby': [
|
||||
for (final (start, end, reading) in ruby) [start, end, reading],
|
||||
],
|
||||
if (!transcription.reviewed) 'rubyUnreviewed': true,
|
||||
};
|
||||
}
|
||||
|
||||
final body = entry['entry'];
|
||||
if (body is! Map<String, Object?> || body['senses'] is! List) {
|
||||
yield '$line\n';
|
||||
continue;
|
||||
}
|
||||
final senses = [
|
||||
for (final sense in body['senses']! as List)
|
||||
if (sense case {'examples': final List<Object?> examples})
|
||||
{
|
||||
...sense as Map<String, Object?>,
|
||||
'examples': [
|
||||
for (final example in examples)
|
||||
if (example is Map<String, Object?>)
|
||||
withRuby(example)
|
||||
else
|
||||
example,
|
||||
],
|
||||
}
|
||||
else
|
||||
sense,
|
||||
];
|
||||
yield changed
|
||||
? '${jsonEncode({
|
||||
...entry,
|
||||
'entry': {...body, 'senses': senses},
|
||||
})}\n'
|
||||
: '$line\n';
|
||||
}
|
||||
}
|
||||
|
|
@ -10,6 +10,7 @@ import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
|
|||
import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/krdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba_readings.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
|
||||
import 'package:halkyon_learn_curator/src/wordfreq.dart';
|
||||
|
|
@ -31,8 +32,9 @@ const updatedFiles = ['jmdict-en.jsonl.gz', 'kanjidic-en.jsonl.gz'];
|
|||
/// curator writes something else from the same download (a label, a field),
|
||||
/// so that an instance converts again as soon as the new curator runs
|
||||
/// rather than at its next update, up to a week later. 2: kanji labelled
|
||||
/// by their radical.
|
||||
const conversion = 2;
|
||||
/// by their radical. 3: furigana over JMdict's examples, from Tatoeba's
|
||||
/// transcriptions.
|
||||
const conversion = 3;
|
||||
|
||||
/// The file in the dictionaries' folder that says which [conversion] made
|
||||
/// them. The server reads only dictionary files (`.jsonl`, `.jsonl.gz`).
|
||||
|
|
@ -59,7 +61,8 @@ Future<void> httpFetch(Uri uri, File file) async {
|
|||
}
|
||||
|
||||
/// Brings the dictionary files in [into] up to date from EDRDG's files of
|
||||
/// the day: JMdict with its sentences and KANJIDIC2 with the kana, the
|
||||
/// the day: JMdict with its sentences, furigana over them from Tatoeba's
|
||||
/// transcriptions (`addTatoebaReadings`), and KANJIDIC2 with the kana, the
|
||||
/// JLPT levels from the curator's [data] folder (`data/README.md`). Each
|
||||
/// file is written beside its place first and then put there at once, so
|
||||
/// the server never reads half of one; should anything fail, the files
|
||||
|
|
@ -87,6 +90,12 @@ Future<List<File>> updateDictionaries({
|
|||
examples: true,
|
||||
),
|
||||
);
|
||||
final jmdictRead = await addTatoebaReadings(
|
||||
jmdict,
|
||||
code: 'jpn',
|
||||
downloads: downloads,
|
||||
fetch: fetch,
|
||||
);
|
||||
final kanjidic = await _write(
|
||||
File('${downloads.path}/${updatedFiles[1]}'),
|
||||
Stream.fromIterable(
|
||||
|
|
@ -98,7 +107,8 @@ Future<List<File>> updateDictionaries({
|
|||
);
|
||||
await into.create(recursive: true);
|
||||
final written = [
|
||||
for (final file in [jmdict, kanjidic]) await _putInPlace(file, into),
|
||||
await _putInPlace(jmdictRead, into, name: updatedFiles[0]),
|
||||
await _putInPlace(kanjidic, into),
|
||||
];
|
||||
await File('${into.path}/$conversionFile').writeAsString('$conversion');
|
||||
return written;
|
||||
|
|
@ -495,9 +505,84 @@ Uri tatoebaLinksOf(String from, String to) => Uri.parse(
|
|||
'$from-${to}_links.tsv.bz2',
|
||||
);
|
||||
|
||||
/// Tatoeba's weekly export of the transcriptions of a language's sentences
|
||||
/// ([code], ISO 639-3).
|
||||
Uri tatoebaTranscriptionsOf(String code) => Uri.parse(
|
||||
'https://downloads.tatoeba.org/exports/per_language/$code/'
|
||||
'${code}_transcriptions.tsv.bz2',
|
||||
);
|
||||
|
||||
/// The lines of Tatoeba's export at [uri], fetched into [downloads] and
|
||||
/// unpacked there once.
|
||||
Future<Stream<String>> _tatoebaLines(
|
||||
Uri uri, {
|
||||
required Directory downloads,
|
||||
required Fetch fetch,
|
||||
}) async {
|
||||
final name = uri.pathSegments.last;
|
||||
final packed = File('${downloads.path}/$name');
|
||||
final unpacked = File('${downloads.path}/${name.replaceAll('.bz2', '')}');
|
||||
if (!unpacked.existsSync()) {
|
||||
await fetch(uri, packed);
|
||||
final input = InputFileStream(packed.path);
|
||||
final output = OutputFileStream(unpacked.path);
|
||||
try {
|
||||
BZip2Decoder().decodeStream(input, output);
|
||||
} finally {
|
||||
await input.close();
|
||||
await output.close();
|
||||
}
|
||||
await packed.delete();
|
||||
}
|
||||
return unpacked
|
||||
.openRead()
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter());
|
||||
}
|
||||
|
||||
/// [dictionary], a dictionary file just written (gzipped), with readings
|
||||
/// over its examples from Tatoeba's transcriptions of the sentences in
|
||||
/// [code] (`withTatoebaReadings`: furigana over Japanese, pinyin over
|
||||
/// Chinese), downloaded into [downloads]; returns the file it wrote beside
|
||||
/// it, or [dictionary] as it was should Tatoeba fail: readings are worth
|
||||
/// having, never worth an update.
|
||||
Future<File> addTatoebaReadings(
|
||||
File dictionary, {
|
||||
required String code,
|
||||
required Directory downloads,
|
||||
Fetch fetch = httpFetch,
|
||||
}) async {
|
||||
try {
|
||||
final transcriptions = await readTatoebaTranscriptions(
|
||||
await _tatoebaLines(
|
||||
tatoebaTranscriptionsOf(code),
|
||||
downloads: downloads,
|
||||
fetch: fetch,
|
||||
),
|
||||
);
|
||||
return await _write(
|
||||
File('${dictionary.path}.readings.gz'),
|
||||
withTatoebaReadings(
|
||||
dictionary
|
||||
.openRead()
|
||||
.transform(gzip.decoder)
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter()),
|
||||
transcriptions,
|
||||
),
|
||||
);
|
||||
} on Object catch (error) {
|
||||
stdout.writeln(
|
||||
'Readings left out of ${dictionary.uri.pathSegments.last}: $error',
|
||||
);
|
||||
return dictionary;
|
||||
}
|
||||
}
|
||||
|
||||
/// [dictionary], a dictionary file just written (gzipped), with Tatoeba's
|
||||
/// sentences in its language that have a translation into its meaning
|
||||
/// language as its entries' first examples (`withTatoebaExamples`),
|
||||
/// language as its entries' first examples (`withTatoebaExamples`), and
|
||||
/// for Chinese pinyin over them (`addTatoebaReadings`),
|
||||
/// downloaded into [downloads]; returns the file it wrote beside it. A
|
||||
/// language Tatoeba has no code for here is left as it is.
|
||||
Future<File> addTatoebaExamples(
|
||||
|
|
@ -518,27 +603,8 @@ Future<File> addTatoebaExamples(
|
|||
if (from == null || to == null) {
|
||||
return dictionary;
|
||||
}
|
||||
Future<Stream<String>> download(Uri uri) async {
|
||||
final name = uri.pathSegments.last;
|
||||
final packed = File('${downloads.path}/$name');
|
||||
final unpacked = File('${downloads.path}/${name.replaceAll('.bz2', '')}');
|
||||
if (!unpacked.existsSync()) {
|
||||
await fetch(uri, packed);
|
||||
final input = InputFileStream(packed.path);
|
||||
final output = OutputFileStream(unpacked.path);
|
||||
try {
|
||||
BZip2Decoder().decodeStream(input, output);
|
||||
} finally {
|
||||
await input.close();
|
||||
await output.close();
|
||||
}
|
||||
await packed.delete();
|
||||
}
|
||||
return unpacked
|
||||
.openRead()
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter());
|
||||
}
|
||||
Future<Stream<String>> download(Uri uri) =>
|
||||
_tatoebaLines(uri, downloads: downloads, fetch: fetch);
|
||||
|
||||
final links = await readTatoebaLinks(
|
||||
await download(tatoebaLinksOf(from, to)),
|
||||
|
|
@ -552,7 +618,7 @@ Future<File> addTatoebaExamples(
|
|||
await download(tatoebaSentencesOf(to)),
|
||||
wanted: linked.contains,
|
||||
);
|
||||
return await _write(
|
||||
final examples = await _write(
|
||||
File('${dictionary.path}.tatoeba.gz'),
|
||||
withTatoebaExamples(
|
||||
dictionary,
|
||||
|
|
@ -561,6 +627,16 @@ Future<File> addTatoebaExamples(
|
|||
links: links,
|
||||
),
|
||||
);
|
||||
// Pinyin over Chinese; Tatoeba transcribes no other language of these
|
||||
// into readings.
|
||||
return from == 'cmn'
|
||||
? await addTatoebaReadings(
|
||||
examples,
|
||||
code: from,
|
||||
downloads: downloads,
|
||||
fetch: fetch,
|
||||
)
|
||||
: examples;
|
||||
}
|
||||
|
||||
/// KRDict's page of exports, which names the current one's download.
|
||||
|
|
|
|||
8
apps/learn/halkyon_learn_curator/test/fixtures/tatoeba-cmn_transcriptions.tsv
vendored
Normal file
8
apps/learn/halkyon_learn_curator/test/fixtures/tatoeba-cmn_transcriptions.tsv
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
750344 cmn Hant 我學習。
|
||||
750344 cmn Latn Wo3 xue2xi2.
|
||||
332662 cmn Hant 您學習嗎？
|
||||
332662 cmn Latn Yorwba Nin2 xue2xi2 ma5?
|
||||
4456365 cmn Hant 她學習數學
|
||||
4456365 cmn Latn Ta1 xue2xi2 shu4xue2
|
||||
333376 cmn Hant 我妹妹目前不學習。
|
||||
333376 cmn Latn Yorwba Wo3 mei4mei5 mu4qian2 bu4 xue2xi2.
|
||||
|
3
apps/learn/halkyon_learn_curator/test/fixtures/tatoeba-jpn_transcriptions.tsv
vendored
Normal file
3
apps/learn/halkyon_learn_curator/test/fixtures/tatoeba-jpn_transcriptions.tsv
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
193344 jpn Hrkt Waldelfe もっと[果物|くだ|もの]を[食|た]べるべきです。
|
||||
81964 jpn Hrkt [僕|ぼく]は[脚本|きゃく|ほん][家|か]で[食|た]べていく[決心|けっ|しん]をした。
|
||||
232489 jpn Hrkt あなたはどこでテレビを[見|み]ますか。
|
||||
|
167
apps/learn/halkyon_learn_curator/test/tatoeba_readings_test.dart
Normal file
167
apps/learn/halkyon_learn_curator/test/tatoeba_readings_test.dart
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/sources/tatoeba_readings.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
void main() {
|
||||
Stream<String> lines(String fixture) =>
|
||||
File('test/fixtures/$fixture')
|
||||
.openRead()
|
||||
.transform(utf8.decoder)
|
||||
.transform(const LineSplitter());
|
||||
|
||||
group('Japanese', () {
|
||||
test('a reading over each kanji where Tatoeba gives one each, over the '
|
||||
'whole word where it gives one for all', () {
|
||||
expect(rubyOf('ja', '僕は脚本家で食べる。', '[僕|ぼく]は[脚本|きゃく|ほん][家|か]で[食|た]べる。'), [
|
||||
(0, 1, 'ぼく'),
|
||||
(2, 3, 'きゃく'),
|
||||
(3, 4, 'ほん'),
|
||||
(4, 5, 'か'),
|
||||
(6, 7, 'た'),
|
||||
]);
|
||||
expect(rubyOf('ja', '一人いくら？', '[一人|ひとり]いくら？'), [(0, 2, 'ひとり')]);
|
||||
});
|
||||
|
||||
test('none for a transcription of another sentence than the example, '
|
||||
'nor for one without any reading', () {
|
||||
expect(rubyOf('ja', '僕は行く。', '[俺|おれ]は[行|い]く。'), isNull);
|
||||
expect(rubyOf('ja', 'ごめん。', 'ごめん。'), isNull);
|
||||
expect(rubyOf('ja', '僕。', '[僕|ぼく。'), isNull);
|
||||
});
|
||||
});
|
||||
|
||||
group('Chinese', () {
|
||||
test('pinyin with tone marks over each word, its syllables counted '
|
||||
"against the sentence's characters, whichever script", () {
|
||||
expect(rubyOf('zh', '我学习。', 'Wo3 xue2xi2.'), [
|
||||
(0, 1, 'wǒ'),
|
||||
(1, 3, 'xuéxí'),
|
||||
]);
|
||||
expect(rubyOf('zh', '您學習嗎？', 'Nin2 xue2xi2 ma5?'), [
|
||||
(0, 1, 'nín'),
|
||||
(1, 3, 'xuéxí'),
|
||||
(3, 4, 'ma'),
|
||||
]);
|
||||
expect(rubyOf('zh', '我妹妹去了', 'wo3 mei4mei5 qu4 le5'), [
|
||||
(0, 1, 'wǒ'),
|
||||
(1, 3, 'mèimei'),
|
||||
(3, 4, 'qù'),
|
||||
(4, 5, 'le'),
|
||||
]);
|
||||
});
|
||||
|
||||
test('marks the vowel pinyin marks: a or e first, the o of ou, else the '
|
||||
'last; ü written u: or v', () {
|
||||
expect(rubyOf('zh', '好够对女绿', 'hao3 gou4 dui4 nu:3 lv4'), [
|
||||
(0, 1, 'hǎo'),
|
||||
(1, 2, 'gòu'),
|
||||
(2, 3, 'duì'),
|
||||
(3, 4, 'nǚ'),
|
||||
(4, 5, 'lǜ'),
|
||||
]);
|
||||
});
|
||||
|
||||
test("none where the syllables don't match the characters in number", () {
|
||||
expect(rubyOf('zh', '我有3个', 'wo3 you3 san1 ge4'), isNull);
|
||||
expect(rubyOf('zh', '我学习', 'wo3 xue2'), isNull);
|
||||
});
|
||||
});
|
||||
|
||||
test('another language has none', () {
|
||||
expect(rubyOf('ko', '집', 'jip'), isNull);
|
||||
});
|
||||
|
||||
test("reads Tatoeba's transcriptions of one script, those no one wrote or "
|
||||
'checked (MeCab, or the automatic pinyin) unreviewed', () async {
|
||||
final japanese = await readTatoebaTranscriptions(
|
||||
lines('tatoeba-jpn_transcriptions.tsv'),
|
||||
);
|
||||
expect(japanese[193344], (
|
||||
text: 'もっと[果物|くだ|もの]を[食|た]べるべきです。',
|
||||
reviewed: true,
|
||||
));
|
||||
expect(japanese[81964]?.reviewed, isFalse);
|
||||
|
||||
final chinese = await readTatoebaTranscriptions(
|
||||
lines('tatoeba-cmn_transcriptions.tsv'),
|
||||
);
|
||||
expect(chinese.keys, unorderedEquals([750344, 332662, 4456365, 333376]));
|
||||
expect(chinese[750344], (text: 'Wo3 xue2xi2.', reviewed: false));
|
||||
expect(chinese[332662]?.reviewed, isTrue);
|
||||
});
|
||||
|
||||
test("puts readings on a dictionary's examples by their Tatoeba ID, those "
|
||||
'unreviewed marked, other lines as they were', () async {
|
||||
final header = jsonEncode({
|
||||
'format': 'halkyon-dictionary',
|
||||
'language': 'zh-Hans',
|
||||
'credits': <Object?>[],
|
||||
});
|
||||
String entry(List<Map<String, Object?>> examples) => jsonEncode({
|
||||
'headword': '学习',
|
||||
'entry': {
|
||||
'senses': [
|
||||
{
|
||||
'gloss': ['to learn'],
|
||||
'examples': examples,
|
||||
},
|
||||
{
|
||||
'gloss': ['study'],
|
||||
'examples': [
|
||||
{'text': '学习', 'zh-Hans': '您学习吗？', 'tatoeba': '332662'},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
final written = [
|
||||
for (final line in await withTatoebaReadings(
|
||||
Stream.fromIterable([
|
||||
header,
|
||||
entry([
|
||||
{
|
||||
'text': '学习',
|
||||
'zh-Hans': '我学习。',
|
||||
'en': 'I study.',
|
||||
'tatoeba': 750344,
|
||||
},
|
||||
{'text': '学习', 'zh-Hans': '我学。', 'tatoeba': 4456365},
|
||||
{'text': '学习', 'zh-Hans': '我学习。'},
|
||||
]),
|
||||
]),
|
||||
{
|
||||
750344: (text: 'Wo3 xue2xi2.', reviewed: false),
|
||||
332662: (text: 'Nin2 xue2xi2 ma5?', reviewed: true),
|
||||
4456365: (text: 'Ta1 xue2xi2 shu4xue2', reviewed: false),
|
||||
},
|
||||
).toList())
|
||||
jsonDecode(line) as Map<String, Object?>,
|
||||
];
|
||||
|
||||
expect(written.first, jsonDecode(header));
|
||||
final senses = (written[1]['entry']! as Map)['senses']! as List;
|
||||
final [first, second, third] = (senses[0] as Map)['examples']! as List;
|
||||
expect(first, {
|
||||
'text': '学习',
|
||||
'zh-Hans': '我学习。',
|
||||
'en': 'I study.',
|
||||
'tatoeba': 750344,
|
||||
'ruby': [
|
||||
[0, 1, 'wǒ'],
|
||||
[1, 3, 'xuéxí'],
|
||||
],
|
||||
'rubyUnreviewed': true,
|
||||
});
|
||||
expect(second, isNot(contains('ruby')));
|
||||
expect(third, isNot(contains('ruby')));
|
||||
final [other] = (senses[1] as Map)['examples']! as List;
|
||||
expect((other as Map)['ruby'], [
|
||||
[0, 1, 'nín'],
|
||||
[1, 3, 'xuéxí'],
|
||||
[3, 4, 'ma'],
|
||||
]);
|
||||
expect(other, isNot(contains('rubyUnreviewed')));
|
||||
});
|
||||
}
|
||||
|
|
@ -14,14 +14,6 @@ void main() {
|
|||
setUp(() => into = Directory.systemTemp.createTempSync('dictionaries'));
|
||||
tearDown(() => into.deleteSync(recursive: true));
|
||||
|
||||
/// Fetches the samples in place of EDRDG's files of the day.
|
||||
Future<void> samples(Uri uri, File file) async {
|
||||
final sample = uri == jmdictSource
|
||||
? 'test/fixtures/jmdict-e-examp-sample.xml'
|
||||
: 'test/fixtures/kanjidic2-sample.xml';
|
||||
await file.writeAsBytes(gzip.encode(await File(sample).readAsBytes()));
|
||||
}
|
||||
|
||||
/// Tatoeba's exports, as the fixtures hold a few of their lines, packed
|
||||
/// as Tatoeba packs them; null for any other address.
|
||||
List<int>? tatoeba(Uri uri) {
|
||||
|
|
@ -32,6 +24,33 @@ void main() {
|
|||
: null;
|
||||
}
|
||||
|
||||
/// Fetches the samples in place of EDRDG's files of the day, and
|
||||
/// Tatoeba's transcriptions.
|
||||
Future<void> samples(Uri uri, File file) async {
|
||||
if (tatoeba(uri) case final packed?) {
|
||||
await file.writeAsBytes(packed);
|
||||
return;
|
||||
}
|
||||
final sample = uri == jmdictSource
|
||||
? 'test/fixtures/jmdict-e-examp-sample.xml'
|
||||
: 'test/fixtures/kanjidic2-sample.xml';
|
||||
await file.writeAsBytes(gzip.encode(await File(sample).readAsBytes()));
|
||||
}
|
||||
|
||||
/// The examples of every sense in the dictionary file [name] in [into].
|
||||
List<Map<String, Object?>> examplesIn(String name) => [
|
||||
for (final line
|
||||
in utf8
|
||||
.decode(gzip.decode(File('${into.path}/$name').readAsBytesSync()))
|
||||
.split('\n')
|
||||
.skip(1))
|
||||
if (line.isNotEmpty)
|
||||
for (final sense
|
||||
in ((jsonDecode(line) as Map)['entry'] as Map)['senses'] as List)
|
||||
for (final example in (sense as Map)['examples'] as List? ?? const [])
|
||||
example as Map<String, Object?>,
|
||||
];
|
||||
|
||||
/// The header of the dictionary file [name] in [into].
|
||||
Map<String, Object?> header(String name) => jsonDecode(
|
||||
utf8
|
||||
|
|
@ -61,6 +80,19 @@ void main() {
|
|||
['JMdict', 'JLPT vocabulary lists', 'Tatoeba'],
|
||||
);
|
||||
expect(header('kanjidic-en.jsonl.gz')['kind'], 'characters');
|
||||
// Furigana over the sentences Tatoeba transcribed, those MeCab's alone
|
||||
// marked.
|
||||
final examples = {
|
||||
for (final example in examplesIn('jmdict-en.jsonl.gz'))
|
||||
'${example['tatoeba']}': example,
|
||||
};
|
||||
expect(examples['193344']?['ruby'], [
|
||||
[3, 4, 'くだ'],
|
||||
[4, 5, 'もの'],
|
||||
[6, 7, 'た'],
|
||||
]);
|
||||
expect(examples['193344'], isNot(contains('rubyUnreviewed')));
|
||||
expect(examples['81964']?['rubyUnreviewed'], isTrue);
|
||||
// Nothing half-written left behind; which conversion made them.
|
||||
expect(
|
||||
[for (final entity in into.listSync()) entity.uri.pathSegments.last]
|
||||
|
|
@ -242,6 +274,17 @@ void main() {
|
|||
(credit as Map)['name'],
|
||||
], contains('Tatoeba'));
|
||||
expect(cedictUpdated(into), isNotNull);
|
||||
// Pinyin over the sentences Tatoeba transcribed.
|
||||
expect(
|
||||
[
|
||||
for (final example in examplesIn('cc-cedict-hans-en.jsonl.gz'))
|
||||
if (example['tatoeba'] == 750344) example['ruby'],
|
||||
].first,
|
||||
[
|
||||
[0, 1, 'wǒ'],
|
||||
[1, 3, 'xuéxí'],
|
||||
],
|
||||
);
|
||||
// Nothing of the downloads left beside the files.
|
||||
expect(
|
||||
[for (final entity in into.listSync()) entity.uri.pathSegments.last]
|
||||
|
|
|
|||
|
|
@ -2172,5 +2172,9 @@
|
|||
"libraryShelfLastWeek": "Letzte 7 Tage",
|
||||
"libraryShelfLastMonth": "Letzter Monat",
|
||||
"libraryShelfOlder": "Älter",
|
||||
"studysetSideIntroduction": "Einführung"
|
||||
"studysetSideIntroduction": "Einführung",
|
||||
"cardHintsFurigana": "Furigana",
|
||||
"cardHintsPinyin": "Pinyin",
|
||||
"cardHintsReadings": "Text darüber",
|
||||
"cardHintsReadingsUnreviewed": "Manche Lesungen wurden automatisch erstellt und nicht geprüft."
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2670,5 +2670,13 @@
|
|||
"libraryShelfOlder": "Older",
|
||||
"@libraryShelfOlder": {"description": "The library's heading over the studysets studied or changed longer ago."},
|
||||
"studysetSideIntroduction": "Introduction",
|
||||
"@studysetSideIntroduction": {"description": "The part of the split beside a studyset's material showing its description, as Tomorrow University names it."}
|
||||
"@studysetSideIntroduction": {"description": "The part of the split beside a studyset's material showing its description, as Tomorrow University names it."},
|
||||
"cardHintsFurigana": "Furigana",
|
||||
"@cardHintsFurigana": {"description": "The hints sheet's switch showing the readings over Japanese sentences."},
|
||||
"cardHintsPinyin": "Pinyin",
|
||||
"@cardHintsPinyin": {"description": "The hints sheet's switch showing the pinyin over Chinese sentences."},
|
||||
"cardHintsReadings": "Text above",
|
||||
"@cardHintsReadings": {"description": "The hints sheet's switch showing the text above its sentences, where it is neither furigana nor pinyin."},
|
||||
"cardHintsReadingsUnreviewed": "Some readings were made automatically and not reviewed.",
|
||||
"@cardHintsReadingsUnreviewed": {"description": "Below a hint whose readings (from Tatoeba, made by MeCab or automatic pinyin) no one reviewed."}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12817,6 +12817,30 @@ abstract class AppLocalizations {
|
|||
/// In en, this message translates to:
|
||||
/// **'Introduction'**
|
||||
String get studysetSideIntroduction;
|
||||
|
||||
/// The hints sheet's switch showing the readings over Japanese sentences.
|
||||
///
|
||||
/// In en, this message translates to:
|
||||
/// **'Furigana'**
|
||||
String get cardHintsFurigana;
|
||||
|
||||
/// The hints sheet's switch showing the pinyin over Chinese sentences.
|
||||
///
|
||||
/// In en, this message translates to:
|
||||
/// **'Pinyin'**
|
||||
String get cardHintsPinyin;
|
||||
|
||||
/// The hints sheet's switch showing the text above its sentences, where it is neither furigana nor pinyin.
|
||||
///
|
||||
/// In en, this message translates to:
|
||||
/// **'Text above'**
|
||||
String get cardHintsReadings;
|
||||
|
||||
/// Below a hint whose readings (from Tatoeba, made by MeCab or automatic pinyin) no one reviewed.
|
||||
///
|
||||
/// In en, this message translates to:
|
||||
/// **'Some readings were made automatically and not reviewed.'**
|
||||
String get cardHintsReadingsUnreviewed;
|
||||
}
|
||||
|
||||
class _AppLocalizationsDelegate
|
||||
|
|
|
|||
|
|
@ -8191,4 +8191,17 @@ class AppLocalizationsDe extends AppLocalizations {
|
|||
|
||||
@override
|
||||
String get studysetSideIntroduction => 'Einführung';
|
||||
|
||||
@override
|
||||
String get cardHintsFurigana => 'Furigana';
|
||||
|
||||
@override
|
||||
String get cardHintsPinyin => 'Pinyin';
|
||||
|
||||
@override
|
||||
String get cardHintsReadings => 'Text darüber';
|
||||
|
||||
@override
|
||||
String get cardHintsReadingsUnreviewed =>
|
||||
'Manche Lesungen wurden automatisch erstellt und nicht geprüft.';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8118,6 +8118,19 @@ class AppLocalizationsEn extends AppLocalizations {
|
|||
|
||||
@override
|
||||
String get studysetSideIntroduction => 'Introduction';
|
||||
|
||||
@override
|
||||
String get cardHintsFurigana => 'Furigana';
|
||||
|
||||
@override
|
||||
String get cardHintsPinyin => 'Pinyin';
|
||||
|
||||
@override
|
||||
String get cardHintsReadings => 'Text above';
|
||||
|
||||
@override
|
||||
String get cardHintsReadingsUnreviewed =>
|
||||
'Some readings were made automatically and not reviewed.';
|
||||
}
|
||||
|
||||
/// The translations for English, as used in the United Kingdom (`en_GB`).
|
||||
|
|
|
|||
|
|
@ -50,6 +50,9 @@ class DeviceSettingsBloc(
|
|||
on<DeviceSettingsEventLibrarySubjectChosen>(
|
||||
(event, emit) => _apply(emit, state.withLibrarySubject(event.subject)),
|
||||
);
|
||||
on<DeviceSettingsEventHintReadingsChosen>(
|
||||
(event, emit) => _apply(emit, state.withHintReadings(shown: event.shown)),
|
||||
);
|
||||
on<DeviceSettingsEventSidebarChosen>(
|
||||
(event, emit) => _apply(emit, state.withSidebar(open: event.open)),
|
||||
);
|
||||
|
|
@ -162,6 +165,14 @@ final class const DeviceSettingsEventLibrarySubjectChosen(final String? subject)
|
|||
List<Object?> get props => [subject];
|
||||
}
|
||||
|
||||
/// The person shows hints' readings from the start, or hides them
|
||||
/// ([shown]).
|
||||
final class const DeviceSettingsEventHintReadingsChosen(final bool shown)
|
||||
extends DeviceSettingsEvent {
|
||||
@override
|
||||
List<Object?> get props => [shown];
|
||||
}
|
||||
|
||||
/// The person opens the rail into the sidebar, or closes it ([open]).
|
||||
final class const DeviceSettingsEventSidebarChosen(final bool open)
|
||||
extends DeviceSettingsEvent {
|
||||
|
|
|
|||
|
|
@ -82,6 +82,7 @@ class FileDeviceSettingsStore(final File _file) implements DeviceSettingsStore {
|
|||
final String id => id,
|
||||
_ => null,
|
||||
},
|
||||
hintReadings: json['hintReadings'] != false,
|
||||
);
|
||||
} on FormatException catch (error) {
|
||||
log('The device settings are unreadable', name: 'Settings', error: error);
|
||||
|
|
@ -112,6 +113,7 @@ class FileDeviceSettingsStore(final File _file) implements DeviceSettingsStore {
|
|||
'checkGrammar': settings.checkGrammar,
|
||||
if (settings.sidebarOpen) 'sidebarOpen': true,
|
||||
'librarySubject': ?settings.librarySubject,
|
||||
if (!settings.hintReadings) 'hintReadings': false,
|
||||
}),
|
||||
flush: true,
|
||||
);
|
||||
|
|
|
|||
|
|
@ -12,8 +12,8 @@ export 'package:halkyon_app/halkyon_app.dart' show Appearance;
|
|||
/// whether studying shows the Pomodoro timer, which studysets' card
|
||||
/// browsers show one card at a time, whether and how the person searches
|
||||
/// the web from the app, who answered the field last and at which effort,
|
||||
/// whether writing is checked as it is written, and whether the rail is
|
||||
/// open into the sidebar.
|
||||
/// whether writing is checked as it is written, whether the rail is
|
||||
/// open into the sidebar, and whether hints show their readings.
|
||||
/// Unlike the person's settings, they don't follow anyone to other devices,
|
||||
/// and they apply before anyone is signed in.
|
||||
final class const DeviceSettings({
|
||||
|
|
@ -75,6 +75,11 @@ final class const DeviceSettings({
|
|||
/// as the person writes (ADR 0024, Settings › Writing): on unless turned
|
||||
/// off; nothing is checked where the instance checks nothing.
|
||||
final bool checkGrammar = true,
|
||||
|
||||
/// Whether the hints sheet shows readings over its sentences (furigana,
|
||||
/// pinyin) from the start, as its switch was last left (decided 10 Nov
|
||||
/// 2026): on unless turned off.
|
||||
final bool hintReadings = true,
|
||||
}) with Equatable {
|
||||
/// These settings with [appearance] instead.
|
||||
DeviceSettings withAppearance(Appearance appearance) =>
|
||||
|
|
@ -130,6 +135,10 @@ final class const DeviceSettings({
|
|||
/// These settings with writing checked as it is written [on] or off.
|
||||
DeviceSettings withCheckGrammar(bool on) => _copy(checkGrammar: on);
|
||||
|
||||
/// These settings with hints' readings [shown] from the start or not.
|
||||
DeviceSettings withHintReadings({required bool shown}) =>
|
||||
_copy(hintReadings: shown);
|
||||
|
||||
DeviceSettings _copy({
|
||||
Appearance? appearance,
|
||||
Object? language = _keep,
|
||||
|
|
@ -145,6 +154,7 @@ final class const DeviceSettings({
|
|||
bool? checkGrammar,
|
||||
bool? sidebarOpen,
|
||||
Object? librarySubject = _keep,
|
||||
bool? hintReadings,
|
||||
}) => DeviceSettings(
|
||||
appearance: appearance ?? this.appearance,
|
||||
language: identical(language, _keep) ? this.language : language as String?,
|
||||
|
|
@ -166,6 +176,7 @@ final class const DeviceSettings({
|
|||
librarySubject: identical(librarySubject, _keep)
|
||||
? this.librarySubject
|
||||
: librarySubject as String?,
|
||||
hintReadings: hintReadings ?? this.hintReadings,
|
||||
);
|
||||
|
||||
@override
|
||||
|
|
@ -184,6 +195,7 @@ final class const DeviceSettings({
|
|||
checkGrammar,
|
||||
sidebarOpen,
|
||||
librarySubject,
|
||||
hintReadings,
|
||||
];
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import 'package:appflowy_editor/appflowy_editor.dart';
|
|||
import 'package:flutter/gestures.dart';
|
||||
import 'package:halkyon_editor/halkyon_editor.dart';
|
||||
import 'package:halkyon_learn_flutter/l10n/app_localizations.dart';
|
||||
import 'package:halkyon_learn_flutter/src/core/editor/ruby_marks.dart';
|
||||
import 'package:halkyon_learn_flutter/src/core/widget/content_languages.dart';
|
||||
import 'package:halkyon_ui/halkyon_ui.dart';
|
||||
import 'package:lucide_icons_flutter/lucide_icons.dart';
|
||||
|
|
@ -17,15 +18,51 @@ import 'package:material_ui/material_ui.dart';
|
|||
/// How large a reading is beside the text it reads.
|
||||
const _rubyScale = 0.5;
|
||||
|
||||
/// Shows the readings below it over their text from the start ([shown]),
|
||||
/// rather than under the pointer or after a long press alone: the hints
|
||||
/// sheet's switch (decided 10 Nov 2026).
|
||||
class const ReadingsShown({
|
||||
required final bool shown,
|
||||
required super.child,
|
||||
super.key,
|
||||
}) extends InheritedWidget {
|
||||
/// Whether readings below [context] show from the start; not unless a
|
||||
/// [ReadingsShown] says so.
|
||||
static bool of(BuildContext context) =>
|
||||
context.dependOnInheritedWidgetOfExactType<ReadingsShown>()?.shown ??
|
||||
false;
|
||||
|
||||
@override
|
||||
bool updateShouldNotify(ReadingsShown oldWidget) => shown != oldWidget.shown;
|
||||
}
|
||||
|
||||
final _kana = RegExp(r'[\u3040-\u30ff]');
|
||||
|
||||
final _latin = RegExp('[a-zA-ZÀ-ɏ]');
|
||||
|
||||
/// What the readings in [content] are: furigana where they are kana,
|
||||
/// pinyin where they are Latin letters, else any.
|
||||
RubyKind rubyKindOf(RichContent content) {
|
||||
for (final reading in readingsOf(content)) {
|
||||
if (_kana.hasMatch(reading)) {
|
||||
return RubyKind.furigana;
|
||||
}
|
||||
if (_latin.hasMatch(reading)) {
|
||||
return RubyKind.pinyin;
|
||||
}
|
||||
}
|
||||
return RubyKind.any;
|
||||
}
|
||||
|
||||
/// What text above is called where content is written in a language that
|
||||
/// has a name for it: furigana over Japanese, pinyin over Chinese, the
|
||||
/// front's language first.
|
||||
enum _RubyKind() {
|
||||
/// has a name for it: furigana over Japanese, pinyin over Chinese.
|
||||
enum RubyKind() {
|
||||
furigana,
|
||||
pinyin,
|
||||
any;
|
||||
|
||||
static _RubyKind of(BuildContext context) {
|
||||
/// The kind of the content around [context], the front's language first.
|
||||
static RubyKind of(BuildContext context) {
|
||||
final (:front, :back) = ContentLanguages.of(context);
|
||||
for (final tag in [front, back]) {
|
||||
switch (tag?.split('-').first.toLowerCase()) {
|
||||
|
|
@ -43,10 +80,10 @@ enum _RubyKind() {
|
|||
/// the content around [context].
|
||||
String rubyButtonName(BuildContext context) {
|
||||
final l10n = AppLocalizations.of(context);
|
||||
return switch (_RubyKind.of(context)) {
|
||||
_RubyKind.furigana => l10n.formatRubyFurigana,
|
||||
_RubyKind.pinyin => l10n.formatRubyPinyin,
|
||||
_RubyKind.any => l10n.formatRuby,
|
||||
return switch (RubyKind.of(context)) {
|
||||
RubyKind.furigana => l10n.formatRubyFurigana,
|
||||
RubyKind.pinyin => l10n.formatRubyPinyin,
|
||||
RubyKind.any => l10n.formatRuby,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -82,7 +119,7 @@ Future<void> editRuby(BuildContext context, EditorState editorState) async {
|
|||
rubyKey,
|
||||
selection,
|
||||
);
|
||||
final kind = _RubyKind.of(context);
|
||||
final kind = RubyKind.of(context);
|
||||
final reading = await showDialog<String>(
|
||||
context: context,
|
||||
builder: (dialogContext) =>
|
||||
|
|
@ -156,7 +193,7 @@ TextSpanDecoratorForAttribute withRuby(
|
|||
),
|
||||
reading: reading,
|
||||
style: style,
|
||||
hidden: true,
|
||||
hidden: !ReadingsShown.of(context),
|
||||
);
|
||||
return WidgetSpan(
|
||||
alignment: PlaceholderAlignment.baseline,
|
||||
|
|
@ -401,7 +438,7 @@ class _RubyTextState() extends State<RubyText> {
|
|||
/// completes with it, empty to take it away, or with null when cancelled.
|
||||
class const _ReadingDialog({
|
||||
required final String initial,
|
||||
required final _RubyKind kind,
|
||||
required final RubyKind kind,
|
||||
}) extends StatefulWidget {
|
||||
@override
|
||||
State<_ReadingDialog> createState() => _ReadingDialogState();
|
||||
|
|
@ -420,17 +457,17 @@ class _ReadingDialogState() extends State<_ReadingDialog> {
|
|||
Widget build(BuildContext context) {
|
||||
final l10n = AppLocalizations.of(context);
|
||||
final (title, label, hint) = switch (widget.kind) {
|
||||
_RubyKind.furigana => (
|
||||
RubyKind.furigana => (
|
||||
l10n.rubyTitleFurigana,
|
||||
l10n.rubyReading,
|
||||
l10n.rubyReadingHint,
|
||||
),
|
||||
_RubyKind.pinyin => (
|
||||
RubyKind.pinyin => (
|
||||
l10n.rubyTitlePinyin,
|
||||
l10n.rubyTitlePinyin,
|
||||
l10n.rubyHintPinyin,
|
||||
),
|
||||
_RubyKind.any => (l10n.rubyTitle, l10n.rubyText, l10n.rubyHint),
|
||||
RubyKind.any => (l10n.rubyTitle, l10n.rubyText, l10n.rubyHint),
|
||||
};
|
||||
return AlertDialog(
|
||||
title: Text(title),
|
||||
|
|
|
|||
|
|
@ -0,0 +1,46 @@
|
|||
import 'package:halkyon_editor/halkyon_editor.dart';
|
||||
|
||||
// What content says of its readings (text above, `rubyKey`), in pure Dart
|
||||
// for the models; the editor draws them (ruby.dart).
|
||||
|
||||
/// The key a paragraph's data marks readings with that no one wrote or
|
||||
/// checked: Tatoeba's made by MeCab, or its automatic pinyin (decided
|
||||
/// 9 Nov 2026).
|
||||
const rubyUnreviewedKey = 'rubyUnreviewed';
|
||||
|
||||
/// The blocks of [content], and the blocks within them.
|
||||
Iterable<Map<Object?, Object?>> _blocksOf(RichContent content) sync* {
|
||||
Iterable<Map<Object?, Object?>> walk(Object? node) sync* {
|
||||
if (node is! Map) {
|
||||
return;
|
||||
}
|
||||
yield node;
|
||||
for (final child in node['children'] as List? ?? const []) {
|
||||
yield* walk(child);
|
||||
}
|
||||
}
|
||||
|
||||
yield* walk(content.editorJson['document']);
|
||||
}
|
||||
|
||||
/// The readings over text in [content], in order.
|
||||
Iterable<String> readingsOf(RichContent content) sync* {
|
||||
for (final block in _blocksOf(content)) {
|
||||
if (block['data'] case {'delta': final List<Object?> delta}) {
|
||||
for (final op in delta) {
|
||||
if (op case {'attributes': {rubyKey: final String reading}}
|
||||
when reading.isNotEmpty) {
|
||||
yield reading;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether [content] has text with a reading over it.
|
||||
bool hasReadings(RichContent content) => readingsOf(content).isNotEmpty;
|
||||
|
||||
/// Whether [content] holds readings no one reviewed ([rubyUnreviewedKey]).
|
||||
bool readingsUnreviewed(RichContent content) => _blocksOf(
|
||||
content,
|
||||
).any((block) => (block['data'] as Map?)?[rubyUnreviewedKey] == true);
|
||||
|
|
@ -1,6 +1,10 @@
|
|||
import 'package:flutter_bloc/flutter_bloc.dart';
|
||||
import 'package:halkyon_editor/halkyon_editor.dart';
|
||||
import 'package:halkyon_learn_flutter/l10n/app_localizations.dart';
|
||||
import 'package:halkyon_learn_flutter/src/core/device_settings/bloc/device_settings_bloc.dart';
|
||||
import 'package:halkyon_learn_flutter/src/core/editor/learn_content.dart';
|
||||
import 'package:halkyon_learn_flutter/src/core/editor/ruby.dart';
|
||||
import 'package:halkyon_learn_flutter/src/core/editor/ruby_marks.dart';
|
||||
import 'package:halkyon_ui/halkyon_ui.dart';
|
||||
import 'package:lucide_icons_flutter/lucide_icons.dart';
|
||||
import 'package:material_ui/material_ui.dart';
|
||||
|
|
@ -156,10 +160,58 @@ class _HintsSheetState() extends State<_HintsSheet> {
|
|||
],
|
||||
)
|
||||
: null,
|
||||
body: LearnContentView(
|
||||
key: ValueKey(_index),
|
||||
content: widget.hints[_index],
|
||||
),
|
||||
body: _Hint(key: ValueKey(_index), hint: widget.hints[_index]),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// One [hint]; with readings over it, a switch above showing them from the
|
||||
/// start or not, as the device keeps it (decided 10 Nov 2026), and below
|
||||
/// it, while they show, whether no one reviewed them.
|
||||
class const _Hint({required final RichContent hint, super.key})
|
||||
extends StatelessWidget {
|
||||
@override
|
||||
Widget build(BuildContext context) {
|
||||
if (!hasReadings(hint)) {
|
||||
return LearnContentView(content: hint);
|
||||
}
|
||||
final l10n = AppLocalizations.of(context);
|
||||
final theme = Theme.of(context);
|
||||
final shown = context.select<DeviceSettingsBloc, bool>(
|
||||
(settings) => settings.state.hintReadings,
|
||||
);
|
||||
return Column(
|
||||
crossAxisAlignment: CrossAxisAlignment.stretch,
|
||||
mainAxisSize: MainAxisSize.min,
|
||||
children: [
|
||||
SwitchListTile(
|
||||
contentPadding: EdgeInsets.zero,
|
||||
title: Text(switch (rubyKindOf(hint)) {
|
||||
RubyKind.furigana => l10n.cardHintsFurigana,
|
||||
RubyKind.pinyin => l10n.cardHintsPinyin,
|
||||
RubyKind.any => l10n.cardHintsReadings,
|
||||
}),
|
||||
value: shown,
|
||||
onChanged: (on) => context.read<DeviceSettingsBloc>().add(
|
||||
DeviceSettingsEventHintReadingsChosen(on),
|
||||
),
|
||||
),
|
||||
ReadingsShown(
|
||||
shown: shown,
|
||||
// Drawn anew when switched: the editor keeps the spans it drew.
|
||||
child: LearnContentView(key: ValueKey(shown), content: hint),
|
||||
),
|
||||
if (shown && readingsUnreviewed(hint))
|
||||
Padding(
|
||||
padding: const EdgeInsets.only(top: 8),
|
||||
child: Text(
|
||||
l10n.cardHintsReadingsUnreviewed,
|
||||
style: theme.textTheme.bodySmall?.copyWith(
|
||||
color: theme.colorScheme.onSurfaceVariant,
|
||||
),
|
||||
),
|
||||
),
|
||||
],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue