feat(learn): furigana and pinyin over hints, from Tatoeba's transcriptions

The curator puts Tatoeba's readings on a dictionary's examples by their
Tatoeba ID (`withTatoebaReadings`, `curate readings`, and in `update`):
furigana over JMdict's Japanese from its jpn transcriptions, pinyin with
tone marks over Chinese from its cmn ones, each word over as many Han
characters as it has syllables. An example gets `ruby`, `[start, end,
reading]` each, and `rubyUnreviewed` where no one wrote or checked them
(MeCab's furigana, the automatic pinyin). JMdict's conversion goes to 3,
so instances convert again at once.

A word's example hints carry them as ruby marks, the word still in bold.
The hints sheet shows them from the start with a switch above
(Furigana, Pinyin), kept on the device (`DeviceSettings.hintReadings`),
and below a hint whose readings no one reviewed says so.

Whats-New: Example sentences in a word's hints now show furigana or pinyin over them, with a switch to hide them.
Whats-New-De: Beispielsätze in den Hinweisen eines Worts zeigen jetzt Furigana oder Pinyin darüber, mit einem Schalter zum Ausblenden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-11-10 20:28:30 +01:00
commit 3257905536
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64

View file

@ -14,14 +14,6 @@ void main() {
setUp(() => into = Directory.systemTemp.createTempSync('dictionaries'));
tearDown(() => into.deleteSync(recursive: true));
/// Fetches the samples in place of EDRDG's files of the day.
Future<void> samples(Uri uri, File file) async {
final sample = uri == jmdictSource
? 'test/fixtures/jmdict-e-examp-sample.xml'
: 'test/fixtures/kanjidic2-sample.xml';
await file.writeAsBytes(gzip.encode(await File(sample).readAsBytes()));
}
/// Tatoeba's exports, as the fixtures hold a few of their lines, packed
/// as Tatoeba packs them; null for any other address.
List<int>? tatoeba(Uri uri) {
@ -32,6 +24,33 @@ void main() {
: null;
}
/// Fetches the samples in place of EDRDG's files of the day, and
/// Tatoeba's transcriptions.
Future<void> samples(Uri uri, File file) async {
if (tatoeba(uri) case final packed?) {
await file.writeAsBytes(packed);
return;
}
final sample = uri == jmdictSource
? 'test/fixtures/jmdict-e-examp-sample.xml'
: 'test/fixtures/kanjidic2-sample.xml';
await file.writeAsBytes(gzip.encode(await File(sample).readAsBytes()));
}
/// The examples of every sense in the dictionary file [name] in [into].
List<Map<String, Object?>> examplesIn(String name) => [
for (final line
in utf8
.decode(gzip.decode(File('${into.path}/$name').readAsBytesSync()))
.split('\n')
.skip(1))
if (line.isNotEmpty)
for (final sense
in ((jsonDecode(line) as Map)['entry'] as Map)['senses'] as List)
for (final example in (sense as Map)['examples'] as List? ?? const [])
example as Map<String, Object?>,
];
/// The header of the dictionary file [name] in [into].
Map<String, Object?> header(String name) => jsonDecode(
utf8
@ -61,6 +80,19 @@ void main() {
['JMdict', 'JLPT vocabulary lists', 'Tatoeba'],
);
expect(header('kanjidic-en.jsonl.gz')['kind'], 'characters');
// Furigana over the sentences Tatoeba transcribed, those MeCab's alone
// marked.
final examples = {
for (final example in examplesIn('jmdict-en.jsonl.gz'))
'${example['tatoeba']}': example,
};
expect(examples['193344']?['ruby'], [
[3, 4, 'くだ'],
[4, 5, 'もの'],
[6, 7, 'た'],
]);
expect(examples['193344'], isNot(contains('rubyUnreviewed')));
expect(examples['81964']?['rubyUnreviewed'], isTrue);
// Nothing half-written left behind; which conversion made them.
expect(
[for (final entity in into.listSync()) entity.uri.pathSegments.last]
@ -242,6 +274,17 @@ void main() {
(credit as Map)['name'],
], contains('Tatoeba'));
expect(cedictUpdated(into), isNotNull);
// Pinyin over the sentences Tatoeba transcribed.
expect(
[
for (final example in examplesIn('cc-cedict-hans-en.jsonl.gz'))
if (example['tatoeba'] == 750344) example['ruby'],
].first,
[
[0, 1, 'wǒ'],
[1, 3, 'xuéxí'],
],
);
// Nothing of the downloads left beside the files.
expect(
[for (final entity in into.listSync()) entity.uri.pathSegments.last]