halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 37 Packages Wiki Activity Actions

feat(learn): furigana and pinyin over hints, from Tatoeba's transcriptions

Browse source 
The curator puts Tatoeba's readings on a dictionary's examples by their
Tatoeba ID (`withTatoebaReadings`, `curate readings`, and in `update`):
furigana over JMdict's Japanese from its jpn transcriptions, pinyin with
tone marks over Chinese from its cmn ones, each word over as many Han
characters as it has syllables. An example gets `ruby`, `[start, end,
reading]` each, and `rubyUnreviewed` where no one wrote or checked them
(MeCab's furigana, the automatic pinyin). JMdict's conversion goes to 3,
so instances convert again at once.

A word's example hints carry them as ruby marks, the word still in bold.
The hints sheet shows them from the start with a switch above
(Furigana, Pinyin), kept on the device (`DeviceSettings.hintReadings`),
and below a hint whose readings no one reviewed says so.

Whats-New: Example sentences in a word's hints now show furigana or pinyin over them, with a switch to hide them.
Whats-New-De: Beispielsätze in den Hinweisen eines Worts zeigen jetzt Furigana oder Pinyin darüber, mit einem Schalter zum Ausblenden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-11-10 20:28:30 +01:00
commit 3257905536
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
1 changed files with 1083 additions and 78 deletions
Download patch file Download diff file Expand all files Collapse all files

53
apps/learn/halkyon_learn_curator/bin/curate.dart
View file

@ -14,6 +14,7 @@ import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
import 'package:halkyon_learn_curator/src/sources/krdict.dart';
import 'package:halkyon_learn_curator/src/sources/makemeahanzi.dart';
import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
import 'package:halkyon_learn_curator/src/sources/tatoeba_readings.dart';
import 'package:halkyon_learn_curator/src/sources/unihan.dart';
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
import 'package:halkyon_learn_curator/src/update.dart';
@ -92,6 +93,16 @@ import 'package:halkyon_learn_curator/src/wordfreq.dart';
/// --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
/// --output wiktionary-de-en.jsonl.gz
///
/// Readings over a dictionary file's examples, from Tatoeba's
/// transcriptions of their sentences (furigana over Japanese, pinyin over
/// Chinese; its `jpn_transcriptions.tsv` or `cmn_transcriptions.tsv`, bzip2
/// or not):
///
/// dart run halkyon_learn_curator:curate readings \
/// --input jmdict-en.jsonl.gz \
/// --transcriptions jpn_transcriptions.tsv.bz2 \
/// --output jmdict-en.jsonl.gz
///
/// Or brings a server's dictionaries up to date from EDRDG's files of the
/// day and, for the languages named, Wiktionary's latest dump and MDBG's
/// CC-CEDICT of the day (Chinese), once or every so many days:
@ -160,6 +171,10 @@ Future<void> main(List<String> args) async {
'links',
help: "tatoeba: Tatoeba's links between the two languages.",
)
..addOption(
'transcriptions',
help: "readings: Tatoeba's transcriptions of the dictionary's language.",
)
..addOption(
'cedict',
help:
@ -281,6 +296,44 @@ Future<void> main(List<String> args) async {
);
return;
}
if (source == 'readings') {
final input = options.option('input');
final output = options.option('output');
final transcriptions = options.option('transcriptions');
if (input == null || output == null || transcriptions == null) {
_fail(
'Usage: curate readings --input <dictionary> --transcriptions <file> '
'--output <dictionary>\n${parser.usage}',
);
}
final read = await readTatoebaTranscriptions(
(transcriptions.endsWith('.bz2')
? Stream.value(
BZip2Decoder().decodeBytes(
File(transcriptions).readAsBytesSync(),
),
)
: File(transcriptions).openRead())
.transform(utf8.decoder)
.transform(const LineSplitter()),
);
final bytes = File(input).openRead();
final written = File('$output.part');
final lines = withTatoebaReadings(
(input.endsWith('.gz') ? bytes.transform(gzip.decoder) : bytes)
.transform(utf8.decoder)
.transform(const LineSplitter()),
read,
).transform(utf8.encoder);
await (output.endsWith('.gz') ? lines.transform(gzip.encoder) : lines).pipe(
written.openWrite(),
);
await written.rename(output);
stdout.writeln(
'Wrote $output with ${read.length} transcriptions to read from.',
);
return;
}
if (source == 'cedict') {
final input = options.option('input');
final output = options.option('output');