halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 40 Packages Wiki Activity Actions

feat(curator): Tatoeba's sentences as examples in every new language

Browse source 
JMdict brings Tatoeba's sentences for Japanese; the other languages'
dictionaries had none (CC-CEDICT) or Wiktionary's own, often
quotations. Each file now takes Tatoeba's sentences of the week (its
per-language exports, CC BY 2.0 FR) as its entries' first examples,
which become a card's hints: each sentence with a translation into the
dictionary's meaning language goes to the entries its words find, cut
as the app cuts a line of subtitles (at spaces, Korean's particles off,
Chinese by its longest words), up to three for each, those using the
word as its headword writes it first (Haus rather than zu Hause; 学习
rather than 學習 in the simplified dictionary), then those nearest seven
words long. German's common words nearly all get some (8,849 of
9,292), in ten seconds and under 300 MB; a dictionary whose sentences
can't be had keeps its words. `curate tatoeba` does the same for a file
made by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-14 07:49:34 +02:00
commit 222175584b
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
1 changed files with 811 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

73
halkyon_learn_curator/bin/curate.dart
View file

@ -1,6 +1,7 @@
import 'dart:convert';
import 'dart:io';
import 'package:archive/archive.dart';
import 'package:args/args.dart';
import 'package:halkyon_learn_curator/src/sources/cedict.dart';
import 'package:halkyon_learn_curator/src/sources/jlpt.dart';
@ -8,6 +9,7 @@ import 'package:halkyon_learn_curator/src/sources/jmdict.dart';
import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
import 'package:halkyon_learn_curator/src/update.dart';
import 'package:halkyon_learn_curator/src/wordfreq.dart';
@ -40,6 +42,15 @@ import 'package:halkyon_learn_curator/src/wordfreq.dart';
/// --input cedict_1_0_ts_utf-8_mdbg.txt.gz \
/// --output ../halkyon_learn_server/dictionaries
///
/// Tatoeba's sentences as examples into a dictionary file of words (its
/// exports of the dictionary's language, of its meaning language, and of
/// the links between them, bzip2 or not):
///
/// dart run halkyon_learn_curator:curate tatoeba \
/// --input wiktionary-de-en.jsonl.gz --sentences deu_sentences.tsv.bz2 \
/// --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
/// --output wiktionary-de-en.jsonl.gz
///
/// Or brings a server's dictionaries up to date from EDRDG's files of the
/// day and, for the languages named, Wiktionary's latest dump and MDBG's
/// CC-CEDICT of the day (Chinese), once or every so many days:
@ -96,6 +107,18 @@ Future<void> main(List<String> args) async {
'to keep current (de,el,en,zh; Chinese from CC-CEDICT, the rest '
"from Wiktionary's dump), none without.",
)
..addOption(
'sentences',
help: "tatoeba: Tatoeba's sentences in the dictionary's language.",
)
..addOption(
'translations',
help: "tatoeba: Tatoeba's sentences in the meaning language.",
)
..addOption(
'links',
help: "tatoeba: Tatoeba's links between the two languages.",
)
..addOption(
'version',
help:
@ -127,6 +150,56 @@ Future<void> main(List<String> args) async {
);
return;
}
if (source == 'tatoeba') {
final input = options.option('input');
final output = options.option('output');
final files = [
for (final name in ['sentences', 'translations', 'links'])
options.option(name),
];
if (input == null || output == null || files.contains(null)) {
_fail(
'Usage: curate tatoeba --input <dictionary> --sentences <file> '
'--translations <file> --links <file> --output <dictionary>'
'\n${parser.usage}',
);
}
Stream<String> lines(String path) {
final bytes = path.endsWith('.bz2')
? Stream.value(
BZip2Decoder().decodeBytes(File(path).readAsBytesSync()),
)
: File(path).openRead();
return bytes.transform(utf8.decoder).transform(const LineSplitter());
}
final links = await readTatoebaLinks(lines(files[2]!));
final sentences = await readTatoebaSentences(
lines(files[0]!),
wanted: links.containsKey,
);
final linked = {for (final ids in links.values) ...ids};
final translations = await readTatoebaSentences(
lines(files[1]!),
wanted: linked.contains,
);
final written = File('$output.part');
final bytes = withTatoebaExamples(
File(input),
sentences: sentences,
translations: translations,
links: links,
).transform(utf8.encoder);
await (output.endsWith('.gz') ? bytes.transform(gzip.encoder) : bytes).pipe(
written.openWrite(),
);
await written.rename(output);
stdout.writeln(
'Wrote $output with ${sentences.length} sentences of Tatoeba to '
'choose from.',
);
return;
}
if (source == 'cedict') {
final input = options.option('input');
final output = options.option('output');