feat(curator): Tatoeba's sentences as examples in every new language

JMdict brings Tatoeba's sentences for Japanese; the other languages'
dictionaries had none (CC-CEDICT) or Wiktionary's own, often
quotations. Each file now takes Tatoeba's sentences of the week (its
per-language exports, CC BY 2.0 FR) as its entries' first examples,
which become a card's hints: each sentence with a translation into the
dictionary's meaning language goes to the entries its words find, cut
as the app cuts a line of subtitles (at spaces, Korean's particles off,
Chinese by its longest words), up to three for each, those using the
word as its headword writes it first (Haus rather than zu Hause; 学习
rather than 學習 in the simplified dictionary), then those nearest seven
words long. German's common words nearly all get some (8,849 of
9,292), in ten seconds and under 300 MB; a dictionary whose sentences
can't be had keeps its words. `curate tatoeba` does the same for a file
made by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

docs/adr/0008-languages-beyond-japanese.md

@@ -69,7 +69,14 @@ made from it be published under CC BY-SA, as studysets are.
 10. **A noun goes onto its card with its article** where the language has gendered
     articles and the noun one gender (`das Haus`, `το σπίτι`): learners learn the gender
     with the word, and the article says it.
-11. **Words are looked up on the web** on Jisho in Japanese, built on JMdict, and on
+11. **Tatoeba's sentences are the examples** (CC BY 2.0 FR, weekly per-language
+    exports), as JMdict's examples edition brings them for Japanese: each sentence with
+    a translation into the meaning language goes to the entries its words find, cut as
+    the app cuts subtitles, up to three for each, those using the word as its headword
+    writes it first (Haus, not zu Hause; simplified characters in the simplified
+    dictionary), then those nearest seven words long. Wiktionary's own examples, often
+    quotations, follow them. A dictionary whose sentences can't be had keeps its words.
+12. **Words are looked up on the web** on Jisho in Japanese, built on JMdict, and on
     Wiktionary at the language's section otherwise, which the dictionaries are drawn from.
 
 ## Consequences

halkyon_learn_curator/README.md

@@ -102,6 +102,24 @@ forms):
    apart, each given to the entry read as the syllabus reads the word (的 *de*,
    not 的 *dì*), and rank them, HSK 1 first; wordfreq ranks the rest.
 
+Example sentences from Tatoeba for those dictionaries (JMdict brings its own),
+as hints on the cards made from them:
+
+```sh
+dart run halkyon_learn_curator:curate tatoeba \
+  --input wiktionary-de-en.jsonl.gz --sentences deu_sentences.tsv.bz2 \
+  --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
+  --output wiktionary-de-en.jsonl.gz
+```
+
+from [Tatoeba's weekly exports](https://downloads.tatoeba.org/exports/per_language/)
+of the dictionary's language, of its meaning language and of the links between
+them (CC BY 2.0 FR). Each sentence with a translation goes to the entries its
+words find by their text keys, as the app finds words in a line of subtitles,
+up to three for each: those using the word as its headword writes it first, then
+those nearest seven words long, the shortest then; they come first in an entry's
+first sense, the word as it stands there beside. Some ten seconds for German.
+
 Put the files into the server's `dictionaries` folder. The server loads each on
 its next start, and again whenever it changes (see the server's README).
 
@@ -130,8 +148,10 @@ so in a container of its own (the server's README). Beside the files,
 from the same download (a label, a field), and an instance converts again as
 soon as the new curator runs, not a week later.
 
-With `--languages de,el,en,ko,zh` it also keeps those languages current: Chinese
+With `--languages de,el,en,ko,zh` it also keeps those languages current, each
-from MDBG's CC-CEDICT of the day as often as EDRDG's files (below), the rest from
+file with Tatoeba's sentences of the week as examples (a dictionary whose
+sentences can't be had keeps its words without them): Chinese from MDBG's
+CC-CEDICT of the day as often as EDRDG's files (below), the rest from
 Wiktionary: each time it wakes it asks kaikki.org when its dump last changed
 (`Last-Modified`, without downloading it), and converts it again only when it is
 newer than the files' (about monthly), when the languages named changed, or when

halkyon_learn_curator/bin/curate.dart

@@ -1,6 +1,7 @@
 import 'dart:convert';
 import 'dart:io';
 
+import 'package:archive/archive.dart';
 import 'package:args/args.dart';
 import 'package:halkyon_learn_curator/src/sources/cedict.dart';
 import 'package:halkyon_learn_curator/src/sources/jlpt.dart';
@@ -8,6 +9,7 @@ import 'package:halkyon_learn_curator/src/sources/jmdict.dart';
 import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
 import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
 import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
+import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
 import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
 import 'package:halkyon_learn_curator/src/update.dart';
 import 'package:halkyon_learn_curator/src/wordfreq.dart';
@@ -40,6 +42,15 @@ import 'package:halkyon_learn_curator/src/wordfreq.dart';
 ///       --input cedict_1_0_ts_utf-8_mdbg.txt.gz \
 ///       --output ../halkyon_learn_server/dictionaries
 ///
+/// Tatoeba's sentences as examples into a dictionary file of words (its
+/// exports of the dictionary's language, of its meaning language, and of
+/// the links between them, bzip2 or not):
+///
+///     dart run halkyon_learn_curator:curate tatoeba \
+///       --input wiktionary-de-en.jsonl.gz --sentences deu_sentences.tsv.bz2 \
+///       --translations eng_sentences.tsv.bz2 --links deu-eng_links.tsv.bz2 \
+///       --output wiktionary-de-en.jsonl.gz
+///
 /// Or brings a server's dictionaries up to date from EDRDG's files of the
 /// day and, for the languages named, Wiktionary's latest dump and MDBG's
 /// CC-CEDICT of the day (Chinese), once or every so many days:
@@ -96,6 +107,18 @@ Future<void> main(List<String> args) async {
           'to keep current (de,el,en,zh; Chinese from CC-CEDICT, the rest '
           "from Wiktionary's dump), none without.",
     )
+    ..addOption(
+      'sentences',
+      help: "tatoeba: Tatoeba's sentences in the dictionary's language.",
+    )
+    ..addOption(
+      'translations',
+      help: "tatoeba: Tatoeba's sentences in the meaning language.",
+    )
+    ..addOption(
+      'links',
+      help: "tatoeba: Tatoeba's links between the two languages.",
+    )
     ..addOption(
       'version',
       help:
@@ -127,6 +150,56 @@ Future<void> main(List<String> args) async {
     );
     return;
   }
+  if (source == 'tatoeba') {
+    final input = options.option('input');
+    final output = options.option('output');
+    final files = [
+      for (final name in ['sentences', 'translations', 'links'])
+        options.option(name),
+    ];
+    if (input == null || output == null || files.contains(null)) {
+      _fail(
+        'Usage: curate tatoeba --input <dictionary> --sentences <file> '
+        '--translations <file> --links <file> --output <dictionary>'
+        '\n${parser.usage}',
+      );
+    }
+    Stream<String> lines(String path) {
+      final bytes = path.endsWith('.bz2')
+          ? Stream.value(
+              BZip2Decoder().decodeBytes(File(path).readAsBytesSync()),
+            )
+          : File(path).openRead();
+      return bytes.transform(utf8.decoder).transform(const LineSplitter());
+    }
+
+    final links = await readTatoebaLinks(lines(files[2]!));
+    final sentences = await readTatoebaSentences(
+      lines(files[0]!),
+      wanted: links.containsKey,
+    );
+    final linked = {for (final ids in links.values) ...ids};
+    final translations = await readTatoebaSentences(
+      lines(files[1]!),
+      wanted: linked.contains,
+    );
+    final written = File('$output.part');
+    final bytes = withTatoebaExamples(
+      File(input),
+      sentences: sentences,
+      translations: translations,
+      links: links,
+    ).transform(utf8.encoder);
+    await (output.endsWith('.gz') ? bytes.transform(gzip.encoder) : bytes).pipe(
+      written.openWrite(),
+    );
+    await written.rename(output);
+    stdout.writeln(
+      'Wrote $output with ${sentences.length} sentences of Tatoeba to '
+      'choose from.',
+    );
+    return;
+  }
   if (source == 'cedict') {
     final input = options.option('input');
     final output = options.option('output');

halkyon_learn_curator/lib/src/sources/tatoeba.dart

@@ -0,0 +1,312 @@
+import 'dart:convert';
+import 'dart:io';
+
+import 'package:halkyon_learn_curator/src/dictionary_file.dart';
+
+/// Who wrote Tatoeba's sentences.
+const tatoebaSentencesCredit = Credit(
+  name: 'Tatoeba',
+  by: 'Tatoeba contributors',
+  license: 'CC-BY-2.0-FR',
+  url: 'https://tatoeba.org/en/terms_of_use',
+);
+
+/// Tatoeba's code (ISO 639-3) of a language by its BCP 47 tag, for the
+/// languages with dictionaries beside Japanese.
+String? tatoebaCode(String language) =>
+    switch (language.split('-').first.toLowerCase()) {
+      'de' => 'deu',
+      'el' => 'ell',
+      'en' => 'eng',
+      'ko' => 'kor',
+      'zh' => 'cmn',
+      _ => null,
+    };
+
+/// How many of Tatoeba's sentences an entry takes at most.
+const tatoebaPerEntry = 3;
+
+/// Each sentence's translations by its id (Tatoeba's
+/// `<code>-<code>_links.tsv`: a sentence's id and its translation's).
+Future<Map<int, List<int>>> readTatoebaLinks(Stream<String> lines) async {
+  final links = <int, List<int>>{};
+  await for (final line in lines) {
+    final fields = line.split('\t');
+    if (fields.length < 2) {
+      continue;
+    }
+    if ((int.tryParse(fields[0]), int.tryParse(fields[1])) case (
+      final from?,
+      final to?,
+    )) {
+      (links[from] ??= []).add(to);
+    }
+  }
+  return links;
+}
+
+/// The sentences of one language by their id (Tatoeba's
+/// `<code>_sentences.tsv`: id, language, text), only those [wanted] says.
+Future<Map<int, String>> readTatoebaSentences(
+  Stream<String> lines, {
+  bool Function(int id)? wanted,
+}) async {
+  final sentences = <int, String>{};
+  await for (final line in lines) {
+    final first = line.indexOf('\t');
+    final second = first < 0 ? -1 : line.indexOf('\t', first + 1);
+    if (second < 0) {
+      continue;
+    }
+    final id = int.tryParse(line.substring(0, first));
+    if (id != null && (wanted == null || wanted(id))) {
+      sentences[id] = line.substring(second + 1);
+    }
+  }
+  return sentences;
+}
+
+/// The lines of [dictionary] (a dictionary file, gzipped when its name ends
+/// in `.gz`), as a stream.
+Stream<String> _linesOf(File dictionary) {
+  final bytes = dictionary.openRead();
+  return (dictionary.path.endsWith('.gz')
+          ? bytes.transform(gzip.decoder)
+          : bytes)
+      .transform(utf8.decoder)
+      .transform(const LineSplitter());
+}
+
+/// [dictionary]'s lines with sentences of Tatoeba as examples of its
+/// entries: each of [sentences] (in the dictionary's language) with a
+/// translation among [translations] (in its meaning language, by [links])
+/// goes to the entries its words find by their text keys, up to
+/// [tatoebaPerEntry] for each: those using the word as its headword writes
+/// it first (Haus rather than zu Hause; 学习 rather than 學習 in simplified
+/// characters), then those nearest seven words long, the shortest then; they come first in the entry's first sense, the word as
+/// it stands there beside (the app sets it in bold), as JMdict's examples
+/// are. The words of a sentence are found as the app finds them in a line
+/// of subtitles: at the spaces, Korean's particles taken off where needed,
+/// and Chinese by the longest keys. The header credits Tatoeba. The
+/// dictionary is read twice, so it is never held whole.
+Stream<String> withTatoebaExamples(
+  File dictionary, {
+  required Map<int, String> sentences,
+  required Map<int, String> translations,
+  required Map<int, List<int>> links,
+}) async* {
+  Map<String, Object?>? header;
+  // The entries each text key finds, by their place in the file, and their
+  // headwords in lower case.
+  final byKey = <String, List<int>>{};
+  final headwords = <String>[];
+  await for (final line in _linesOf(dictionary)) {
+    if (header == null) {
+      header = jsonDecode(line) as Map<String, Object?>;
+      continue;
+    }
+    final entry = jsonDecode(line) as Map<String, Object?>;
+    headwords.add((entry['headword']! as String).toLowerCase());
+    for (final key
+        in (entry['textKeys'] as List?)?.cast<String>() ?? const <String>[]) {
+      (byKey[key.toLowerCase()] ??= []).add(headwords.length - 1);
+    }
+  }
+  if (header == null) {
+    return;
+  }
+  final language = header['language']! as String;
+  final glossLanguage = header['glossLanguage']! as String;
+  final wordsOf = switch (language.split('-').first) {
+    'zh' => (String sentence) => _longestWords(sentence, byKey),
+    'ko' => (String sentence) => _koreanWords(sentence, byKey),
+    _ => _spacedWords,
+  };
+  final best = <int, List<({int id, int score, String form})>>{};
+  for (final MapEntry(key: id, value: sentence) in sentences.entries) {
+    if (!(links[id] ?? const []).any(translations.containsKey)) {
+      continue;
+    }
+    final words = wordsOf(sentence);
+    final length = (words.length - 7).abs() * 1000 + sentence.length;
+    final seen = <int>{};
+    for (final (:key, :text) in words) {
+      for (final entry in byKey[key] ?? const <int>[]) {
+        if (!seen.add(entry)) {
+          continue;
+        }
+        final score =
+            length + (text.toLowerCase() == headwords[entry] ? 0 : 100000);
+        final kept = (best[entry] ??= [])
+          ..add((id: id, score: score, form: text))
+          ..sort(
+            (a, b) => a.score != b.score
+                ? a.score.compareTo(b.score)
+                : a.id.compareTo(b.id),
+          );
+        if (kept.length > tatoebaPerEntry) {
+          kept.removeLast();
+        }
+      }
+    }
+  }
+  yield '${jsonEncode({
+    ...header,
+    'credits': [...(header['credits']! as List), if (best.isNotEmpty) tatoebaSentencesCredit.toJson()],
+  })}\n';
+  // The header (-1), then each entry by its place.
+  var place = -1;
+  await for (final line in _linesOf(dictionary)) {
+    if (place++ < 0) {
+      continue;
+    }
+    final kept = best[place - 1];
+    final entry = kept == null
+        ? null
+        : jsonDecode(line) as Map<String, Object?>;
+    final body = entry?['entry'] as Map<String, Object?>?;
+    final senses = [
+      for (final sense in (body?['senses'] as List? ?? const []))
+        Map<String, Object?>.of(sense as Map<String, Object?>),
+    ];
+    if (entry == null || body == null || kept == null || senses.isEmpty) {
+      yield '$line\n';
+      continue;
+    }
+    senses.first['examples'] = [
+      for (final (:id, score: _, :form) in kept)
+        {
+          'text': form,
+          language: sentences[id],
+          glossLanguage: [for (final other in links[id]!) ?translations[other]]
+              .first,
+          'tatoeba': id,
+        },
+      ...(senses.first['examples'] as List? ?? const []),
+    ];
+    yield '${jsonEncode({
+      ...entry,
+      'entry': {...body, 'senses': senses},
+    })}\n';
+  }
+}
+
+/// A word of a sentence: the key it is found by, and the text it is there.
+typedef _Word = ({String key, String text});
+
+final _letters = RegExp(
+  r"[\p{L}\p{M}]+(?:['’\-][\p{L}\p{M}]+)*",
+  unicode: true,
+);
+
+/// A language written with spaces: each word, lower case.
+List<_Word> _spacedWords(String sentence) => [
+  for (final match in _letters.allMatches(sentence))
+    (key: match[0]!.toLowerCase().replaceAll('’', "'"), text: match[0]!),
+];
+
+/// Korean: each word between spaces as it stands if a key finds it, else
+/// without the longest particle or ending that leaves one.
+List<_Word> _koreanWords(String sentence, Map<String, List<int>> byKey) => [
+  for (final match in _letters.allMatches(sentence))
+    ?[
+          match[0]!,
+          for (final particle in koreanParticles)
+            if (match[0]!.length > particle.length &&
+                match[0]!.endsWith(particle))
+              match[0]!.substring(0, match[0]!.length - particle.length),
+        ]
+        .where(byKey.containsKey)
+        .map((word) => (key: word, text: word))
+        .firstOrNull,
+];
+
+/// Chinese: from the start, each time the longest run of up to eight
+/// characters a key finds, then on after it.
+List<_Word> _longestWords(String sentence, Map<String, List<int>> byKey) {
+  final characters = sentence.runes.toList();
+  final words = <_Word>[];
+  var start = 0;
+  while (start < characters.length) {
+    var length = 8;
+    for (; length > 0; length--) {
+      if (start + length <= characters.length &&
+          byKey.containsKey(
+            String.fromCharCodes(characters.sublist(start, start + length)),
+          )) {
+        break;
+      }
+    }
+    if (length == 0) {
+      start++;
+      continue;
+    }
+    final text = String.fromCharCodes(
+      characters.sublist(start, start + length),
+    );
+    words.add((key: text, text: text));
+    start += length;
+  }
+  return words;
+}
+
+/// The particles and the copula's endings Korean attaches to a word, the
+/// longest first, as the app takes them off (`korean_words.dart`).
+const koreanParticles = [
+  '에서부터',
+  '으로부터',
+  '이라고는',
+  '에게서',
+  '한테서',
+  '으로서',
+  '으로써',
+  '으로는',
+  '에서는',
+  '에서도',
+  '에게는',
+  '까지는',
+  '부터는',
+  '이라고',
+  '이에요',
+  '입니다',
+  '이었다',
+  '이었어',
+  '이지만',
+  '처럼',
+  '같이',
+  '보다',
+  '부터',
+  '까지',
+  '에서',
+  '에게',
+  '한테',
+  '께서',
+  '하고',
+  '이랑',
+  '이나',
+  '이다',
+  '이야',
+  '예요',
+  '였다',
+  '라고',
+  '으로',
+  '에는',
+  '에도',
+  '은',
+  '는',
+  '이',
+  '가',
+  '을',
+  '를',
+  '의',
+  '에',
+  '도',
+  '만',
+  '로',
+  '와',
+  '과',
+  '랑',
+  '나',
+  '야',
+  '요',
+];

halkyon_learn_curator/lib/src/update.dart

@@ -1,10 +1,13 @@
 import 'dart:convert';
 import 'dart:io';
 
+import 'package:archive/archive_io.dart';
+
 import 'package:halkyon_learn_curator/src/sources/cedict.dart';
 import 'package:halkyon_learn_curator/src/sources/jlpt.dart';
 import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
 import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
+import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
 import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
 import 'package:halkyon_learn_curator/src/wordfreq.dart';
 
@@ -110,11 +113,11 @@ Future<File> _write(File file, Stream<String> lines) async {
   return file;
 }
 
-/// [file] moved into [folder] under its name: copied beside its place
+/// [file] moved into [folder] under its name (or [name]): copied beside
-/// (`.part`, which the server doesn't read), then renamed over it, which
+/// its place (`.part`, which the server doesn't read), then renamed over
-/// replaces it at once.
+/// it, which replaces it at once.
-Future<File> _putInPlace(File file, Directory folder) async {
+Future<File> _putInPlace(File file, Directory folder, {String? name}) async {
-  final name = file.uri.pathSegments.last;
+  name ??= file.uri.pathSegments.last;
   final part = await file.copy('${folder.path}/.$name.part');
   return await part.rename('${folder.path}/$name');
 }
@@ -219,14 +222,16 @@ DateTime _modifiedOf(Uri uri, HttpClientResponse response) {
 /// the curator converts it ([wiktionaryStateFile]). The dump is converted
 /// as it downloads, never kept (some 3 GB); each file is written beside
 /// its place and renamed over it. The word frequencies are in the
-/// curator's [data] folder (`wordfreq/`). Returns the files written, none
+/// curator's [data] folder (`wordfreq/`). Each file takes Tatoeba's
-/// when they were up to date.
+/// sentences as examples before it is put in place. Returns the files
+/// written, none when they were up to date.
 Future<List<File>> updateWiktionary({
   required Directory into,
   required Directory data,
   required Set<String> languages,
   Modified modified = httpModified,
   Open open = httpOpen,
+  Fetch fetch = httpFetch,
 }) async {
   if (languages.isEmpty) {
     return const [];
@@ -245,19 +250,48 @@ Future<List<File>> updateWiktionary({
           case final file when file.existsSync())
         language: await WordFrequencies.read(file),
   };
-  final written = await wiktionaryDictionaries(
+  final work = await Directory.systemTemp.createTemp('wiktionary-update');
+  try {
+    final converted = await wiktionaryDictionaries(
       download.bytes
           .transform(gzip.decoder)
           .transform(utf8.decoder)
           .transform(const LineSplitter()),
-    into: into,
+      into: work,
       sourceVersion: download.modified.toIso8601String().substring(0, 10),
       languages: outputs.languages,
       translatedInto: outputs.translatedInto,
       frequencies: frequencies,
     );
+    await into.create(recursive: true);
+    final written = [
+      for (final file in converted)
+        await _putInPlace(
+          await _withExamples(file, work: work, fetch: fetch),
+          into,
+          name: file.uri.pathSegments.last,
+        ),
+    ];
     await state.writeAsString(_wiktionaryState(download.modified, languages));
     return written;
+  } finally {
+    await work.delete(recursive: true);
+  }
+}
+
+/// [file] with Tatoeba's sentences as examples, or as it is should
+/// Tatoeba fail: examples are worth having, never worth an update.
+Future<File> _withExamples(
+  File file, {
+  required Directory work,
+  required Fetch fetch,
+}) async {
+  try {
+    return await addTatoebaExamples(file, downloads: work, fetch: fetch);
+  } on Object catch (error) {
+    stdout.writeln('Tatoeba left out of ${file.uri.pathSegments.last}: $error');
+    return file;
+  }
 }
 
 /// What [wiktionaryStateFile] says of files made from the dump of
@@ -276,8 +310,9 @@ final cedictSource = Uri.parse(
 
 /// Brings CC-CEDICT's two files in [into] (simplified and traditional
 /// characters) up to date from MDBG's file of the day, with the HSK levels
-/// and word frequencies in the curator's [data] folder; each written
+/// and word frequencies in the curator's [data] folder, and Tatoeba's
-/// beside its place and renamed over it. Returns the files written.
+/// sentences as examples; each written beside its place and renamed over
+/// it. Returns the files written.
 Future<List<File>> updateCedict({
   required Directory into,
   required Directory data,
@@ -308,7 +343,14 @@ Future<List<File>> updateCedict({
         ),
     ];
     await into.create(recursive: true);
-    return [for (final file in written) await _putInPlace(file, into)];
+    return [
+      for (final file in written)
+        await _putInPlace(
+          await _withExamples(file, work: downloads, fetch: fetch),
+          into,
+          name: file.uri.pathSegments.last,
+        ),
+    ];
   } finally {
     await downloads.delete(recursive: true);
   }
@@ -348,3 +390,83 @@ DateTime? cedictUpdated(Directory folder) {
     chinese: languages.any(isChinese),
   );
 }
+
+/// Tatoeba's weekly export of a language's sentences ([code], ISO 639-3).
+Uri tatoebaSentencesOf(String code) => Uri.parse(
+  'https://downloads.tatoeba.org/exports/per_language/$code/'
+  '${code}_sentences.tsv.bz2',
+);
+
+/// Tatoeba's weekly export of the translations between two languages.
+Uri tatoebaLinksOf(String from, String to) => Uri.parse(
+  'https://downloads.tatoeba.org/exports/per_language/$from/'
+  '$from-${to}_links.tsv.bz2',
+);
+
+/// [dictionary], a dictionary file just written (gzipped), with Tatoeba's
+/// sentences in its language that have a translation into its meaning
+/// language as its entries' first examples (`withTatoebaExamples`),
+/// downloaded into [downloads]; returns the file it wrote beside it. A
+/// language Tatoeba has no code for here is left as it is.
+Future<File> addTatoebaExamples(
+  File dictionary, {
+  required Directory downloads,
+  Fetch fetch = httpFetch,
+}) async {
+  final header = jsonDecode(
+    await dictionary
+        .openRead()
+        .transform(gzip.decoder)
+        .transform(utf8.decoder)
+        .transform(const LineSplitter())
+        .first,
+  ) as Map<String, Object?>;
+  final from = tatoebaCode(header['language']! as String);
+  final to = tatoebaCode(header['glossLanguage']! as String);
+  if (from == null || to == null) {
+    return dictionary;
+  }
+  Future<Stream<String>> download(Uri uri) async {
+    final name = uri.pathSegments.last;
+    final packed = File('${downloads.path}/$name');
+    final unpacked = File('${downloads.path}/${name.replaceAll('.bz2', '')}');
+    if (!unpacked.existsSync()) {
+      await fetch(uri, packed);
+      final input = InputFileStream(packed.path);
+      final output = OutputFileStream(unpacked.path);
+      try {
+        BZip2Decoder().decodeStream(input, output);
+      } finally {
+        await input.close();
+        await output.close();
+      }
+      await packed.delete();
+    }
+    return unpacked
+        .openRead()
+        .transform(utf8.decoder)
+        .transform(const LineSplitter());
+  }
+
+  final links = await readTatoebaLinks(
+    await download(tatoebaLinksOf(from, to)),
+  );
+  final sentences = await readTatoebaSentences(
+    await download(tatoebaSentencesOf(from)),
+    wanted: links.containsKey,
+  );
+  final linked = {for (final ids in links.values) ...ids};
+  final translations = await readTatoebaSentences(
+    await download(tatoebaSentencesOf(to)),
+    wanted: linked.contains,
+  );
+  return await _write(
+    File('${dictionary.path}.tatoeba.gz'),
+    withTatoebaExamples(
+      dictionary,
+      sentences: sentences,
+      translations: translations,
+      links: links,
+    ),
+  );
+}

halkyon_learn_curator/pubspec.yaml

@@ -10,6 +10,7 @@ environment:
 resolution: workspace
 
 dependencies:
+  archive: ^4.0.7
   args: ^2.7.0
   xml: ^7.0.1
 

halkyon_learn_curator/test/fixtures/README.md

@@ -44,3 +44,10 @@ the extraction is Tatu Ylonen's Wiktextract (<https://kaikki.org>).
 2026-10-13): 中国, 了, 女儿, 学习, 书, 熊猫, 略, 的, 说, 银行, 和 and 咊, an old
 variant of 和, with their readings. CC-CEDICT is by MDBG and its contributors, CC BY-SA 4.0
 (<https://www.mdbg.net/chinese/dictionary?page=cc-cedict>).
+
+`tatoeba-*.tsv` hold a few lines of Tatoeba's exports of 2026-10-10
+(<https://downloads.tatoeba.org/exports/per_language/>): German, Chinese and
+Korean sentences using Haus, aß, Straße, 学习, 집에서 and 학생이 (and one of each
+without an English translation), the links to their English translations, and
+those translations. The sentences are by Tatoeba's contributors, CC BY 2.0 FR
+(<https://tatoeba.org/en/terms_of_use>).

halkyon_learn_curator/test/fixtures/tatoeba-cmn-eng_links.tsv

@@ -0,0 +1,4 @@
+750344	750342
+750344	9848192
+332662	68954
+4456365	309992

halkyon_learn_curator/test/fixtures/tatoeba-cmn_sentences.tsv

@@ -0,0 +1,4 @@
+750344	cmn	我学习。
+332662	cmn	您学习吗？
+4456365	cmn	她学习数学
+333376	cmn	我妹妹目前不学习。

halkyon_learn_curator/test/fixtures/tatoeba-deu-eng_links.tsv

@@ -0,0 +1,10 @@
+1550095	1549913
+9984782	8257152
+605418	24278
+10051184	8475487
+1107645	1098492
+2205383	2203598
+2491619	1364167
+2491619	5222666
+13139663	13139893
+910906	462934

halkyon_learn_curator/test/fixtures/tatoeba-deu_sentences.tsv

@@ -0,0 +1,10 @@
+1550095	deu	Na, altes Haus!
+9984782	deu	Er ist im Haus.
+605418	deu	Das Haus brennt.
+10051184	deu	Er aß.
+1107645	deu	Ich aß.
+2205383	deu	Wer aß?
+2491619	deu	In welcher Straße?
+13139663	deu	Das ist die Straße.
+910906	deu	Die Straße ist nass.
+345389	deu	Während meiner Abwesenheit wurde in meinem Haus eingebrochen.

halkyon_learn_curator/test/fixtures/tatoeba-eng_sentences.tsv

@@ -0,0 +1,17 @@
+24278	eng	The house is on fire.
+68954	eng	Are you studying?
+69521	eng	Are you a student?
+309992	eng	She studies mathematics.
+462934	eng	The street is wet.
+750342	eng	I study.
+1098492	eng	I ate.
+1364167	eng	In which street?
+1549913	eng	Hello, old friend!
+2203598	eng	Who ate?
+5222666	eng	On what street?
+8105184	eng	Mary works at a flower shop.
+8257152	eng	He is in the house.
+8475487	eng	He ate.
+9848192	eng	I learn.
+10268745	eng	I'll do it at home.
+13139893	eng	It is the street.

halkyon_learn_curator/test/fixtures/tatoeba-kor-eng_links.tsv

@@ -0,0 +1,4 @@
+13191814	10268745
+8533585	8105184
+13172354	69521
+13172353	69521

halkyon_learn_curator/test/fixtures/tatoeba-kor_sentences.tsv

@@ -0,0 +1,5 @@
+13191814	kor	집에서 할게.
+8533585	kor	메리는 꽃집에서 일해.
+13172354	kor	학생이야?
+13172353	kor	학생이에요?
+507619	kor	그는 이모의 집에서 머물렀다.

halkyon_learn_curator/test/tatoeba_test.dart

@@ -0,0 +1,133 @@
+import 'dart:convert';
+import 'dart:io';
+
+import 'package:halkyon_learn_curator/src/sources/cedict.dart';
+import 'package:halkyon_learn_curator/src/sources/tatoeba.dart';
+import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
+import 'package:test/test.dart';
+
+void main() {
+  late Directory into;
+
+  setUp(() => into = Directory.systemTemp.createTempSync('tatoeba'));
+  tearDown(() => into.deleteSync(recursive: true));
+
+  Stream<String> lines(String fixture) =>
+      File('test/fixtures/$fixture')
+          .openRead()
+          .transform(utf8.decoder)
+          .transform(const LineSplitter());
+
+  /// [file]'s lines with the fixtures' sentences of [code] as examples.
+  Future<List<Map<String, Object?>>> enriched(File file, String code) async {
+    final links = await readTatoebaLinks(lines('tatoeba-$code-eng_links.tsv'));
+    final sentences = await readTatoebaSentences(
+      lines('tatoeba-${code}_sentences.tsv'),
+      wanted: links.containsKey,
+    );
+    final linked = {for (final ids in links.values) ...ids};
+    final translations = await readTatoebaSentences(
+      lines('tatoeba-eng_sentences.tsv'),
+      wanted: linked.contains,
+    );
+    return [
+      for (final line in await withTatoebaExamples(
+        file,
+        sentences: sentences,
+        translations: translations,
+        links: links,
+      ).join().then((text) => text.split('\n')))
+        if (line.isNotEmpty) jsonDecode(line) as Map<String, Object?>,
+    ];
+  }
+
+  List<Map<String, Object?>> examplesOf(Map<String, Object?> line) => [
+    for (final example
+        in ((((line['entry']! as Map)['senses']! as List).first
+                    as Map)['examples']
+                as List? ??
+            const []))
+      example as Map<String, Object?>,
+  ];
+
+  test("a German word's sentences come first in its first sense, those "
+      'without a translation left out, the header crediting Tatoeba', () async {
+    final [german, ...] = await wiktionaryDictionaries(
+      lines('wiktionary-sample.jsonl'),
+      into: into,
+      sourceVersion: '2026-10-03',
+      languages: {'de'},
+    );
+
+    final result = await enriched(german, 'deu');
+
+    expect([
+      for (final credit in result.first['credits']! as List)
+        (credit as Map)['name'],
+    ], contains('Tatoeba'));
+    final haus = result
+        .skip(1)
+        .singleWhere((line) => line['id'] == 'Haus/noun');
+    final examples = examplesOf(haus);
+    // Three of Tatoeba's, before Wiktionary's own.
+    // Those nearest seven words long first, then the shortest.
+    expect(examples.take(3).map((example) => example['tatoeba']), [
+      9984782,
+      1550095,
+      605418,
+    ]);
+    expect(examples.first['text'], 'Haus');
+    expect(examples.first['de'], 'Er ist im Haus.');
+    expect(examples.first['en'], isA<String>());
+    expect(examples[3]['de'], 'In dem Haus haben wir mal gewohnt.');
+    // A form finds its word: aß is essen's.
+    final essen = result
+        .skip(1)
+        .singleWhere((line) => line['id'] == 'essen/verb');
+    expect(examplesOf(essen).first['text'], 'aß');
+  });
+
+  test(
+    'Chinese sentences are cut by the longest words the dictionary has',
+    () async {
+      final file = File('${into.path}/cc-cedict-hans-en.jsonl');
+      await file.writeAsString(
+        cedictDictionary(
+          await File('test/fixtures/cedict-sample.txt').readAsString(),
+          traditional: false,
+        ).join(),
+      );
+
+      final result = await enriched(file, 'cmn');
+
+      final xuexi = result
+          .skip(1)
+          .singleWhere((line) => line['id'] == '學習 学习 [xue2 xi2]');
+      expect(examplesOf(xuexi), isNotEmpty);
+      expect(examplesOf(xuexi).first['text'], '学习');
+      expect(examplesOf(xuexi).first['zh-Hans'], contains('学习'));
+    },
+  );
+
+  test('Korean words are found with their particles taken off', () async {
+    final [korean, ...] = await wiktionaryDictionaries(
+      lines('wiktionary-sample.jsonl'),
+      into: into,
+      sourceVersion: '2026-10-03',
+      languages: {'ko'},
+    );
+
+    final result = await enriched(korean, 'kor');
+
+    final home = result.skip(1).singleWhere((line) => line['id'] == '집/noun/1');
+    expect(examplesOf(home), isNotEmpty);
+    expect(examplesOf(home).first['text'], '집');
+    expect(examplesOf(home).first['ko'], contains('집에서'));
+  });
+
+  test("Tatoeba's codes of the languages", () {
+    expect(tatoebaCode('de'), 'deu');
+    expect(tatoebaCode('zh-Hant'), 'cmn');
+    expect(tatoebaCode('ja'), isNull);
+  });
+}

halkyon_learn_curator/test/update_test.dart

@@ -1,6 +1,8 @@
 import 'dart:convert';
 import 'dart:io';
 
+import 'package:archive/archive.dart';
+
 import 'package:halkyon_learn_curator/src/update.dart';
 import 'package:test/test.dart';
 
@@ -18,6 +20,16 @@ void main() {
     await file.writeAsBytes(gzip.encode(await File(sample).readAsBytes()));
   }
 
+  /// Tatoeba's exports, as the fixtures hold a few of their lines, packed
+  /// as Tatoeba packs them; null for any other address.
+  List<int>? tatoeba(Uri uri) {
+    final name = uri.pathSegments.last.replaceAll('.bz2', '');
+    final fixture = File('test/fixtures/tatoeba-$name');
+    return fixture.existsSync()
+        ? BZip2Encoder().encodeBytes(fixture.readAsBytesSync())
+        : null;
+  }
+
   /// The header of the dictionary file [name] in [into].
   Map<String, Object?> header(String name) => jsonDecode(
     utf8
@@ -128,6 +140,9 @@ void main() {
       languages: languages,
       modified: modified,
       open: open,
+      fetch: (uri, file) => file.writeAsBytes(
+        tatoeba(uri) ?? (throw HttpException('$uri', uri: uri)),
+      ),
     );
 
     test(
@@ -186,22 +201,54 @@ void main() {
     final written = await updateCedict(
       into: into,
       data: Directory('data'),
-      fetch: (uri, file) async {
+      fetch: (uri, file) => file.writeAsBytes(
-        expect(uri, cedictSource);
+        uri == cedictSource
-        await file.writeAsBytes(
+            ? gzip.encode(
-          gzip.encode(
                 File('test/fixtures/cedict-sample.txt').readAsBytesSync(),
+              )
+            : tatoeba(uri) ?? (throw HttpException('$uri', uri: uri)),
       ),
     );
-      },
-    );
 
     expect(
       [for (final file in written) file.uri.pathSegments.last],
       ['cc-cedict-hans-en.jsonl.gz', 'cc-cedict-hant-en.jsonl.gz'],
     );
     expect(header('cc-cedict-hant-en.jsonl.gz')['language'], 'zh-Hant');
+    expect([
+      for (final credit
+          in header('cc-cedict-hans-en.jsonl.gz')['credits']! as List)
+        (credit as Map)['name'],
+    ], contains('Tatoeba'));
     expect(cedictUpdated(into), isNotNull);
+    // Nothing of the downloads left beside the files.
+    expect(
+      [for (final entity in into.listSync()) entity.uri.pathSegments.last]
+        ..sort(),
+      ['cc-cedict-hans-en.jsonl.gz', 'cc-cedict-hant-en.jsonl.gz'],
+    );
+  });
+
+  test("a dictionary keeps its words without Tatoeba's sentences when they "
+      "can't be had", () async {
+    final written = await updateCedict(
+      into: into,
+      data: Directory('data'),
+      fetch: (uri, file) => uri == cedictSource
+          ? file.writeAsBytes(
+              gzip.encode(
+                File('test/fixtures/cedict-sample.txt').readAsBytesSync(),
+              ),
+            )
+          : throw HttpException('$uri answered 503.', uri: uri),
+    );
+
+    expect(written, hasLength(2));
+    expect([
+      for (final credit
+          in header('cc-cedict-hans-en.jsonl.gz')['credits']! as List)
+        (credit as Map)['name'],
+    ], isNot(contains('Tatoeba')));
   });
 
   test('Chinese comes from CC-CEDICT, the other languages from Wiktionary', () {