feat(curator): dictionaries of German, Greek and English from Wiktionary

The languages besides Japanese come from the English Wiktionary as
Wiktextract reads it (decided 2026-10-14: German, Greek and English
first, then Chinese, then Korean): kaikki.org's raw dump, every language
in one file, as its per-language files are being withdrawn.
`curate wiktionary --languages de,el,en` reads it once, streamed, and
writes German and Greek with Wiktionary's English meanings and English
with what its translation tables give in German, the app's other
language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources
`wiktionary-<language>`, as the server holds one source per meaning
language). Inflected forms, and Wiktionary's entries for forms alone
("Häusern: dative plural of Haus"), find their word instead of being
entries; names and affixes are left out, as are articles that tables
give cells of their own, abbreviations and rare spellings, which would
match common words. Entries keep their part of speech, genders, IPA,
principal forms and examples keyed by language as JMdict's.

wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank
the words by how often they are used and label the 15,000 most used
`common`. Typing without accents or umlauts finds a word (`strasse`,
`haeuser`, `spiti`).

The format gains `textKeys`: what finds an entry in running text, its
own forms, never its meanings, which would match words of the meaning
language in a subtitle. `curate update --wiktionary de,el,en` keeps the
languages current: it asks kaikki.org when the dump last changed and
converts it again only when it is newer, or the languages or the
conversion changed. The whole dump converts in about ten minutes in
under 300 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

halkyon_learn_curator/README.md

@@ -49,11 +49,38 @@ KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:
 
    About 12,800 characters, in some 5 seconds.
 
+Wiktionary, for the languages other than Japanese (decided 2026-10-14: German,
+Greek and English first, then Chinese, then Korean):
+
+1. Download `raw-wiktextract-data.jsonl.gz` from
+   [kaikki.org](https://kaikki.org/dictionary/rawdata.html): the English
+   Wiktionary as Wiktextract reads it, every language in one file (some 3 GB,
+   24 GB unpacked; kaikki.org's files of single languages are being withdrawn).
+2. Convert, in this package, naming the languages:
+
+   ```sh
+   dart run halkyon_learn_curator:curate wiktionary \
+     --input raw-wiktextract-data.jsonl.gz --languages de,el,en \
+     --version 2026-10-03 --output ../halkyon_learn_server/dictionaries
+   ```
+
+   Each language but English becomes its words with Wiktionary's English
+   meanings (`wiktionary-de-en.jsonl.gz`, source `wiktionary-de`); English
+   becomes its words with what their translation tables give in German
+   (`wiktionary-en-de.jsonl.gz`), German being the app's other language. An
+   inflected form, and an entry Wiktionary has for one alone ("Häusern: dative
+   plural of Haus"), finds its word rather than being an entry of its own;
+   names and affixes are left out. Word frequencies from `data/wordfreq/` rank
+   the words and say which are `common` (the 15,000 most used). Reading the
+   whole dump takes a while; it is read once, streamed, keeping in memory only
+   which forms belong to which word.
+
 Put the files into the server's `dictionaries` folder. The server loads each on
 its next start, and again whenever it changes (see the server's README).
 
 ## Keeping a server's dictionaries current
 
+
 EDRDG's licence asks of a server showing its dictionaries that it update them
 at least once a month. `curate update` does it from the sources alone:
 
@@ -76,6 +103,14 @@ so in a container of its own (the server's README). Beside the files,
 from the same download (a label, a field), and an instance converts again as
 soon as the new curator runs, not a week later.
 
+With `--wiktionary de,el,en` it also keeps those languages current from
+Wiktionary: each time it wakes it asks kaikki.org when its dump last changed
+(`Last-Modified`, without downloading it), and converts it again only when it is
+newer than the files' (about monthly), when the languages named changed, or when
+`wiktionaryConversion` in `lib/src/update.dart` was raised; `.wiktionary` beside
+the files says which. The dump is converted as it downloads, never stored.
+Files of a language no longer named stay until removed by hand.
+
 ## The format (`halkyon-dictionary`, version 1)
 
 [JSON Lines](https://jsonlines.org/), gzipped when the name ends in `.gz`.
@@ -97,11 +132,12 @@ The **first line** says what the file holds:
 |---|---|
 | `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
 | `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
-| `labels` | What it can be picked by: `common`, `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
+| `labels` | What it can be picked by: `common` (JMdict's own; for Wiktionary the 15,000 words wordfreq finds most used), `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji), a kanji's Kangxi radical (`radical:1` … `radical:214`, KANJIDIC2's classical one) and the script (`hiragana`, `katakana`). |
 | `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
 | `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
-| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし). Left out when empty. |
+| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし; a word without its accents or umlauts, `strasse`, `haeuser`, `spiti`). Left out when empty. |
-| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. |
+| `textKeys` | What finds the entry in running text (subtitles, lyrics): its own forms, lower case (`haus`, `häuser`, `häusern`), never its meanings, which would match words of the meaning language in a text. Left out: every key without a space finds it, as for JMdict, whose keys in Japanese script never meet a meaning's. |
+| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. A Wiktionary word's: `pos` (Wiktionary's part of speech for the whole entry: `noun`, `verb`, `adj`), `genders` (`m`, `f`, `n` of a German or Greek noun), `romanization` (Greek's), `ipa`, `inflections` (the forms on the word's own line: a noun's genitive and plural, a verb's principal parts; `text` and Wiktionary's `tags`), and `senses` with `glosses`, `tags` (how and where a sense is used: `colloquial`, `Austria`), `info` (for English with German translations, the English meaning a translation is of) and `examples` keyed by language as JMdict's (`de`, the sentence; `en`, its translation; `text`, the word as it stands there). |
 
 ## The app's stroke files
 

halkyon_learn_curator/bin/curate.dart

@@ -7,7 +7,9 @@ import 'package:halkyon_learn_curator/src/sources/jmdict.dart';
 import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
 import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
 import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
+import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
 import 'package:halkyon_learn_curator/src/update.dart';
+import 'package:halkyon_learn_curator/src/wordfreq.dart';
 
 /// Turns a dictionary download into a dictionary file for the server's
 /// `dictionaries` folder, or KanjiVG into the app's stroke files
@@ -23,11 +25,21 @@ import 'package:halkyon_learn_curator/src/update.dart';
 ///       --input kanjivg-20260714.xml.gz \
 ///       --output ../halkyon_learn_flutter/assets/kanjivg
 ///
+/// Wiktionary, as kaikki.org's dump of it, into dictionary files of the
+/// languages named (German and Greek with English meanings, English with
+/// German ones):
+///
+///     dart run halkyon_learn_curator:curate wiktionary \
+///       --input raw-wiktextract-data.jsonl.gz --languages de,el,en \
+///       --output ../halkyon_learn_server/dictionaries
+///
 /// Or brings a server's dictionaries up to date from EDRDG's files of the
-/// day, once or every so many days:
+/// day and, for the languages named, Wiktionary's latest dump, once or
+/// every so many days:
 ///
 ///     dart run halkyon_learn_curator:curate update \
-///       --into ../halkyon_learn_server/dictionaries [--every 7]
+///       --into ../halkyon_learn_server/dictionaries [--every 7] \
+///       [--wiktionary de,el,en]
 Future<void> main(List<String> args) async {
   final parser = ArgParser()
     ..addOption(
@@ -68,6 +80,24 @@ Future<void> main(List<String> args) async {
       help:
           'update: keep running, and update once the files are this many '
           'days old.',
+    )
+    ..addOption(
+      'languages',
+      help:
+          'wiktionary: the languages to write dictionaries of, by their '
+          'BCP 47 tags (de,el,en).',
+    )
+    ..addOption(
+      'version',
+      help:
+          "wiktionary: the dump's date (2026-10-03); the input file's date "
+          'without.',
+    )
+    ..addOption(
+      'wiktionary',
+      help:
+          "update: the languages to keep current from Wiktionary's dump "
+          '(de,el,en); none without.',
     );
   final ArgResults options;
   try {
@@ -83,10 +113,52 @@ Future<void> main(List<String> args) async {
     if (into == null || (every != null && (days == null || days < 1))) {
       _fail(
         'Usage: curate update --into <folder> [--data <folder>] '
-        '[--every <days>]\n${parser.usage}',
+        '[--every <days>] [--wiktionary <languages>]\n${parser.usage}',
       );
     }
-    await _update(Directory(into), Directory(options.option('data')!), days);
+    await _update(
+      Directory(into),
+      Directory(options.option('data')!),
+      days,
+      wiktionary: _languages(options.option('wiktionary')),
+    );
+    return;
+  }
+  if (source == 'wiktionary') {
+    final input = options.option('input');
+    final output = options.option('output');
+    final languages = _languages(options.option('languages'));
+    if (input == null || output == null || languages.isEmpty) {
+      _fail(
+        'Usage: curate wiktionary --input <file> --languages <languages> '
+        '--output <folder> [--version <date>] [--data <folder>]'
+        '\n${parser.usage}',
+      );
+    }
+    final read = File(input).openRead();
+    final outputs = wiktionaryOutputs(languages);
+    final data = options.option('data')!;
+    final written = await wiktionaryDictionaries(
+      (input.endsWith('.gz') ? read.transform(gzip.decoder) : read)
+          .transform(utf8.decoder)
+          .transform(const LineSplitter()),
+      into: Directory(output),
+      sourceVersion:
+          options.option('version') ??
+          File(input).lastModifiedSync().toIso8601String().substring(0, 10),
+      languages: outputs.languages,
+      translatedInto: outputs.translatedInto,
+      frequencies: {
+        for (final language in {...languages, 'en'})
+          if (File('$data/wordfreq/small_$language.msgpack.gz') case final file
+              when file.existsSync())
+            language: await WordFrequencies.read(file),
+      },
+    );
+    stdout.writeln(
+      'Wrote ${[for (final file in written) file.uri.pathSegments.last].join(', ')} '
+      'to $output.',
+    );
     return;
   }
   final input = options.option('input');
@@ -162,16 +234,23 @@ Future<void> main(List<String> args) async {
 /// Updates the dictionaries in [into]; with [every], keeps running and
 /// updates them whenever they are that many days old (as the files' dates
 /// say, so a restart doesn't download them again), trying again after six
-/// hours when an update fails.
+/// hours when an update fails. The languages of [wiktionary] are looked at
-Future<void> _update(Directory into, Directory data, int? every) async {
+/// each time it wakes, and converted again only when Wiktionary's dump is
+/// newer than theirs.
+Future<void> _update(
+  Directory into,
+  Directory data,
+  int? every, {
+  Set<String> wiktionary = const {},
+}) async {
   void say(String message) =>
       stdout.writeln('${DateTime.now().toUtc().toIso8601String()} $message');
+  void written(List<File> files) => say(
+    'Updated ${[for (final file in files) file.uri.pathSegments.last].join(', ')}.',
+  );
   Future<bool> once() async {
     try {
-      final written = await updateDictionaries(into: into, data: data);
+      written(await updateDictionaries(into: into, data: data));
-      say(
-        'Updated ${[for (final file in written) file.uri.pathSegments.last].join(', ')}.',
-      );
       return true;
     } on Object catch (error) {
       say('Updating the dictionaries failed: $error');
@@ -179,8 +258,25 @@ Future<void> _update(Directory into, Directory data, int? every) async {
     }
   }
 
+  Future<bool> wiktionaryOnce() async {
+    try {
+      final files = await updateWiktionary(
+        into: into,
+        data: data,
+        languages: wiktionary,
+      );
+      if (files.isNotEmpty) {
+        written(files);
+      }
+      return true;
+    } on Object catch (error) {
+      say('Updating Wiktionary failed: $error');
+      return false;
+    }
+  }
+
   if (every == null) {
-    if (!await once()) {
+    if (!await once() || !await wiktionaryOnce()) {
       exit(1);
     }
     return;
@@ -194,11 +290,21 @@ Future<void> _update(Directory into, Directory data, int? every) async {
       }
       continue;
     }
-    say('Up to date; the next update ${due.toUtc().toIso8601String()}.');
+    final retry = !await wiktionaryOnce();
-    await Future<void>.delayed(due.difference(now));
+    final next = retry && due.difference(now) > const Duration(hours: 6)
+        ? DateTime.now().add(const Duration(hours: 6))
+        : due;
+    say('Up to date; the next look ${next.toUtc().toIso8601String()}.');
+    await Future<void>.delayed(next.difference(DateTime.now()));
   }
 }
 
+/// The languages a comma-separated list names (`de,el,en`).
+Set<String> _languages(String? list) => {
+  for (final language in (list ?? '').split(','))
+    if (language.trim().isNotEmpty) language.trim(),
+};
+
 Never _fail(String message) {
   stderr.writeln(message);
   exit(64);

halkyon_learn_curator/data/README.md

@@ -1,7 +1,7 @@
 # Data the curator brings along
 
 What the dictionaries draw on besides their sources, and never changes, so
-`curate update` needs only EDRDG's files of the day.
+`curate update` needs only EDRDG's files of the day and Wiktionary's dump.
 
 `jlpt/n1.csv` … `n5.csv` are the JLPT vocabulary lists of
 [yomitan-jlpt-vocab](https://github.com/stephenmk/yomitan-jlpt-vocab)
@@ -17,3 +17,12 @@ for N1: the `jlpt_new` of
 
 The lists are over ten years old and won't change; replace them only for a list
 of the same shape.
+
+`wordfreq/small_<language>.msgpack.gz` are wordfreq's lists of how often words
+are used ([wordfreq](https://github.com/rspeer/wordfreq) v3.0, as of 2021-03-29,
+the last change to its data), unchanged: CC BY-SA 4.0, by Robyn Speer, drawn from
+SUBTLEX (Marc Brysbaert et al., whose authors wordfreq's licence asks to be
+credited, and who made SUBTLEX freely available), OpenSubtitles, Wikipedia,
+Google Books Ngrams and others, as wordfreq's README lists them. The Wiktionary
+dictionaries take their ranks and their `common` label from them and credit
+both. wordfreq's data won't change: it was sunset in 2024.

halkyon_learn_curator/lib/src/dictionary_file.dart

@@ -66,8 +66,9 @@ final class const DictionaryHeader({
 /// to choose entries by (`common`, `jlpt:n5`), a [rank] to order them by
 /// (lower first), the [keys] a search matches against (lower case), the
 /// [looseKeys] it matches after them (spellings people type that aren't
-/// quite right, as `sushi` for すうし), and the [entry] itself as the app
+/// quite right, as `sushi` for すうし), the [textKeys] that find it in
-/// turns it into a card.
+/// running text (its own forms; empty: every key without a space), and
+/// the [entry] itself as the app turns it into a card.
 final class const DictionaryEntry({
   required final String id,
   required final String headword,
@@ -77,6 +78,7 @@ final class const DictionaryEntry({
   required final List<String> keys,
   required final Map<String, Object?> entry,
   final List<String> looseKeys = const [],
+  final List<String> textKeys = const [],
 }) {
   Map<String, Object?> toJson() => {
     'id': id,
@@ -86,6 +88,7 @@ final class const DictionaryEntry({
     'rank': rank,
     'keys': keys,
     if (looseKeys.isNotEmpty) 'looseKeys': looseKeys,
+    if (textKeys.isNotEmpty) 'textKeys': textKeys,
     'entry': entry,
   };
 }

halkyon_learn_curator/lib/src/fold.dart

@@ -0,0 +1,67 @@
+/// [text] in lower case as frequency lists and people typing compare
+/// words: `ß` as `ss` and a word-final `ς` as `σ`, as Unicode's case
+/// folding does (Python's `str.casefold`, which wordfreq's lists are made
+/// with).
+String casefold(String text) =>
+    text.toLowerCase().replaceAll('ß', 'ss').replaceAll('ς', 'σ');
+
+/// [text] case-folded ([casefold]) and without its accents: what people
+/// type without a keyboard for the language (`strasse` for `Straße`,
+/// `spiti` for `σπίτι`, `cafe` for `café`). Letters of the Latin and Greek
+/// alphabets only; others stay as they are.
+String withoutAccents(String text) {
+  final folded = casefold(text);
+  final out = StringBuffer();
+  for (final rune in folded.runes) {
+    final char = String.fromCharCode(rune);
+    out.write(_bare[char] ?? char);
+  }
+  return out.toString();
+}
+
+/// German's umlauts written out (`ae`, `oe`, `ue`), as typed on keyboards
+/// without them; null for a word without one.
+String? umlautsWrittenOut(String text) {
+  final folded = casefold(text);
+  if (!folded.contains(RegExp('[äöü]'))) {
+    return null;
+  }
+  return folded
+      .replaceAll('ä', 'ae')
+      .replaceAll('ö', 'oe')
+      .replaceAll('ü', 'ue');
+}
+
+/// Each accented letter (lower case) and its bare one.
+final _bare = {
+  for (final (accented, bare) in const [
+    ('àáâãäåāăą', 'a'),
+    ('çćĉċč', 'c'),
+    ('ďđ', 'd'),
+    ('èéêëēĕėęě', 'e'),
+    ('ĝğġģ', 'g'),
+    ('ĥħ', 'h'),
+    ('ìíîïĩīĭįı', 'i'),
+    ('ĵ', 'j'),
+    ('ķ', 'k'),
+    ('ĺļľŀł', 'l'),
+    ('ñńņňʼn', 'n'),
+    ('òóôõöøōŏő', 'o'),
+    ('ŕŗř', 'r'),
+    ('śŝşšș', 's'),
+    ('ţťŧț', 't'),
+    ('ùúûüũūŭůűų', 'u'),
+    ('ŵ', 'w'),
+    ('ýÿŷ', 'y'),
+    ('źżž', 'z'),
+    ('άἀἁἄἅὰᾶ', 'α'),
+    ('έἐἑἔἕὲ', 'ε'),
+    ('ήἠἡἤἥὴῆ', 'η'),
+    ('ίϊΐἰἱἴἵὶῖ', 'ι'),
+    ('όὀὁὄὅὸ', 'ο'),
+    ('ύϋΰὐὑὔὕὺῦ', 'υ'),
+    ('ώὠὡὤὥὼῶ', 'ω'),
+    ('ῥ', 'ρ'),
+  ])
+    for (final char in accented.split('')) char: bare,
+};

halkyon_learn_curator/lib/src/gloss_keys.dart

@@ -1,8 +1,10 @@
 /// What a search for a meaning matches in [gloss], lower case: the gloss
 /// without its remarks in brackets, and without a leading "to" or article
 /// too (`to eat` gives `to eat` and `eat`; `the sun (in the sky)` gives
-/// `the sun` and `sun`). Empty for a gloss of remarks only.
+/// `the sun` and `sun`). Empty for a gloss of remarks only. Meanings in
-List<String> glossKeys(String gloss) {
+/// German ([language] `de`) lose their article or `sich` instead
+/// (`das Haus` gives `das haus` and `haus`).
+List<String> glossKeys(String gloss, {String language = 'en'}) {
   final plain = gloss
       .toLowerCase()
       .replaceAll(RegExp(r'\([^)]*\)'), ' ')
@@ -11,6 +13,12 @@ List<String> glossKeys(String gloss) {
   if (plain.isEmpty) {
     return const [];
   }
-  final bare = plain.replaceFirst(RegExp('^(to|a|an|the) '), '');
+  final bare = plain.replaceFirst(_leading[language] ?? _leading['en']!, '');
   return {plain, bare}.toList();
 }
+
+/// The words a meaning may start with that a search does without.
+final _leading = {
+  'en': RegExp('^(to|a|an|the) '),
+  'de': RegExp('^(der|die|das|den|dem|des|ein|eine|einen|einem|einer|sich) '),
+};

halkyon_learn_curator/lib/src/sources/wiktionary.dart

@@ -0,0 +1,631 @@
+import 'dart:convert';
+import 'dart:io';
+
+import 'package:halkyon_learn_curator/src/dictionary_file.dart';
+import 'package:halkyon_learn_curator/src/fold.dart';
+import 'package:halkyon_learn_curator/src/gloss_keys.dart';
+import 'package:halkyon_learn_curator/src/wordfreq.dart';
+
+/// Who wrote Wiktionary, and who made it readable here.
+const wiktionaryCredits = [
+  Credit(
+    name: 'Wiktionary',
+    by: 'Wiktionary contributors',
+    license: 'CC-BY-SA-4.0',
+    url: 'https://en.wiktionary.org/wiki/Wiktionary:Copyrights',
+  ),
+  Credit(
+    name: 'Wiktextract (kaikki.org)',
+    by: 'Tatu Ylonen',
+    license: 'CC-BY-SA-4.0',
+    url: 'https://kaikki.org/dictionary/rawdata.html',
+  ),
+];
+
+/// The name of the dictionary file of [language]'s words with meanings in
+/// [glossLanguage] (`wiktionary-de-en.jsonl.gz`).
+String wiktionaryFileName(String language, String glossLanguage) =>
+    'wiktionary-$language-$glossLanguage.jsonl.gz';
+
+/// Parts of speech Wiktionary has entries for that are no words of running
+/// text, or names: left out.
+const _notWords = {
+  'name',
+  'character',
+  'symbol',
+  'punct',
+  'romanization',
+  'prefix',
+  'suffix',
+  'infix',
+  'interfix',
+  'circumfix',
+  'affix',
+  'combining_form',
+};
+
+/// Tags of a form that make it no inflection worth showing beside the
+/// headword (it may still find the word).
+const _notShown = {
+  'romanization',
+  'auxiliary',
+  'table-tags',
+  'inflection-template',
+  'class',
+  'diminutive',
+  'augmentative',
+  'alternative',
+  'obsolete',
+  'archaic',
+  'rare',
+  'dialectal',
+  'nonstandard',
+  'misspelling',
+  'error-unrecognized-form',
+};
+
+/// Tags of a form that make it no form of the word as written: a
+/// romanisation, a note about the inflection table, or the auxiliary verb
+/// a verb forms its perfect with (`haben` beside `essen`).
+const _notForms = {
+  'auxiliary',
+  'romanization',
+  'table-tags',
+  'inflection-template',
+  'class',
+};
+
+/// Tags of a sense worth keeping beside its meaning: how and where it is
+/// used.
+const _senseTags = {
+  'colloquial',
+  'informal',
+  'formal',
+  'slang',
+  'vulgar',
+  'derogatory',
+  'offensive',
+  'humorous',
+  'figuratively',
+  'literally',
+  'archaic',
+  'obsolete',
+  'dated',
+  'rare',
+  'poetic',
+  'literary',
+  'transitive',
+  'intransitive',
+  'reflexive',
+  'impersonal',
+  'countable',
+  'uncountable',
+  'plural',
+  'Austria',
+  'Switzerland',
+  'Germany',
+  'US',
+  'UK',
+  'Cyprus',
+};
+
+/// The gender tags of German and Greek nouns, as an entry keeps them.
+const _genders = {'masculine': 'm', 'feminine': 'f', 'neuter': 'n'};
+
+/// How many senses, examples a sense, and inflections an entry keeps.
+const _maxSenses = 16;
+const _maxExamples = 3;
+const _maxInflections = 6;
+
+/// Words with a place in the frequency list below this are `common`.
+const commonPlaces = 15000;
+
+/// Converts the English Wiktionary as Wiktextract reads it (kaikki.org's
+/// raw dump, a JSON object per line) into dictionary files in [into]: for
+/// each of [languages] (`de`, `el`), that language's words with
+/// Wiktionary's English meanings ([wiktionaryFileName] `wiktionary-de-en`);
+/// for each of [translatedInto] (`de`), English words with what the
+/// translation tables of their entries give in that language
+/// (`wiktionary-en-de`). An entry's inflected forms, and the entries
+/// Wiktionary has for them alone ("Häusern: dative plural of Haus"), find
+/// it, in a search and in running text, without being entries of their
+/// own; names and affixes are left out. [frequencies] by language rank the
+/// words, the most used first, and say which are `common`.
+///
+/// Reads the dump once, keeping in memory only which forms belong to which
+/// word; the entries wait in temporary files until it is read. Returns
+/// the files written.
+Future<List<File>> wiktionaryDictionaries(
+  Stream<String> lines, {
+  required Directory into,
+  required String sourceVersion,
+  Set<String> languages = const {},
+  Set<String> translatedInto = const {},
+  Map<String, WordFrequencies> frequencies = const {},
+}) async {
+  final work = await Directory.systemTemp.createTemp('wiktionary');
+  try {
+    final outputs = {
+      for (final language in languages)
+        (language, 'en'): _Output(work, language, 'en'),
+      for (final gloss in translatedInto)
+        ('en', gloss): _Output(work, 'en', gloss),
+    };
+    // Each language's forms, by the word they are forms of.
+    final formsOf = <String, Map<String, Set<String>>>{};
+    final wanted = {...languages, if (translatedInto.isNotEmpty) 'en'};
+    final marks = [for (final code in wanted) '"lang_code": "$code"'];
+    await for (final line in lines) {
+      if (!marks.any(line.contains)) {
+        continue;
+      }
+      final json = jsonDecode(line);
+      if (json is! Map<String, Object?>) {
+        continue;
+      }
+      final language = json['lang_code'];
+      if (language is! String || !wanted.contains(language)) {
+        continue;
+      }
+      final word = json['word'];
+      final pos = json['pos'];
+      if (word is! String || pos is! String || _notWords.contains(pos)) {
+        continue;
+      }
+      final senses = _maps(json['senses']);
+      // Senses saying the word is a form of another ("dative plural of
+      // Haus") make it a key of that one.
+      for (final sense in senses) {
+        if (_strings(sense['tags']).any(_shortForms.contains)) {
+          continue;
+        }
+        for (final target in [
+          ..._maps(sense['form_of']),
+          ..._maps(sense['alt_of']),
+        ]) {
+          if (target['word'] case final String lemma) {
+            (formsOf[language] ??= {}).putIfAbsent(lemma, () => {}).add(word);
+          }
+        }
+      }
+      final own = [
+        for (final sense in senses)
+          if (_maps(sense['form_of']).isEmpty &&
+              _maps(sense['alt_of']).isEmpty &&
+              _strings(sense['glosses']).isNotEmpty)
+            sense,
+      ];
+      if (outputs[(language, 'en')] case final output? when own.isNotEmpty) {
+        output.add(_glossedEntry(json, word, pos, own, language));
+      }
+      if (language == 'en') {
+        for (final gloss in translatedInto) {
+          if (_translatedEntry(json, word, pos, gloss) case final entry?) {
+            outputs[('en', gloss)]!.add(entry);
+          }
+        }
+      }
+    }
+    await into.create(recursive: true);
+    return [
+      for (final output in outputs.values)
+        await output.finish(
+          into,
+          sourceVersion: sourceVersion,
+          formsOf: formsOf[output.language] ?? const {},
+          frequencies: frequencies[output.language] ?? WordFrequencies.none,
+        ),
+    ];
+  } finally {
+    await work.delete(recursive: true);
+  }
+}
+
+/// The entries of one dictionary file, waiting in a temporary file until
+/// the dump is read.
+final class _Output(
+  final Directory _work,
+  final String language,
+  final String glossLanguage,
+) {
+  late final _pending = File('${_work.path}/$language-$glossLanguage.jsonl');
+  late final _sink = _pending.openWrite();
+
+  void add(Map<String, Object?> entry) => _sink.writeln(jsonEncode(entry));
+
+  /// The dictionary file in [into], each entry with the forms [formsOf]
+  /// gives its word among the keys, ranked by [frequencies].
+  Future<File> finish(
+    Directory into, {
+    required String sourceVersion,
+    required Map<String, Set<String>> formsOf,
+    required WordFrequencies frequencies,
+  }) async {
+    await _sink.close();
+    final file = File(
+      '${into.path}/.${wiktionaryFileName(language, glossLanguage)}.part',
+    );
+    final ids = <String>{};
+    final header = DictionaryHeader(
+      source: 'wiktionary-$language',
+      sourceVersion: sourceVersion,
+      language: language,
+      glossLanguage: glossLanguage,
+      license: 'CC-BY-SA-4.0',
+      credits: [
+        ...wiktionaryCredits,
+        if (frequencies.length > 0) ...wordfreqCredits,
+      ],
+    );
+    Stream<String> lines() async* {
+      yield headerLine(header);
+      await for (final line
+          in _pending
+              .openRead()
+              .transform(utf8.decoder)
+              .transform(const LineSplitter())) {
+        final pending = jsonDecode(line) as Map<String, Object?>;
+        final word = pending['headword']! as String;
+        // One id for each entry, even should Wiktionary give two the same.
+        var id = pending['id']! as String;
+        for (var n = 2; !ids.add(id); n++) {
+          id = '${pending['id']}~$n';
+        }
+        final forms = {
+          ...(pending['textKeys']! as List).cast<String>(),
+          for (final form in formsOf[word] ?? const <String>{})
+            if (!form.contains(' ')) form.toLowerCase(),
+        };
+        final keys = {...(pending['keys']! as List).cast<String>(), ...forms};
+        final loose = {
+          for (final key in forms) ...[
+            withoutAccents(key),
+            ?umlautsWrittenOut(key),
+          ],
+        }.difference(keys);
+        // The word's own frequency; else, half as high, its most used
+        // form's. Words in capitals (`ΝΑ`, an acronym; `Η`, a letter) are
+        // others than the lower-case words the lists count.
+        final capitals =
+            word == word.toUpperCase() && word != word.toLowerCase();
+        final formPlaces = [
+          for (final form in forms) ?frequencies.placeOf(form),
+        ];
+        final place = capitals
+            ? null
+            : frequencies.placeOf(word) ??
+                  (formPlaces.isEmpty
+                      ? null
+                      : formPlaces.reduce((a, b) => a < b ? a : b) * 2);
+        yield entryLine(
+          DictionaryEntry(
+            id: id,
+            headword: word,
+            reading: null,
+            labels: [if (place != null && place < commonPlaces) 'common'],
+            rank: (place ?? 900000) * 16 + _posOrder(pending['pos']),
+            keys: keys.toList(),
+            looseKeys: loose.toList(),
+            textKeys: forms.toList(),
+            entry: pending['entry']! as Map<String, Object?>,
+          ),
+        );
+      }
+    }
+
+    await lines()
+        .transform(utf8.encoder)
+        .transform(gzip.encoder)
+        .pipe(file.openWrite());
+    return await file.rename(
+      '${into.path}/${wiktionaryFileName(language, glossLanguage)}',
+    );
+  }
+}
+
+/// Parts of speech in the order an entry of each comes among entries of
+/// one word.
+int _posOrder(Object? pos) => switch (pos) {
+  'noun' => 0,
+  'verb' => 1,
+  'adj' => 2,
+  'adv' => 3,
+  'pron' => 4,
+  'det' || 'article' => 5,
+  'prep' || 'postp' => 6,
+  'conj' => 7,
+  'num' => 8,
+  'particle' => 9,
+  'intj' => 10,
+  _ => 15,
+};
+
+/// A word of [language] with its English meanings ([senses], those that
+/// are no form of another word), as the entry waits for its keys.
+Map<String, Object?> _glossedEntry(
+  Map<String, Object?> json,
+  String word,
+  String pos,
+  List<Map<String, Object?>> senses,
+  String language,
+) {
+  final forms = _maps(json['forms']);
+  final genders = {
+    for (final sense in senses)
+      for (final tag in _strings(sense['tags'])) ?_genders[tag],
+  };
+  final romanization = [
+    for (final form in forms)
+      if (_strings(form['tags']).contains('romanization'))
+        ?form['form'] as String?,
+  ].firstOrNull;
+  final glosses = [for (final sense in senses) _strings(sense['glosses']).last];
+  final textKeys = _textKeys(word, pos, language, forms);
+  return {
+    'id': _idOf(json, word, pos),
+    'headword': word,
+    'pos': pos,
+    'textKeys': textKeys,
+    'keys': [
+      ...textKeys,
+      ?romanization?.toLowerCase(),
+      for (final gloss in glosses) ..._meaningKeys(gloss),
+    ],
+    'entry': _withoutEmpty({
+      'pos': pos,
+      'genders': [
+        for (final gender in const ['m', 'f', 'n'])
+          if (genders.contains(gender)) gender,
+      ],
+      'romanization': romanization,
+      'ipa': _ipaOf(json),
+      'inflections': _inflections(word, forms),
+      'senses': [
+        for (final sense in senses.take(_maxSenses))
+          _withoutEmpty({
+            'glosses': [_strings(sense['glosses']).last],
+            'tags': [
+              for (final tag in _strings(sense['tags']))
+                if (_senseTags.contains(tag)) tag,
+            ],
+            'examples': [
+              for (final example in _maps(sense['examples']))
+                ?_exampleOf(example, language),
+            ].take(_maxExamples).toList(),
+          }),
+      ],
+    }),
+  };
+}
+
+/// An English word with what the translation tables of its entry give in
+/// [gloss] (`de`), a sense for each meaning the table names; null without
+/// any.
+Map<String, Object?>? _translatedEntry(
+  Map<String, Object?> json,
+  String word,
+  String pos,
+  String gloss,
+) {
+  // The meanings in the order the table lists them, each with its words.
+  final senses = <String, List<String>>{};
+  for (final translation in _maps(json['translations'])) {
+    final text = translation['word'];
+    if (translation['lang_code'] != gloss ||
+        text is! String ||
+        text.trim().isEmpty ||
+        // Regional varieties (`Alemannic-German`, `Austria`) are tagged
+        // with capitals, grammar in lower case.
+        _strings(translation['tags'])
+            .any((tag) => tag.startsWith(RegExp('[A-Z]')))) {
+      continue;
+    }
+    final words = senses.putIfAbsent(
+      translation['sense'] as String? ?? '',
+      () => [],
+    );
+    if (!words.contains(text)) {
+      words.add(text);
+    }
+  }
+  if (senses.isEmpty) {
+    return null;
+  }
+  final forms = _maps(json['forms']);
+  final textKeys = _textKeys(word, pos, 'en', forms);
+  return {
+    'id': _idOf(json, word, pos),
+    'headword': word,
+    'pos': pos,
+    'textKeys': textKeys,
+    'keys': [
+      ...textKeys,
+      for (final words in senses.values)
+        for (final text in words) ...glossKeys(text, language: gloss),
+    ],
+    'entry': _withoutEmpty({
+      'pos': pos,
+      'ipa': _ipaOf(json),
+      'inflections': _inflections(word, forms),
+      'senses': [
+        for (final MapEntry(key: meaning, value: words) in senses.entries.take(
+          _maxSenses,
+        ))
+          _withoutEmpty({
+            'glosses': words,
+            'info': [if (meaning.isNotEmpty) meaning],
+          }),
+      ],
+    }),
+  };
+}
+
+/// What a search for a meaning matches in [gloss]: the whole, and each of
+/// the meanings it lists (`house, building` gives `house` and `building`
+/// too).
+Set<String> _meaningKeys(String gloss) => {
+  ...glossKeys(gloss),
+  for (final part
+      in gloss.replaceAll(RegExp(r'\([^)]*\)'), ' ').split(RegExp('[,;]')))
+    ...glossKeys(part),
+};
+
+/// The id an entry keeps for good: its word and part of speech, and which
+/// of the word's etymologies it is where Wiktionary has several.
+String _idOf(Map<String, Object?> json, String word, String pos) => [
+  word,
+  pos,
+  // A number, though the dump writes it as text.
+  if (json['etymology_number'] case final Object number
+      when '$number'.isNotEmpty)
+    '$number',
+].join('/');
+
+/// What finds the word in running text: itself and every form it takes
+/// as one word, lower case.
+List<String> _textKeys(
+  String word,
+  String pos,
+  String language,
+  List<Map<String, Object?>> forms,
+) => {
+  word.toLowerCase(),
+  for (final form in forms)
+    if (form['form'] case final String text
+        when !text.contains(' ') &&
+            text.isNotEmpty &&
+            !_strings(form['tags']).any(_notForms.contains) &&
+            !_strings(form['tags']).any(_unusual.contains) &&
+            // Greek's tables give the article a cell of its own (`ο`,
+            // `του` beside `βάτος`): no form of the noun.
+            (_articleWords.contains(pos) ||
+                !(_articles[language] ?? const {}).contains(
+                  text.toLowerCase(),
+                )))
+      text.toLowerCase(),
+}.toList();
+
+/// Tags of a form too rare to find the word by in running text, where it
+/// may be another, common word (`the`, an obsolete spelling of `thee`).
+const _unusual = {
+  'obsolete',
+  'archaic',
+  'rare',
+  'dialectal',
+  'nonstandard',
+  'misspelling',
+  'abbreviation',
+};
+
+/// Parts of speech whose forms may be articles.
+const _articleWords = {'article', 'det', 'pron'};
+
+/// The articles of each language, which inflection tables may list as
+/// cells of their own.
+const _articles = {
+  'de': {
+    'der',
+    'die',
+    'das',
+    'des',
+    'dem',
+    'den',
+    'ein',
+    'eine',
+    'einer',
+    'eines',
+    'einem',
+    'einen',
+  },
+  'el': {
+    'ο',
+    'η',
+    'το',
+    'του',
+    'της',
+    'τον',
+    'την',
+    'τη',
+    'οι',
+    'τα',
+    'των',
+    'τους',
+    'τις',
+    'ένας',
+    'μια',
+    'μία',
+    'ένα',
+    'ενός',
+    'μιας',
+    'έναν',
+  },
+  'en': {'the', 'a', 'an'},
+};
+
+/// Tags of a sense making its word a short form of another, which finds
+/// that word in no text (`TO`, turnover; `h.`, Haus).
+const _shortForms = {'abbreviation', 'initialism', 'acronym', 'clipping'};
+
+/// The forms shown beside the headword: those Wiktionary gives on the
+/// word's own line (genitive and plural, a verb's principal parts), not
+/// the whole table, nor rare or regional ones.
+List<Map<String, Object?>> _inflections(
+  String word,
+  List<Map<String, Object?>> forms,
+) => [
+  for (final form in forms)
+    if (form['source'] == null &&
+        form['form'] is String &&
+        form['form'] != word &&
+        !_strings(form['tags']).any(_notShown.contains))
+      {'text': form['form'], 'tags': _strings(form['tags'])},
+].take(_maxInflections).toList();
+
+/// The word's pronunciation in IPA, its first.
+String? _ipaOf(Map<String, Object?> json) =>
+    [for (final sound in _maps(json['sounds'])) ?sound['ipa'] as String?]
+        .firstOrNull;
+
+/// A sentence using the word, in [language], with its English, as an
+/// entry keeps it: the form the word takes there (`text`, Wiktionary's
+/// bold), the sentence under the language's tag, its English under `en`.
+/// Null without a translation.
+Map<String, Object?>? _exampleOf(
+  Map<String, Object?> example,
+  String language,
+) {
+  final text = example['text'];
+  final english = example['translation'] ?? example['english'];
+  if (text is! String || english is! String || text.isEmpty) {
+    return null;
+  }
+  final bold = switch (example['bold_text_offsets']) {
+    [[final int start, final int end], ...]
+        when start >= 0 && end <= text.length && start < end =>
+      text.substring(start, end),
+    _ => '',
+  };
+  return {'text': bold, language: text, 'en': english};
+}
+
+/// [map] without the keys whose value is null or an empty list.
+Map<String, Object?> _withoutEmpty(Map<String, Object?> map) => {
+  for (final MapEntry(:key, :value) in map.entries)
+    if (value != null && !(value is List && value.isEmpty)) key: value,
+};
+
+List<Map<String, Object?>> _maps(Object? json) => switch (json) {
+  final List<Object?> list => [
+    for (final item in list)
+      if (item is Map<String, Object?>) item,
+  ],
+  _ => const [],
+};
+
+List<String> _strings(Object? json) => switch (json) {
+  final List<Object?> list => [
+    for (final item in list)
+      if (item is String) item,
+  ],
+  _ => const [],
+};

halkyon_learn_curator/lib/src/update.dart

@@ -4,6 +4,8 @@ import 'dart:io';
 import 'package:halkyon_learn_curator/src/sources/jlpt.dart';
 import 'package:halkyon_learn_curator/src/sources/jmdict_xml.dart';
 import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
+import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
+import 'package:halkyon_learn_curator/src/wordfreq.dart';
 
 /// JMdict in English with Tatoeba's sentences, as EDRDG makes it every day.
 final jmdictSource = Uri.parse(
@@ -137,3 +139,131 @@ DateTime? lastUpdated(Directory folder) {
   }
   return oldest;
 }
+
+/// Wiktionary in English as Wiktextract reads it, every language in one
+/// file, made anew about monthly (kaikki.org).
+final wiktionarySource = Uri.parse(
+  'https://kaikki.org/dictionary/raw-wiktextract-data.jsonl.gz',
+);
+
+/// How the curator converts Wiktionary now; raise it as [conversion].
+const wiktionaryConversion = 1;
+
+/// The file in the dictionaries' folder that says which dump, languages
+/// and [wiktionaryConversion] made the Wiktionary files.
+const wiktionaryStateFile = '.wiktionary';
+
+/// A download as it begins: when the file was last changed, and its bytes.
+typedef Download = ({DateTime modified, Stream<List<int>> bytes});
+
+/// When the file at a URI was last changed, without downloading it.
+typedef Modified = Future<DateTime> Function(Uri uri);
+
+/// Starts downloading the file at a URI.
+typedef Open = Future<Download> Function(Uri uri);
+
+HttpClient _client() => HttpClient()
+  ..connectionTimeout = const Duration(seconds: 30)
+  ..userAgent = 'halkyon_learn_curator (dictionary update)';
+
+/// When [uri] last changed, as its `Last-Modified` says; throws for any
+/// answer but 200 or without the date.
+Future<DateTime> httpModified(Uri uri) async {
+  final client = _client();
+  try {
+    final response = await (await client.headUrl(uri)).close();
+    await response.drain<void>();
+    return _modifiedOf(uri, response);
+  } finally {
+    client.close();
+  }
+}
+
+/// Downloads [uri] as a stream; throws for any answer but 200 or without
+/// a date.
+Future<Download> httpOpen(Uri uri) async {
+  final client = _client();
+  try {
+    final response = await (await client.getUrl(uri)).close();
+    return (modified: _modifiedOf(uri, response), bytes: response);
+  } finally {
+    // Closes once the download is read.
+    client.close();
+  }
+}
+
+DateTime _modifiedOf(Uri uri, HttpClientResponse response) {
+  if (response.statusCode != HttpStatus.ok) {
+    throw HttpException('$uri answered ${response.statusCode}.', uri: uri);
+  }
+  final modified = response.headers.value(HttpHeaders.lastModifiedHeader);
+  if (modified == null) {
+    throw HttpException('$uri says no date.', uri: uri);
+  }
+  return HttpDate.parse(modified);
+}
+
+/// The dictionary files [languages] (`de`, `el`, `en`) are learnt with from
+/// Wiktionary: each language with English meanings, and English with
+/// German ones, German being the app's other language.
+({Set<String> languages, Set<String> translatedInto}) wiktionaryOutputs(
+  Set<String> languages,
+) => (
+  languages: languages.difference({'en'}),
+  translatedInto: languages.contains('en') ? {'de'} : const {},
+);
+
+/// Brings the Wiktionary files for [languages] in [into] up to date: only
+/// when the dump changed since they were made, or the languages, or how
+/// the curator converts it ([wiktionaryStateFile]). The dump is converted
+/// as it downloads, never kept (some 3 GB); each file is written beside
+/// its place and renamed over it. The word frequencies are in the
+/// curator's [data] folder (`wordfreq/`). Returns the files written, none
+/// when they were up to date.
+Future<List<File>> updateWiktionary({
+  required Directory into,
+  required Directory data,
+  required Set<String> languages,
+  Modified modified = httpModified,
+  Open open = httpOpen,
+}) async {
+  if (languages.isEmpty) {
+    return const [];
+  }
+  final state = File('${into.path}/$wiktionaryStateFile');
+  final dumped = await modified(wiktionarySource);
+  final wanted = _wiktionaryState(dumped, languages);
+  if (state.existsSync() && state.readAsStringSync() == wanted) {
+    return const [];
+  }
+  final download = await open(wiktionarySource);
+  final outputs = wiktionaryOutputs(languages);
+  final frequencies = {
+    for (final language in {...languages, 'en'})
+      if (File('${data.path}/wordfreq/small_$language.msgpack.gz')
+          case final file when file.existsSync())
+        language: await WordFrequencies.read(file),
+  };
+  final written = await wiktionaryDictionaries(
+    download.bytes
+        .transform(gzip.decoder)
+        .transform(utf8.decoder)
+        .transform(const LineSplitter()),
+    into: into,
+    sourceVersion: download.modified.toIso8601String().substring(0, 10),
+    languages: outputs.languages,
+    translatedInto: outputs.translatedInto,
+    frequencies: frequencies,
+  );
+  await state.writeAsString(_wiktionaryState(download.modified, languages));
+  return written;
+}
+
+/// What [wiktionaryStateFile] says of files made from the dump of
+/// [dumped] for [languages].
+String _wiktionaryState(DateTime dumped, Set<String> languages) =>
+    '${jsonEncode({
+      'conversion': wiktionaryConversion,
+      'dump': dumped.toUtc().toIso8601String(),
+      'languages': [...languages]..sort(),
+    })}\n';

halkyon_learn_curator/lib/src/wordfreq.dart

@@ -0,0 +1,120 @@
+import 'dart:convert';
+import 'dart:io';
+import 'dart:typed_data';
+
+import 'package:halkyon_learn_curator/src/dictionary_file.dart';
+import 'package:halkyon_learn_curator/src/fold.dart';
+
+/// Who made the word frequencies, and what they are drawn from (wordfreq's
+/// licence asks that the SUBTLEX authors be credited).
+const wordfreqCredits = [
+  Credit(
+    name: 'wordfreq',
+    by: 'Robyn Speer',
+    license: 'CC-BY-SA-4.0',
+    url: 'https://github.com/rspeer/wordfreq',
+  ),
+  Credit(
+    name: 'SUBTLEX',
+    by: 'Marc Brysbaert et al.',
+    license: 'CC-BY-SA-4.0',
+    url: 'http://crr.ugent.be/programs-data/subtitle-frequencies',
+  ),
+];
+
+/// How often words are used in a language, as wordfreq's lists say: each
+/// word's place among them, the most used first. Words compare case-folded
+/// ([casefold]), as the lists are made.
+final class WordFrequencies(final Map<String, int> _places) {
+  /// No word known.
+  static final none = WordFrequencies(const {});
+
+  /// wordfreq's list [file] (`small_de.msgpack.gz`): msgpack of a header
+  /// and then a list of words for each frequency, the most used first,
+  /// each a tenth of a decibel (a centibel) rarer than the one before.
+  static Future<WordFrequencies> read(File file) async {
+    final bytes = gzip.decode(await file.readAsBytes());
+    final lists = switch (_Msgpack(Uint8List.fromList(bytes)).read()) {
+      [{'format': 'cB', 'version': 1}, ...final lists] => lists,
+      _ => throw FormatException('Not a list of wordfreq ($file).'),
+    };
+    final places = <String, int>{};
+    for (final words in lists) {
+      for (final word in (words! as List).cast<String>()) {
+        places.putIfAbsent(casefold(word), () => places.length);
+      }
+    }
+    return WordFrequencies(places);
+  }
+
+  /// [word]'s place, the most used word 0; null for a word the list lacks.
+  int? placeOf(String word) => _places[casefold(word)];
+
+  /// How many words the list holds.
+  int get length => _places.length;
+}
+
+/// Just enough of msgpack (https://msgpack.org) to read wordfreq's lists:
+/// lists, maps, strings, whole numbers, nil and booleans.
+final class _Msgpack(final Uint8List _bytes) {
+  late final _data = ByteData.sublistView(_bytes);
+  var _at = 0;
+
+  Object? read() {
+    final byte = _bytes[_at++];
+    if (byte <= 0x7f) {
+      return byte;
+    }
+    if (byte >= 0xe0) {
+      return byte - 0x100;
+    }
+    if (byte & 0xf0 == 0x80) {
+      return _map(byte & 0x0f);
+    }
+    if (byte & 0xf0 == 0x90) {
+      return _list(byte & 0x0f);
+    }
+    if (byte & 0xe0 == 0xa0) {
+      return _string(byte & 0x1f);
+    }
+    return switch (byte) {
+      0xc0 => null,
+      0xc2 => false,
+      0xc3 => true,
+      0xcc => _bytes[_at++],
+      0xcd => _uint(2),
+      0xce => _uint(4),
+      0xd9 => _string(_bytes[_at++]),
+      0xda => _string(_uint(2)),
+      0xdb => _string(_uint(4)),
+      0xdc => _list(_uint(2)),
+      0xdd => _list(_uint(4)),
+      0xde => _map(_uint(2)),
+      0xdf => _map(_uint(4)),
+      _ => throw FormatException(
+        'msgpack type 0x${byte.toRadixString(16)} not read here.',
+      ),
+    };
+  }
+
+  int _uint(int length) {
+    final value = switch (length) {
+      2 => _data.getUint16(_at),
+      _ => _data.getUint32(_at),
+    };
+    _at += length;
+    return value;
+  }
+
+  String _string(int length) {
+    final text = utf8.decode(Uint8List.sublistView(_bytes, _at, _at + length));
+    _at += length;
+    return text;
+  }
+
+  List<Object?> _list(int length) => [for (var i = 0; i < length; i++) read()];
+
+  Map<Object?, Object?> _map(int length) => {
+    for (var i = 0; i < length; i++) read(): read(),
+  };
+}

halkyon_learn_curator/test/fixtures/README.md

@@ -29,3 +29,13 @@ of 2026-10-09 (JMdict with Tatoeba's sentences, CC BY 2.0 FR) and the
 declarations of the entities they use: JMdict is © the Electronic Dictionary
 Research and Development Group, used under its licence, CC BY-SA 4.0
 (<https://www.edrdg.org/edrdg/licence.html>).
+
+`wiktionary-sample.jsonl` holds 25 lines of kaikki.org's Wiktextract data of the
+English Wiktionary (the dump of 2026-10-03, read per word from kaikki.org): the
+German Haus, Häuser, Häusern, essen, aß, gegessen and Straße, the Greek σπίτι,
+σπιτιού and γράφω, the English house, houses, go and went, and the French
+maison, each shortened to the fields the curator reads (no categories,
+etymologies or related words; at most six senses, two examples, 40 forms, two
+sounds, and 14 translations, those into German and Greek). Wiktionary's text is
+by its contributors, CC BY-SA 4.0 (<https://en.wiktionary.org/wiki/Wiktionary:Copyrights>);
+the extraction is Tatu Ylonen's Wiktextract (<https://kaikki.org>).

halkyon_learn_curator/test/update_test.dart

@@ -97,4 +97,85 @@ void main() {
   test('a folder missing a file was never brought up to date', () {
     expect(lastUpdated(into), isNull);
   });
+
+  group("Wiktionary's dump", () {
+    var dumped = DateTime.utc(2026, 10, 3, 8, 24);
+    var opened = 0;
+
+    setUp(() {
+      dumped = DateTime.utc(2026, 10, 3, 8, 24);
+      opened = 0;
+    });
+
+    Future<DateTime> modified(Uri uri) async => dumped;
+
+    /// The sample in place of kaikki.org's dump.
+    Future<Download> open(Uri uri) async {
+      opened++;
+      return (
+        modified: dumped,
+        bytes: Stream.value(
+          gzip.encode(
+            File('test/fixtures/wiktionary-sample.jsonl').readAsBytesSync(),
+          ),
+        ),
+      );
+    }
+
+    Future<List<File>> update(Set<String> languages) => updateWiktionary(
+      into: into,
+      data: Directory('data'),
+      languages: languages,
+      modified: modified,
+      open: open,
+    );
+
+    test(
+      'is converted for the languages named, German and Greek with English '
+      'meanings, English with German ones, and ranked by frequency',
+      () async {
+        final written = await update({'de', 'el', 'en'});
+
+        expect(
+          [for (final file in written) file.uri.pathSegments.last]..sort(),
+          [
+            'wiktionary-de-en.jsonl.gz',
+            'wiktionary-el-en.jsonl.gz',
+            'wiktionary-en-de.jsonl.gz',
+          ],
+        );
+        expect(
+          header('wiktionary-de-en.jsonl.gz')['sourceVersion'],
+          '2026-10-03',
+        );
+        expect([
+          for (final credit
+              in header('wiktionary-de-en.jsonl.gz')['credits']! as List)
+            (credit as Map)['name'],
+        ], contains('wordfreq'));
+        expect(File('${into.path}/$wiktionaryStateFile').existsSync(), isTrue);
+      },
+    );
+
+    test(
+      'is downloaded again only once it changed, or the languages did',
+      () async {
+        await update({'de'});
+        expect(await update({'de'}), isEmpty);
+        expect(opened, 1);
+
+        dumped = DateTime.utc(2026, 11, 2);
+        expect(await update({'de'}), hasLength(1));
+        expect(opened, 2);
+
+        expect(await update({'de', 'el'}), hasLength(2));
+        expect(opened, 3);
+      },
+    );
+
+    test('nothing is asked for without languages', () async {
+      expect(await update({}), isEmpty);
+      expect(opened, 0);
+    });
+  });
 }

halkyon_learn_curator/test/wiktionary_test.dart

@@ -0,0 +1,247 @@
+import 'dart:convert';
+import 'dart:io';
+
+import 'package:halkyon_learn_curator/src/fold.dart';
+import 'package:halkyon_learn_curator/src/sources/wiktionary.dart';
+import 'package:halkyon_learn_curator/src/wordfreq.dart';
+import 'package:test/test.dart';
+
+void main() {
+  late Directory into;
+
+  setUp(() => into = Directory.systemTemp.createTempSync('wiktionary'));
+  tearDown(() => into.deleteSync(recursive: true));
+
+  /// The sample's lines, as the dump's.
+  Stream<String> sample() =>
+      File('test/fixtures/wiktionary-sample.jsonl')
+          .openRead()
+          .transform(utf8.decoder)
+          .transform(const LineSplitter());
+
+  /// The lines of the dictionary file [name] in [into], decoded.
+  List<Map<String, Object?>> linesOf(String name) => [
+    for (final line
+        in utf8
+            .decode(gzip.decode(File('${into.path}/$name').readAsBytesSync()))
+            .split('\n'))
+      if (line.isNotEmpty) jsonDecode(line) as Map<String, Object?>,
+  ];
+
+  Map<String, Object?> entryOf(String name, String id) =>
+      linesOf(name).skip(1).singleWhere((line) => line['id'] == id);
+
+  Future<List<File>> convert({
+    Set<String> languages = const {'de', 'el'},
+    Set<String> translatedInto = const {'de'},
+    Map<String, WordFrequencies> frequencies = const {},
+  }) => wiktionaryDictionaries(
+    sample(),
+    into: into,
+    sourceVersion: '2026-10-03',
+    languages: languages,
+    translatedInto: translatedInto,
+    frequencies: frequencies,
+  );
+
+  test('writes a file for each language, and for English with what its '
+      'translation tables give', () async {
+    final written = await convert();
+
+    expect([for (final file in written) file.uri.pathSegments.last]..sort(), [
+      'wiktionary-de-en.jsonl.gz',
+      'wiktionary-el-en.jsonl.gz',
+      'wiktionary-en-de.jsonl.gz',
+    ]);
+    final header = linesOf('wiktionary-de-en.jsonl.gz').first;
+    expect(header['source'], 'wiktionary-de');
+    expect(header['language'], 'de');
+    expect(header['glossLanguage'], 'en');
+    expect(header['sourceVersion'], '2026-10-03');
+    expect(header['license'], 'CC-BY-SA-4.0');
+    expect(
+      [
+        for (final credit in header['credits']! as List)
+          (credit as Map)['name'],
+      ],
+      ['Wiktionary', 'Wiktextract (kaikki.org)'],
+    );
+    expect(
+      linesOf('wiktionary-en-de.jsonl.gz').first['source'],
+      'wiktionary-en',
+    );
+    // Nothing half-written left behind.
+    expect(
+      [for (final entity in into.listSync()) entity.uri.pathSegments.last]
+        ..sort(),
+      [
+        'wiktionary-de-en.jsonl.gz',
+        'wiktionary-el-en.jsonl.gz',
+        'wiktionary-en-de.jsonl.gz',
+      ],
+    );
+  });
+
+  test('a German noun: its gender, its plural beside it, its meanings with '
+      'their examples, every form finding it in text', () async {
+    await convert();
+
+    final haus = entryOf('wiktionary-de-en.jsonl.gz', 'Haus/noun');
+    expect(haus['headword'], 'Haus');
+    expect(haus.containsKey('reading'), isFalse);
+    final entry = haus['entry']! as Map<String, Object?>;
+    expect(entry['pos'], 'noun');
+    expect(entry['genders'], ['n']);
+    expect(entry['ipa'], '[haʊ̯s]');
+    expect(
+      [for (final form in entry['inflections']! as List) (form as Map)['text']],
+      ['Hauses', 'Häuser'],
+    );
+    final senses = (entry['senses']! as List).cast<Map<String, Object?>>();
+    expect(senses.first['glosses'], ['house, building']);
+    expect((senses.first['examples']! as List).first, {
+      'text': 'Haus',
+      'de': 'In dem Haus haben wir mal gewohnt.',
+      'en': 'We used to live in that house.',
+    });
+    // Its own forms, and those Wiktionary has entries for alone.
+    final textKeys = (haus['textKeys']! as List).cast<String>();
+    expect(textKeys, containsAll(['haus', 'hauses', 'häuser', 'häusern']));
+    final keys = (haus['keys']! as List).cast<String>();
+    expect(keys, containsAll(['haus', 'häusern', 'house, building', 'house']));
+    // Typed without umlauts.
+    expect(
+      (haus['looseKeys']! as List).cast<String>(),
+      containsAll(['hauser', 'haeuser']),
+    );
+  });
+
+  test(
+    'forms, names and other languages are no entries of their own',
+    () async {
+      await convert();
+
+      final ids = [
+        for (final line in linesOf('wiktionary-de-en.jsonl.gz').skip(1))
+          line['id'],
+      ];
+      // Häuser, Häusern, aß and gegessen are forms; Haus and Häusern are also
+      // places, which are names.
+      expect(ids, containsAll(['Haus/noun', 'essen/verb', 'Straße/noun']));
+      expect(ids, isNot(contains('Häusern/noun')));
+      expect(ids, isNot(contains('aß/verb')));
+      expect(ids, isNot(contains('Haus/name')));
+      expect([
+        for (final line in linesOf('wiktionary-de-en.jsonl.gz').skip(1))
+          line['headword'],
+      ], isNot(contains('maison')));
+      final essen = entryOf('wiktionary-de-en.jsonl.gz', 'essen/verb');
+      expect(
+        (essen['textKeys']! as List).cast<String>(),
+        containsAll(['isst', 'aß', 'gegessen']),
+      );
+      // Typed with `ss`.
+      expect(
+        (entryOf('wiktionary-de-en.jsonl.gz', 'Straße/noun')['looseKeys']!
+                as List)
+            .cast<String>(),
+        contains('strasse'),
+      );
+    },
+  );
+
+  test(
+    'a Greek word: found by its romanisation and without its accents',
+    () async {
+      await convert();
+
+      final spiti = entryOf('wiktionary-el-en.jsonl.gz', 'σπίτι/noun');
+      final entry = spiti['entry']! as Map<String, Object?>;
+      expect(entry['romanization'], 'spíti');
+      expect(entry['genders'], ['n']);
+      expect(
+        (spiti['textKeys']! as List).cast<String>(),
+        containsAll(['σπίτι', 'σπιτιού']),
+      );
+      expect((spiti['keys']! as List).cast<String>(), contains('spíti'));
+    // The articles its table gives cells of their own are no forms of it.
+    expect(
+      (spiti['textKeys']! as List).cast<String>(),
+      isNot(anyOf(contains('το'), contains('του'), contains('τα'))),
+    );
+      expect(
+        (spiti['looseKeys']! as List).cast<String>(),
+        containsAll(['σπιτι', 'σπιτιου']),
+      );
+    },
+  );
+
+  test('an English word with its German translations, a meaning each, found '
+      'by its forms and by the German words', () async {
+    await convert();
+
+    final house = entryOf('wiktionary-en-de.jsonl.gz', 'house/noun/1');
+    final entry = house['entry']! as Map<String, Object?>;
+    final senses = (entry['senses']! as List).cast<Map<String, Object?>>();
+    expect(senses.first, {
+      'glosses': ['Haus'],
+      'info': ['human abode'],
+    });
+    // Regional words (Alemannic German) left out.
+    expect([
+      for (final sense in senses) ...sense['glosses']! as List,
+    ], isNot(contains('Huus')));
+    expect(
+      (house['textKeys']! as List).cast<String>(),
+      containsAll(['house', 'houses']),
+    );
+    expect((house['keys']! as List).cast<String>(), contains('haus'));
+    final go = entryOf('wiktionary-en-de.jsonl.gz', 'go/verb/1');
+    expect(
+      (go['textKeys']! as List).cast<String>(),
+      containsAll(['goes', 'went', 'gone']),
+    );
+  });
+
+  test('frequencies rank the words, the most used first, and say which are '
+      'common', () async {
+    final german = await WordFrequencies.read(
+      File('data/wordfreq/small_de.msgpack.gz'),
+    );
+    await convert(languages: {'de'}, frequencies: {'de': german});
+
+    final header = linesOf('wiktionary-de-en.jsonl.gz').first;
+    expect(
+      [
+        for (final credit in header['credits']! as List)
+          (credit as Map)['name'],
+      ],
+      ['Wiktionary', 'Wiktextract (kaikki.org)', 'wordfreq', 'SUBTLEX'],
+    );
+    final ranked = [
+      for (final line in linesOf('wiktionary-de-en.jsonl.gz').skip(1))
+        (line['headword'], line['rank']! as int, line['labels']),
+    ]..sort((a, b) => a.$2.compareTo(b.$2));
+    expect(ranked.first.$1, 'Haus');
+    expect(ranked.first.$3, ['common']);
+  });
+
+  group('folding', () {
+    test('case-folds as Unicode does', () {
+      expect(casefold('Straße'), 'strasse');
+      expect(casefold('ΣΠΊΤΙΣ'), 'σπίτισ');
+    });
+
+    test('drops accents of Latin and Greek letters', () {
+      expect(withoutAccents('Häuser'), 'hauser');
+      expect(withoutAccents('σπιτιού'), 'σπιτιου');
+      expect(withoutAccents('café'), 'cafe');
+      expect(withoutAccents('学習'), '学習');
+    });
+
+    test('writes umlauts out', () {
+      expect(umlautsWrittenOut('Häuser'), 'haeuser');
+      expect(umlautsWrittenOut('Haus'), isNull);
+    });
+  });
+}