fix(curator): HSK levels by the syllabus's tones; variants find their word

The syllabus's reading is matched with its tones first, so 和 hé gets
HSK 1 and 和 hè doesn't; only where no entry of a word reads it so (the
syllabus marks the tone changes of 一 and 不, 一下 yíxià, CC-CEDICT
doesn't) are the letters alone compared. CC-CEDICT's 2,885 entries that
are only "variant of …" (咊, an old variant of 和) are no entries of
their own any more, which repeated their word in the simplified file,
but written forms of that word, finding it in text and searches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

halkyon_learn_curator/lib/src/sources/cedict.dart

@@ -27,15 +27,32 @@ const hskCredit = Credit(
 String cedictFileName({required bool traditional}) =>
     'cc-cedict-${traditional ? 'hant' : 'hans'}-en.jsonl.gz';
 
-/// HSK levels by word (simplified) and pinyin without tones: 1 to 6, and 7
-/// for the advanced levels 7 to 9, which the syllabus doesn't tell apart.
-typedef HskLevels = Map<(String, String), int>;
+/// HSK levels by word (simplified) and its pinyin, with its tones (`toned`,
+/// marks, no spaces: `xuéxí`) and without (`toneless`: `xuexi`): 1 to 6,
+/// and 7 for the advanced levels 7 to 9, which the syllabus doesn't tell
+/// apart.
+typedef HskLevels = ({
+  Map<(String, String), int> toned,
+  Map<(String, String), int> toneless,
+});
+
+/// No HSK levels.
+const noHskLevels = (
+  toned: <(String, String), int>{},
+  toneless: <(String, String), int>{},
+);
+
+/// [pinyin] with tone marks, as a key: lower case, letters and marks
+/// alone (`xué xí`, `xuéxí` → `xuéxí`).
+String _tonedKey(String pinyin) =>
+    pinyin.toLowerCase().replaceAll(RegExp(r"[\s\-'’·]"), '');
 
 /// The HSK levels of the 2025 examination syllabus, as `hsk30` extracted
 /// them ([file]: word, level, another level of a homograph, pinyin, part of
 /// speech). A word graded twice keeps its lower level.
 Future<HskLevels> readHskLevels(File file) async {
-  final levels = <(String, String), int>{};
+  final toned = <(String, String), int>{};
+  final toneless = <(String, String), int>{};
   for (final line in await file.readAsLines()) {
     final [word, level, ..., pinyin, _] = switch (line.split('\t')) {
       final fields when fields.length >= 5 => fields,
@@ -45,11 +62,15 @@ Future<HskLevels> readHskLevels(File file) async {
     if (number == null || word.isEmpty) {
       continue;
     }
-    final key = (word, tonelessPinyin(pinyin));
+    for (final (levels, key) in [
+      (toned, (word, _tonedKey(pinyin))),
+      (toneless, (word, tonelessPinyin(pinyin))),
+    ]) {
       final known = levels[key];
       levels[key] = known == null || number < known ? number : known;
     }
-  return levels;
+  }
+  return (toned: toned, toneless: toneless);
 }
 
 /// CC-CEDICT ([text], its file as MDBG publishes it) as the lines of a
@@ -63,7 +84,7 @@ Future<HskLevels> readHskLevels(File file) async {
 Iterable<String> cedictDictionary(
   String text, {
   required bool traditional,
-  HskLevels hsk = const {},
+  HskLevels hsk = noHskLevels,
   WordFrequencies? frequencies,
 }) sync* {
   final version = RegExp(r'^#! date=(\S+)', multiLine: true).firstMatch(text);
@@ -76,7 +97,7 @@ Iterable<String> cedictDictionary(
       license: 'CC-BY-SA-4.0',
       credits: [
         cedictCredit,
-        if (hsk.isNotEmpty) hskCredit,
+        if (hsk.toned.isNotEmpty) hskCredit,
         if (frequencies != null && frequencies.length > 0) ...wordfreqCredits,
       ],
     ),
@@ -94,7 +115,45 @@ Iterable<String> cedictDictionary(
       if (match[3]!.toLowerCase() == match[3])
         (match[2]!, tonelessPinyin(match[3]!)),
   };
+  // Words of the syllabus some entry reads with its tones: only those
+  // entries get the level (和 hé, not 和 hè); the others' by their letters,
+  // as the syllabus marks tone changes (一下 yíxià) CC-CEDICT doesn't.
+  final readWithTones = {
+    for (final match in matches)
+      if (hsk.toned.containsKey((
+        match[2]!,
+        _tonedKey(pinyinWithToneMarks(match[3]!)),
+      )))
+        match[2]!,
+  };
+  // Entries that are only variants of another (咊, an old variant of 和
+  // hé): no entries of their own, but written forms finding that one.
+  final variantsOf = <(String, String), Set<String>>{};
+  final variantsOnly = <RegExpMatch>{};
   for (final match in matches) {
+    final definitions = [
+      for (final definition in match[4]!.split('/'))
+        if (definition.trim().isNotEmpty) definition.trim(),
+    ];
+    final targets = [
+      for (final definition in definitions)
+        ?_variantTarget.firstMatch(definition),
+    ];
+    if (definitions.isEmpty || targets.length != definitions.length) {
+      continue;
+    }
+    variantsOnly.add(match);
+    for (final target in targets) {
+      final written = target[1]!.split('|');
+      variantsOf
+          .putIfAbsent((written.last, target[2]!.toLowerCase()), () => {})
+          .addAll({match[1]!, match[2]!});
+    }
+  }
+  for (final match in matches) {
+    if (variantsOnly.contains(match)) {
+      continue;
+    }
     final named = match[3]!.toLowerCase() != match[3];
     final key = (match[2]!, tonelessPinyin(match[3]!));
     yield entryLine(
@@ -104,28 +163,44 @@ Iterable<String> cedictDictionary(
         pinyin: match[3]!,
         definitions: match[4]!.split('/'),
         traditional: traditional,
-        hsk: named && common.contains(key) ? const {} : hsk,
+        hsk: named && common.contains(key) ? noHskLevels : hsk,
+        byTones: readWithTones.contains(match[2]),
         frequencies: frequencies,
+        variants: variantsOf[(match[2]!, match[3]!.toLowerCase())] ?? const {},
       ),
     );
   }
 }
 
-/// One line of CC-CEDICT as a dictionary entry.
+/// A meaning that only says a word is a variant of another, as
+/// `variant of 和[he2]` and `old variant of 個|个[ge4]` (the target in
+/// traditional|simplified characters).
+final _variantTarget = RegExp(
+  r'^(?:old |archaic |Japanese )?variant of (\S+?)\[([^\]]+)\]$',
+);
+
+/// One line of CC-CEDICT as a dictionary entry; [variants], written forms
+/// of entries that are only its variants, find it in text and searches.
 DictionaryEntry cedictEntry({
   required String traditionalForm,
   required String simplifiedForm,
   required String pinyin,
   required List<String> definitions,
   required bool traditional,
-  HskLevels hsk = const {},
+  HskLevels hsk = noHskLevels,
   WordFrequencies? frequencies,
+
+  /// Whether the syllabus's reading of the word is matched with its tones.
+  bool byTones = false,
+  Set<String> variants = const {},
 }) {
   final headword = traditional ? traditionalForm : simplifiedForm;
   final other = traditional ? simplifiedForm : traditionalForm;
   final marked = pinyinWithToneMarks(pinyin);
   final toneless = tonelessPinyin(pinyin);
-  final level = hsk[(simplifiedForm, toneless)];
+  final level = byTones
+      ? hsk.toned[(simplifiedForm, _tonedKey(marked))]
+      : hsk.toneless[(simplifiedForm, toneless)];
   final place = frequencies?.placeOf(simplifiedForm);
   // Measure words are a remark on the word, not a meaning.
   final classifiers = [
@@ -155,6 +230,7 @@ DictionaryEntry cedictEntry({
     keys: {
       headword,
       other,
+      ...variants,
       marked.replaceAll(' ', '').toLowerCase(),
       toneless,
       pinyin.replaceAll(' ', '').toLowerCase(),
@@ -162,7 +238,7 @@ DictionaryEntry cedictEntry({
         for (final gloss in sense.split(';')) ...glossKeys(gloss),
     }.toList(),
     looseKeys: [marked.toLowerCase(), pinyin.toLowerCase()],
-    textKeys: {headword, other}.toList(),
+    textKeys: {headword, other, ...variants}.toList(),
     entry: {
       if (other != headword)
         'forms': [

halkyon_learn_curator/test/cedict_test.dart

@@ -82,6 +82,9 @@ void main() {
       'de, not dì; 书 shū, not the Book of Documents, Shū', () {
     expect(entryOf('的 的 [de5]')['labels'], contains('hsk:1'));
     expect(entryOf('的 的 [di4]')['labels'], isNot(contains('hsk:1')));
+    // Read with its tones: 说 shuō, not shuì.
+    expect(entryOf('說 说 [shuo1]')['labels'], contains('hsk:1'));
+    expect(entryOf('說 说 [shui4]')['labels'], isEmpty);
     expect(entryOf('書 书 [shu1]')['labels'], contains('hsk:1'));
     expect(entryOf('書 书 [Shu1]')['labels'], isEmpty);
     expect(entryOf('中國 中国 [Zhong1 guo2]')['labels'], contains('hsk:1'));
@@ -97,6 +100,18 @@ void main() {
     );
   });
 
+  test('an entry that is only a variant of another is no entry, but finds '
+      'that one', () {
+    final ids = [
+      for (final line in linesOf(traditional: false).skip(1)) line['id'],
+    ];
+    expect(ids, isNot(contains('咊 和 [he2]')));
+    expect(
+      (entryOf('和 和 [he2]')['textKeys']! as List).cast<String>(),
+      contains('咊'),
+    );
+  });
+
   test('measure words are a remark beside the meanings, pinyin marked', () {
     final book = entryOf('書 书 [shu1]')['entry']! as Map;
     expect(book['classifiers'], ['本 běn', '冊|册 cè', '部 bù']);

halkyon_learn_curator/test/fixtures/README.md

@@ -40,7 +40,7 @@ sounds, and 14 translations, those into German and Greek). Wiktionary's text is
 by its contributors, CC BY-SA 4.0 (<https://en.wiktionary.org/wiki/Wiktionary:Copyrights>);
 the extraction is Tatu Ylonen's Wiktextract (<https://kaikki.org>).
 
-`cedict-sample.txt` holds CC-CEDICT's header and 16 of its lines (MDBG's file of
-2026-10-13): 中国, 了, 女儿, 学习, 书, 熊猫, 略, 的, 说 and 银行, with their
-readings. CC-CEDICT is by MDBG and its contributors, CC BY-SA 4.0
+`cedict-sample.txt` holds CC-CEDICT's header and 23 of its lines (MDBG's file of
+2026-10-13): 中国, 了, 女儿, 学习, 书, 熊猫, 略, 的, 说, 银行, 和 and 咊, an old
+variant of 和, with their readings. CC-CEDICT is by MDBG and its contributors, CC BY-SA 4.0
 (<https://www.mdbg.net/chinese/dictionary?page=cc-cedict>).

halkyon_learn_curator/test/fixtures/cedict-sample.txt

@@ -44,3 +44,10 @@
 說 说 [shui4] /to persuade/
 說 说 [shuo1] /to speak; to talk; to say/to explain; to comment/to scold; to tell off/(bound form) theory; doctrine/
 銀行 银行 [yin2 hang2] /bank/CL:家[jia1],個|个[ge4]/
+咊 和 [he2] /old variant of 和[he2]/
+和 和 [He2] /surname He/
+和 和 [he2] /(joining two nouns) and; together with; with (Taiwan pr. [han4])/(math.) sum/to make peace/(sports) to draw; to tie/(bound form) harmonious/(bound form) Japan; Japanese/
+和 和 [he4] /to compose a poem in reply (to sb's poem) using the same rhyme sequence/to join in the singing/to chime in with others/
+和 和 [hu2] /to complete a set in mahjong or playing cards/
+和 和 [huo2] /to combine a powdery substance (flour, plaster etc) with water/Taiwan pr. [huo4]/
+和 和 [huo4] /to mix (ingredients) together/to blend/classifier for rinses of clothes/classifier for boilings of medicinal herbs/