feat(curator): KANJIDIC2 and the kana as a dictionary of characters; KanjiVG's strokes for the app
A dictionary file now says whether it holds words or characters (`kind`). The kanjidic adapter turns KANJIDIC2 into one entry per kanji that has a Japanese reading or an English meaning (12,633), with its readings, meanings, school grade, strokes, frequency and JLPT level (Jonathan Waller's, through kanji-data's kanji.json), labelled `jlpt:nX`, `grade:N`/`grade:secondary`, followed by the 71 hiragana and katakana of the kana table with their romaji. A kun reading is found whole; its kanji part only as a loose key, so typing "ta" brings た before every kanji read so. `curate kanjivg` writes KanjiVG's strokes and components as the app's stroke files, one gzipped JSON file per 256 code points, keyed by the code point in each entry's id (KanjiVG's 橅 names itself 機). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
42c57bd40a
commit
e42cdfbfcb
GPG key ID:
07F6FCF405488B64
12 changed files with 1036 additions and 29 deletions
|
|
@ -4,6 +4,7 @@ Turns dictionary downloads into the dictionary files a Halkyon Learn server
|
|||
loads. It reads each source with an adapter of its own and writes one format
|
||||
for all of them, so a new source or language needs a new adapter, never a
|
||||
change to the server or the app. It doesn't depend on Serverpod or the server.
|
||||
It also makes the stroke files the app writes characters with (below).
|
||||
|
||||
## Making a dictionary file
|
||||
|
||||
|
|
@ -27,8 +28,25 @@ JMdict in English, with the JLPT levels and example sentences on top:
|
|||
The whole of JMdict (about 219,000 entries) takes some 20 seconds and
|
||||
300 MB of memory: entries are read one at a time.
|
||||
|
||||
Put the file into the server's `dictionaries` folder. The server loads it on
|
||||
its next start, and again whenever the file changes (see the server's README).
|
||||
KANJIDIC2 in English, with the JLPT levels of its kanji, and the kana:
|
||||
|
||||
1. Download `kanjidic2.xml.gz` from
|
||||
[the EDRDG](https://www.edrdg.org/wiki/index.php/KANJIDIC_Project).
|
||||
2. Download `kanji.json` from
|
||||
[kanji-data](https://github.com/davidluzgouveia/kanji-data): Jonathan
|
||||
Waller's JLPT levels by kanji (`jlpt_new`; nothing else is read).
|
||||
3. Convert, in this package:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate kanjidic \
|
||||
--input kanjidic2.xml.gz --jlpt kanji.json \
|
||||
--output kanjidic-en.jsonl.gz
|
||||
```
|
||||
|
||||
About 12,800 characters, in some 5 seconds.
|
||||
|
||||
Put the files into the server's `dictionaries` folder. The server loads each on
|
||||
its next start, and again whenever it changes (see the server's README).
|
||||
|
||||
## The format (`halkyon-dictionary`, version 1)
|
||||
|
||||
|
|
@ -39,6 +57,7 @@ The **first line** says what the file holds:
|
|||
| Field | |
|
||||
|---|---|
|
||||
| `format`, `formatVersion` | `"halkyon-dictionary"`, `1` |
|
||||
| `kind` | `words` (a dictionary of words, JMdict) or `characters` (of the characters words are written in, KANJIDIC2). The app shows them apart. |
|
||||
| `source`, `sourceVersion` | The source's name (`jmdict`) and version (`3.6.2 (2026-10-05)`). An instance holds one version of a source for each meaning language. |
|
||||
| `language`, `glossLanguage` | BCP 47 tags: the words' language (`ja`) and the meanings' (`en`). |
|
||||
| `license` | The data's licence, an SPDX identifier (`CC-BY-SA-4.0`). |
|
||||
|
|
@ -50,11 +69,33 @@ The **first line** says what the file holds:
|
|||
|---|---|
|
||||
| `id` | Its id in the source; with the source and meaning language, it names the entry for good. |
|
||||
| `headword`, `reading` | How the word is shown, and how it is read; no `reading` when the headword is its own (a word written in kana). |
|
||||
| `labels` | What it can be picked by: `common`, `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`). |
|
||||
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest. |
|
||||
| `labels` | What it can be picked by: `common`, `jlpt:n5` … `jlpt:n1` (later e.g. `hsk:3`, `topik:1`); for characters also the school grade (`grade:1` … `grade:6`, `grade:secondary` for the rest of the Jōyō kanji) and the script (`hiragana`, `katakana`). |
|
||||
| `rank` | Its order among the source's entries, lower first; JMdict: N5 to N1 in their lists' order, then other common words, then the rest; KANJIDIC2: hiragana and katakana in the kana table's order, then kanji by JLPT level, then by school grade, then the rest, each by frequency. |
|
||||
| `keys` | What a search matches, lower case: written forms, readings in their own script and romanised, meanings. Everything specific to a language happens here, so the server's search knows no language. |
|
||||
| `looseKeys` | What a search matches after every `keys` match: spellings people type that aren't quite right (romaji with long vowels typed once, `sushi` for すうし). Left out when empty. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. |
|
||||
| `entry` | The entry itself, as cards take it: `forms` and `readings` (`text`, `common`, the source's `tags`; for a reading the `forms` it belongs to), `senses` (`glosses`; the source's codes in `pos`, `misc`, `field`, `dialect`; `info`; the `forms` and `readings` a sense is limited to; `examples`, sentences using the word in that sense: `text`, the form it takes there, `ja`, the sentence, `en`, its translation, and the source's id, e.g. `tatoeba`). Lists are left out when empty. A character's: `meanings`; `on` and `kun`, its readings as KANJIDIC2 writes them (a kun reading's okurigana after a `.`, a dash where it attaches to others); `grade`, `strokes`, `frequency` (its place among the 2,500 most used in newspapers) and `jlpt` where it has them. A kana's: `romaji` and `script`. |
|
||||
|
||||
## The app's stroke files
|
||||
|
||||
How every kanji and kana is written comes from
|
||||
[KanjiVG](https://kanjivg.tagaini.net) and is bundled with the app, so writing
|
||||
works without a server. To update it, download `kanjivg-<date>.xml.gz` from
|
||||
[KanjiVG's releases](https://github.com/KanjiVG/kanjivg/releases) and, in this
|
||||
package:
|
||||
|
||||
```sh
|
||||
dart run halkyon_learn_curator:curate kanjivg \
|
||||
--input kanjivg-20260714.xml.gz \
|
||||
--output ../halkyon_learn_flutter/assets/kanjivg
|
||||
```
|
||||
|
||||
It writes one gzipped JSON file for every 256 code points that have any,
|
||||
named by the code point's upper bits in hexadecimal (食, U+98DF, is in
|
||||
`098.json.gz`; 94 files, 2.7 MB), so the app reads only the part it needs.
|
||||
Each maps a character to its strokes (`s`, SVG paths in KanjiVG's 109 × 109
|
||||
box, in the order they're written) and the components KanjiVG first splits it
|
||||
into (`c`, when it has any). Then update the release named in the folder's
|
||||
`ATTRIBUTION.md`.
|
||||
|
||||
## Adding a source
|
||||
|
||||
|
|
|
|||
|
|
@ -4,19 +4,34 @@ import 'dart:io';
|
|||
import 'package:args/args.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jlpt.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/jmdict.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
|
||||
import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
|
||||
|
||||
/// Turns a dictionary download into a dictionary file for the server's
|
||||
/// `dictionaries` folder (README.md):
|
||||
/// `dictionaries` folder, or KanjiVG into the app's stroke files
|
||||
/// (README.md):
|
||||
///
|
||||
/// dart run halkyon_learn_curator:curate jmdict \
|
||||
/// --input jmdict-examples-eng-3.6.2.json --jlpt original_data \
|
||||
/// --output jmdict-en.jsonl.gz
|
||||
/// dart run halkyon_learn_curator:curate kanjidic \
|
||||
/// --input kanjidic2.xml.gz --jlpt kanji.json \
|
||||
/// --output kanjidic-en.jsonl.gz
|
||||
/// dart run halkyon_learn_curator:curate kanjivg \
|
||||
/// --input kanjivg-20260714.xml.gz \
|
||||
/// --output ../halkyon_learn_flutter/assets/kanjivg
|
||||
Future<void> main(List<String> args) async {
|
||||
final parser = ArgParser()
|
||||
..addOption('input', abbr: 'i', help: 'The source file, unpacked.')
|
||||
..addOption(
|
||||
'input',
|
||||
abbr: 'i',
|
||||
help: 'The source file, unpacked or gzipped (.gz).',
|
||||
)
|
||||
..addOption(
|
||||
'jlpt',
|
||||
help: "yomitan-jlpt-vocab's original_data folder (n1.csv … n5.csv).",
|
||||
help:
|
||||
"jmdict: yomitan-jlpt-vocab's original_data folder (n1.csv … "
|
||||
"n5.csv); kanjidic: kanji-data's kanji.json.",
|
||||
)
|
||||
..addFlag(
|
||||
'examples',
|
||||
|
|
@ -29,7 +44,9 @@ Future<void> main(List<String> args) async {
|
|||
..addOption(
|
||||
'output',
|
||||
abbr: 'o',
|
||||
help: 'The dictionary file to write; gzipped when it ends in .gz.',
|
||||
help:
|
||||
'The dictionary file to write, gzipped when it ends in .gz; for '
|
||||
'kanjivg the folder to write the stroke files into.',
|
||||
);
|
||||
final ArgResults options;
|
||||
try {
|
||||
|
|
@ -40,36 +57,70 @@ Future<void> main(List<String> args) async {
|
|||
final [source, ...] = options.rest.isEmpty ? [''] : options.rest;
|
||||
final input = options.option('input');
|
||||
final output = options.option('output');
|
||||
if (source != 'jmdict' || input == null || output == null) {
|
||||
if (!['jmdict', 'kanjidic', 'kanjivg'].contains(source) ||
|
||||
input == null ||
|
||||
output == null) {
|
||||
_fail(
|
||||
'Usage: curate jmdict --input <file> [--jlpt <folder>] --output '
|
||||
'<file>\n${parser.usage}',
|
||||
'Usage: curate jmdict|kanjidic|kanjivg --input <file> [--jlpt <path>] '
|
||||
'--output <path>\n${parser.usage}',
|
||||
);
|
||||
}
|
||||
final read = File(input).openRead();
|
||||
final text = (input.endsWith('.gz') ? read.transform(gzip.decoder) : read)
|
||||
.transform(utf8.decoder);
|
||||
final jlptPath = options.option('jlpt');
|
||||
|
||||
final jlpt = switch (options.option('jlpt')) {
|
||||
final folder? => await readJlptLevels(Directory(folder)),
|
||||
null => const <String, JlptLevel>{},
|
||||
};
|
||||
if (source == 'kanjivg') {
|
||||
final strokes = kanjivgStrokes(await text.join());
|
||||
final folder = await Directory(output).create(recursive: true);
|
||||
final files = strokeFiles(strokes);
|
||||
for (final MapEntry(key: name, value: characters) in files.entries) {
|
||||
await File('${folder.path}/$name.json.gz')
|
||||
.writeAsBytes(gzip.encode(utf8.encode(jsonEncode(characters))));
|
||||
}
|
||||
stdout.writeln(
|
||||
'Wrote the strokes of ${strokes.length} characters to '
|
||||
'${files.length} files in $output.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
final Stream<String> lines;
|
||||
final int withLevel;
|
||||
if (source == 'kanjidic') {
|
||||
final jlpt = jlptPath == null
|
||||
? const <String, int>{}
|
||||
: await readKanjiJlpt(File(jlptPath));
|
||||
lines = Stream.fromIterable(
|
||||
kanjidicDictionary(await text.join(), jlpt: jlpt),
|
||||
);
|
||||
withLevel = jlpt.length;
|
||||
} else {
|
||||
final jlpt = jlptPath == null
|
||||
? const <String, JlptLevel>{}
|
||||
: await readJlptLevels(Directory(jlptPath));
|
||||
lines = jmdictDictionary(
|
||||
text,
|
||||
jlpt: jlpt,
|
||||
examples: options.wasParsed('examples')
|
||||
? options.flag('examples')
|
||||
: File(input).uri.pathSegments.last.contains('examples'),
|
||||
);
|
||||
withLevel = jlpt.length;
|
||||
}
|
||||
var entries = -1;
|
||||
final lines =
|
||||
jmdictDictionary(
|
||||
File(input).openRead().transform(utf8.decoder),
|
||||
jlpt: jlpt,
|
||||
examples: options.wasParsed('examples')
|
||||
? options.flag('examples')
|
||||
: File(input).uri.pathSegments.last.contains('examples'),
|
||||
).map((line) {
|
||||
final bytes = lines
|
||||
.map((line) {
|
||||
entries++;
|
||||
return line;
|
||||
});
|
||||
final bytes = lines.transform(utf8.encoder);
|
||||
})
|
||||
.transform(utf8.encoder);
|
||||
await (output.endsWith('.gz') ? bytes.transform(gzip.encoder) : bytes).pipe(
|
||||
File(output).openWrite(),
|
||||
);
|
||||
stdout.writeln(
|
||||
'Wrote $entries entries to $output'
|
||||
'${jlpt.isEmpty ? '' : ' (${jlpt.length} with a JLPT level)'}.',
|
||||
'${withLevel == 0 ? '' : ' ($withLevel with a JLPT level)'}.',
|
||||
);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -29,9 +29,16 @@ final class const Credit({
|
|||
};
|
||||
}
|
||||
|
||||
/// What a dictionary holds: words, or characters (kanji and kana).
|
||||
enum DictionaryKind() {
|
||||
words,
|
||||
characters,
|
||||
}
|
||||
|
||||
/// What a dictionary file holds: entries of [source] at [sourceVersion],
|
||||
/// words of [language] with meanings in [glossLanguage] (BCP 47 tags), under
|
||||
/// [license], drawn from what [credits] names.
|
||||
/// words (or characters, [kind]) of [language] with meanings in
|
||||
/// [glossLanguage] (BCP 47 tags), under [license], drawn from what [credits]
|
||||
/// names.
|
||||
final class const DictionaryHeader({
|
||||
required final String source,
|
||||
required final String sourceVersion,
|
||||
|
|
@ -39,11 +46,13 @@ final class const DictionaryHeader({
|
|||
required final String glossLanguage,
|
||||
required final String license,
|
||||
required final List<Credit> credits,
|
||||
final DictionaryKind kind = DictionaryKind.words,
|
||||
}) {
|
||||
Map<String, Object?> toJson() => {
|
||||
'format': dictionaryFormat,
|
||||
'formatVersion': dictionaryFormatVersion,
|
||||
'source': source,
|
||||
'kind': kind.name,
|
||||
'sourceVersion': sourceVersion,
|
||||
'language': language,
|
||||
'glossLanguage': glossLanguage,
|
||||
|
|
|
|||
210
halkyon_learn_curator/lib/src/sources/kanjidic.dart
Normal file
210
halkyon_learn_curator/lib/src/sources/kanjidic.dart
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/dictionary_file.dart';
|
||||
import 'package:halkyon_learn_curator/src/gloss_keys.dart';
|
||||
import 'package:halkyon_learn_curator/src/japanese/kana.dart';
|
||||
import 'package:xml/xml.dart';
|
||||
|
||||
/// Who made KANJIDIC2 and the JLPT kanji lists.
|
||||
const kanjidicCredits = [
|
||||
Credit(
|
||||
name: 'KANJIDIC2',
|
||||
by: 'Electronic Dictionary Research and Development Group (EDRDG)',
|
||||
license: 'CC-BY-SA-4.0',
|
||||
url: 'https://www.edrdg.org/edrdg/licence.html',
|
||||
),
|
||||
Credit(
|
||||
name: 'JLPT kanji lists',
|
||||
by:
|
||||
"Jonathan Waller (tanos.co.uk); by character in David Gouveia's "
|
||||
'kanji-data',
|
||||
license: 'CC-BY-4.0',
|
||||
url: 'https://www.tanos.co.uk/jlpt/',
|
||||
),
|
||||
];
|
||||
|
||||
/// The JLPT levels of kanji-data's `kanji.json` ([file]), 5 for N5 to 1
|
||||
/// for N1, by the character: its `jlpt_new`, Waller's levels since 2010.
|
||||
Future<Map<String, int>> readKanjiJlpt(File file) async {
|
||||
final kanji = jsonDecode(await file.readAsString()) as Map<String, Object?>;
|
||||
return {
|
||||
for (final MapEntry(:key, :value) in kanji.entries)
|
||||
if (value case {'jlpt_new': final int level}) key: level,
|
||||
};
|
||||
}
|
||||
|
||||
/// KANJIDIC2's XML ([text]) as the lines of a dictionary file of Japanese
|
||||
/// characters with English meanings: every kanji with its on and kun
|
||||
/// readings, meanings, school grade, stroke count and frequency, and its JLPT
|
||||
/// level from [jlpt] (N5 to N1 as 5 to 1, by the character); then hiragana
|
||||
/// and katakana with their romaji. Labels pick a level (`jlpt:n5`), a school
|
||||
/// grade (`grade:1` … `grade:6`, `grade:secondary`) or a script (`hiragana`,
|
||||
/// `katakana`); kanji rank by JLPT level, then grade, then frequency.
|
||||
Iterable<String> kanjidicDictionary(
|
||||
String text, {
|
||||
Map<String, int> jlpt = const {},
|
||||
}) sync* {
|
||||
final root = XmlDocument.parse(text).rootElement;
|
||||
final version =
|
||||
root
|
||||
.getElement('header')
|
||||
?.getElement('database_version')
|
||||
?.innerText
|
||||
.trim() ??
|
||||
'';
|
||||
yield headerLine(
|
||||
DictionaryHeader(
|
||||
source: 'kanjidic',
|
||||
kind: DictionaryKind.characters,
|
||||
sourceVersion: version,
|
||||
language: 'ja',
|
||||
glossLanguage: 'en',
|
||||
license: 'CC-BY-SA-4.0',
|
||||
credits: kanjidicCredits,
|
||||
),
|
||||
);
|
||||
for (final character in root.findElements('character')) {
|
||||
if (kanjiEntry(character, jlpt: jlpt) case final entry?) {
|
||||
yield entryLine(entry);
|
||||
}
|
||||
}
|
||||
for (final entry in kanaEntries()) {
|
||||
yield entryLine(entry);
|
||||
}
|
||||
}
|
||||
|
||||
/// KANJIDIC2's `<character>` [character] as an entry; null for one with
|
||||
/// neither a Japanese reading nor an English meaning.
|
||||
DictionaryEntry? kanjiEntry(
|
||||
XmlElement character, {
|
||||
Map<String, int> jlpt = const {},
|
||||
}) {
|
||||
final literal = character.getElement('literal')?.innerText.trim() ?? '';
|
||||
if (literal.isEmpty) {
|
||||
return null;
|
||||
}
|
||||
final misc = character.getElement('misc');
|
||||
int? number(String name) =>
|
||||
int.tryParse(misc?.getElement(name)?.innerText.trim() ?? '');
|
||||
final grade = number('grade');
|
||||
final strokes = number('stroke_count');
|
||||
final frequency = number('freq');
|
||||
final level = jlpt[literal];
|
||||
|
||||
final groups =
|
||||
character
|
||||
.getElement('reading_meaning')
|
||||
?.findElements('rmgroup')
|
||||
.toList() ??
|
||||
const <XmlElement>[];
|
||||
List<String> readings(String type) => [
|
||||
for (final group in groups)
|
||||
for (final reading in group.findElements('reading'))
|
||||
if (reading.getAttribute('r_type') == type) reading.innerText.trim(),
|
||||
];
|
||||
final on = readings('ja_on');
|
||||
final kun = readings('ja_kun');
|
||||
final meanings = [
|
||||
for (final group in groups)
|
||||
for (final meaning in group.findElements('meaning'))
|
||||
if (meaning.getAttribute('m_lang') == null) meaning.innerText.trim(),
|
||||
];
|
||||
if (on.isEmpty && kun.isEmpty && meanings.isEmpty) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// Kun readings mark where the okurigana start (`た.べる`) and where the
|
||||
// reading attaches to others (`-び`). The whole reading is a key; the
|
||||
// character's own part (`た`) only a loose one, or typing `ta` would
|
||||
// bring every kanji read た before the kana.
|
||||
String whole(String reading) => reading.replaceAll(RegExp('[.-]'), '');
|
||||
final kunWhole = {for (final reading in kun) whole(reading)};
|
||||
final kunStems = {
|
||||
for (final reading in kun) reading.split('.').first.replaceAll('-', ''),
|
||||
}.difference(kunWhole);
|
||||
final keys = <String>{
|
||||
literal,
|
||||
for (final reading in on.map(whole)) ...[
|
||||
reading,
|
||||
toHiragana(reading),
|
||||
toRomaji(reading),
|
||||
],
|
||||
for (final reading in kunWhole) ...[reading, toRomaji(reading)],
|
||||
for (final meaning in meanings) ...glossKeys(meaning),
|
||||
}..remove('');
|
||||
final looseKeys = {
|
||||
for (final stem in kunStems) ...[stem, toRomaji(stem)],
|
||||
}.difference(keys)..remove('');
|
||||
return DictionaryEntry(
|
||||
id: literal,
|
||||
headword: literal,
|
||||
reading: null,
|
||||
labels: [
|
||||
if (level != null) 'jlpt:n$level',
|
||||
?switch (grade) {
|
||||
final grade? when grade >= 1 && grade <= 6 => 'grade:$grade',
|
||||
8 => 'grade:secondary',
|
||||
_ => null,
|
||||
},
|
||||
],
|
||||
rank: switch ((level, grade)) {
|
||||
(final level?, _) => (6 - level) * 10000 + (frequency ?? 9999),
|
||||
(null, final grade?) when grade <= 8 =>
|
||||
100000 + grade * 10000 + (frequency ?? 9999),
|
||||
_ => 900000 + (frequency ?? 9999),
|
||||
},
|
||||
keys: keys.toList(),
|
||||
looseKeys: looseKeys.toList(),
|
||||
entry: {
|
||||
'meanings': meanings,
|
||||
'on': on,
|
||||
'kun': kun,
|
||||
'grade': ?grade,
|
||||
'strokes': ?strokes,
|
||||
'frequency': ?frequency,
|
||||
'jlpt': ?level,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
/// Hiragana in the order of the kana table: the plain sounds, ん, then
|
||||
/// those with dakuten and handakuten.
|
||||
const _hiragana =
|
||||
'あいうえおかきくけこさしすせそたちつてとなにぬねのはひふへほまみむめも'
|
||||
'やゆよらりるれろわをんがぎぐげござじずぜぞだぢづでどばびぶべぼぱぴぷぺぽ';
|
||||
|
||||
/// The romaji kana tables give where they differ from how the word is
|
||||
/// read: を as `wo`, telling it from お.
|
||||
const _tableRomaji = {'を': 'wo'};
|
||||
|
||||
/// The hiragana and katakana, in the kana table's order, as entries: each
|
||||
/// with its romaji, found by it and by how it's read, labelled by its
|
||||
/// script, ranked before every kanji.
|
||||
List<DictionaryEntry> kanaEntries() => [
|
||||
for (final (script, offset) in const [('hiragana', 0), ('katakana', 0x60)])
|
||||
for (final (place, code) in _hiragana.runes.indexed)
|
||||
_kana(
|
||||
String.fromCharCode(code + offset),
|
||||
romaji:
|
||||
_tableRomaji[String.fromCharCode(code)] ??
|
||||
toRomaji(String.fromCharCode(code)),
|
||||
script: script,
|
||||
rank: (offset == 0 ? 0 : 1000) + place,
|
||||
),
|
||||
];
|
||||
|
||||
DictionaryEntry _kana(
|
||||
String kana, {
|
||||
required String romaji,
|
||||
required String script,
|
||||
required int rank,
|
||||
}) => DictionaryEntry(
|
||||
id: kana,
|
||||
headword: kana,
|
||||
reading: null,
|
||||
labels: [script],
|
||||
rank: rank,
|
||||
keys: {kana, toHiragana(kana), romaji, toRomaji(kana)}.toList(),
|
||||
entry: {'romaji': romaji, 'script': script},
|
||||
);
|
||||
60
halkyon_learn_curator/lib/src/sources/kanjivg.dart
Normal file
60
halkyon_learn_curator/lib/src/sources/kanjivg.dart
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
import 'package:xml/xml.dart';
|
||||
|
||||
/// How a character is written, from KanjiVG: its `strokes` in the order
|
||||
/// they are written, each an SVG path in a 109 × 109 box, and the
|
||||
/// `components` it is built from, as KanjiVG first splits it (語: 言, 吾).
|
||||
typedef CharacterStrokes = ({List<String> strokes, List<String> components});
|
||||
|
||||
/// KanjiVG's combined file ([text], `kanjivg-<date>.xml`) as each
|
||||
/// character's strokes, by the character. The character is the code point
|
||||
/// in its id (`kvg:kanji_098df`), which KanjiVG's own `kvg:element` doesn't
|
||||
/// always match (its 橅 says 機).
|
||||
Map<String, CharacterStrokes> kanjivgStrokes(String text) => {
|
||||
for (final kanji in XmlDocument.parse(text).rootElement.findElements('kanji'))
|
||||
if ((kanji.getAttribute('id') ?? '').split('_') case [_, final code])
|
||||
if (int.tryParse(code, radix: 16) case final code?)
|
||||
if (kanji.getElement('g') case final group?)
|
||||
String.fromCharCode(code): (
|
||||
strokes: [
|
||||
for (final path in group.findAllElements('path'))
|
||||
?path.getAttribute('d'),
|
||||
],
|
||||
components: _components(group).toList(),
|
||||
),
|
||||
};
|
||||
|
||||
/// The characters [group]'s children are, looking through the groups that
|
||||
/// are none (a stroke's or a position's).
|
||||
Iterable<String> _components(XmlElement group) sync* {
|
||||
for (final child in group.childElements.where(
|
||||
(child) => child.name.local == 'g',
|
||||
)) {
|
||||
if (child.getAttribute('kvg:element') case final element?) {
|
||||
yield element;
|
||||
} else {
|
||||
yield* _components(child);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Which of the app's stroke files holds [character]: those of each 256
|
||||
/// code points, named by the code point's upper bits in hexadecimal
|
||||
/// (食, U+98DF, in `098`).
|
||||
String strokeFileOf(String character) =>
|
||||
(character.runes.first >> 8).toRadixString(16).padLeft(3, '0');
|
||||
|
||||
/// [strokes] as the app's stroke files, by name: for each character its
|
||||
/// strokes (`s`) and, when it has any, its components (`c`).
|
||||
Map<String, Map<String, Object>> strokeFiles(
|
||||
Map<String, CharacterStrokes> strokes,
|
||||
) {
|
||||
final files = <String, Map<String, Object>>{};
|
||||
for (final MapEntry(key: character, value: (:strokes, :components))
|
||||
in strokes.entries) {
|
||||
files.putIfAbsent(strokeFileOf(character), () => {})[character] = {
|
||||
's': strokes,
|
||||
if (components.isNotEmpty) 'c': components,
|
||||
};
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
|
@ -11,6 +11,7 @@ resolution: workspace
|
|||
|
||||
dependencies:
|
||||
args: ^2.7.0
|
||||
xml: ^7.0.1
|
||||
|
||||
dev_dependencies:
|
||||
test: ^1.25.5
|
||||
|
|
|
|||
14
halkyon_learn_curator/test/fixtures/README.md
vendored
14
halkyon_learn_curator/test/fixtures/README.md
vendored
|
|
@ -9,3 +9,17 @@ its licence, CC BY-SA 4.0 (<https://www.edrdg.org/edrdg/licence.html>).
|
|||
(<https://github.com/stephenmk/yomitan-jlpt-vocab>), CC BY-SA 4.0, from Jonathan
|
||||
Waller's lists (<http://www.tanos.co.uk/jlpt/>), CC BY; one row without an id is
|
||||
made up.
|
||||
|
||||
`kanjidic2-sample.xml` holds five characters of KANJIDIC2 (database version
|
||||
2026-282) with its header and a shortened DTD: KANJIDIC2 is © the Electronic
|
||||
Dictionary Research and Development Group, used under its licence, CC BY-SA 4.0
|
||||
(<https://www.edrdg.org/edrdg/licence.html>).
|
||||
|
||||
`kanji-data-sample.json` holds four entries of kanji-data's `kanji.json`
|
||||
(<https://github.com/davidluzgouveia/kanji-data>), cut to their numbers, from
|
||||
KANJIDIC2 and from Jonathan Waller's JLPT levels (<http://www.tanos.co.uk/jlpt/>),
|
||||
CC BY.
|
||||
|
||||
`kanjivg-sample.xml` holds four characters of KanjiVG (the release of
|
||||
2026-07-14) with its notice, among them 橅, whose entry names itself 機:
|
||||
KanjiVG is © Ulrich Apel, CC BY-SA 3.0 (<https://kanjivg.tagaini.net>).
|
||||
|
|
|
|||
30
halkyon_learn_curator/test/fixtures/kanji-data-sample.json
vendored
Normal file
30
halkyon_learn_curator/test/fixtures/kanji-data-sample.json
vendored
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
{
|
||||
"一": {
|
||||
"strokes": 1,
|
||||
"grade": 1,
|
||||
"freq": 2,
|
||||
"jlpt_old": 4,
|
||||
"jlpt_new": 5
|
||||
},
|
||||
"食": {
|
||||
"strokes": 9,
|
||||
"grade": 2,
|
||||
"freq": 328,
|
||||
"jlpt_old": 4,
|
||||
"jlpt_new": 5
|
||||
},
|
||||
"亜": {
|
||||
"strokes": 7,
|
||||
"grade": 8,
|
||||
"freq": 1509,
|
||||
"jlpt_old": 1,
|
||||
"jlpt_new": 1
|
||||
},
|
||||
"唖": {
|
||||
"strokes": 10,
|
||||
"grade": null,
|
||||
"freq": null,
|
||||
"jlpt_old": null,
|
||||
"jlpt_new": null
|
||||
}
|
||||
}
|
||||
313
halkyon_learn_curator/test/fixtures/kanjidic2-sample.xml
vendored
Normal file
313
halkyon_learn_curator/test/fixtures/kanjidic2-sample.xml
vendored
Normal file
|
|
@ -0,0 +1,313 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE kanjidic2 [
|
||||
<!ELEMENT kanjidic2 (header,character*)>
|
||||
]>
|
||||
<kanjidic2>
|
||||
<header>
|
||||
<!-- KANJIDIC 2 - XML format kanji database combining the KANJIDIC
|
||||
and KANJD212 files plus the kanji from JIS X 0213.
|
||||
-->
|
||||
<file_version>4</file_version>
|
||||
<database_version>2026-282</database_version>
|
||||
<date_of_creation>2026-10-09</date_of_creation>
|
||||
</header>
|
||||
<character>
|
||||
<literal>一</literal>
|
||||
<codepoint>
|
||||
<cp_value cp_type="ucs">4e00</cp_value>
|
||||
<cp_value cp_type="jis208">1-16-76</cp_value>
|
||||
</codepoint>
|
||||
<radical>
|
||||
<rad_value rad_type="classical">1</rad_value>
|
||||
</radical>
|
||||
<misc>
|
||||
<grade>1</grade>
|
||||
<stroke_count>1</stroke_count>
|
||||
<variant var_type="jis208">1-48-01</variant>
|
||||
<freq>2</freq>
|
||||
<jlpt>4</jlpt>
|
||||
</misc>
|
||||
<dic_number>
|
||||
<dic_ref dr_type="nelson_c">1</dic_ref>
|
||||
<dic_ref dr_type="nelson_n">1</dic_ref>
|
||||
<dic_ref dr_type="halpern_njecd">3341</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkd">4148</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkld">2105</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkld_2ed">2850</dic_ref>
|
||||
<dic_ref dr_type="heisig">1</dic_ref>
|
||||
<dic_ref dr_type="heisig6">1</dic_ref>
|
||||
<dic_ref dr_type="gakken">4</dic_ref>
|
||||
<dic_ref dr_type="oneill_names">3</dic_ref>
|
||||
<dic_ref dr_type="oneill_kk">1</dic_ref>
|
||||
<dic_ref dr_type="moro" m_vol="1" m_page="0001">1</dic_ref>
|
||||
<dic_ref dr_type="henshall">1</dic_ref>
|
||||
<dic_ref dr_type="sh_kk">2</dic_ref>
|
||||
<dic_ref dr_type="sh_kk2">2</dic_ref>
|
||||
<dic_ref dr_type="sakade">1</dic_ref>
|
||||
<dic_ref dr_type="jf_cards">1</dic_ref>
|
||||
<dic_ref dr_type="henshall3">1</dic_ref>
|
||||
<dic_ref dr_type="tutt_cards">1</dic_ref>
|
||||
<dic_ref dr_type="crowley">1</dic_ref>
|
||||
<dic_ref dr_type="kanji_in_context">1</dic_ref>
|
||||
<dic_ref dr_type="busy_people">1.A</dic_ref>
|
||||
<dic_ref dr_type="kodansha_compact">1</dic_ref>
|
||||
<dic_ref dr_type="maniette">1</dic_ref>
|
||||
</dic_number>
|
||||
<query_code>
|
||||
<q_code qc_type="skip">4-1-4</q_code>
|
||||
<q_code qc_type="sh_desc">0a1.1</q_code>
|
||||
<q_code qc_type="four_corner">1000.0</q_code>
|
||||
<q_code qc_type="deroo">3072</q_code>
|
||||
</query_code>
|
||||
<reading_meaning>
|
||||
<rmgroup>
|
||||
<reading r_type="pinyin">yi1</reading>
|
||||
<reading r_type="korean_r">il</reading>
|
||||
<reading r_type="korean_h">일</reading>
|
||||
<reading r_type="vietnam">Nhất</reading>
|
||||
<reading r_type="ja_on">イチ</reading>
|
||||
<reading r_type="ja_on">イツ</reading>
|
||||
<reading r_type="ja_kun">ひと-</reading>
|
||||
<reading r_type="ja_kun">ひと.つ</reading>
|
||||
<meaning>one</meaning>
|
||||
<meaning>one radical (no.1)</meaning>
|
||||
<meaning m_lang="fr">un</meaning>
|
||||
<meaning m_lang="fr">radical un (no. 1)</meaning>
|
||||
<meaning m_lang="es">uno</meaning>
|
||||
<meaning m_lang="es">1</meaning>
|
||||
<meaning m_lang="pt">um</meaning>
|
||||
</rmgroup>
|
||||
<nanori>かず</nanori>
|
||||
<nanori>い</nanori>
|
||||
<nanori>いっ</nanori>
|
||||
<nanori>いる</nanori>
|
||||
<nanori>かつ</nanori>
|
||||
<nanori>かづ</nanori>
|
||||
<nanori>てん</nanori>
|
||||
<nanori>はじめ</nanori>
|
||||
<nanori>ひ</nanori>
|
||||
<nanori>ひとつ</nanori>
|
||||
<nanori>まこと</nanori>
|
||||
</reading_meaning>
|
||||
</character>
|
||||
<character>
|
||||
<literal>食</literal>
|
||||
<codepoint>
|
||||
<cp_value cp_type="ucs">98df</cp_value>
|
||||
<cp_value cp_type="jis208">1-31-09</cp_value>
|
||||
</codepoint>
|
||||
<radical>
|
||||
<rad_value rad_type="classical">184</rad_value>
|
||||
</radical>
|
||||
<misc>
|
||||
<grade>2</grade>
|
||||
<stroke_count>9</stroke_count>
|
||||
<freq>328</freq>
|
||||
<jlpt>4</jlpt>
|
||||
</misc>
|
||||
<dic_number>
|
||||
<dic_ref dr_type="nelson_c">5154</dic_ref>
|
||||
<dic_ref dr_type="nelson_n">6674</dic_ref>
|
||||
<dic_ref dr_type="halpern_njecd">2075</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkd">2578</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkld">1316</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkld_2ed">1787</dic_ref>
|
||||
<dic_ref dr_type="heisig">1472</dic_ref>
|
||||
<dic_ref dr_type="heisig6">1582</dic_ref>
|
||||
<dic_ref dr_type="gakken">269</dic_ref>
|
||||
<dic_ref dr_type="oneill_names">1159</dic_ref>
|
||||
<dic_ref dr_type="oneill_kk">283</dic_ref>
|
||||
<dic_ref dr_type="moro" m_vol="12" m_page="0371">44014</dic_ref>
|
||||
<dic_ref dr_type="henshall">146</dic_ref>
|
||||
<dic_ref dr_type="sh_kk">322</dic_ref>
|
||||
<dic_ref dr_type="sh_kk2">322</dic_ref>
|
||||
<dic_ref dr_type="sakade">253</dic_ref>
|
||||
<dic_ref dr_type="jf_cards">99</dic_ref>
|
||||
<dic_ref dr_type="henshall3">163</dic_ref>
|
||||
<dic_ref dr_type="tutt_cards">180</dic_ref>
|
||||
<dic_ref dr_type="crowley">207</dic_ref>
|
||||
<dic_ref dr_type="kanji_in_context">131</dic_ref>
|
||||
<dic_ref dr_type="busy_people">2.13</dic_ref>
|
||||
<dic_ref dr_type="kodansha_compact">1900</dic_ref>
|
||||
<dic_ref dr_type="maniette">1484</dic_ref>
|
||||
</dic_number>
|
||||
<query_code>
|
||||
<q_code qc_type="skip">2-2-7</q_code>
|
||||
<q_code qc_type="sh_desc">8b0.1</q_code>
|
||||
<q_code qc_type="four_corner">8073.2</q_code>
|
||||
<q_code qc_type="deroo">2865</q_code>
|
||||
</query_code>
|
||||
<reading_meaning>
|
||||
<rmgroup>
|
||||
<reading r_type="pinyin">shi2</reading>
|
||||
<reading r_type="pinyin">si4</reading>
|
||||
<reading r_type="korean_r">sig</reading>
|
||||
<reading r_type="korean_r">sa</reading>
|
||||
<reading r_type="korean_h">식</reading>
|
||||
<reading r_type="korean_h">사</reading>
|
||||
<reading r_type="vietnam">Thực</reading>
|
||||
<reading r_type="vietnam">Tự</reading>
|
||||
<reading r_type="ja_on">ショク</reading>
|
||||
<reading r_type="ja_on">ジキ</reading>
|
||||
<reading r_type="ja_kun">く.う</reading>
|
||||
<reading r_type="ja_kun">く.らう</reading>
|
||||
<reading r_type="ja_kun">た.べる</reading>
|
||||
<reading r_type="ja_kun">は.む</reading>
|
||||
<meaning>eat</meaning>
|
||||
<meaning>food</meaning>
|
||||
<meaning m_lang="fr">manger</meaning>
|
||||
<meaning m_lang="fr">nourriture</meaning>
|
||||
<meaning m_lang="es">alimento</meaning>
|
||||
<meaning m_lang="es">comida</meaning>
|
||||
<meaning m_lang="es">eclipse</meaning>
|
||||
<meaning m_lang="es">comer</meaning>
|
||||
<meaning m_lang="pt">comer</meaning>
|
||||
<meaning m_lang="pt">comida</meaning>
|
||||
<meaning m_lang="pt">alimento</meaning>
|
||||
</rmgroup>
|
||||
<nanori>ぐい</nanori>
|
||||
</reading_meaning>
|
||||
</character>
|
||||
<character>
|
||||
<literal>亜</literal>
|
||||
<codepoint>
|
||||
<cp_value cp_type="ucs">4e9c</cp_value>
|
||||
<cp_value cp_type="jis208">1-16-01</cp_value>
|
||||
</codepoint>
|
||||
<radical>
|
||||
<rad_value rad_type="classical">7</rad_value>
|
||||
<rad_value rad_type="nelson_c">1</rad_value>
|
||||
</radical>
|
||||
<misc>
|
||||
<grade>8</grade>
|
||||
<stroke_count>7</stroke_count>
|
||||
<variant var_type="jis208">1-48-19</variant>
|
||||
<freq>1509</freq>
|
||||
<jlpt>1</jlpt>
|
||||
</misc>
|
||||
<dic_number>
|
||||
<dic_ref dr_type="nelson_c">43</dic_ref>
|
||||
<dic_ref dr_type="nelson_n">81</dic_ref>
|
||||
<dic_ref dr_type="halpern_njecd">3540</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkd">4354</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkld">2204</dic_ref>
|
||||
<dic_ref dr_type="halpern_kkld_2ed">2966</dic_ref>
|
||||
<dic_ref dr_type="heisig">1809</dic_ref>
|
||||
<dic_ref dr_type="heisig6">1950</dic_ref>
|
||||
<dic_ref dr_type="gakken">1331</dic_ref>
|
||||
<dic_ref dr_type="oneill_names">525</dic_ref>
|
||||
<dic_ref dr_type="oneill_kk">1788</dic_ref>
|
||||
<dic_ref dr_type="moro" m_vol="1" m_page="0525">272</dic_ref>
|
||||
<dic_ref dr_type="henshall">997</dic_ref>
|
||||
<dic_ref dr_type="sh_kk">1616</dic_ref>
|
||||
<dic_ref dr_type="sh_kk2">1724</dic_ref>
|
||||
<dic_ref dr_type="jf_cards">1032</dic_ref>
|
||||
<dic_ref dr_type="tutt_cards">1092</dic_ref>
|
||||
<dic_ref dr_type="kanji_in_context">1818</dic_ref>
|
||||
<dic_ref dr_type="kodansha_compact">35</dic_ref>
|
||||
<dic_ref dr_type="maniette">1827</dic_ref>
|
||||
</dic_number>
|
||||
<query_code>
|
||||
<q_code qc_type="skip">4-7-1</q_code>
|
||||
<q_code qc_type="sh_desc">0a7.14</q_code>
|
||||
<q_code qc_type="four_corner">1010.6</q_code>
|
||||
<q_code qc_type="deroo">3273</q_code>
|
||||
</query_code>
|
||||
<reading_meaning>
|
||||
<rmgroup>
|
||||
<reading r_type="pinyin">ya4</reading>
|
||||
<reading r_type="korean_r">a</reading>
|
||||
<reading r_type="korean_h">아</reading>
|
||||
<reading r_type="vietnam">A</reading>
|
||||
<reading r_type="vietnam">Á</reading>
|
||||
<reading r_type="ja_on">ア</reading>
|
||||
<reading r_type="ja_kun">つ.ぐ</reading>
|
||||
<meaning>Asia</meaning>
|
||||
<meaning>rank next</meaning>
|
||||
<meaning>come after</meaning>
|
||||
<meaning>-ous</meaning>
|
||||
<meaning m_lang="fr">Asie</meaning>
|
||||
<meaning m_lang="fr">suivant</meaning>
|
||||
<meaning m_lang="fr">sub-</meaning>
|
||||
<meaning m_lang="fr">sous-</meaning>
|
||||
<meaning m_lang="es">pref. para indicar</meaning>
|
||||
<meaning m_lang="es">venir después de</meaning>
|
||||
<meaning m_lang="es">Asia</meaning>
|
||||
<meaning m_lang="pt">Ásia</meaning>
|
||||
<meaning m_lang="pt">próxima</meaning>
|
||||
<meaning m_lang="pt">o que vem depois</meaning>
|
||||
<meaning m_lang="pt">-ous</meaning>
|
||||
</rmgroup>
|
||||
<nanori>や</nanori>
|
||||
<nanori>つぎ</nanori>
|
||||
<nanori>つぐ</nanori>
|
||||
</reading_meaning>
|
||||
</character>
|
||||
<character>
|
||||
<literal>唖</literal>
|
||||
<codepoint>
|
||||
<cp_value cp_type="ucs">5516</cp_value>
|
||||
<cp_value cp_type="jis208">1-16-02</cp_value>
|
||||
</codepoint>
|
||||
<radical>
|
||||
<rad_value rad_type="classical">30</rad_value>
|
||||
</radical>
|
||||
<misc>
|
||||
<stroke_count>10</stroke_count>
|
||||
<variant var_type="jis212">1-21-64</variant>
|
||||
<variant var_type="jis212">1-45-68</variant>
|
||||
</misc>
|
||||
<dic_number>
|
||||
<dic_ref dr_type="nelson_c">939</dic_ref>
|
||||
<dic_ref dr_type="nelson_n">795</dic_ref>
|
||||
<dic_ref dr_type="heisig">2958</dic_ref>
|
||||
<dic_ref dr_type="heisig6">2964</dic_ref>
|
||||
<dic_ref dr_type="moro" m_vol="2" m_page="1066">3743</dic_ref>
|
||||
</dic_number>
|
||||
<query_code>
|
||||
<q_code qc_type="skip">1-3-7</q_code>
|
||||
<q_code qc_type="sh_desc">3d8.3</q_code>
|
||||
<q_code qc_type="four_corner">6101.7</q_code>
|
||||
</query_code>
|
||||
<reading_meaning>
|
||||
<rmgroup>
|
||||
<reading r_type="pinyin">ya1</reading>
|
||||
<reading r_type="korean_r">a</reading>
|
||||
<reading r_type="korean_h">아</reading>
|
||||
<reading r_type="vietnam">Á</reading>
|
||||
<reading r_type="vietnam">Ớ</reading>
|
||||
<reading r_type="vietnam">Ứ</reading>
|
||||
<reading r_type="ja_on">ア</reading>
|
||||
<reading r_type="ja_on">アク</reading>
|
||||
<reading r_type="ja_kun">おし</reading>
|
||||
<meaning>mute</meaning>
|
||||
<meaning>dumb</meaning>
|
||||
</rmgroup>
|
||||
</reading_meaning>
|
||||
</character>
|
||||
<character>
|
||||
<literal>僲</literal>
|
||||
<codepoint>
|
||||
<cp_value cp_type="ucs">50f2</cp_value>
|
||||
<cp_value cp_type="jis212">1-18-49</cp_value>
|
||||
<cp_value cp_type="jis213">1-14-38</cp_value>
|
||||
</codepoint>
|
||||
<radical>
|
||||
<rad_value rad_type="classical">9</rad_value>
|
||||
</radical>
|
||||
<misc>
|
||||
<stroke_count>16</stroke_count>
|
||||
</misc>
|
||||
<dic_number>
|
||||
<dic_ref dr_type="moro">0</dic_ref>
|
||||
</dic_number>
|
||||
<query_code>
|
||||
<q_code qc_type="skip">1-2-14</q_code>
|
||||
</query_code>
|
||||
<reading_meaning>
|
||||
<rmgroup>
|
||||
<reading r_type="pinyin">xian1</reading>
|
||||
</rmgroup>
|
||||
</reading_meaning>
|
||||
</character>
|
||||
</kanjidic2>
|
||||
105
halkyon_learn_curator/test/fixtures/kanjivg-sample.xml
vendored
Normal file
105
halkyon_learn_curator/test/fixtures/kanjivg-sample.xml
vendored
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!--
|
||||
Copyright (C) 2009-2013 Ulrich Apel.
|
||||
This work is distributed under the conditions of the Creative Commons
|
||||
Attribution-Share Alike 3.0 Licence. This means you are free:
|
||||
* to Share - to copy, distribute and transmit the work
|
||||
* to Remix - to adapt the work
|
||||
|
||||
Under the following conditions:
|
||||
* Attribution. You must attribute the work by stating your use of KanjiVG in
|
||||
your own copyright header and linking to KanjiVG's website
|
||||
(http://kanjivg.tagaini.net)
|
||||
* Share Alike. If you alter, transform, or build upon this work, you may
|
||||
distribute the resulting work only under the same or similar license to this
|
||||
one.
|
||||
|
||||
See http://creativecommons.org/licenses/by-sa/3.0/ for more details.
|
||||
This file was generated on 2026-07-14 from the most recent KanjiVG data.
|
||||
-->
|
||||
<kanjivg xmlns:kvg='http://kanjivg.tagaini.net'>
|
||||
<kanji id="kvg:kanji_03042">
|
||||
<g id="kvg:03042" kvg:element="あ">
|
||||
<path id="kvg:03042-s1" d="M31.01,33c0.88,0.88,2.75,1.82,5.25,1.75c8.62-0.25,20-2.12,29.5-4.25c1.51-0.34,4.62-0.88,6.62-0.5"/>
|
||||
<path id="kvg:03042-s2" d="M49.76,17.62c0.88,1,1.82,3.26,1.38,5.25c-3.75,16.75-6.25,38.13-5.13,53.63c0.41,5.7,1.88,10.88,3.38,13.62"/>
|
||||
<path id="kvg:03042-s3" d="M65.63,44.12c0.75,1.12,1.16,4.39,0.5,6.12c-4.62,12.26-11.24,23.76-25.37,35.76c-6.86,5.83-15.88,3.75-16.25-8.38c-0.34-10.87,13.38-23.12,32.38-26.74c12.42-2.37,27,1.38,30.5,12.75c4.05,13.18-3.76,26.37-20.88,30.49"/>
|
||||
</g>
|
||||
</kanji>
|
||||
<kanji id="kvg:kanji_098df">
|
||||
<g id="kvg:098df" kvg:element="食" kvg:radical="general">
|
||||
<path id="kvg:098df-s1" kvg:type="㇒" d="M52.75,10.5c0.11,0.98-0.19,2.67-0.97,3.93C45,25.34,31.75,41.19,14,51.5"/>
|
||||
<path id="kvg:098df-s2" kvg:type="㇏" d="M52.75,16.25c5.09,4.8,25.71,19.61,33.7,24.9c2.68,1.78,5.37,2.79,8.55,3.35"/>
|
||||
<path id="kvg:098df-s3" kvg:type="㇑a" d="M52.25,29.25c1,1,1.5,2.25,1.5,3.5c0,2,0,3,0,5.5"/>
|
||||
<path id="kvg:098df-s4" kvg:type="㇕" d="M38,40c0.83,0.47,2.19,1,3.86,0.83c9.39-0.96,21.95-2.76,23.25-2.84c1.67-0.1,3.14,0.88,3.11,2.53C68.2,41.8,67,53.25,66.34,62.4c-0.07,0.94-0.13,1.36-0.13,1.99"/>
|
||||
<path id="kvg:098df-s5" kvg:type="㇐a" d="M40.83,51.73C47.25,51.25,59.5,50,66,49.75"/>
|
||||
<path id="kvg:098df-s6" kvg:type="㇐a" d="M40.69,63.9c7.04-0.52,16.55-1.62,24.6-2.04"/>
|
||||
<path id="kvg:098df-s7" kvg:type="㇙" d="M38.25,40.25c1.12,1.12,1.5,2.62,1.5,4c0,9.12,0,43.62,0,47.25c0,4,1,4.88,4.12,2.88c2.93-1.87,6.75-5.25,10.88-8.38"/>
|
||||
<path id="kvg:098df-s8" kvg:type="㇒" d="M74,64c0.25,1.25,0.09,2.57-0.75,3.5c-3.5,3.88-4.5,4.88-7.25,7.5"/>
|
||||
<path id="kvg:098df-s9" kvg:type="㇏" d="M51.5,71C55.75,71,77,90,81,92.75c2.49,1.71,4.62,2.62,7.5,3.5"/>
|
||||
</g>
|
||||
</kanji>
|
||||
<kanji id="kvg:kanji_08a9e">
|
||||
<g id="kvg:08a9e" kvg:element="語">
|
||||
<g id="kvg:08a9e-g1" kvg:element="言" kvg:position="left" kvg:radical="general">
|
||||
<path id="kvg:08a9e-s1" kvg:type="㇔" d="M26,15.25c2.82,1.41,7.29,5.8,8,8"/>
|
||||
<path id="kvg:08a9e-s2" kvg:type="㇐" d="M12.37,32.97c1.25,0.28,2.88,0.66,4.36,0.53c7.02-0.59,17.78-1.75,25.95-3c1.52-0.23,3.57-0.38,5.16,0.03"/>
|
||||
<path id="kvg:08a9e-s3" kvg:type="㇐" d="M18.73,45.76c0.38,0.18,2.71,0.2,3.1,0.18c3.97-0.21,9.79-1.19,14.46-2.31c1.67-0.4,2.71-0.38,3.86-0.08"/>
|
||||
<path id="kvg:08a9e-s4" kvg:type="㇐" d="M18.73,58.89c0.89,0.23,1.89,0.36,3.35,0.15c3.89-0.54,10.71-1.51,14.85-2.29c0.7-0.13,1.82-0.26,2.61-0.1"/>
|
||||
<g id="kvg:08a9e-g2" kvg:element="口">
|
||||
<path id="kvg:08a9e-s5" kvg:type="㇑" d="M17.14,71.9c0.63,0.62,1.12,1.65,1.23,2.57c0.63,5.03,1.51,10.28,2.23,15.59c0.14,1.03,0.27,2.02,0.41,2.93"/>
|
||||
<path id="kvg:08a9e-s6" kvg:type="㇕b" d="M19.37,73.6c5.67-0.94,15.47-2.73,20.36-3.48c1.49-0.22,2.39,1.05,2.18,2.08c-0.71,3.44-2.27,9.75-3.23,13.89"/>
|
||||
<path id="kvg:08a9e-s7" kvg:type="㇐b" d="M21.47,89.02c3.95-0.45,10.71-1.19,16.28-1.61c1.21-0.09,2.36-0.17,3.41-0.22"/>
|
||||
</g>
|
||||
</g>
|
||||
<g id="kvg:08a9e-g3" kvg:element="吾" kvg:position="right" kvg:phon="吾">
|
||||
<g id="kvg:08a9e-g4" kvg:element="五">
|
||||
<g id="kvg:08a9e-g5" kvg:element="二" kvg:part="1">
|
||||
<path id="kvg:08a9e-s8" kvg:type="㇐" d="M51.79,17.49c1.38,0.26,3.91,0.28,5.27,0.15C63.88,17,72.62,15.62,80,15.32c2.3-0.1,3.67,0.04,4.81,0.15"/>
|
||||
</g>
|
||||
<path id="kvg:08a9e-s9" kvg:type="㇑a" d="M67.75,20.25c0.37,1.25,0.5,2.38,0.23,3.75c-0.75,3.78-6.03,23.83-7.96,31.58"/>
|
||||
<path id="kvg:08a9e-s10" kvg:type="㇕c" d="M52.18,36.96c1.82,0.66,4.17,0.95,5.84,0.66c8.48-1.5,16.13-3.06,22.74-4.1c2.49-0.39,4.05,1.27,3.71,2.93c-0.6,2.93-2.48,11.43-3.74,17.86"/>
|
||||
<g id="kvg:08a9e-g6" kvg:element="二" kvg:part="2">
|
||||
<path id="kvg:08a9e-s11" kvg:type="㇐" d="M46.33,58.46c1.13,0.24,3.94,0.2,5.07,0.08c12.34-1.29,19.11-2.39,40.88-4.02c1.88-0.14,3.75-0.02,4.69,0.09"/>
|
||||
</g>
|
||||
</g>
|
||||
<g id="kvg:08a9e-g7" kvg:element="口">
|
||||
<path id="kvg:08a9e-s12" kvg:type="㇑" d="M52.5,69.88c0.93,0.93,1.42,2.28,1.54,3.31c0.71,6.06,1.42,12.65,2.06,19.3c0.15,1.5,0.28,2.44,0.4,3.75"/>
|
||||
<path id="kvg:08a9e-s13" kvg:type="㇕b" d="M54.99,71.67c9.47-1.45,23.75-3.41,28.85-3.9c2.14-0.21,3.28,0.98,2.86,2.93c-0.84,3.88-3.08,12.57-4.39,17.58"/>
|
||||
<path id="kvg:08a9e-s14" kvg:type="㇐b" d="M57.2,91.49c5.94-0.55,14.67-1.24,23.54-1.76c1.3-0.08,2.63-0.13,3.97-0.2"/>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
</kanji>
|
||||
<kanji id="kvg:kanji_06a45">
|
||||
<g id="kvg:06a45" kvg:element="機">
|
||||
<g id="kvg:06a45-g1" kvg:element="木" kvg:position="left" kvg:radical="general">
|
||||
<path id="kvg:06a45-s1" kvg:type="㇐" d="M10.28,39.47c1.6,0.66,3.76,0.28,4.98,0.09c4.46-0.7,12.85-2.31,18.53-3.4c1.23-0.23,2.99-0.8,4.23-0.29"/>
|
||||
<path id="kvg:06a45-s2" kvg:type="㇑" d="M25.36,15c1.26,1.26,1.79,2.75,1.79,4.42c0,0.82-0.04,50.17-0.05,69.95c0,3.14,0,5.54,0,6.88"/>
|
||||
<path id="kvg:06a45-s3" kvg:type="㇒" d="M26.19,39.53c0,0.84-0.1,1.58-0.34,2.33C22,54.25,19.33,62.31,12.11,75.34"/>
|
||||
<path id="kvg:06a45-s4" kvg:type="㇔/㇏" d="M29.88,45.75c2.33,1.62,6.16,6.86,8.12,10"/>
|
||||
</g>
|
||||
<g id="kvg:06a45-g2" kvg:element="無" kvg:position="right">
|
||||
<g id="kvg:06a45-g3" kvg:position="top">
|
||||
<g id="kvg:06a45-g4" kvg:element="丿">
|
||||
<path id="kvg:06a45-s5" kvg:type="㇒" d="M54.51,12.5c0.04,0.56,0.15,1.62-0.08,2.25c-2.72,7.91-6.75,15.16-13.84,21.54"/>
|
||||
</g>
|
||||
<path id="kvg:06a45-s6" kvg:type="㇐" d="M52.83,25c0.72-0.03,2.13-0.1,3.17-0.21c8.01-0.86,23.7-3.89,29.02-4.4c1.4-0.13,2.24,0.17,2.94,0.35"/>
|
||||
<g id="kvg:06a45-g5" kvg:element="一">
|
||||
<path id="kvg:06a45-s7" kvg:type="㇐" d="M37.25,48.7c1.24,0.54,3.52,0.7,4.77,0.54C52,48,80.69,45.25,94.15,44.51c2.07-0.11,3.32,0.26,4.35,0.53"/>
|
||||
</g>
|
||||
<path id="kvg:06a45-s8" kvg:type="㇑" d="M47.09,37.25c0.89,0.5,1.43,2.25,1.6,3.25c0.18,1,0.31,15.75,0.48,27.5"/>
|
||||
<path id="kvg:06a45-s9" kvg:type="㇑" d="M58.37,30.75c0.89,0.5,1.62,2.23,1.6,3.25c-0.07,5.5,0.14,19.5,0.14,33.75"/>
|
||||
<path id="kvg:06a45-s10" kvg:type="㇑" d="M71.65,29.75c0.89,0.5,1.67,2.23,1.6,3.25c-0.47,8-0.88,17.5-1.5,33.25"/>
|
||||
<path id="kvg:06a45-s11" kvg:type="㇑" d="M85.34,30.25c0.89,0.5,1.87,2.28,1.6,4.25c-1.09,8-1.5,15.25-3.54,30.75"/>
|
||||
<path id="kvg:06a45-s12" kvg:type="㇐" d="M37.16,68.7c1.32,0.54,3.74,0.67,5.05,0.54c7.53-0.74,37.49-2.99,51.76-3.73c2.19-0.11,3.52,0.26,4.61,0.53"/>
|
||||
</g>
|
||||
<g id="kvg:06a45-g6" kvg:element="灬" kvg:variant="true" kvg:original="火" kvg:position="bottom">
|
||||
<path id="kvg:06a45-s13" kvg:type="㇔" d="M40.94,80.5c0,5.25-4.71,13-5.94,14.5"/>
|
||||
<path id="kvg:06a45-s14" kvg:type="㇔" d="M53.75,79.08c2.49,2.57,4.85,9.64,5.47,13.63"/>
|
||||
<path id="kvg:06a45-s15" kvg:type="㇔" d="M71.51,78.83c2.4,2.27,6.19,9.35,6.79,12.88"/>
|
||||
<path id="kvg:06a45-s16" kvg:type="㇔" d="M87.89,77.08c3.73,2.89,9.63,11.89,10.56,16.38"/>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
</kanji>
|
||||
</kanjivg>
|
||||
124
halkyon_learn_curator/test/kanjidic_test.dart
Normal file
124
halkyon_learn_curator/test/kanjidic_test.dart
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/sources/kanjidic.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
void main() {
|
||||
late List<Map<String, Object?>> lines;
|
||||
|
||||
setUpAll(() async {
|
||||
final jlpt = await readKanjiJlpt(
|
||||
File('test/fixtures/kanji-data-sample.json'),
|
||||
);
|
||||
lines = [
|
||||
for (final line in kanjidicDictionary(
|
||||
await File('test/fixtures/kanjidic2-sample.xml').readAsString(),
|
||||
jlpt: jlpt,
|
||||
))
|
||||
jsonDecode(line) as Map<String, Object?>,
|
||||
];
|
||||
});
|
||||
|
||||
Map<String, Object?> entry(String character) =>
|
||||
lines.skip(1).firstWhere((line) => line['id'] == character);
|
||||
Map<String, Object?> fields(String character) =>
|
||||
entry(character)['entry']! as Map<String, Object?>;
|
||||
|
||||
test('the header says the file holds characters, and whose they are', () {
|
||||
final header = lines.first;
|
||||
expect(header['source'], 'kanjidic');
|
||||
expect(header['kind'], 'characters');
|
||||
expect(header['sourceVersion'], '2026-282');
|
||||
expect((header['language'], header['glossLanguage']), ('ja', 'en'));
|
||||
expect(header['license'], 'CC-BY-SA-4.0');
|
||||
expect(
|
||||
[
|
||||
for (final credit in header['credits']! as List)
|
||||
(credit as Map)['name'],
|
||||
],
|
||||
['KANJIDIC2', 'JLPT kanji lists'],
|
||||
);
|
||||
});
|
||||
|
||||
test('a kanji: its readings, English meanings only, grade, strokes, '
|
||||
'frequency and JLPT level', () {
|
||||
final eat = entry('食');
|
||||
expect(eat['headword'], '食');
|
||||
expect(eat['reading'], isNull);
|
||||
expect(eat['labels'], ['jlpt:n5', 'grade:2']);
|
||||
expect(fields('食'), {
|
||||
'meanings': ['eat', 'food'],
|
||||
'on': ['ショク', 'ジキ'],
|
||||
'kun': ['く.う', 'く.らう', 'た.べる', 'は.む'],
|
||||
'grade': 2,
|
||||
'strokes': 9,
|
||||
'frequency': 328,
|
||||
'jlpt': 5,
|
||||
});
|
||||
expect(fields('一')['meanings'], isNot(contains('un')));
|
||||
});
|
||||
|
||||
test('a search finds a kanji by its on readings in either kana and in '
|
||||
'romaji, by whole kun readings and by its meanings; the kanji part '
|
||||
'of a kun reading only loosely', () {
|
||||
final eat = entry('食');
|
||||
expect(
|
||||
eat['keys'],
|
||||
containsAll([
|
||||
'食',
|
||||
'ショク',
|
||||
'しょく',
|
||||
'shoku',
|
||||
'たべる',
|
||||
'taberu',
|
||||
'くらう',
|
||||
'eat',
|
||||
'food',
|
||||
]),
|
||||
);
|
||||
expect(eat['keys'], isNot(contains('た')));
|
||||
expect(eat['looseKeys'], containsAll(['た', 'ta', 'く', 'ku']));
|
||||
// ひと- and ひと.つ: the suffix's dash goes, its kanji part is whole.
|
||||
expect(entry('一')['keys'], containsAll(['ひと', 'ひとつ', 'hitotsu']));
|
||||
expect(entry('一')['looseKeys'], isNot(contains('ひと')));
|
||||
});
|
||||
|
||||
test('levels and grades label the kanji and rank them: JLPT level, '
|
||||
'then school grade, then frequency', () {
|
||||
expect(entry('亜')['labels'], ['jlpt:n1', 'grade:secondary']);
|
||||
expect(entry('唖')['labels'], isEmpty);
|
||||
final ranks = [
|
||||
for (final kanji in ['一', '食', '亜', '唖']) entry(kanji)['rank']! as int,
|
||||
];
|
||||
expect(ranks, orderedEquals([...ranks]..sort()));
|
||||
});
|
||||
|
||||
test('a character with no Japanese reading and no English meaning is '
|
||||
'left out', () {
|
||||
expect(lines.skip(1).where((line) => line['id'] == '僲'), isEmpty);
|
||||
});
|
||||
|
||||
test('hiragana and katakana follow, the kana table each, before every '
|
||||
'kanji, found by their romaji', () {
|
||||
final kana = [
|
||||
for (final line in lines.skip(1))
|
||||
if ((line['labels']! as List).any(['hiragana', 'katakana'].contains))
|
||||
line,
|
||||
];
|
||||
expect(kana, hasLength(142));
|
||||
expect(kana.first['id'], 'あ');
|
||||
expect(kana[71]['id'], 'ア');
|
||||
expect(kana.last['id'], 'ポ');
|
||||
expect(entry('ア')['labels'], ['katakana']);
|
||||
expect(entry('ア')['keys'], containsAll(['ア', 'あ', 'a']));
|
||||
expect(fields('ぢ'), {'romaji': 'ji', 'script': 'hiragana'});
|
||||
// を is "wo" in the table, found by "o" too.
|
||||
expect(fields('を')['romaji'], 'wo');
|
||||
expect(entry('を')['keys'], containsAll(['wo', 'o']));
|
||||
final lastKana = kana
|
||||
.map((line) => line['rank']! as int)
|
||||
.reduce((a, b) => a > b ? a : b);
|
||||
expect(entry('食')['rank']! as int, greaterThan(lastKana));
|
||||
});
|
||||
}
|
||||
49
halkyon_learn_curator/test/kanjivg_test.dart
Normal file
49
halkyon_learn_curator/test/kanjivg_test.dart
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
import 'dart:io';
|
||||
|
||||
import 'package:halkyon_learn_curator/src/sources/kanjivg.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
void main() {
|
||||
late Map<String, CharacterStrokes> strokes;
|
||||
|
||||
setUpAll(() async {
|
||||
strokes = kanjivgStrokes(
|
||||
await File('test/fixtures/kanjivg-sample.xml').readAsString(),
|
||||
);
|
||||
});
|
||||
|
||||
test("a character's strokes are its paths in the order they're written", () {
|
||||
final eat = strokes['食']!;
|
||||
expect(eat.strokes, hasLength(9));
|
||||
expect(eat.strokes.first, startsWith('M52.75,10.5c'));
|
||||
expect(eat.strokes.last, startsWith('M51.5,71C'));
|
||||
expect(strokes['あ']!.strokes, hasLength(3));
|
||||
});
|
||||
|
||||
test('its components are the characters KanjiVG first splits it into, '
|
||||
'looking through groups that are none', () {
|
||||
expect(strokes['語']!.components, ['言', '吾']);
|
||||
expect(strokes['食']!.components, isEmpty);
|
||||
});
|
||||
|
||||
test("the character is the code point in the entry's id, not KanjiVG's "
|
||||
'element name, which can be wrong', () {
|
||||
// KanjiVG's U+6A45 (橅) names itself 機.
|
||||
expect(strokes.keys, containsAll(['橅', 'あ', '食', '語']));
|
||||
expect(strokes.keys, isNot(contains('機')));
|
||||
});
|
||||
|
||||
test('the stroke files hold every 256 code points, named by their upper '
|
||||
'bits', () {
|
||||
expect(strokeFileOf('食'), '098');
|
||||
expect(strokeFileOf('あ'), '030');
|
||||
expect(strokeFileOf('𠀋'), '200');
|
||||
final files = strokeFiles(strokes);
|
||||
expect(files.keys, containsAll(['030', '098', '08a', '06a']));
|
||||
expect(files['08a']!['語'], {
|
||||
's': strokes['語']!.strokes,
|
||||
'c': ['言', '吾'],
|
||||
});
|
||||
expect(files['098']!['食'], {'s': strokes['食']!.strokes});
|
||||
});
|
||||
}
|
||||
Loading…
Reference in a new issue