halkyon/halkyon_learn_flutter/integration_test/japanese_analyser_test.dart

64 lines
2.6 KiB
Dart
Raw Normal View History

feat(dictionary): Japanese cut into words and looked up on the device Once the person agrees, the device holds MeCab's IPADIC and the instance's JMdict and cuts lines as Jisho.org does, so their text never leaves it (ADR 0012). MeCab comes through mecab_for_dart, whose build hook compiles it; its tokens are grouped by Ve's rules, written anew in Dart, with に kept a word of its own. Each word is found as it stands or as a form of a word, else by its dictionary form, else as the longest words it holds from token to token; longer words over token boundaries are offered as before. The curator packs Ubuntu's compiled IPADIC beside the dictionary files (curate ipadic, and every update run). The server exports its JMdict from its tables, so the device finds the same keys and keeps the same versions, and serves both at /device-dictionaries/, named in dictionary.deviceFiles('ja') with their hashes; Caddy forwards the path. The app checks each file against its hash, unpacks IPADIC, builds a SQLite database from the export on an isolate, and asks the instance for newer files every 30 days, downloading only what changed. The Watch screen offers the download once per device with its size (34 MB, some 150 MB once set up); Settings downloads and removes them. Without them, or should cutting on the device fail, lines go to the instance as before. MeCab's, IPADIC's and Ve's licences join the licence list. Every build now needs a C++ compiler; mecab_for_dart holds code_assets at 1.x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
2026-10-15 21:55:00 +02:00
import 'dart:io';
import 'package:flutter_test/flutter_test.dart';
import 'package:halkyon_learn_flutter/src/feature/dictionary/data/device_dictionaries.dart';
import 'package:halkyon_learn_flutter/src/feature/dictionary/data/device_dictionary_store.dart';
import 'package:halkyon_learn_flutter/src/feature/dictionary/data/japanese_analyser.dart';
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/sentence_words.dart';
import 'package:integration_test/integration_test.dart';
// Japanese cut on the device for real (ADR 0012): MeCab as the package's
// build hook compiles it for the platform, IPADIC as the curator packs it,
// and JMdict as an instance exports it. On the desktop,
//
// flutter test integration_test/japanese_analyser_test.dart -d linux \
// --dart-define=IPADIC_PACK=<mecab-ipadic.tar.gz> \
// --dart-define=JMDICT_EXPORT=<jmdict-en-….jsonl.gz>
//
// with the files an instance serves at /device-dictionaries/ (both named in
// `dictionary.deviceFiles('ja')`); the curator's `curate ipadic` writes the
// pack.
const _pack = String.fromEnvironment('IPADIC_PACK');
const _export = String.fromEnvironment('JMDICT_EXPORT');
void main() {
IntegrationTestWidgetsFlutterBinding.ensureInitialized();
late Directory folder;
setUpAll(() async {
folder = Directory.systemTemp.createTempSync('japanese');
unpackIpadic(File(_pack), Directory('${folder.path}/ipadic'));
await buildDeviceDictionary(
File(_export),
File('${folder.path}/jmdict.sqlite'),
);
});
tearDownAll(() => folder.deleteSync(recursive: true));
String cut(List<SentencePart> parts) => [
for (final part in parts)
if (part.isWord) part.text else '(${part.text})',
].join(' ');
testWidgets('cuts lines into words as Jisho.org does, and finds each in '
'JMdict', (tester) async {
final lines = await tester.runAsync(
() => IsolateJapaneseAnalyser().wordsOf(
ipadic: '${folder.path}/ipadic',
database: '${folder.path}/jmdict.sqlite',
sentences: ['今日はいい天気ですね。', 'お前がもうろくだよ', '勉強している', '静かな部屋'],
),
);
final [weather, senile, study, room] = lines!;
expect(cut(weather), '今日 は いい 天気 です ね (。)');
expect(weather.first.entries.first.headword, '今日');
expect(cut(senile), 'お前 が もうろく だ よ');
expect(senile[2].entries.first.headword, '耄碌');
expect(study.single.entries.first.headword, '勉強');
expect(study.single.steps, isNotEmpty);
expect(cut(room), '静かな 部屋');
expect(room.first.entries.first.headword, '静か');
});
}