feat(curator): dictionaries of German, Greek and English from Wiktionary
The languages besides Japanese come from the English Wiktionary as
Wiktextract reads it (decided 2026-10-14: German, Greek and English
first, then Chinese, then Korean): kaikki.org's raw dump, every language
in one file, as its per-language files are being withdrawn.
`curate wiktionary --languages de,el,en` reads it once, streamed, and
writes German and Greek with Wiktionary's English meanings and English
with what its translation tables give in German, the app's other
language (`wiktionary-de-en`, `-el-en`, `-en-de`; sources
`wiktionary-<language>`, as the server holds one source per meaning
language). Inflected forms, and Wiktionary's entries for forms alone
("Häusern: dative plural of Haus"), find their word instead of being
entries; names and affixes are left out, as are articles that tables
give cells of their own, abbreviations and rare spellings, which would
match common words. Entries keep their part of speech, genders, IPA,
principal forms and examples keyed by language as JMdict's.
wordfreq's small lists (CC BY-SA 4.0, crediting SUBTLEX) in data/ rank
the words by how often they are used and label the 15,000 most used
`common`. Typing without accents or umlauts finds a word (`strasse`,
`haeuser`, `spiti`).
The format gains `textKeys`: what finds an entry in running text, its
own forms, never its meanings, which would match words of the meaning
language in a subtitle. `curate update --wiktionary de,el,en` keeps the
languages current: it asks kaikki.org when the dump last changed and
converts it again only when it is newer, or the languages or the
conversion changed. The whole dump converts in about ten minutes in
under 300 MB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
5683a1f33e
commit
6d57f50b16
1 changed files with 1495 additions and 22 deletions
|
|
@ -97,4 +97,85 @@ void main() {
|
|||
test('a folder missing a file was never brought up to date', () {
|
||||
expect(lastUpdated(into), isNull);
|
||||
});
|
||||
|
||||
group("Wiktionary's dump", () {
|
||||
var dumped = DateTime.utc(2026, 10, 3, 8, 24);
|
||||
var opened = 0;
|
||||
|
||||
setUp(() {
|
||||
dumped = DateTime.utc(2026, 10, 3, 8, 24);
|
||||
opened = 0;
|
||||
});
|
||||
|
||||
Future<DateTime> modified(Uri uri) async => dumped;
|
||||
|
||||
/// The sample in place of kaikki.org's dump.
|
||||
Future<Download> open(Uri uri) async {
|
||||
opened++;
|
||||
return (
|
||||
modified: dumped,
|
||||
bytes: Stream.value(
|
||||
gzip.encode(
|
||||
File('test/fixtures/wiktionary-sample.jsonl').readAsBytesSync(),
|
||||
),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
Future<List<File>> update(Set<String> languages) => updateWiktionary(
|
||||
into: into,
|
||||
data: Directory('data'),
|
||||
languages: languages,
|
||||
modified: modified,
|
||||
open: open,
|
||||
);
|
||||
|
||||
test(
|
||||
'is converted for the languages named, German and Greek with English '
|
||||
'meanings, English with German ones, and ranked by frequency',
|
||||
() async {
|
||||
final written = await update({'de', 'el', 'en'});
|
||||
|
||||
expect(
|
||||
[for (final file in written) file.uri.pathSegments.last]..sort(),
|
||||
[
|
||||
'wiktionary-de-en.jsonl.gz',
|
||||
'wiktionary-el-en.jsonl.gz',
|
||||
'wiktionary-en-de.jsonl.gz',
|
||||
],
|
||||
);
|
||||
expect(
|
||||
header('wiktionary-de-en.jsonl.gz')['sourceVersion'],
|
||||
'2026-10-03',
|
||||
);
|
||||
expect([
|
||||
for (final credit
|
||||
in header('wiktionary-de-en.jsonl.gz')['credits']! as List)
|
||||
(credit as Map)['name'],
|
||||
], contains('wordfreq'));
|
||||
expect(File('${into.path}/$wiktionaryStateFile').existsSync(), isTrue);
|
||||
},
|
||||
);
|
||||
|
||||
test(
|
||||
'is downloaded again only once it changed, or the languages did',
|
||||
() async {
|
||||
await update({'de'});
|
||||
expect(await update({'de'}), isEmpty);
|
||||
expect(opened, 1);
|
||||
|
||||
dumped = DateTime.utc(2026, 11, 2);
|
||||
expect(await update({'de'}), hasLength(1));
|
||||
expect(opened, 2);
|
||||
|
||||
expect(await update({'de', 'el'}), hasLength(2));
|
||||
expect(opened, 3);
|
||||
},
|
||||
);
|
||||
|
||||
test('nothing is asked for without languages', () async {
|
||||
expect(await update({}), isEmpty);
|
||||
expect(opened, 0);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue