feat(dictionary): German, Greek, English, Chinese and Korean words
A studyset in German, Greek, English, Chinese or Korean finds its dictionary as one in Japanese does (ADR 0008), and its subtitles are cut into words by its language (`WordCutting`): Chinese by its longest words as they stand, the longer words over each cut offered as for Japanese; Korean at its spaces, a word the dictionary lacks as it stands taken without the particle it ends with (집에서 is 집, 아이 stays a child); every other language at its spaces, a word written exactly so first (the noun Essen before the verb essen). A studyset finds the dictionary of its language tag, else of its language alone (zh for zh-Hans). A Wiktionary word shows its genders, principal forms, IPA, romanisation and Hanja; a German or Greek noun of one gender goes onto its card with its article (das Haus, το σπίτι); its tag is its word class, named in full (Noun, Adjective, Preposition), as Wiktionary's adj and adv would read as JMdict's codes. Articles, pronouns, prepositions and conjunctions are left out of a video's words. Chinese words show their pinyin over each character on the Watch screen, carry their HSK level as a tag (HSK-1 … HSK-7-9), and the dictionary adds a level whole; the Watch screen filters by HSK levels for Chinese, JLPT's for Japanese, and by none for a language without. A reading is typed in a session only when it is kana, so pinyin isn't. Words are looked up on the web on Jisho in Japanese, on Wiktionary at the language's section otherwise, and a search's hint speaks of kana and romaji only in Japanese. `DICTIONARY_LANGUAGES` in deploy/.env names the languages an instance keeps (unset: de,el,en,ko,zh; empty: none). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
292e369ed3
commit
a2441ecdf2
GPG key ID:
07F6FCF405488B64
32 changed files with 1639 additions and 99 deletions
File diff suppressed because one or more lines are too long
|
|
@ -38,6 +38,14 @@ GITHUB_CLIENT_SECRET=
|
|||
# Running modified code, set it to where your modified source is published.
|
||||
SOURCE_URL=
|
||||
|
||||
# The languages besides Japanese whose dictionaries this instance keeps (the dictionaries
|
||||
# service): Chinese from CC-CEDICT, weekly (some 4 MB), in simplified and traditional
|
||||
# characters; the rest from Wiktionary (German, Greek and Korean with English meanings,
|
||||
# English with German ones), whose whole dump (some 3 GB) it downloads whenever
|
||||
# kaikki.org makes a new one, about monthly, and converts in ten minutes. Unset:
|
||||
# de,el,en,ko,zh. Empty: none.
|
||||
#DICTIONARY_LANGUAGES=de,el,en,ko,zh
|
||||
|
||||
# The Jellyfin server the app suggests to this instance's people, who watch and listen
|
||||
# to learn from it (Halkyon Learn's Watch screen). Empty: `https://jellyfin.` on the
|
||||
# domain registered for this instance (learn.example.org → jellyfin.example.org), if a
|
||||
|
|
|
|||
|
|
@ -95,9 +95,12 @@ services:
|
|||
|
||||
# Keeps the dictionary files current from EDRDG's files of the day, once they are a
|
||||
# week old (EDRDG's licence asks it of servers showing its dictionaries at least
|
||||
# monthly; the curator's README): in a container of its own, the curator never in
|
||||
# the server's image or process. It only downloads (some 25 MB a week) and writes
|
||||
# into the volume; the files are never backed up, as they can be had again.
|
||||
# monthly; the curator's README), and the languages of DICTIONARY_LANGUAGES: Chinese
|
||||
# from MDBG's CC-CEDICT as often, the rest from Wiktionary's dump whenever kaikki.org
|
||||
# makes a new one (about monthly, some 3 GB, converted as it downloads): in a
|
||||
# container of its own, the curator never in the server's image or process. It only
|
||||
# downloads and writes into the volume; the files are never backed up, as they can be
|
||||
# had again.
|
||||
dictionaries:
|
||||
build:
|
||||
context: ..
|
||||
|
|
@ -106,8 +109,11 @@ services:
|
|||
restart: unless-stopped
|
||||
# Docker's stop signal reaches the curator through a small init.
|
||||
init: true
|
||||
# A conversion takes some 400 MB for a minute a week.
|
||||
# JMdict's conversion takes some 400 MB for a minute a week, Wiktionary's some
|
||||
# 300 MB for ten minutes a month.
|
||||
mem_limit: 1g
|
||||
# Unset: German, Greek, English, Korean and Chinese; empty: none.
|
||||
command: ["--every", "7", "--languages", "${DICTIONARY_LANGUAGES-de,el,en,ko,zh}"]
|
||||
volumes:
|
||||
- dictionaries:/dictionaries
|
||||
logging: *logging
|
||||
|
|
|
|||
90
docs/adr/0008-languages-beyond-japanese.md
Normal file
90
docs/adr/0008-languages-beyond-japanese.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
# ADR 0008: Dictionaries and word cutting for languages beyond Japanese
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-10-14
|
||||
|
||||
## Context
|
||||
|
||||
Halkyon Learn's language support was built for Japanese: JMdict and KANJIDIC2 as
|
||||
dictionaries, a longest-match splitter with Japanese conjugations for subtitles,
|
||||
furigana, JLPT levels, Jisho links. Studysets can already be in 22 languages, and the
|
||||
app speaks English and German, so learners of German, Greek, English, Chinese and Korean
|
||||
find no dictionary to add words from, and their subtitles aren't cut into words.
|
||||
|
||||
What a dictionary for learners needs, per language: meanings in the learner's language,
|
||||
the forms a word takes in running text, readings where the script doesn't say them,
|
||||
some order of usefulness (frequency, exam levels), and a licence that lets the cards
|
||||
made from it be published under CC BY-SA, as studysets are.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Order** (the developer's choice): German, Greek and English first, sharing one
|
||||
source and one way of cutting text; then Chinese; then Korean, which needs morphology
|
||||
(from Wiktionary for now, with its particles taken off by rule).
|
||||
2. **Wiktionary for the languages without a dedicated dictionary**, as Wiktextract reads
|
||||
the English Wiktionary (kaikki.org, CC BY-SA 4.0 and GFDL; Wiktextract's code MIT). Its
|
||||
per-language files are being withdrawn, so the curator reads the raw dump of every
|
||||
language (some 3 GB), streamed, once. German and Greek get Wiktionary's English
|
||||
meanings; English gets German ones from the translation tables of its entries, which
|
||||
name the English meaning each translation is of (the German Wiktionary has German
|
||||
*definitions* of English words, worse for cards than translations). German glosses for
|
||||
Greek, Chinese or Korean would only come from inverted translation tables, weak and
|
||||
noisy: those studysets fall back to English meanings, as the app's dictionary choice
|
||||
already does.
|
||||
3. **Each instance converts the dump itself** (the developer's choice, as it does EDRDG's
|
||||
files), whenever kaikki.org makes a new one (about monthly): the curator asks for its
|
||||
date first and downloads only when it is newer. Ten minutes in under 300 MB on
|
||||
learn.halkyon.eu's server. `DICTIONARY_LANGUAGES` (unset: `de,el,en,ko,zh`) names
|
||||
the languages.
|
||||
4. **Forms find their word; they aren't entries.** Wiktionary has entries for inflected
|
||||
forms alone ("Häusern: dative plural of Haus"); they become keys of their word, as do
|
||||
the forms of a word's inflection tables. Left out of the keys: articles that tables
|
||||
give cells of their own (Greek's `ο`, `του`), abbreviations (`TO` for turnover) and
|
||||
rare or obsolete spellings (`the` for thee), as they would match common words.
|
||||
5. **Text keys** (`textKeys`, a field of the dictionary format): what finds an entry in
|
||||
running text, its own forms, never its meanings, which would match words of the meaning
|
||||
language in a subtitle. JMdict's files name none and stay as they were.
|
||||
6. **Frequency from wordfreq** (its small lists, CC BY-SA 4.0, crediting SUBTLEX):
|
||||
Wiktionary and CC-CEDICT have no commonness of their own; the 15,000 most used words
|
||||
are `common`, and words rank by their own frequency (a form's only when the word has
|
||||
none, at half its rank), so a rare word never ranks by a frequent form.
|
||||
7. **Chinese from CC-CEDICT** (CC BY-SA 4.0, made for learners, daily), as two
|
||||
dictionaries of the same entries: simplified (`zh-Hans`) and traditional (`zh-Hant`), so
|
||||
a studyset finds words as it writes them; pinyin with tone marks, a syllable to a
|
||||
character; entries that are only variants find their word instead of repeating it.
|
||||
8. **HSK levels of the 2025 syllabus** (the developer's choice: level tags, credited),
|
||||
which HSK 3.0's examination follows since July 2026, as hsk30 extracted them (MIT). The
|
||||
Ministry claims copyright on the standard's documents; only each word's level reaches
|
||||
the dictionaries, a fact about it. Levels 7 to 9 are one, as the syllabus has them. A
|
||||
word's level goes to the entry read as the syllabus reads it, with its tones (和 hé, not
|
||||
hè), else by its letters (the syllabus marks the tone changes of 一 and 不).
|
||||
9. **Text is cut by its language** (`WordCutting`): Japanese by its longest words with
|
||||
conjugations, Chinese by its longest words as they stand (with the longer words
|
||||
running over each cut offered beside it, as for Japanese), Korean at its spaces, a word
|
||||
the dictionary lacks as it stands taken without the particle or copula ending it ends
|
||||
with (집에서 is 집, 아이 stays a child; Korean's own grammar, no analyser: a verb's
|
||||
endings are among its forms in Wiktionary), every other language at its spaces, each
|
||||
word looked up as it stands, an entry written exactly so first (German's noun `Essen`
|
||||
before its verb `essen`).
|
||||
10. **A noun goes onto its card with its article** where the language has gendered
|
||||
articles and the noun one gender (`das Haus`, `το σπίτι`): learners learn the gender
|
||||
with the word, and the article says it.
|
||||
11. **Words are looked up on the web** on Jisho in Japanese, built on JMdict, and on
|
||||
Wiktionary at the language's section otherwise, which the dictionaries are drawn from.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An instance downloads some 3 GB a month more, and holds some 180,000 entries more
|
||||
(German 84,000, English 66,000, Greek 27,000) besides CC-CEDICT's 122,000 in each script.
|
||||
- Wiktionary has no stable ids: an entry's id is its word, part of speech and etymology
|
||||
number (`Haus/noun`). Cards keep it as a soft reference, as they keep JMdict's; an entry
|
||||
Wiktionary splits or renames is no longer found by a card made from it, which keeps
|
||||
what it copied.
|
||||
- German compounds Wiktionary lacks (`Haustürschlüssel`) and separable verbs split in a
|
||||
sentence (`rufe … an`) aren't found; a Korean verb in a form Wiktionary's table lacks
|
||||
isn't either. KRDict (the National Institute of Korean Language's learners'
|
||||
dictionary, CC BY-SA 2.0 KR, with levels and Hanja) would do better for Korean; its
|
||||
download needs a person on its site, or an API key.
|
||||
- Not bundled, as their licences forbid what studysets need: Ding and FreeDict's German
|
||||
(GPL), the Goethe-Institut's and the English Vocabulary Profile's level lists
|
||||
(copyrighted), the Kelly lists (non-commercial), KRDict's sound files.
|
||||
|
|
@ -635,6 +635,7 @@
|
|||
"studysetFromDictionary": "Aus dem Wörterbuch",
|
||||
"dictionaryTitle": "Wörterbuch",
|
||||
"dictionarySearchHint": "Ein Wort in Kanji, Kana oder Romaji, oder eine Bedeutung",
|
||||
"dictionarySearchHintWords": "Ein Wort, in jeder Form, oder eine Bedeutung",
|
||||
"dictionaryNothingFound": "Nichts gefunden für „{query}“.",
|
||||
"dictionaryUnreachable": "Das Wörterbuch ist gerade nicht erreichbar.",
|
||||
"dictionaryRetry": "Erneut versuchen",
|
||||
|
|
@ -1804,7 +1805,7 @@
|
|||
"watchProgressLearning": "Am Lernen",
|
||||
"watchProgressDue": "Fällig",
|
||||
"watchProgressLearned": "Gelernt · {days, plural, =0{heute wieder} =1{morgen wieder} other{in {days} Tagen wieder}}",
|
||||
"watchJisho": "Auf Jisho nachschlagen",
|
||||
"watchLookUpOn": "Auf {site} nachschlagen",
|
||||
"watchWider": "Längere Wörter hier",
|
||||
"watchWordsTitleSong": "Wörter in diesem Lied",
|
||||
"watchStudyWordsSong": "Die Wörter dieses Lieds lernen",
|
||||
|
|
@ -1845,7 +1846,7 @@
|
|||
"watchAdded": "{word} hinzugefügt",
|
||||
"watchStretchNone": "„{text}“ steht nicht als ein Wort im Wörterbuch.",
|
||||
"watchStretchFailed": "Das Nachschlagen hat nicht geklappt. Wähle es noch einmal, um es erneut zu versuchen.",
|
||||
"watchStretchJisho": "Auf Jisho suchen",
|
||||
"watchStretchSearch": "Auf {site} suchen",
|
||||
"watchTapAWord": "Tipp auf ein Wort der Untertitel, um es nachzuschlagen, oder zieh über Zeichen, um einen beliebigen Teil einer Zeile nachzuschlagen (auf einem Touchscreen erst gedrückt halten).",
|
||||
"watchFormOf": "Eine Form von {word}",
|
||||
"watchFailed": "Das Video konnte nicht abgespielt werden."
|
||||
|
|
|
|||
|
|
@ -719,6 +719,7 @@
|
|||
"studysetFromDictionary": "From the dictionary",
|
||||
"dictionaryTitle": "Dictionary",
|
||||
"dictionarySearchHint": "A word in kanji, kana or romaji, or a meaning",
|
||||
"dictionarySearchHintWords": "A word, any form of it, or a meaning",
|
||||
"dictionaryNothingFound": "Nothing found for “{query}”.",
|
||||
"@dictionaryNothingFound": {"placeholders": {"query": {"type": "String"}}},
|
||||
"dictionaryUnreachable": "The dictionary can't be reached right now.",
|
||||
|
|
@ -2122,7 +2123,8 @@
|
|||
"watchProgressDue": "Due",
|
||||
"watchProgressLearned": "Learned · {days, plural, =0{again today} =1{again tomorrow} other{again in {days} days}}",
|
||||
"@watchProgressLearned": {"placeholders": {"days": {"type": "int"}}},
|
||||
"watchJisho": "Look up on Jisho",
|
||||
"watchLookUpOn": "Look up on {site}",
|
||||
"@watchLookUpOn": {"placeholders": {"site": {"type": "String"}}},
|
||||
"watchWider": "Longer words here",
|
||||
"watchWordsTitleSong": "Words in this song",
|
||||
"watchStudyWordsSong": "Study this song's words",
|
||||
|
|
@ -2168,7 +2170,8 @@
|
|||
"watchStretchNone": "{text} isn't in the dictionary as one word.",
|
||||
"@watchStretchNone": {"placeholders": {"text": {"type": "String"}}},
|
||||
"watchStretchFailed": "Couldn't look it up. Choose it again to try once more.",
|
||||
"watchStretchJisho": "Search Jisho for it",
|
||||
"watchStretchSearch": "Search {site} for it",
|
||||
"@watchStretchSearch": {"placeholders": {"site": {"type": "String"}}},
|
||||
"@watchAdded": {"placeholders": {"word": {"type": "String"}}},
|
||||
"watchTapAWord": "Tap a word of the subtitles to look it up, or drag across characters to look up any part of a line (on a touch screen, hold first).",
|
||||
"watchFormOf": "A form of {word}",
|
||||
|
|
|
|||
|
|
@ -3796,6 +3796,12 @@ abstract class AppLocalizations {
|
|||
/// **'A word in kanji, kana or romaji, or a meaning'**
|
||||
String get dictionarySearchHint;
|
||||
|
||||
/// No description provided for @dictionarySearchHintWords.
|
||||
///
|
||||
/// In en, this message translates to:
|
||||
/// **'A word, any form of it, or a meaning'**
|
||||
String get dictionarySearchHintWords;
|
||||
|
||||
/// No description provided for @dictionaryNothingFound.
|
||||
///
|
||||
/// In en, this message translates to:
|
||||
|
|
@ -10665,11 +10671,11 @@ abstract class AppLocalizations {
|
|||
/// **'Learned · {days, plural, =0{again today} =1{again tomorrow} other{again in {days} days}}'**
|
||||
String watchProgressLearned(int days);
|
||||
|
||||
/// No description provided for @watchJisho.
|
||||
/// No description provided for @watchLookUpOn.
|
||||
///
|
||||
/// In en, this message translates to:
|
||||
/// **'Look up on Jisho'**
|
||||
String get watchJisho;
|
||||
/// **'Look up on {site}'**
|
||||
String watchLookUpOn(String site);
|
||||
|
||||
/// No description provided for @watchWider.
|
||||
///
|
||||
|
|
@ -10911,11 +10917,11 @@ abstract class AppLocalizations {
|
|||
/// **'Couldn\'t look it up. Choose it again to try once more.'**
|
||||
String get watchStretchFailed;
|
||||
|
||||
/// No description provided for @watchStretchJisho.
|
||||
/// No description provided for @watchStretchSearch.
|
||||
///
|
||||
/// In en, this message translates to:
|
||||
/// **'Search Jisho for it'**
|
||||
String get watchStretchJisho;
|
||||
/// **'Search {site} for it'**
|
||||
String watchStretchSearch(String site);
|
||||
|
||||
/// No description provided for @watchTapAWord.
|
||||
///
|
||||
|
|
|
|||
|
|
@ -2311,6 +2311,10 @@ class AppLocalizationsDe extends AppLocalizations {
|
|||
String get dictionarySearchHint =>
|
||||
'Ein Wort in Kanji, Kana oder Romaji, oder eine Bedeutung';
|
||||
|
||||
@override
|
||||
String get dictionarySearchHintWords =>
|
||||
'Ein Wort, in jeder Form, oder eine Bedeutung';
|
||||
|
||||
@override
|
||||
String dictionaryNothingFound(String query) {
|
||||
return 'Nichts gefunden für „$query“.';
|
||||
|
|
@ -6720,7 +6724,9 @@ class AppLocalizationsDe extends AppLocalizations {
|
|||
}
|
||||
|
||||
@override
|
||||
String get watchJisho => 'Auf Jisho nachschlagen';
|
||||
String watchLookUpOn(String site) {
|
||||
return 'Auf $site nachschlagen';
|
||||
}
|
||||
|
||||
@override
|
||||
String get watchWider => 'Längere Wörter hier';
|
||||
|
|
@ -6879,7 +6885,9 @@ class AppLocalizationsDe extends AppLocalizations {
|
|||
'Das Nachschlagen hat nicht geklappt. Wähle es noch einmal, um es erneut zu versuchen.';
|
||||
|
||||
@override
|
||||
String get watchStretchJisho => 'Auf Jisho suchen';
|
||||
String watchStretchSearch(String site) {
|
||||
return 'Auf $site suchen';
|
||||
}
|
||||
|
||||
@override
|
||||
String get watchTapAWord =>
|
||||
|
|
|
|||
|
|
@ -2293,6 +2293,10 @@ class AppLocalizationsEn extends AppLocalizations {
|
|||
String get dictionarySearchHint =>
|
||||
'A word in kanji, kana or romaji, or a meaning';
|
||||
|
||||
@override
|
||||
String get dictionarySearchHintWords =>
|
||||
'A word, any form of it, or a meaning';
|
||||
|
||||
@override
|
||||
String dictionaryNothingFound(String query) {
|
||||
return 'Nothing found for “$query”.';
|
||||
|
|
@ -6661,7 +6665,9 @@ class AppLocalizationsEn extends AppLocalizations {
|
|||
}
|
||||
|
||||
@override
|
||||
String get watchJisho => 'Look up on Jisho';
|
||||
String watchLookUpOn(String site) {
|
||||
return 'Look up on $site';
|
||||
}
|
||||
|
||||
@override
|
||||
String get watchWider => 'Longer words here';
|
||||
|
|
@ -6818,7 +6824,9 @@ class AppLocalizationsEn extends AppLocalizations {
|
|||
'Couldn\'t look it up. Choose it again to try once more.';
|
||||
|
||||
@override
|
||||
String get watchStretchJisho => 'Search Jisho for it';
|
||||
String watchStretchSearch(String site) {
|
||||
return 'Search $site for it';
|
||||
}
|
||||
|
||||
@override
|
||||
String get watchTapAWord =>
|
||||
|
|
|
|||
|
|
@ -11,7 +11,8 @@ import 'package:halkyon_learn_flutter/src/feature/kanji/model/radical.dart';
|
|||
/// The labels a whole level is added by, in the order offered: for
|
||||
/// Japanese words the JLPT levels, easiest first, then the other common
|
||||
/// words; for its characters the kana, then the JLPT levels, then the
|
||||
/// school grades.
|
||||
/// school grades; for Chinese words the HSK levels, then the other common
|
||||
/// words; for any other language its common words.
|
||||
List<String> levelLabelsOf(Dictionary dictionary) => switch (dictionary) {
|
||||
Dictionary(kind: DictionaryKind.characters) => const [
|
||||
'hiragana',
|
||||
|
|
@ -37,6 +38,17 @@ List<String> levelLabelsOf(Dictionary dictionary) => switch (dictionary) {
|
|||
'jlpt:n1',
|
||||
'common',
|
||||
],
|
||||
// The HSK levels of 2025, the last for 7 to 9.
|
||||
Dictionary(:final language) when language.startsWith('zh') => const [
|
||||
'hsk:1',
|
||||
'hsk:2',
|
||||
'hsk:3',
|
||||
'hsk:4',
|
||||
'hsk:5',
|
||||
'hsk:6',
|
||||
'hsk:7',
|
||||
'common',
|
||||
],
|
||||
_ => const ['common'],
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import 'dart:math' as math;
|
|||
import 'package:halkyon_learn_flutter/src/feature/dictionary/data/dictionary_gateway.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/dictionary.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/sentence_words.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/text_words.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/studyset/model/studyset_tree.dart';
|
||||
|
||||
/// The instance's dictionaries: which there are, words looked up in one,
|
||||
|
|
@ -20,9 +21,10 @@ class DictionaryRepository(final DictionaryGateway _gateway) {
|
|||
_known ??= await _gateway.dictionaries();
|
||||
|
||||
/// The dictionary of [kind] (words unless given) for a studyset in
|
||||
/// [languages]: of its front's language, with meanings in its back's,
|
||||
/// else in English, else in any; null when there is none, or no front
|
||||
/// language.
|
||||
/// [languages]: of its front's language (the dictionary's tag as it is,
|
||||
/// else its language alone: `zh` for `zh-Hans`), with meanings in its
|
||||
/// back's, else in English, else in any; null when there is none, or no
|
||||
/// front language.
|
||||
Future<Dictionary?> dictionaryFor(
|
||||
StudysetLanguages languages, {
|
||||
DictionaryKind kind = DictionaryKind.words,
|
||||
|
|
@ -31,12 +33,28 @@ class DictionaryRepository(final DictionaryGateway _gateway) {
|
|||
if (front == null) {
|
||||
return null;
|
||||
}
|
||||
final ofLanguage = [
|
||||
String primary(String tag) => tag.split('-').first.toLowerCase();
|
||||
final ofKind = [
|
||||
for (final dictionary in await dictionaries())
|
||||
if (dictionary.language == front && dictionary.kind == kind) dictionary,
|
||||
if (dictionary.kind == kind) dictionary,
|
||||
];
|
||||
final exactly = [
|
||||
for (final dictionary in ofKind)
|
||||
if (dictionary.language == front) dictionary,
|
||||
];
|
||||
final ofLanguage = exactly.isNotEmpty
|
||||
? exactly
|
||||
: [
|
||||
for (final dictionary in ofKind)
|
||||
if (primary(dictionary.language) == primary(front)) dictionary,
|
||||
];
|
||||
final back = languages.back;
|
||||
return ofLanguage
|
||||
.where((dictionary) => dictionary.glossLanguage == languages.back)
|
||||
.where(
|
||||
(dictionary) =>
|
||||
back != null &&
|
||||
primary(dictionary.glossLanguage) == primary(back),
|
||||
)
|
||||
.firstOrNull ??
|
||||
ofLanguage
|
||||
.where((dictionary) => dictionary.glossLanguage == 'en')
|
||||
|
|
@ -98,19 +116,21 @@ class DictionaryRepository(final DictionaryGateway _gateway) {
|
|||
final _parts = <(String, String, String), List<SentencePart>>{};
|
||||
static const _sentencesKept = 5000;
|
||||
|
||||
/// [sentences] each cut into the words of [dictionary] (`sentenceParts`),
|
||||
/// those cut before as they were; the rest asked for in as few requests
|
||||
/// as [keysAtOnce] allows, one sentence never split over two.
|
||||
/// [sentences] each cut into the words of [dictionary], as its language
|
||||
/// is cut ([WordCutting]), those cut before as they were; the rest asked
|
||||
/// for in as few requests as [keysAtOnce] allows, one sentence never
|
||||
/// split over two.
|
||||
Future<List<List<SentencePart>>> wordsOf(
|
||||
Dictionary dictionary,
|
||||
List<String> sentences,
|
||||
) async {
|
||||
(String, String, String) keyOf(String sentence) =>
|
||||
(dictionary.source, dictionary.glossLanguage, sentence);
|
||||
final cutting = WordCutting.of(dictionary.language);
|
||||
final asking = <String, Set<String>>{
|
||||
for (final sentence in sentences.toSet())
|
||||
if (!_parts.containsKey(keyOf(sentence)))
|
||||
sentence: sentenceKeys(sentence),
|
||||
sentence: cutting.keysOf(sentence),
|
||||
};
|
||||
final batch = <String>[];
|
||||
final keys = <String>{};
|
||||
|
|
@ -120,7 +140,7 @@ class DictionaryRepository(final DictionaryGateway _gateway) {
|
|||
}
|
||||
final found = await _gateway.withKeys(dictionary, keys.toList());
|
||||
for (final sentence in batch) {
|
||||
_parts[keyOf(sentence)] = sentenceParts(sentence, found);
|
||||
_parts[keyOf(sentence)] = cutting.partsOf(sentence, found);
|
||||
}
|
||||
batch.clear();
|
||||
keys.clear();
|
||||
|
|
@ -148,8 +168,8 @@ class DictionaryRepository(final DictionaryGateway _gateway) {
|
|||
final _stretches = <(String, String, String), SentencePart>{};
|
||||
|
||||
/// [text], a stretch of a line chosen by hand from [start], looked up in
|
||||
/// [dictionary] as one word (`stretchPart`): one request for its
|
||||
/// [stretchKeys], none for one asked before.
|
||||
/// [dictionary] as one word, as its language is cut ([WordCutting]): one
|
||||
/// request for its keys, none for one asked before.
|
||||
Future<SentencePart> wordOfStretch(
|
||||
Dictionary dictionary,
|
||||
String text, {
|
||||
|
|
@ -164,11 +184,12 @@ class DictionaryRepository(final DictionaryGateway _gateway) {
|
|||
steps: known.steps,
|
||||
);
|
||||
}
|
||||
final keys = stretchKeys(text);
|
||||
final cutting = WordCutting.of(dictionary.language);
|
||||
final keys = cutting.stretchKeysOf(text);
|
||||
final found = keys.isEmpty
|
||||
? const <String, List<DictionaryItem>>{}
|
||||
: await _gateway.withKeys(dictionary, keys.toList());
|
||||
final part = _stretches[key] = stretchPart(text, start, found);
|
||||
final part = _stretches[key] = cutting.stretchPartOf(text, start, found);
|
||||
while (_stretches.length > _sentencesKept) {
|
||||
_stretches.remove(_stretches.keys.first);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -121,6 +121,8 @@ class ServerpodDictionaryGateway(final protocol.Client _client)
|
|||
labels: hit.labels,
|
||||
entry: hit.entry,
|
||||
version: hit.version,
|
||||
language: dictionary.language,
|
||||
glossLanguage: dictionary.glossLanguage,
|
||||
),
|
||||
DictionaryKind.characters => DictionaryCharacter.fromEntry(
|
||||
ref: (source: hit.source, id: hit.sourceId),
|
||||
|
|
|
|||
|
|
@ -0,0 +1,144 @@
|
|||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/dictionary.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/sentence_words.dart';
|
||||
|
||||
/// The longest a Chinese word of a sentence is taken to be, in characters:
|
||||
/// CC-CEDICT's longer entries are idioms and names of four to eight.
|
||||
const longestChineseWord = 8;
|
||||
|
||||
/// Every key a dictionary is asked for to cut [sentence], Chinese, into
|
||||
/// words: from each Han character, each run of Han characters up to
|
||||
/// [longestChineseWord] long. Chinese doesn't inflect, so a word is looked
|
||||
/// up only as it stands.
|
||||
Set<String> chineseSentenceKeys(String sentence) {
|
||||
final characters = sentence.runes.toList();
|
||||
return {
|
||||
for (var start = 0; start < characters.length; start++)
|
||||
if (_isHan(characters[start]))
|
||||
for (
|
||||
var end = start + 1;
|
||||
end <= characters.length &&
|
||||
end - start <= longestChineseWord &&
|
||||
_isHan(characters[end - 1]);
|
||||
end++
|
||||
)
|
||||
String.fromCharCodes(characters.sublist(start, end)),
|
||||
};
|
||||
}
|
||||
|
||||
/// [sentence], Chinese, cut into words, as [found] has them by their keys
|
||||
/// ([chineseSentenceKeys]): from the start, each time the longest run of
|
||||
/// characters that is a word, then on after it; what no word starts with
|
||||
/// stays text. Each word knows the longer words running over it into its
|
||||
/// neighbours ([SentencePart.wider]), which the longest match may have
|
||||
/// missed.
|
||||
List<SentencePart> chineseSentenceParts(
|
||||
String sentence,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) {
|
||||
final characters = sentence.runes.toList();
|
||||
final offsets = <int>[0];
|
||||
for (final character in characters) {
|
||||
offsets.add(offsets.last + (character > 0xFFFF ? 2 : 1));
|
||||
}
|
||||
String text(int from, int length) =>
|
||||
String.fromCharCodes(characters.sublist(from, from + length));
|
||||
final at = <int, List<({int length, List<DictionaryWord> entries})>>{};
|
||||
List<({int length, List<DictionaryWord> entries})> wordsAt(int from) =>
|
||||
at[from] ??= [
|
||||
for (var length = longestChineseWord; length > 0; length--)
|
||||
if (from + length <= characters.length &&
|
||||
characters.sublist(from, from + length).every(_isHan))
|
||||
if (_entries(found[text(from, length)]) case final entries
|
||||
when entries.isNotEmpty)
|
||||
(length: length, entries: entries),
|
||||
];
|
||||
|
||||
final parts = <SentencePart>[];
|
||||
var plainFrom = 0;
|
||||
var start = 0;
|
||||
void plainUntil(int end) {
|
||||
if (end > plainFrom) {
|
||||
parts.add(
|
||||
SentencePart(
|
||||
text: text(plainFrom, end - plainFrom),
|
||||
start: offsets[plainFrom],
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
while (start < characters.length) {
|
||||
final word = wordsAt(start).firstOrNull;
|
||||
if (word == null) {
|
||||
start++;
|
||||
continue;
|
||||
}
|
||||
plainUntil(start);
|
||||
final to = start + word.length;
|
||||
final wider = <({int from, int length, List<DictionaryWord> entries})>[
|
||||
for (
|
||||
var other = start;
|
||||
other >= 0 && start - other < longestChineseWord;
|
||||
other--
|
||||
)
|
||||
for (final candidate in wordsAt(other))
|
||||
if (other + candidate.length >= to &&
|
||||
(other < start || other + candidate.length > to))
|
||||
(
|
||||
from: other,
|
||||
length: candidate.length,
|
||||
entries: candidate.entries,
|
||||
),
|
||||
]..sort((a, b) => b.length.compareTo(a.length));
|
||||
parts.add(
|
||||
SentencePart(
|
||||
text: text(start, word.length),
|
||||
start: offsets[start],
|
||||
entries: word.entries,
|
||||
wider: [
|
||||
for (final longer in wider.take(widerWords))
|
||||
SentencePart(
|
||||
text: text(longer.from, longer.length),
|
||||
start: offsets[longer.from],
|
||||
entries: longer.entries,
|
||||
),
|
||||
],
|
||||
),
|
||||
);
|
||||
start = to;
|
||||
plainFrom = start;
|
||||
}
|
||||
plainUntil(characters.length);
|
||||
return parts;
|
||||
}
|
||||
|
||||
/// The keys a dictionary is asked for to look up [text], a stretch of a
|
||||
/// Chinese line chosen by hand, as one word: it, if it is Han characters.
|
||||
Set<String> chineseStretchKeys(String text) => {
|
||||
if (text.isNotEmpty && text.runes.every(_isHan)) text,
|
||||
};
|
||||
|
||||
/// [text], a stretch of a Chinese line chosen by hand from [start], as one
|
||||
/// word, as [found] has it by its [chineseStretchKeys].
|
||||
SentencePart chineseStretchPart(
|
||||
String text,
|
||||
int start,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) => SentencePart(
|
||||
text: text,
|
||||
start: start,
|
||||
entries: chineseStretchKeys(text).isEmpty ? const [] : _entries(found[text]),
|
||||
);
|
||||
|
||||
List<DictionaryWord> _entries(List<DictionaryItem>? items) => [
|
||||
for (final item in items ?? const <DictionaryItem>[])
|
||||
if (item is DictionaryWord) item,
|
||||
];
|
||||
|
||||
/// Whether [character] is a Han character (or 〇, the zero).
|
||||
bool _isHan(int character) =>
|
||||
(character >= 0x3400 && character <= 0x4DBF) ||
|
||||
(character >= 0x4E00 && character <= 0x9FFF) ||
|
||||
(character >= 0xF900 && character <= 0xFAFF) ||
|
||||
(character >= 0x20000 && character <= 0x3134F) ||
|
||||
character == 0x3007;
|
||||
|
|
@ -48,9 +48,18 @@ final class const Dictionary({
|
|||
];
|
||||
}
|
||||
|
||||
/// A sentence using a word: in Japanese, with the word in the form it takes
|
||||
/// there (`食べて`), and its translation into English.
|
||||
typedef WordExample = ({String form, String japanese, String? english});
|
||||
/// A sentence using a word, in the word's language, with the word in the
|
||||
/// form it takes there (`食べて`, `isst`), and its translation into the
|
||||
/// dictionary's meaning language.
|
||||
typedef WordExample = ({String form, String sentence, String? translation});
|
||||
|
||||
/// A form of a word shown beside it: its [text], and what it is
|
||||
/// (Wiktionary's [tags]: `genitive`, `plural`, `past`).
|
||||
final class const WordInflection(final String text, final List<String> tags)
|
||||
with Equatable {
|
||||
@override
|
||||
List<Object?> get props => [text, tags];
|
||||
}
|
||||
|
||||
/// An entry of a dictionary as the app shows it and turns it into a card:
|
||||
/// a word, or a character.
|
||||
|
|
@ -79,6 +88,13 @@ sealed class const DictionaryItem() with Equatable {
|
|||
/// The tags a card made from it gets, wherever it is made.
|
||||
List<String> get tagNames;
|
||||
|
||||
/// Its HSK level, 1 to 6, and 7 for 7 to 9; null when it has none.
|
||||
int? get hsk => [
|
||||
for (final label in labels)
|
||||
if (RegExp(r'^hsk:([1-7])$').firstMatch(label) case final match?)
|
||||
int.parse(match[1]!),
|
||||
].firstOrNull;
|
||||
|
||||
/// Its JLPT level, 5 for N5 to 1 for N1; null when it has none.
|
||||
int? get jlpt => [
|
||||
for (final label in labels)
|
||||
|
|
@ -102,8 +118,34 @@ final class const DictionaryWord({
|
|||
/// Sentences using it, its first meaning's first.
|
||||
final List<WordExample> examples = const [],
|
||||
@override final String? version,
|
||||
|
||||
/// Its part of speech as Wiktionary names it for the whole entry, in
|
||||
/// full (`noun`, `adverb`); null for JMdict's, whose senses have theirs.
|
||||
final String? partOfSpeech,
|
||||
|
||||
/// A noun's genders (`m`, `f`, `n`), German's and Greek's.
|
||||
final List<String> genders = const [],
|
||||
|
||||
/// The forms shown beside it: a noun's genitive and plural, a verb's
|
||||
/// principal parts.
|
||||
final List<WordInflection> inflections = const [],
|
||||
|
||||
/// How it sounds, in IPA.
|
||||
final String? ipa,
|
||||
|
||||
/// It in Latin letters, for a language written otherwise (Greek).
|
||||
final String? romanization,
|
||||
|
||||
/// A Korean word's Hanja, the characters it is written in from Chinese.
|
||||
final List<String> hanja = const [],
|
||||
|
||||
/// Its language, as its dictionary's (`ja`, `de`).
|
||||
final String language = 'ja',
|
||||
}) extends DictionaryItem {
|
||||
/// The word from its [entry] as the dictionary file holds it (JSON).
|
||||
/// The word from its [entry] as the dictionary file holds it (JSON), of
|
||||
/// [language] with meanings in [glossLanguage]: JMdict's, or
|
||||
/// Wiktionary's (an entry naming its part of speech for the whole of it,
|
||||
/// with genders, inflections and IPA beside its senses).
|
||||
factory fromEntry({
|
||||
required DictionaryRef ref,
|
||||
required String headword,
|
||||
|
|
@ -111,26 +153,60 @@ final class const DictionaryWord({
|
|||
required List<String> labels,
|
||||
required String entry,
|
||||
String? version,
|
||||
String language = 'ja',
|
||||
String glossLanguage = 'en',
|
||||
}) {
|
||||
final json = jsonDecode(entry);
|
||||
final senses = json is Map<String, Object?> ? json['senses'] : null;
|
||||
final fields = json is Map<String, Object?>
|
||||
? json
|
||||
: const <String, Object?>{};
|
||||
final senses = fields['senses'];
|
||||
List<String> strings(Object? value) => [
|
||||
if (value is List)
|
||||
for (final item in value) item.toString(),
|
||||
];
|
||||
final partOfSpeech = switch (fields['pos']) {
|
||||
final String pos => _partsOfSpeech[pos] ?? pos,
|
||||
_ => null,
|
||||
};
|
||||
return DictionaryWord(
|
||||
ref: ref,
|
||||
headword: headword,
|
||||
reading: reading,
|
||||
labels: labels,
|
||||
version: version,
|
||||
language: language,
|
||||
partOfSpeech: partOfSpeech,
|
||||
genders: strings(fields['genders']),
|
||||
inflections: [
|
||||
if (fields['inflections'] case final List<Object?> forms)
|
||||
for (final form in forms.whereType<Map<String, Object?>>())
|
||||
if (form['text'] case final String text)
|
||||
WordInflection(text, strings(form['tags'])),
|
||||
],
|
||||
ipa: fields['ipa'] as String?,
|
||||
romanization: fields['romanization'] as String?,
|
||||
hanja: strings(fields['hanja']),
|
||||
senses: [
|
||||
if (senses is List)
|
||||
for (final sense in senses.whereType<Map<String, Object?>>())
|
||||
for (final (index, sense)
|
||||
in senses.whereType<Map<String, Object?>>().indexed)
|
||||
VocabularySense(
|
||||
glosses: strings(sense['glosses']),
|
||||
pos: strings(sense['pos']),
|
||||
misc: strings(sense['misc']),
|
||||
label: switch (strings(sense['info'])) {
|
||||
// Wiktionary's English meaning a translation is of.
|
||||
final info when partOfSpeech != null && info.isNotEmpty =>
|
||||
info.join('; '),
|
||||
_ => null,
|
||||
},
|
||||
// Wiktionary names the part of speech once, for every sense;
|
||||
// the first carries it, as JMdict's senses carry theirs.
|
||||
pos: partOfSpeech == null
|
||||
? strings(sense['pos'])
|
||||
: [if (index == 0) partOfSpeech],
|
||||
misc: partOfSpeech == null
|
||||
? strings(sense['misc'])
|
||||
: strings(sense['tags']),
|
||||
field: strings(sense['field']),
|
||||
dialect: strings(sense['dialect']),
|
||||
),
|
||||
|
|
@ -140,30 +216,50 @@ final class const DictionaryWord({
|
|||
for (final sense in senses.whereType<Map<String, Object?>>())
|
||||
if (sense['examples'] case final List<Object?> examples)
|
||||
for (final example in examples.whereType<Map<String, Object?>>())
|
||||
if (example case {
|
||||
'ja': final String japanese,
|
||||
'text': final String form,
|
||||
})
|
||||
if (example case {'text': final String form}
|
||||
when example[language] is String)
|
||||
(
|
||||
form: form,
|
||||
japanese: japanese,
|
||||
english: example['en'] as String?,
|
||||
sentence: example[language]! as String,
|
||||
translation:
|
||||
(example[glossLanguage] ?? example['en']) as String?,
|
||||
),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
/// The article a card puts before a noun of one gender, in German and
|
||||
/// Greek (`das Haus`, `το σπίτι`); null otherwise.
|
||||
String? get article => switch ((language.split('-').first, genders)) {
|
||||
('de', ['m']) => 'der',
|
||||
('de', ['f']) => 'die',
|
||||
('de', ['n']) => 'das',
|
||||
('el', ['m']) => 'ο',
|
||||
('el', ['f']) => 'η',
|
||||
('el', ['n']) => 'το',
|
||||
_ => null,
|
||||
};
|
||||
|
||||
/// What a card studies: the headword, a noun with its article.
|
||||
String get front => switch (article) {
|
||||
final article? when partOfSpeech == 'noun' => '$article $headword',
|
||||
_ => headword,
|
||||
};
|
||||
|
||||
/// How many of its [examples] a card made from it takes as hints.
|
||||
static const examplesOnCard = 3;
|
||||
|
||||
/// The tags a card made from it gets, wherever it is made: its JLPT level
|
||||
/// (`JLPT-N5`) and its word class by its first sense (JMdict's codes,
|
||||
/// grouped: `Verb`, `Noun`, …), in English as the dictionary's codes are,
|
||||
/// so a level or a class can be studied alone.
|
||||
/// grouped: `Verb`, `Noun`, …; Wiktionary's part of speech), in English as
|
||||
/// the dictionary's codes are, so a level or a class can be studied alone.
|
||||
@override
|
||||
List<String> get tagNames => [
|
||||
if (jlpt case final level?) 'JLPT-N$level',
|
||||
...{
|
||||
if (hsk case final level?) hskTagName(level),
|
||||
if (partOfSpeech case final pos?)
|
||||
?_wiktionaryClasses[pos]
|
||||
else ...{
|
||||
for (final code in senses.firstOrNull?.pos ?? const <String>[])
|
||||
?_wordClass(code),
|
||||
},
|
||||
|
|
@ -175,7 +271,7 @@ final class const DictionaryWord({
|
|||
@override
|
||||
CardContent get card => CardContent(
|
||||
type: CardType.vocabulary,
|
||||
front: RichContent.plain(headword),
|
||||
front: RichContent.plain(front),
|
||||
back: RichContent.empty(),
|
||||
reading: reading ?? '',
|
||||
senses: senses,
|
||||
|
|
@ -189,7 +285,7 @@ final class const DictionaryWord({
|
|||
@override
|
||||
CardContent updated(CardContent base) => base
|
||||
.copyWith(
|
||||
front: RichContent.plain(headword),
|
||||
front: RichContent.plain(front),
|
||||
reading: reading ?? '',
|
||||
senses: senses,
|
||||
entry: ref,
|
||||
|
|
@ -212,9 +308,52 @@ final class const DictionaryWord({
|
|||
senses,
|
||||
examples,
|
||||
version,
|
||||
partOfSpeech,
|
||||
genders,
|
||||
inflections,
|
||||
ipa,
|
||||
romanization,
|
||||
hanja,
|
||||
language,
|
||||
];
|
||||
}
|
||||
|
||||
/// Wiktionary's parts of speech, in full as a word's grammar shows them:
|
||||
/// its own short names (`adj`, `adv`) would read as JMdict's codes.
|
||||
const _partsOfSpeech = {
|
||||
'adj': 'adjective',
|
||||
'adv': 'adverb',
|
||||
'pron': 'pronoun',
|
||||
'prep': 'preposition',
|
||||
'postp': 'postposition',
|
||||
'conj': 'conjunction',
|
||||
'det': 'determiner',
|
||||
'num': 'numeral',
|
||||
'intj': 'interjection',
|
||||
'prep_phrase': 'phrase',
|
||||
'abbrev': 'abbreviation',
|
||||
'contraction': 'contraction',
|
||||
};
|
||||
|
||||
/// The word class a card gets as its tag, by Wiktionary's part of speech
|
||||
/// in full.
|
||||
const _wiktionaryClasses = {
|
||||
'noun': 'Noun',
|
||||
'verb': 'Verb',
|
||||
'adjective': 'Adjective',
|
||||
'adverb': 'Adverb',
|
||||
'pronoun': 'Pronoun',
|
||||
'preposition': 'Preposition',
|
||||
'postposition': 'Preposition',
|
||||
'conjunction': 'Conjunction',
|
||||
'determiner': 'Determiner',
|
||||
'article': 'Article',
|
||||
'numeral': 'Numeral',
|
||||
'interjection': 'Interjection',
|
||||
'particle': 'Particle',
|
||||
'phrase': 'Expression',
|
||||
};
|
||||
|
||||
/// A character of a dictionary of characters: a kanji with its meanings,
|
||||
/// readings, school grade, stroke count and frequency, or a kana with its
|
||||
/// romaji.
|
||||
|
|
@ -343,7 +482,7 @@ const _secondarySchool = 'Secondary school';
|
|||
/// [example] as a hint: the sentence, the word in it in bold, and its
|
||||
/// English on a line below.
|
||||
RichContent _hint(WordExample example) {
|
||||
final (:form, :japanese, :english) = example;
|
||||
final (:form, sentence: japanese, translation: english) = example;
|
||||
final at = form.isEmpty ? -1 : japanese.indexOf(form);
|
||||
return RichContent({
|
||||
'document': {
|
||||
|
|
@ -387,9 +526,13 @@ RichContent _hint(WordExample example) {
|
|||
bool isDictionaryTagName(String name) =>
|
||||
_dictionaryTagNames.contains(canonicalTagName(name));
|
||||
|
||||
/// The tag of HSK level [level] (`HSK-1`, `HSK-7-9`).
|
||||
String hskTagName(int level) => level == 7 ? 'HSK-7-9' : 'HSK-$level';
|
||||
|
||||
final _dictionaryTagNames = {
|
||||
for (final name in [
|
||||
for (var level = 1; level <= 5; level++) 'JLPT-N$level',
|
||||
for (var level = 1; level <= 7; level++) hskTagName(level),
|
||||
for (var grade = 1; grade <= 6; grade++) 'Grade $grade',
|
||||
_secondarySchool,
|
||||
'Hiragana',
|
||||
|
|
@ -405,6 +548,11 @@ final _dictionaryTagNames = {
|
|||
'Counter',
|
||||
'Expression',
|
||||
'Verb',
|
||||
'Adjective',
|
||||
'Preposition',
|
||||
'Determiner',
|
||||
'Article',
|
||||
'Numeral',
|
||||
])
|
||||
canonicalTagName(name),
|
||||
};
|
||||
|
|
|
|||
|
|
@ -0,0 +1,170 @@
|
|||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/dictionary.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/sentence_words.dart';
|
||||
|
||||
/// A Korean word as text writes it between spaces: Hangul and other
|
||||
/// letters, a noun with its particles attached (`집에서`).
|
||||
final _word = RegExp(r'[\p{L}\p{M}]+', unicode: true);
|
||||
|
||||
/// The particles and the copula's endings Korean attaches to a word, the
|
||||
/// longest first (decided 2026-10-14: Korean's own grammar, no analyser;
|
||||
/// a verb's endings are among its forms in the dictionary).
|
||||
const koreanParticles = [
|
||||
'에서부터',
|
||||
'으로부터',
|
||||
'이라고는',
|
||||
'에게서',
|
||||
'한테서',
|
||||
'으로서',
|
||||
'으로써',
|
||||
'으로는',
|
||||
'에서는',
|
||||
'에서도',
|
||||
'에게는',
|
||||
'까지는',
|
||||
'부터는',
|
||||
'이라고',
|
||||
'이에요',
|
||||
'입니다',
|
||||
'이었다',
|
||||
'이었어',
|
||||
'이지만',
|
||||
'처럼',
|
||||
'같이',
|
||||
'보다',
|
||||
'부터',
|
||||
'까지',
|
||||
'에서',
|
||||
'에게',
|
||||
'한테',
|
||||
'께서',
|
||||
'하고',
|
||||
'이랑',
|
||||
'이나',
|
||||
'이다',
|
||||
'이야',
|
||||
'예요',
|
||||
'였다',
|
||||
'라고',
|
||||
'으로',
|
||||
'에는',
|
||||
'에도',
|
||||
'은',
|
||||
'는',
|
||||
'이',
|
||||
'가',
|
||||
'을',
|
||||
'를',
|
||||
'의',
|
||||
'에',
|
||||
'도',
|
||||
'만',
|
||||
'로',
|
||||
'와',
|
||||
'과',
|
||||
'랑',
|
||||
'나',
|
||||
'야',
|
||||
'요',
|
||||
];
|
||||
|
||||
/// The words a Korean word written as [word] may be: itself, and itself
|
||||
/// without each particle or ending it ends with ([koreanParticles]).
|
||||
List<({String stem, String particle})> _readings(String word) => [
|
||||
(stem: word, particle: ''),
|
||||
for (final particle in koreanParticles)
|
||||
if (word.length > particle.length && word.endsWith(particle))
|
||||
(
|
||||
stem: word.substring(0, word.length - particle.length),
|
||||
particle: particle,
|
||||
),
|
||||
];
|
||||
|
||||
/// Every key a dictionary is asked for to cut [sentence], Korean, into
|
||||
/// words: each word between spaces as it stands and without each particle
|
||||
/// it may end with.
|
||||
Set<String> koreanSentenceKeys(String sentence) => {
|
||||
for (final match in _word.allMatches(sentence))
|
||||
for (final (:stem, particle: _) in _readings(match[0]!)) stem.toLowerCase(),
|
||||
};
|
||||
|
||||
/// [sentence], Korean, cut into words, as [found] has them by their keys
|
||||
/// ([koreanSentenceKeys]): each word between spaces as it stands if the
|
||||
/// dictionary has it, else without the longest particle that leaves a word
|
||||
/// (the particle staying text); what lies between stays text.
|
||||
List<SentencePart> koreanSentenceParts(
|
||||
String sentence,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) {
|
||||
final parts = <SentencePart>[];
|
||||
var plainFrom = 0;
|
||||
void plainUntil(int end) {
|
||||
if (end > plainFrom) {
|
||||
parts.add(
|
||||
SentencePart(
|
||||
text: sentence.substring(plainFrom, end),
|
||||
start: plainFrom,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (final match in _word.allMatches(sentence)) {
|
||||
for (final (:stem, particle: _) in _readings(match[0]!)) {
|
||||
final entries = _entriesOf(found[stem.toLowerCase()]);
|
||||
if (entries.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
plainUntil(match.start);
|
||||
parts.add(
|
||||
SentencePart(
|
||||
text: stem,
|
||||
start: match.start,
|
||||
entries: entries,
|
||||
),
|
||||
);
|
||||
plainFrom = match.start + stem.length;
|
||||
break;
|
||||
}
|
||||
}
|
||||
plainUntil(sentence.length);
|
||||
return parts;
|
||||
}
|
||||
|
||||
/// The keys a dictionary is asked for to look up [text], a stretch of a
|
||||
/// Korean line chosen by hand, as one word: it and it without its
|
||||
/// particles, if it is one word.
|
||||
Set<String> koreanStretchKeys(String text) => switch (_word.allMatches(text)) {
|
||||
final words when words.length == 1 && words.first[0] == text.trim() => {
|
||||
for (final (:stem, particle: _) in _readings(text.trim()))
|
||||
stem.toLowerCase(),
|
||||
},
|
||||
_ => const {},
|
||||
};
|
||||
|
||||
/// [text], a stretch of a Korean line chosen by hand from [start], as one
|
||||
/// word, as [found] has it by its [koreanStretchKeys].
|
||||
SentencePart koreanStretchPart(
|
||||
String text,
|
||||
int start,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) {
|
||||
if (koreanStretchKeys(text).isEmpty) {
|
||||
return SentencePart(text: text, start: start);
|
||||
}
|
||||
for (final (:stem, particle: _) in _readings(text.trim())) {
|
||||
final entries = _entriesOf(found[stem.toLowerCase()]);
|
||||
if (entries.isNotEmpty) {
|
||||
return SentencePart(
|
||||
text: text,
|
||||
start: start,
|
||||
entries: entries,
|
||||
);
|
||||
}
|
||||
}
|
||||
return SentencePart(text: text, start: start);
|
||||
}
|
||||
|
||||
List<DictionaryWord> _entriesOf(List<DictionaryItem>? items) => [
|
||||
for (final item in items ?? const <DictionaryItem>[])
|
||||
if (item is DictionaryWord) item,
|
||||
];
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/furigana.dart';
|
||||
|
||||
/// [text] (a Chinese word as a sentence has it) in pieces, each character
|
||||
/// with its syllable of [reading] (pinyin, a syllable to a character, as
|
||||
/// the dictionary gives it: `xué xí`), shown above it; the whole word with
|
||||
/// the whole reading when they don't pair up one to one (a letter or a
|
||||
/// number in the word).
|
||||
List<RubyPiece> pinyinRuby(String text, String? reading) {
|
||||
if (reading == null || reading.trim().isEmpty) {
|
||||
return [(text: text, reading: null)];
|
||||
}
|
||||
final characters = text.runes.toList();
|
||||
final syllables = reading.trim().split(RegExp(r'\s+'));
|
||||
if (syllables.length != characters.length) {
|
||||
return [(text: text, reading: reading)];
|
||||
}
|
||||
return [
|
||||
for (final (index, character) in characters.indexed)
|
||||
(text: String.fromCharCode(character), reading: syllables[index]),
|
||||
];
|
||||
}
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/dictionary.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/sentence_words.dart';
|
||||
|
||||
/// A word of a language written with spaces between its words (German,
|
||||
/// Greek, English): letters and their marks, joined by an apostrophe or a
|
||||
/// hyphen within it (`don't`, `E-Mail`).
|
||||
final _word = RegExp(r"[\p{L}\p{M}]+(?:['’\-][\p{L}\p{M}]+)*", unicode: true);
|
||||
|
||||
/// The key a dictionary finds [word] by: lower case, with a typographic
|
||||
/// apostrophe as a plain one, as the dictionary's forms are written.
|
||||
String _keyOf(String word) => word.toLowerCase().replaceAll('’', "'");
|
||||
|
||||
/// Every key a dictionary is asked for to cut [sentence], in a language
|
||||
/// written with spaces between its words, into words: each word as it
|
||||
/// stands, in lower case. The dictionary knows each word's forms (`aß`
|
||||
/// finds `essen`), so nothing is worked out here.
|
||||
Set<String> spacedSentenceKeys(String sentence) => {
|
||||
for (final match in _word.allMatches(sentence)) _keyOf(match[0]!),
|
||||
};
|
||||
|
||||
/// [sentence], in a language written with spaces between its words, cut
|
||||
/// into its words, as [found] has them by their keys
|
||||
/// ([spacedSentenceKeys]); what lies between them, and words no entry has,
|
||||
/// stay text.
|
||||
List<SentencePart> spacedSentenceParts(
|
||||
String sentence,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) {
|
||||
final parts = <SentencePart>[];
|
||||
var plainFrom = 0;
|
||||
void plainUntil(int end) {
|
||||
if (end > plainFrom) {
|
||||
parts.add(
|
||||
SentencePart(
|
||||
text: sentence.substring(plainFrom, end),
|
||||
start: plainFrom,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (final match in _word.allMatches(sentence)) {
|
||||
final entries = _entriesOf(match[0]!, found);
|
||||
if (entries.isEmpty) {
|
||||
continue;
|
||||
}
|
||||
plainUntil(match.start);
|
||||
parts.add(
|
||||
SentencePart(text: match[0]!, start: match.start, entries: entries),
|
||||
);
|
||||
plainFrom = match.end;
|
||||
}
|
||||
plainUntil(sentence.length);
|
||||
return parts;
|
||||
}
|
||||
|
||||
/// The keys a dictionary is asked for to look up [text], a stretch of a
|
||||
/// line chosen by hand, as one word: none when it holds more than one
|
||||
/// (dictionaries find text a word at a time).
|
||||
Set<String> spacedStretchKeys(String text) => switch (_word.allMatches(text)) {
|
||||
final words when words.length == 1 && words.first[0] == text.trim() => {
|
||||
_keyOf(text.trim()),
|
||||
},
|
||||
_ => const {},
|
||||
};
|
||||
|
||||
/// [text], a stretch of a line chosen by hand from [start], as one word, as
|
||||
/// [found] has it by its [spacedStretchKeys]; no entries when it is no word
|
||||
/// the dictionary has.
|
||||
SentencePart spacedStretchPart(
|
||||
String text,
|
||||
int start,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) => SentencePart(
|
||||
text: text,
|
||||
start: start,
|
||||
entries: spacedStretchKeys(text).isEmpty
|
||||
? const []
|
||||
: _entriesOf(text.trim(), found),
|
||||
);
|
||||
|
||||
/// The entries [word] may be, as [found] has them: those written as it
|
||||
/// stands first (German's noun `Essen` before its verb `essen`), then those
|
||||
/// written so in another case (`Haus` at a sentence's start), then those
|
||||
/// it is a form of, each in the dictionary's order.
|
||||
List<DictionaryWord> _entriesOf(
|
||||
String word,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) {
|
||||
final key = _keyOf(word);
|
||||
final entries = [
|
||||
for (final item in found[key] ?? const <DictionaryItem>[])
|
||||
if (item is DictionaryWord) item,
|
||||
];
|
||||
int order(DictionaryWord entry) => entry.headword == word
|
||||
? 0
|
||||
: entry.headword.toLowerCase() == key
|
||||
? 1
|
||||
: 2;
|
||||
final ordered = [...entries.indexed]
|
||||
..sort((a, b) {
|
||||
final byForm = order(a.$2).compareTo(order(b.$2));
|
||||
return byForm != 0 ? byForm : a.$1.compareTo(b.$1);
|
||||
});
|
||||
return [for (final (_, entry) in ordered) entry];
|
||||
}
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/chinese_words.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/dictionary.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/korean_words.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/sentence_words.dart';
|
||||
import 'package:halkyon_learn_flutter/src/feature/dictionary/model/spaced_words.dart';
|
||||
|
||||
/// How text in a language is cut into the words a dictionary has: at its
|
||||
/// spaces (German, Greek, English and most others: `spaced_words.dart`), or,
|
||||
/// for a language writing none between its words, by the longest words the
|
||||
/// dictionary has from each place (Japanese with its conjugations,
|
||||
/// `sentence_words.dart`; Chinese, which doesn't inflect,
|
||||
/// `chinese_words.dart`; decided 2026-10-14). Korean is cut at its spaces,
|
||||
/// its particles taken off a word the dictionary lacks as it stands
|
||||
/// (`korean_words.dart`).
|
||||
enum WordCutting() {
|
||||
/// At the spaces, each word looked up as it stands.
|
||||
spaces,
|
||||
|
||||
/// At the spaces, a word's particles taken off where it needs (Korean).
|
||||
spacesKorean,
|
||||
|
||||
/// By the dictionary's longest words, conjugations worked out (Japanese).
|
||||
longestJapanese,
|
||||
|
||||
/// By the dictionary's longest words, as they stand (Chinese).
|
||||
longestChinese;
|
||||
|
||||
/// How [language] (a BCP 47 tag, `de`, `ja`) is cut.
|
||||
static WordCutting of(String language) =>
|
||||
switch (language.split('-').first.toLowerCase()) {
|
||||
'ja' => longestJapanese,
|
||||
'zh' || 'yue' => longestChinese,
|
||||
'ko' => spacesKorean,
|
||||
_ => spaces,
|
||||
};
|
||||
|
||||
/// The keys a dictionary is asked for to cut [sentence] into words.
|
||||
Set<String> keysOf(String sentence) => switch (this) {
|
||||
spaces => spacedSentenceKeys(sentence),
|
||||
spacesKorean => koreanSentenceKeys(sentence),
|
||||
longestJapanese => sentenceKeys(sentence),
|
||||
longestChinese => chineseSentenceKeys(sentence),
|
||||
};
|
||||
|
||||
/// [sentence] cut into words, as [found] has them by their [keysOf].
|
||||
List<SentencePart> partsOf(
|
||||
String sentence,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) => switch (this) {
|
||||
spaces => spacedSentenceParts(sentence, found),
|
||||
spacesKorean => koreanSentenceParts(sentence, found),
|
||||
longestJapanese => sentenceParts(sentence, found),
|
||||
longestChinese => chineseSentenceParts(sentence, found),
|
||||
};
|
||||
|
||||
/// The keys a dictionary is asked for to look up [text], a stretch of a
|
||||
/// line chosen by hand, as one word.
|
||||
Set<String> stretchKeysOf(String text) => switch (this) {
|
||||
spaces => spacedStretchKeys(text),
|
||||
spacesKorean => koreanStretchKeys(text),
|
||||
longestJapanese => stretchKeys(text),
|
||||
longestChinese => chineseStretchKeys(text),
|
||||
};
|
||||
|
||||
/// [text], a stretch chosen by hand from [start], as one word, as [found]
|
||||
/// has it by its [stretchKeysOf].
|
||||
SentencePart stretchPartOf(
|
||||
String text,
|
||||
int start,
|
||||
Map<String, List<DictionaryItem>> found,
|
||||
) => switch (this) {
|
||||
spaces => spacedStretchPart(text, start, found),
|
||||
spacesKorean => koreanStretchPart(text, start, found),
|
||||
longestJapanese => stretchPart(text, start, found),
|
||||
longestChinese => chineseStretchPart(text, start, found),
|
||||
};
|
||||
}
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
/// Where a word of a language is looked up on the web, once someone taps
|
||||
/// for it (decided 2026-10-14): Jisho for Japanese, built on JMdict as the
|
||||
/// app's Japanese dictionary is, with sentences and kanji parts beside;
|
||||
/// for every other language Wiktionary, which the app's dictionaries of
|
||||
/// those languages are drawn from.
|
||||
typedef WordSite = ({
|
||||
String name,
|
||||
Uri Function(String word) entry,
|
||||
Uri Function(String text) search,
|
||||
});
|
||||
|
||||
/// The site words of [language] (a BCP 47 tag) are looked up on.
|
||||
WordSite wordSiteOf(String language) {
|
||||
final primary = language.split('-').first.toLowerCase();
|
||||
if (primary == 'ja') {
|
||||
return (
|
||||
name: 'Jisho',
|
||||
entry: (word) => Uri.https('jisho.org', '/search/$word'),
|
||||
search: (text) => Uri.https('jisho.org', '/search/$text'),
|
||||
);
|
||||
}
|
||||
final heading = _englishNames[primary];
|
||||
return (
|
||||
name: 'Wiktionary',
|
||||
// The word's page, at the language's section where it has one.
|
||||
entry: (word) => Uri(
|
||||
scheme: 'https',
|
||||
host: 'en.wiktionary.org',
|
||||
path: '/wiki/${word.replaceAll(' ', '_')}',
|
||||
fragment: heading,
|
||||
),
|
||||
search: (text) => Uri.https('en.wiktionary.org', '/w/index.php', {
|
||||
'search': text,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
/// Languages by the English name of their sections on Wiktionary.
|
||||
const _englishNames = {
|
||||
'ar': 'Arabic',
|
||||
'de': 'German',
|
||||
'el': 'Greek',
|
||||
'en': 'English',
|
||||
'es': 'Spanish',
|
||||
'fr': 'French',
|
||||
'he': 'Hebrew',
|
||||
'hi': 'Hindi',
|
||||
'it': 'Italian',
|
||||
'ko': 'Korean',
|
||||
'nl': 'Dutch',
|
||||
'pl': 'Polish',
|
||||
'pt': 'Portuguese',
|
||||
'ru': 'Russian',
|
||||
'sv': 'Swedish',
|
||||
'th': 'Thai',
|
||||
'tr': 'Turkish',
|
||||
'uk': 'Ukrainian',
|
||||
'vi': 'Vietnamese',
|
||||
'zh': 'Chinese',
|
||||
};
|
||||
|
|
@ -102,7 +102,10 @@ class _LookUpState() extends State<_LookUp> {
|
|||
textInputAction: TextInputAction.search,
|
||||
decoration: InputDecoration(
|
||||
hintText: switch (widget.dictionary.kind) {
|
||||
DictionaryKind.words => l10n.dictionarySearchHint,
|
||||
DictionaryKind.words => wordSearchHint(
|
||||
l10n,
|
||||
widget.dictionary,
|
||||
),
|
||||
DictionaryKind.characters => l10n.dictionaryCharacterSearchHint,
|
||||
},
|
||||
prefixIcon: const Icon(LucideIcons.search),
|
||||
|
|
@ -170,3 +173,10 @@ class _LookUpState() extends State<_LookUp> {
|
|||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// What a search of [dictionary]'s words takes: in Japanese also kana and
|
||||
/// romaji, in any language a meaning.
|
||||
String wordSearchHint(AppLocalizations l10n, Dictionary dictionary) =>
|
||||
dictionary.language.split('-').first == 'ja'
|
||||
? l10n.dictionarySearchHint
|
||||
: l10n.dictionarySearchHintWords;
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue