Tizian 「ティツィアーン」
1d2011a676
Once the person agrees, the device holds MeCab's IPADIC and the
instance's JMdict and cuts lines as Jisho.org does, so their text never
leaves it (ADR 0012). MeCab comes through mecab_for_dart, whose build
hook compiles it; its tokens are grouped by Ve's rules, written anew in
Dart, with に kept a word of its own. Each word is found as it stands or
as a form of a word, else by its dictionary form, else as the longest
words it holds from token to token; longer words over token boundaries
are offered as before.
The curator packs Ubuntu's compiled IPADIC beside the dictionary files
(curate ipadic, and every update run). The server exports its JMdict
from its tables, so the device finds the same keys and keeps the same
versions, and serves both at /device-dictionaries/, named in
dictionary.deviceFiles('ja') with their hashes; Caddy forwards the path.
The app checks each file against its hash, unpacks IPADIC, builds a
SQLite database from the export on an isolate, and asks the instance
for newer files every 30 days, downloading only what changed. The Watch
screen offers the download once per device with its size (34 MB, some
150 MB once set up); Settings downloads and removes them. Without them,
or should cutting on the device fail, lines go to the instance as
before. MeCab's, IPADIC's and Ve's licences join the licence list.
Every build now needs a C++ compiler; mecab_for_dart holds code_assets
at 1.x.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
95 lines
6.3 KiB
Markdown
95 lines
6.3 KiB
Markdown
# ADR 0012: Japanese cut into words and looked up on the device
|
||
|
||
- **Status:** accepted
|
||
- **Date:** 2026-10-15
|
||
|
||
## Context
|
||
|
||
Watching from Jellyfin cuts every line of subtitles and lyrics into words. Until now the
|
||
app sent the text of each line, and of each stretch chosen by hand, to the instance,
|
||
which answered from its `dictionary_key` table and kept nothing; the Jellyfin settings
|
||
said so. Two things were wanting:
|
||
|
||
- **Privacy.** The text of what a person watches left the device, line by line.
|
||
- **The cut.** The longest match knows no grammar: 今日は is read as こんにちは, and
|
||
heuristics keep short kana from swallowing particles (が and まだ, not がま and だ).
|
||
Jisho.org cuts sentences with MeCab and IPADIC, grouping MeCab's tokens into the words
|
||
a learner looks up with Ve's rules. AGENTS.md had this planned for after the
|
||
hackathon.
|
||
|
||
The developer decided (2026-10-15) to cut and look up on the device, with MeCab through
|
||
`mecab_for_dart`, IPADIC downloaded from the instance at first use, and JMdict on the
|
||
device too; to ask once, with the size, and to keep the instance's way as the fallback.
|
||
|
||
## Decision
|
||
|
||
1. **MeCab** 0.996 through `mecab_for_dart` (BSD; MeCab itself is offered under the GPL,
|
||
the LGPL or the BSD licence, and the app takes the BSD's). Its build hook compiles
|
||
MeCab's C++ for each platform (`native_toolchain_c`), so every build needs a C++
|
||
compiler, `flutter test` included: the snap's build has clang; CI's runner must have
|
||
one. The package still wants `code_assets` 1.x, which moved `code_assets` from 2.1.0
|
||
to 1.2.1, `native_toolchain_c` from 0.19.5 to 0.19.3 and `objective_c` from 9.6.0 to
|
||
9.5.0. Not chosen: Sudachi and Kuromoji (JVM), Lindera (Rust; a larger library and
|
||
the same dictionaries), a tokenizer in Dart (none mature).
|
||
2. **IPADIC comes from the instance, not with the app.** Bundled, it would add 14 MB to
|
||
every download of the app (51 MB once unpacked), for people who never watch in
|
||
Japanese too. The curator packs Ubuntu's compiled `mecab-ipadic-utf8` (what MeCab
|
||
0.996 reads) into `mecab-ipadic.tar.gz` beside the dictionary files: the files MeCab
|
||
reads, IPADIC's licence and an empty `mecabrc`, the same bytes for the same
|
||
dictionary, left as it is when unchanged (`curate ipadic`; `curate update` on every
|
||
run). 13.6 MB to download.
|
||
3. **JMdict as the instance holds it.** The server exports its JMdict with English
|
||
meanings from its tables: each entry with its keys (`dictionary_key`'s), labels,
|
||
rank, version and the entry as cards take it, as gzipped JSON lines with a header
|
||
(`format` 1). From the tables rather than the curator's file, so that the device finds
|
||
words by the same keys and keeps the same versions, and corrections still reach its
|
||
cards. The server makes the export in its temporary folder as it starts and after
|
||
each load, and serves it and the IPADIC pack on its web server at
|
||
`/device-dictionaries/<name>-<first 16 hex digits of the hash>…` (immutable, so
|
||
cached for good; Caddy forwards the path), only the files it has ready.
|
||
`dictionary.deviceFiles('ja')` names them with their address, SHA-256 and size, once
|
||
both are ready, none for other languages. 21.7 MB.
|
||
4. **The device builds a database** from the export, streamed, on an isolate: the
|
||
entries, and every key with its entry's rank and id, ordered as the instance orders
|
||
them (rank, then id), so a key gives the same first eight entries
|
||
(`DeviceDictionaryStore`). Some 8 seconds on a desktop, 101 MB.
|
||
5. **The cut.** MeCab's tokens are grouped by Ve's rules (Kim Ahlström's Ve, MIT,
|
||
written anew in Dart, `ve_words.dart`; one change: に stays a word of its own, as
|
||
Jisho.org shows it). Each word is found as the longest match finds a stretch (as it
|
||
stands, or as a form of a word whose class fits, `japaneseWordOf`), else by its
|
||
dictionary form (静かな by 静か), else cut into the longest words it holds from token
|
||
to token (二十五歳 into 二十, 五 and 歳). The longer words running over a word into its
|
||
neighbours, from token to token, are offered as before (携帯 and 電話, which IPADIC
|
||
cuts apart, both offer 携帯電話). A stretch chosen by hand is looked up in the device's
|
||
database. Each batch of lines is cut on an isolate of its own, MeCab and the database
|
||
opened anew (about 100 ms for a batch, which is the line shown, then a whole track).
|
||
6. **Asked once, with the size.** The Watch screen offers the download once per device,
|
||
for a dictionary the instance's export would serve, saying how much it downloads
|
||
("34.0 MB, more once set up"); Not now is final there. Settings › Japanese on this
|
||
device downloads them, says how much room they take and removes them. They belong to
|
||
the device (`<app support>/dictionaries/`, a manifest of what is installed and whether
|
||
the person was asked): the same for every person and instance, downloaded through the
|
||
instance in use. Each file is checked against its SHA-256 and size, fetched over HTTPS
|
||
(plain HTTP only to this machine), and the pack unpacks only the files MeCab reads.
|
||
7. **Updates.** Once 30 days have passed, cutting lines asks the instance quietly for
|
||
newer files, and only those whose hash changed are downloaded: IPADIC rarely, JMdict's
|
||
export after the instance loaded a newer JMdict, so at most some 22 MB a month.
|
||
8. **The instance stays the fallback.** Without the dictionaries (not downloaded, another
|
||
language, another source), or should cutting on the device fail, lines go to the
|
||
instance as before. Lines already cut are cut anew once the dictionaries arrive.
|
||
|
||
## Consequences
|
||
|
||
- A device holding the dictionaries never sends the text of what is watched; the
|
||
instance only learns that it downloaded them, and, monthly, that it asked for newer
|
||
ones.
|
||
- Japanese is cut with its grammar, as Jisho.org cuts it; the longest match's
|
||
heuristics stay for the fallback.
|
||
- A device spends 34 MB to download and some 150 MB of room; the offer says the first,
|
||
Settings the second, and removing them is a tap.
|
||
- Every build, CI's too, needs a C++ compiler.
|
||
- Each instance keeps a 22 MB export in its temporary folder, made again after a
|
||
restart, and its web server serves it (production runs one already, for the
|
||
assistant's connector).
|
||
- MeCab's, IPADIC's and Ve's licences join the app's licence list.
|
||
- Other languages are still cut through the instance.
|