halkyon/docs/adr/0012-japanese-on-the-device.md

# ADR 0012: Japanese cut into words and looked up on the device

- **Status:** accepted
- **Date:** 2026-10-15

## Context

Watching from Jellyfin cuts every line of subtitles and lyrics into words. Until now the
app sent the text of each line, and of each stretch chosen by hand, to the instance,
which answered from its `dictionary_key` table and kept nothing; the Jellyfin settings
said so. Two things were wanting:

- **Privacy.** The text of what a person watches left the device, line by line.
- **The cut.** The longest match knows no grammar: 今日は is read as こんにちは, and
  heuristics keep short kana from swallowing particles (が and まだ, not がま and だ).
  Jisho.org cuts sentences with MeCab and IPADIC, grouping MeCab's tokens into the words
  a learner looks up with Ve's rules. AGENTS.md had this planned for after the
  hackathon.

The developer decided (2026-10-15) to cut and look up on the device, with MeCab through
`mecab_for_dart`, IPADIC downloaded from the instance at first use, and JMdict on the
device too; to ask once, with the size, and to keep the instance's way as the fallback.

## Decision

1. **MeCab** 0.996 through `mecab_for_dart` (BSD; MeCab itself is offered under the GPL,
   the LGPL or the BSD licence, and the app takes the BSD's). Its build hook compiles
   MeCab's C++ for each platform (`native_toolchain_c`), so every build needs a C++
   compiler, `flutter test` included: the snap's build has clang; CI's runner must have
   one. The package still wants `code_assets` 1.x, which moved `code_assets` from 2.1.0
   to 1.2.1, `native_toolchain_c` from 0.19.5 to 0.19.3 and `objective_c` from 9.6.0 to
   9.5.0. Not chosen: Sudachi and Kuromoji (JVM), Lindera (Rust; a larger library and
   the same dictionaries), a tokenizer in Dart (none mature).
2. **IPADIC comes from the instance, not with the app.** Bundled, it would add 14 MB to
   every download of the app (51 MB once unpacked), for people who never watch in
   Japanese too. The curator packs Ubuntu's compiled `mecab-ipadic-utf8` (what MeCab
   0.996 reads) into `mecab-ipadic.tar.gz` beside the dictionary files: the files MeCab
   reads, IPADIC's licence and an empty `mecabrc`, the same bytes for the same
   dictionary, left as it is when unchanged (`curate ipadic`; `curate update` on every
   run). 13.6 MB to download.
3. **JMdict as the instance holds it.** The server exports its JMdict with English
   meanings from its tables: each entry with its keys (`dictionary_key`'s), labels,
   rank, version and the entry as cards take it, as gzipped JSON lines with a header
   (`format` 1). From the tables rather than the curator's file, so that the device finds
   words by the same keys and keeps the same versions, and corrections still reach its
   cards. The server makes the export in its temporary folder as it starts and after
   each load, and serves it and the IPADIC pack on its web server at
   `/device-dictionaries/<name>-<first 16 hex digits of the hash>…` (immutable, so
   cached for good; Caddy forwards the path), only the files it has ready.
   `dictionary.deviceFiles('ja')` names them with their address, SHA-256 and size, once
   both are ready, none for other languages. 21.7 MB.
4. **The device builds a database** from the export, streamed, on an isolate: the
   entries, and every key with its entry's rank and id, ordered as the instance orders
   them (rank, then id), so a key gives the same first eight entries
   (`DeviceDictionaryStore`). Some 8 seconds on a desktop, 101 MB.
5. **The cut.** MeCab's tokens are grouped by Ve's rules (Kim Ahlström's Ve, MIT,
   written anew in Dart, `ve_words.dart`; one change: に stays a word of its own, as
   Jisho.org shows it). Each word is found as the longest match finds a stretch (as it
   stands, or as a form of a word whose class fits, `japaneseWordOf`), else by its
   dictionary form (静かな by 静か), else cut into the longest words it holds from token
   to token (二十五歳 into 二十, 五 and 歳). The longer words running over a word into its
   neighbours, from token to token, are offered as before (携帯 and 電話, which IPADIC
   cuts apart, both offer 携帯電話). A stretch chosen by hand is looked up in the device's
   database. Each batch of lines is cut on an isolate of its own, MeCab and the database
   opened anew (about 100 ms for a batch, which is the line shown, then a whole track).
6. **Asked once, with the size.** The Watch screen offers the download once per device,
   for a dictionary the instance's export would serve, saying how much it downloads
   ("34.0 MB, more once set up"); Not now is final there. Settings › Japanese on this
   device downloads them, says how much room they take and removes them. They belong to
   the device (`<app support>/dictionaries/`, a manifest of what is installed and whether
   the person was asked): the same for every person and instance, downloaded through the
   instance in use. Each file is checked against its SHA-256 and size, fetched over HTTPS
   (plain HTTP only to this machine), and the pack unpacks only the files MeCab reads.
7. **Updates.** Once 30 days have passed, cutting lines asks the instance quietly for
   newer files, and only those whose hash changed are downloaded: IPADIC rarely, JMdict's
   export after the instance loaded a newer JMdict, so at most some 22 MB a month.
8. **The instance stays the fallback.** Without the dictionaries (not downloaded, another
   language, another source), or should cutting on the device fail, lines go to the
   instance as before. Lines already cut are cut anew once the dictionaries arrive.

## Consequences

- A device holding the dictionaries never sends the text of what is watched; the
  instance only learns that it downloaded them, and, monthly, that it asked for newer
  ones.
- Japanese is cut with its grammar, as Jisho.org cuts it; the longest match's
  heuristics stay for the fallback.
- A device spends 34 MB to download and some 150 MB of room; the offer says the first,
  Settings the second, and removing them is a tap.
- Every build, CI's too, needs a C++ compiler.
- Each instance keeps a 22 MB export in its temporary folder, made again after a
  restart, and its web server serves it (production runs one already, for the
  assistant's connector).
- MeCab's, IPADIC's and Ve's licences join the app's licence list.
- Other languages are still cut through the instance.