fix: dictionary headwords, search order and a loader that can't crash

The curator:
- A word JMdict says is usually written in kana gets a kana headword,
  even where the JLPT lists give kanji (その, not 其の), and a rare or
  search-only spelling is never the headword while a usual reading
  exists (そして, not 而して); 9,611 headwords change, 96 of them N5.
- Keys hold full-width letters as typed (Tシャツ as tシャツ), デュ/テュ/
  フュ/イェ romanised, long a, i and e typed once as loose keys (kado for
  カード), and a blank JLPT line is skipped.

The server:
- A search is one query ordered by tier, then rank: a loose or prefix
  match no longer pushes an exact one out of the list (75-150 ms over
  all of JMdict, before 150-450).
- Queries find long vowels with macrons spelt おお or ー and typed once
  (ōkii, kōhī), ん typed n' or nn, and full-width letters.
- Pages of a label are ordered by rank and id, so entries that share a
  rank are neither repeated nor skipped.
- The loader never throws (an unreadable folder ended the server from
  the background), refuses two files of one source rather than loading
  both at every start, reads no rows back when replacing a source, and
  takes its time from clock.
- Dictionary files stay out of the Docker build context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-09 12:15:16 +02:00
commit 50ca46f6b1
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64

View file

@ -3,6 +3,9 @@
# Secrets and local state
halkyon_learn_server/config/passwords.yaml
# Dictionary files: tens of megabytes, mounted on an instance, never built in
halkyon_learn_server/dictionaries/
deploy/dictionaries/
halkyon_learn_server/.serverpod/
deploy/.env
**/*.log