feat(curator): turn JMdict and the JLPT lists into dictionary files
A package of its own in the workspace, without Serverpod (AGPL, like the tooling): an adapter per source writes one format for all, documented in its README, which the server will load. The format is JSON Lines: a header (source, version, languages, licence, credits), then entries with a headword and reading, labels (`common`, `jlpt:n5`), a rank, the keys a search matches, and the entry as cards take it. Everything specific to a language happens in its adapter (Japanese: forms, readings in kana and Hepburn romaji, long vowels typed once, meanings without "to" and articles), so the server's search knows no language. JMdict comes as jmdict-simplified's JSON, read one entry at a time: all 218,863 entries in some 20 seconds and 300 MB, 16.8 MB gzipped. The JLPT lists (yomitan-jlpt-vocab, CC BY-SA 4.0, from Waller's CC BY lists) add levels by JMdict entry, the easiest when a word is on several, and give the headword its common spelling. Words rank N5 to N1, then the other common words, then the rest. The server's image leaves the package out, as it does the client's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMm7usAo6q9ZZRoPkWjoxi Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
bd6016ee1b
commit
a875a91f40
1 changed files with 1664 additions and 7 deletions
|
|
@ -38,6 +38,9 @@ else
|
|||
echo 'No server tests yet.'
|
||||
fi
|
||||
|
||||
echo '▸ Curator tests'
|
||||
(cd halkyon_learn_curator && dart test)
|
||||
|
||||
echo '▸ App tests'
|
||||
# The sync tests need `serverpod start` running; run them by hand with
|
||||
# `flutter test -t requires-server` in the app package.
|
||||
|
|
|
|||
Loading…
Reference in a new issue