feat(seed): tag Greek entries with HNC lemma-bulk membership #21

Merged

raisondetredev merged 1 commit from refs/pull/21/head into main

2026-07-01 21:53:24 +02:00

raisondetredev commented

2026-07-01 21:52:20 +02:00

(Migrated from codeberg.org)

What

Populates the hnc_lemma_words_<N> boolean columns on dictionary_entries during Greek seeding, so the app's "Top N most common words" bulk-add (500 … 5000) actually returns words. This is the Greek analogue of the Japanese jlpt join.

Why

The feature was already wired end-to-end except the population step: the 47 nested HNC frequency lists (hnc_lemma_words_bulk_<N>.txt), the 47 hnc_lemma_words_<N> columns, and the count_dictionary_entries / add_dictionary_vocabulary RPCs + HncWordsFilter chips all exist. But nothing ever set the flags — the bilingual seeder emitted them implicitly-absent — so every Greek entry had them false and "Top N" returned nothing.

How

New HncLemmaBulks (lib/src/hnc_lemma_bulks.dart) — reads the nested hnc_lemma_words_bulk_<N>.txt lists (500 … 23500, step 500) into a smallest-bulk-per-lemma map (the lists are supersets, so a word's membership is fully described by the smallest bulk it appears in). flagsFor(word) returns the 47 columns; a word in the top-smallest list is flagged for smallest and every larger bucket. Matching is lowercased and accent-folded (tonos + final sigma ς → σ) so headword accentuation differences don't cause misses. No Unicode package needed — Greek is all BMP.
BilingualSeed.rowsForLine takes an optional HncLemmaBulks and spreads the flags into each row, matched on the Greek expression (the headword for el/el-de). Non-Greek/unlisted words match nothing → all false.
seed_bilingual gains --hnc-bulk-dir <dir> (mirror of --jlpt-dir): loads the lists, logs the lemma count, and reports how many seeded rows matched a bulk (so a bad path / low match-rate is visible).
No migration or RPC change — columns, lists and RPCs are already deployed. Re-seeding the existing Greek data with --hnc-bulk-dir updates the flags in place (upsert merges on (id, language)); no new rows.

Verification

dart analyze clean; dart test green (new hnc_lemma_bulks_test.dart: nested membership → correct flags, boundary buckets, absent → all false, case/accent/final-sigma folding, and the seeder join).
Dry-run against the real 47-file bulk_lists/ (23 432 lemmas loaded): κάνω (top-500) → all 47 flags true; χτύπημα → smallest bulk 1500; ουρανοξύστης → smallest bulk 7500; a German de-el headword matched nothing. Match-count log correct.

Follow-ups (not in this PR)

The deployed add_dictionary_vocabulary RPC SELECTs e.lemma, a column no checked-in migration defines (selected but unused in the loop) — latent inconsistency, unrelated to flag population.
wiktionary_5k is an orphaned column (added + indexed, referenced by neither RPC nor app).

## What Populates the `hnc_lemma_words_<N>` boolean columns on `dictionary_entries` during Greek seeding, so the app's **"Top N most common words"** bulk-add (500 … 5000) actually returns words. This is the Greek analogue of the Japanese `jlpt` join. ## Why The feature was already wired end-to-end **except the population step**: the 47 nested HNC frequency lists (`hnc_lemma_words_bulk_<N>.txt`), the 47 `hnc_lemma_words_<N>` columns, and the `count_dictionary_entries` / `add_dictionary_vocabulary` RPCs + `HncWordsFilter` chips all exist. But nothing ever set the flags — the bilingual seeder emitted them implicitly-absent — so every Greek entry had them false and "Top N" returned nothing. ## How - **New `HncLemmaBulks`** (`lib/src/hnc_lemma_bulks.dart`) — reads the nested `hnc_lemma_words_bulk_<N>.txt` lists (500 … 23500, step 500) into a smallest-bulk-per-lemma map (the lists are supersets, so a word's membership is fully described by the smallest bulk it appears in). `flagsFor(word)` returns the 47 columns; a word in the top-`smallest` list is flagged for `smallest` and every larger bucket. Matching is lowercased and accent-folded (tonos + final sigma `ς → σ`) so headword accentuation differences don't cause misses. No Unicode package needed — Greek is all BMP. - **`BilingualSeed.rowsForLine`** takes an optional `HncLemmaBulks` and spreads the flags into each row, matched on the Greek `expression` (the headword for `el`/`el-de`). Non-Greek/unlisted words match nothing → all false. - **`seed_bilingual`** gains `--hnc-bulk-dir <dir>` (mirror of `--jlpt-dir`): loads the lists, logs the lemma count, and reports how many seeded rows matched a bulk (so a bad path / low match-rate is visible). - **No migration or RPC change** — columns, lists and RPCs are already deployed. Re-seeding the existing Greek data with `--hnc-bulk-dir` updates the flags in place (upsert merges on `(id, language)`); no new rows. ## Verification - `dart analyze` clean; `dart test` green (new `hnc_lemma_bulks_test.dart`: nested membership → correct flags, boundary buckets, absent → all false, case/accent/final-sigma folding, and the seeder join). - Dry-run against the real 47-file `bulk_lists/` (23 432 lemmas loaded): `κάνω` (top-500) → all 47 flags true; `χτύπημα` → smallest bulk 1500; `ουρανοξύστης` → smallest bulk 7500; a German `de-el` headword matched nothing. Match-count log correct. ## Follow-ups (not in this PR) - The deployed `add_dictionary_vocabulary` RPC `SELECT`s `e.lemma`, a column no checked-in migration defines (selected but unused in the loop) — latent inconsistency, unrelated to flag population. - `wiktionary_5k` is an orphaned column (added + indexed, referenced by neither RPC nor app).