feat(seed): HNC frequency rank + merge duplicate dictionary entries #23

Merged

raisondetredev merged 2 commits from refs/pull/23/head into main

2026-07-02 01:18:07 +02:00

raisondetredev commented

2026-07-01 22:58:20 +02:00

(Migrated from codeberg.org)

Two curator-side fixes for Greek vocabulary quality.

1. Frequency rank (most-common-first ordering)

HncLemmaBulks.rankFor(word) → 1-based position in the master HNC list (rank <= band; null when unlisted). Stamped as hnc_rank by BilingualSeed; backfill_hnc_rank sets it on already-seeded rows via full-row merge-upsert. The server RPC staggers due_at by this so bulk-added cards are introduced most-common-first. (Backfilled live: 24,087 ranked.)

2. Merge duplicate (expression, reading) entries

The bilingual source splits one headword into several rows (one per sense-block / part of speech) — e.g. κάποιος as adjective "irgendein" AND pronoun "jemand", or κάνω's 9 senses across two rows. Bulk-add de-dups cards by (expression, reading), so only one split row became a card and the others' meanings were silently dropped; it also made the catalog count exceed the cards actually added (518 vs 505).

mergeDuplicateEntries / mergeEntryGroup collapse rows sharing (expression, reading), unioning meanings/senses/pos (richest row leads → primary meaning first; common OR-ed, most-elementary jlpt, smallest hnc_rank). Applied in BilingualSeed.rowsForLine (future seeds clean) and exposed via a new dedup_dictionary_entries CLI to fix the live catalog in place (PATCH survivor, delete the rest; small delete chunks to avoid HTTP 414).

Applied to the live el catalog: 539 groups merged, 546 rows removed (37,945 → 37,399). Top-500 count is now 505 = cards added; κάνω carries all 9 meanings; re-run finds 0 duplicates.

Verification

dart analyze clean; dart test green (153 tests) incl. rankFor and mergeEntryGroup/mergeDuplicateEntries cases (union meanings richest-first, distinct pos, OR common, min rank, different readings not merged, seeder applies the merge).

Two curator-side fixes for Greek vocabulary quality. ## 1. Frequency rank (most-common-first ordering) `HncLemmaBulks.rankFor(word)` → 1-based position in the master HNC list (`rank <= band`; null when unlisted). Stamped as `hnc_rank` by `BilingualSeed`; `backfill_hnc_rank` sets it on already-seeded rows via full-row merge-upsert. The server RPC staggers `due_at` by this so bulk-added cards are introduced most-common-first. (Backfilled live: 24,087 ranked.) ## 2. Merge duplicate (expression, reading) entries The bilingual source splits one headword into several rows (one per sense-block / part of speech) — e.g. κάποιος as adjective "irgendein" AND pronoun "jemand", or κάνω's 9 senses across two rows. Bulk-add de-dups cards by (expression, reading), so only one split row became a card and the others' **meanings were silently dropped**; it also made the catalog count exceed the cards actually added (518 vs 505). `mergeDuplicateEntries` / `mergeEntryGroup` collapse rows sharing (expression, reading), unioning `meanings`/`senses`/`pos` (richest row leads → primary meaning first; `common` OR-ed, most-elementary `jlpt`, smallest `hnc_rank`). Applied in `BilingualSeed.rowsForLine` (future seeds clean) and exposed via a new `dedup_dictionary_entries` CLI to fix the live catalog in place (PATCH survivor, delete the rest; small delete chunks to avoid HTTP 414). **Applied to the live el catalog:** 539 groups merged, 546 rows removed (37,945 → 37,399). Top-500 count is now **505 = cards added**; κάνω carries all 9 meanings; re-run finds 0 duplicates. ## Verification `dart analyze` clean; `dart test` green (153 tests) incl. `rankFor` and `mergeEntryGroup`/`mergeDuplicateEntries` cases (union meanings richest-first, distinct pos, OR common, min rank, different readings not merged, seeder applies the merge).