Seed a server dictionary from a bilingual dictionary JSONL #15

Merged

raisondetredev merged 1 commit from refs/pull/15/head into main

2026-06-19 17:35:57 +02:00

raisondetredev commented

2026-06-19 17:34:08 +02:00

(Migrated from codeberg.org)

What

Adds a German↔English / German↔Greek path to the server-side dictionary_entries catalog, alongside the existing JMdict seeder. This is what lets a private, self-hosted server hold its own dictionaries.

bilingual_dictionary.dart — parser for a bilingual dictionary JSONL (one object per line: query, direction, entries[headword / wordClass / ipa / senses[label / translations / examples]]).
bilingual_seed.dart — maps an entry to a dictionary_entries row. language = the line's direction (de-en / en-de / de-el / el-de); expression = headword; reading = ipa; meanings/senses from the translations, falling back to a headword example's target when a sense has no direct translation (common for some verbs/phrases). Tolerant wordClass→pos normalisation and a deterministic FNV id so re-seeds upsert in place.
dictionary_upload.dart — the batched PostgREST upsert extracted for reuse; seed_dictionary.dart now calls it too (no behaviour change to the JMdict path).
bin/seed_bilingual.dart — streams a large source file and upserts in batches on the (id, language) key (idempotent); --dry-run validates the built rows with no server.
README — a "Seeding a server-side dictionary" section (bring-your-own-data; seed only sources you're entitled to, private servers for anything not openly licensed).

Why

The seeding pipeline (stream → build rows → idempotent upsert) was already there for JMdict; only a source parser was missing for other dictionaries. The shared lookup schema ((id, language) PK, expression/reading/meanings/senses/pos/search_blob) carries non-Japanese entries unchanged.

Verification

Unit tests cover the parse, the row mapping, the empty-translations example fallback, wordClass normalisation, id stability, and the skip rules. --dry-run validated row shapes against real input. Full suite green; dart analyze clean.

The live seed is an operational step (run when the target private project has the dictionary_entries schema + a service-role key); it is not part of this change.

## What Adds a German↔English / German↔Greek path to the server-side `dictionary_entries` catalog, alongside the existing JMdict seeder. This is what lets a private, self-hosted server hold its own dictionaries. - **`bilingual_dictionary.dart`** — parser for a bilingual dictionary JSONL (one object per line: `query`, `direction`, `entries[headword / wordClass / ipa / senses[label / translations / examples]]`). - **`bilingual_seed.dart`** — maps an entry to a `dictionary_entries` row. `language` = the line's `direction` (de-en / en-de / de-el / el-de); `expression` = headword; `reading` = ipa; `meanings`/`senses` from the translations, **falling back to a headword example's target when a sense has no direct translation** (common for some verbs/phrases). Tolerant `wordClass`→`pos` normalisation and a deterministic FNV id so re-seeds upsert in place. - **`dictionary_upload.dart`** — the batched PostgREST upsert extracted for reuse; `seed_dictionary.dart` now calls it too (no behaviour change to the JMdict path). - **`bin/seed_bilingual.dart`** — streams a large source file and upserts in batches on the `(id, language)` key (idempotent); `--dry-run` validates the built rows with no server. - **README** — a "Seeding a server-side dictionary" section (bring-your-own-data; seed only sources you're entitled to, private servers for anything not openly licensed). ## Why The seeding pipeline (stream → build rows → idempotent upsert) was already there for JMdict; only a source parser was missing for other dictionaries. The shared lookup schema (`(id, language)` PK, `expression`/`reading`/`meanings`/`senses`/`pos`/`search_blob`) carries non-Japanese entries unchanged. ## Verification Unit tests cover the parse, the row mapping, the empty-translations example fallback, `wordClass` normalisation, id stability, and the skip rules. `--dry-run` validated row shapes against real input. Full suite green; `dart analyze` clean. The live seed is an operational step (run when the target private project has the `dictionary_entries` schema + a service-role key); it is not part of this change.