fix(seed): drop duplicate keys instead of failing the whole batch #24

Merged

raisondetredev merged 1 commit from refs/pull/24/head into main

2026-08-03 17:51:37 +02:00

raisondetredev commented

2026-08-03 17:51:05 +02:00

(Migrated from codeberg.org)

Rows are keyed (id, language) and ids derive from
(language, query, entry index), so a corpus holding one query on two
lines produces two rows with the same key. Postgres rejects an ON
CONFLICT command that would touch the same row twice (SQLSTATE 21000),
and it fails the entire batch, not just the repeat — one duplicated
page 500s a thousand good rows with it.

That is not hypothetical: seeding English died at 13,000 rows because a
crawl restart had re-fetched 28 pages, appending a second line for each.
The repeats were identical but for fetchedAt, which no row field
derives from, so keeping the first loses nothing.

Note the failure was also silent to the caller — these CLIs return 70
from Future<int> main(), which Dart ignores, so the run still exited
0. Left alone here as it is a repo-wide pattern, but scripted seeding
cannot trust the exit code; grep the log for "Upsert failed".

Signed-off-by: Tizian 「ティツィアーン」 raisondetredev@pm.me

Rows are keyed `(id, language)` and ids derive from `(language, query, entry index)`, so a corpus holding one query on two lines produces two rows with the same key. Postgres rejects an ON CONFLICT command that would touch the same row twice (SQLSTATE 21000), and it fails the *entire* batch, not just the repeat — one duplicated page 500s a thousand good rows with it. That is not hypothetical: seeding English died at 13,000 rows because a crawl restart had re-fetched 28 pages, appending a second line for each. The repeats were identical but for `fetchedAt`, which no row field derives from, so keeping the first loses nothing. Note the failure was also silent to the caller — these CLIs `return 70` from `Future<int> main()`, which Dart ignores, so the run still exited 0. Left alone here as it is a repo-wide pattern, but scripted seeding cannot trust the exit code; grep the log for "Upsert failed". Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>