perf(cards): index the bulk-add de-duplication probe #84

Merged

raisondetredev merged 1 commit from refs/pull/84/head into main

2026-08-03 17:51:25 +02:00

raisondetredev commented

2026-08-03 17:51:11 +02:00

(Migrated from codeberg.org)

Measured at ~2 us per card already present (a probe against a
7,745-card deck took 13.8 ms), a 114,830-word "All words" add worked out
to ~3.7 hours against the function's own statement_timeout of 120s: it
would abort ~11k cards in and roll the whole transaction back. So the
largest bulk adds could never succeed. English has no frequency bands to
split the work into, and Greek's "All" (37,399) was already past the
ceiling too.

Two indexes rather than one composite, because the probe is an OR of two
independent tests: the planner now covers it with a BitmapOr over both,
taking the same probe from 13.768 ms to 0.115 ms — and, more to the
point, off the quadratic curve.

Applied to the live database.

Signed-off-by: Tizian 「ティツィアーン」 raisondetredev@pm.me

add_dictionary_vocabulary loops over the catalog row by row, asking for each candidate whether the deck already holds it. Neither arm of that EXISTS was indexable — both sides are expressions — so the only usable index was cards_studyset_id_idx and every probe scanned the whole deck. Since the deck grows as the add proceeds, the loop was quadratic in the words added. Measured at ~2 us per card already present (a probe against a 7,745-card deck took 13.8 ms), a 114,830-word "All words" add worked out to ~3.7 hours against the function's own statement_timeout of 120s: it would abort ~11k cards in and roll the whole transaction back. So the largest bulk adds could never succeed. English has no frequency bands to split the work into, and Greek's "All" (37,399) was already past the ceiling too. Two indexes rather than one composite, because the probe is an OR of two independent tests: the planner now covers it with a BitmapOr over both, taking the same probe from 13.768 ms to 0.115 ms — and, more to the point, off the quadratic curve. Applied to the live database. Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>