Collecting pending changes read each change's domain row on its own: one
statement per insert, and one per update for the column it carries. A
device that had never synced a space of about 100,600 field rows made the
server run some 60,000 of them in the collection's one transaction, 143
to 198 s, and since nothing is sent before everything is collected, the
client's connection closed first and the next attempt started over.
Before walking the inserts, and again before walking the updates, the
collection now reads the rows they name ahead, 500 to a statement per
table and space, with the same column expressions, in the same snapshot:
SELECT those columns WHERE "spaceId" = the space AND "id" IN (the rows).
The raw rows are kept for the collection and each found row's owner is
recorded in the owner cache; an insert decodes its row and restores its
attempted values from them as before, and an update takes its column
from them, so the updates of a row collected as an insert read nothing
more. A row the read-ahead didn't find (missing, or owned by another
space) takes the per-row path unchanged, which tells those cases apart;
both end the collection with an integrity violation, so they are left
unbatched. Deletes still look up their owners row by row, but find in
the owner cache every row the same collection read ahead. Chunking and
streaming are unchanged.
Measured on embedded PostgreSQL, a first collection of 30,040 rows and
90,054 updates (90,067 field rows) took 51.5 to 53.6 s before and takes
16.9 to 17.1 s now, its 120,194 changes identical on the wire, byte for
byte, to what the per-row path produces from the same database. On SQLite,
3,594 changes of a fixture with structured columns, blobs, deletes and
attempted foreign keys on inserts and updates are byte for byte the same
before and after. The new test collects 1,200 rows, 600 of them renamed,
for a fresh peer (1,800 reads of the table before, 3 now), and 600 renames
since a peer's checkpoint (600 reads before, 2 now), and checks every
payload against the author's rows and the peer's rows after the merge.
The core, server, integration and simulation suites pass, but for one
cross-space seed with delta delivery (DST_SEED_BASE=1794152224), whose
local update of an organization finds no row before this change as after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
A merge wrote each inserted row one by one: a savepoint for the domain
row, another for its tracker, two statements each, and then read every
tracker back to record attempted values. A device's first push of a
library of 8,000 cards (24,650 changes) took the server 34 to 37 s, nearly
all of it in those round trips.
A batch the projection pass leaves out (nothing in it touches a foreign
key or a unique index, or its inserts are proven to change no decision)
now writes its new rows many to a statement, up to 1,000 at a time, in
one savepoint: each table's domain rows in one insert, parents' tables
first, then their trackers. A row qualifies when the batch inserts it once
and touches it no other way, no tracker exists for it, nothing earlier in
the batch names it, and every row of the batch it names was written
before it. Its values are planned in the batch's order, as the per-row
path plans them. A chunk whose writes fail (a row there already, perhaps
another space's) is rolled back, the presence it claimed for its rows is
forgotten, and it and every row after it are left to the per-row path,
which tells those cases apart. Attempted values read the trackers the
merge holds instead of querying them.
The proof for merged inserts also accepts a parent of the row's own
table inserted earlier in the batch, so a library's nested studysets no
longer send its first push through the pass; a chunk ends before a row
whose parent of its own table it holds. A row naming itself, or a parent
inserted after it, still takes the pass.
Measured on PostgreSQL, the 24,650 changes merge in about 5 s instead of
34 to 37 s, and with the studysets nested instead of 41 s. A device on
SQLite, whose adapter still inserts row by row, receives the library in
44 s instead of 60 s. The new tests cover a batch spanning chunks, a chunk
failing on a row whose tracker was lost and falling back, and a tree of
one table's rows inserted parents first (the shortcut, a chunk per level)
and children first (the pass). The core, server, integration and
simulation suites pass, but for two PostgreSQL tests that can't start
their database while another embedded PostgreSQL runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A merge that deleted anything ran the full projection pass, whose closure
follows foreign keys both ways and so loads every row connected to the
deleted ones. A device deleting a card with its revisions, tags and
reviews cost each device receiving it about 4 s, and the server merging
the push about 5.4 s, in a library of 8,000 cards whose tags join it.
A device deleting a row locally tombstones everything visible below it,
so its batch carries those deletes along. What the pass could still
decide is a row below that the deleting device never saw, inserted here
meanwhile, which it hides by cascade. A batch that only deletes (none
restoring) now leaves the pass out when the deleted tables cascade only,
by the same rules as a local delete (_deletesOnlyCascade, shared with
canSkipProjectionAfterLocalDelete now), no attempted value is pending in
the connected tables, and every visible row below a deleted one, found by
walking down the cascades alone, is deleted by the same batch
(canLeaveProjectionUnchangedByMergedDeletes). A batch both inserting and
deleting, or restoring, takes the pass as before.
Measured on PostgreSQL: such a card delete merges in about 35 ms instead
of 5.4 s; received on a device, 99 ms instead of 4 s. The new tests
cover a leaf deleted through the shortcut and a middle row's delete,
which isn't eligible, hiding its children by the pass; in the app's sync
tests a card deleted while another device rated it declines the shortcut,
and dropping the walk's check fails that test. The core, server,
integration and simulation suites pass, but for two PostgreSQL tests
that can't start their database while another embedded PostgreSQL runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A merge that touched any foreign key ran the planning pass for its new
rows and then the full projection pass, whose closure follows foreign
keys both ways and so loads every row connected to the merged ones. In a
library whose tags join its parts, one merged rating, edit, tag or new
card loaded all of it: about 9.5 s on a device receiving it and 12 s on
the server merging a push, for 8,000 cards.
A batch that only inserts new rows (each once) and updates columns no
projection reads now leaves both passes out when its inserts are proven,
as local inserts are (canLeaveProjectionUnchangedByMergedInserts, on
localProjectionDependenciesAreStable): each row references its parents by
id, each parent visible or inserted earlier in the same batch, nothing
refers to the row yet (no row, no attempted value), and no unique claim
is contested. A parent inserted after its child in the batch would find
the child's reference held back, which only the pass restores, so it
counts only when earlier. Deletes, inserts onto rows there already and
anything unproven take the passes as before. The batch's parents are
still resolved together for the writes.
The new tests cover a child of a visible parent, a parent and child in
one batch (both through the shortcut), and a child of a merged delete
and a parent arriving after its held-back child (both through the
passes); dropping the parent or the reference check fails the matching
test. The core, server, integration and simulation suites pass, but for
two PostgreSQL tests that cannot start their database while another
embedded PostgreSQL runs, with or without this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
insteadOfDelete ran the full foreign key projection after every local
delete. Its closure follows edges both ways, so it loads every row
connected to the deleted one, which in a schema whose tables all hang off
one another is nearly the whole database: one deleted row of a person's
8,000 flashcards loaded 56,000 of 56,350 CRDT rows and took about 8 s on
the calling isolate, most of it deserialising rows. Inserts and updates
have had a proof that the pass is unnecessary
(canLeaveLocalProjectionUnchanged); deletes had none.
canSkipProjectionAfterLocalDelete proves it for a delete: every table
the delete can cascade into is reached by cascade edges on id only, none
of them restricts its parent (a deleted child can release a parent
delete held back by it), none has a unique index (a released claim can
bring a contestant back) or lies in a component with a SET DEFAULT edge,
and no attempted value is pending in the connected tables. The delete's
own cascade (applyDeleteActions) then is the whole projection; otherwise
the full pass runs as before. applyDeleteActions also asks for the
visible children of a whole batch of parents at once for cascades by id,
instead of one query per parent.
On that database a single delete now takes 6-63 ms, a card with its
revisions, tags and reviews 46-145 ms instead of 28-39 s. The new tests
cover a restrict child deleted locally releasing a merged parent delete
(on the person tables, and on the chain tables, where the shortcut is
asked and declines: the test fails with the restrict condition removed)
and a cascade child deleted locally through the shortcut, its parent
then deleted by a merged remote delete. The core, server, integration
and simulation suites pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
