perf: Write a merge's new rows in chunks when the pass is left out

A merge wrote each inserted row one by one: a savepoint for the domain
row, another for its tracker, two statements each, and then read every
tracker back to record attempted values. A device's first push of a
library of 8,000 cards (24,650 changes) took the server 34 to 37 s, nearly
all of it in those round trips.

A batch the projection pass leaves out (nothing in it touches a foreign
key or a unique index, or its inserts are proven to change no decision)
now writes its new rows many to a statement, up to 1,000 at a time, in
one savepoint: each table's domain rows in one insert, parents' tables
first, then their trackers. A row qualifies when the batch inserts it once
and touches it no other way, no tracker exists for it, nothing earlier in
the batch names it, and every row of the batch it names was written
before it. Its values are planned in the batch's order, as the per-row
path plans them. A chunk whose writes fail (a row there already, perhaps
another space's) is rolled back, the presence it claimed for its rows is
forgotten, and it and every row after it are left to the per-row path,
which tells those cases apart. Attempted values read the trackers the
merge holds instead of querying them.

The proof for merged inserts also accepts a parent of the row's own
table inserted earlier in the batch, so a library's nested studysets no
longer send its first push through the pass; a chunk ends before a row
whose parent of its own table it holds. A row naming itself, or a parent
inserted after it, still takes the pass.

Measured on PostgreSQL, the 24,650 changes merge in about 5 s instead of
34 to 37 s, and with the studysets nested instead of 41 s. A device on
SQLite, whose adapter still inserts row by row, receives the library in
44 s instead of 60 s. The new tests cover a batch spanning chunks, a chunk
failing on a row whose tracker was lost and falling back, and a tree of
one table's rows inserted parents first (the shortcut, a chunk per level)
and children first (the pass). The core, server, integration and
simulation suites pass, but for two PostgreSQL tests that can't start
their database while another embedded PostgreSQL runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test/serverpod_offline_sync_test_server/test/integration/merge/merge_new_rows_in_chunks_test.dart

@@ -0,0 +1,225 @@
+import 'dart:typed_data';
+
+import 'package:serverpod/serverpod.dart';
+import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
+import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
+import 'package:test/test.dart';
+
+import '../test_tools/client_session.dart';
+import '../test_tools/crdt_probes.dart';
+
+// A batch the projection pass leaves out writes its new rows many to a
+// statement, up to a thousand at a time, rather than one by one. The rows
+// and their trackers have to come out as one by one, also where a chunk
+// fails and the rows from it on are written one by one after all.
+void main() {
+  initTestClientSession();
+
+  final node = const Uuid().v7obj();
+  final authoredAt = DateTime.now().toUtc();
+  var counter = 0;
+
+  // In the order a device authored them, as a push sends them.
+  CrdtMergeInsert insertOf(TableRow row, String tableName) => CrdtMergeInsert(
+    uuidSpaceId: testCrdtUserId,
+    tableName: tableName,
+    uuidRowId: row.id! as UuidValue,
+    uuidNodeId: node,
+    hlcDatetime: authoredAt,
+    hlcCounter: counter++,
+    data: row,
+  );
+
+  /// [count] middles, each followed by a cascade child naming it.
+  (List<FkChainSetNullMiddle>, List<FkChainSetNullCascadeChild>) family(
+    int count,
+  ) {
+    final middles = [
+      for (var i = 0; i < count; i++)
+        FkChainSetNullMiddle(id: const Uuid().v7obj(), name: 'middle $i'),
+    ];
+    final children = [
+      for (final (i, middle) in middles.indexed)
+        FkChainSetNullCascadeChild(
+          id: const Uuid().v7obj(),
+          name: 'child $i',
+          setNullMiddleId: middle.id,
+        ),
+    ];
+    return (middles, children);
+  }
+
+  List<CrdtMergeChange> inserts(
+    List<FkChainSetNullMiddle> middles,
+    List<FkChainSetNullCascadeChild> children,
+  ) => [
+    for (final (i, middle) in middles.indexed) ...[
+      insertOf(middle, FkChainSetNullMiddle.t.tableName),
+      insertOf(children[i], FkChainSetNullCascadeChild.t.tableName),
+    ],
+  ];
+
+  Future<void> expectWrittenAsAuthored(
+    List<FkChainSetNullMiddle> middles,
+    List<FkChainSetNullCascadeChild> children,
+  ) async {
+    final storedMiddles = {
+      for (final middle in await FkChainSetNullMiddle.db.find(
+        session,
+        where: (t) => t.id.inSet(<UuidValue>{for (final middle in middles) middle.id!}),
+      ))
+        middle.id: middle,
+    };
+    final storedChildren = {
+      for (final child in await FkChainSetNullCascadeChild.db.find(
+        session,
+        where: (t) => t.id.inSet(<UuidValue>{for (final child in children) child.id!}),
+      ))
+        child.id: child,
+    };
+    for (final middle in middles) {
+      expect(storedMiddles[middle.id]?.name, middle.name);
+    }
+    for (final child in children) {
+      expect(storedChildren[child.id]?.setNullMiddleId, child.setNullMiddleId);
+    }
+    final trackers = await CrdtDataRow.db.find(
+      testSession,
+      where: (t) => t.uuidRowId.inSet(<UuidValue>{
+        for (final middle in middles) middle.id!,
+        for (final child in children) child.id!,
+      }),
+    );
+    expect(trackers, hasLength(middles.length + children.length));
+    expect(
+      await attemptedValue(
+        rowId: children.last.id!,
+        columnName: FkChainSetNullCascadeChild.t.setNullMiddleId.columnName,
+      ),
+      isNull,
+    );
+  }
+
+  group('Given a remote batch of 1,200 new middles, each followed by a '
+      'cascade child naming it,', () {
+    late List<FkChainSetNullMiddle> middles;
+    late List<FkChainSetNullCascadeChild> children;
+
+    setUp(() async {
+      (middles, children) = family(1200);
+      await session.db.mergeChanges(
+        inserts(middles, children),
+        spaceId: testCrdtUserId,
+      );
+    });
+
+    test('then every row is there as authored, with one tracker each.', () async {
+      await expectWrittenAsAuthored(middles, children);
+    });
+  });
+
+  group('Given a middle here whose tracker was lost,', () {
+    late List<FkChainSetNullMiddle> middles;
+    late List<FkChainSetNullCascadeChild> children;
+
+    setUp(() async {
+      (middles, children) = family(800);
+      final lost = middles[600];
+      await session.db.transactionForUser(
+        testCrdtUserId,
+        (tx) => FkChainSetNullMiddle.db.insertRow(
+          session,
+          FkChainSetNullMiddle(id: lost.id, name: 'before recovery'),
+          transaction: tx,
+        ),
+      );
+      await CrdtDataRow.db.deleteWhere(
+        testSession,
+        where: (t) => t.uuidRowId.equals(lost.id),
+      );
+    });
+
+    group('when a remote batch of new rows holding it in its second '
+        'thousand is merged,', () {
+      setUp(() async {
+        await session.db.mergeChanges(
+          inserts(middles, children),
+          spaceId: testCrdtUserId,
+        );
+      });
+
+      test('then every row is there as authored, the lost one recovered, '
+          'with one tracker each.', () async {
+        await expectWrittenAsAuthored(middles, children);
+      });
+    });
+  });
+
+  group('Given rows of one table forming a tree,', () {
+    // Each row's parent is the row half as far into the list.
+    late List<Types> tree;
+
+    setUp(() {
+      tree = [];
+      for (var i = 0; i < 40; i++) {
+        tree.add(
+          Types(
+            id: const Uuid().v7obj(),
+            aBool: true,
+            aDateTime: DateTime.utc(2026),
+            aText: 'row $i',
+            anInt: i,
+            anInt64: BigInt.from(i),
+            aReal: 0.5,
+            aBlob: ByteData(1),
+            parentId: i == 0 ? null : tree[(i - 1) ~/ 2].id,
+          ),
+        );
+      }
+    });
+
+    Future<void> expectTreeAsAuthored() async {
+      final stored = {
+        for (final row in await Types.db.find(
+          session,
+          where: (t) => t.id.inSet(<UuidValue>{for (final row in tree) row.id!}),
+        ))
+          row.id: row,
+      };
+      for (final row in tree) {
+        expect(stored[row.id]?.parentId, row.parentId);
+        expect(
+          await attemptedValue(
+            rowId: row.id!,
+            columnName: Types.t.parentId.columnName,
+          ),
+          isNull,
+        );
+      }
+    }
+
+    group('when a remote batch inserting them, parents first, is merged,', () {
+      setUp(() async {
+        await session.db.mergeChanges([
+          for (final row in tree) insertOf(row, Types.t.tableName),
+        ], spaceId: testCrdtUserId);
+      });
+
+      test('then every row is there, holding its parent.', () async {
+        await expectTreeAsAuthored();
+      });
+    });
+
+    group('when a remote batch inserting them, children first, is merged,', () {
+      setUp(() async {
+        await session.db.mergeChanges([
+          for (final row in tree.reversed) insertOf(row, Types.t.tableName),
+        ], spaceId: testCrdtUserId);
+      });
+
+      test('then every row is there, holding its parent.', () async {
+        await expectTreeAsAuthored();
+      });
+    });
+  });
+}