fix: Correct checkpoint progress and add delta DST coverage (#158)

This commit is contained in:
Marcelo Mendonça Soares 2026-10-07 15:49:15 -03:00 • committed by GitHub
commit f59a90bc97
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
24 changed files with 1829 additions and 163 deletions

View file

@ -99,7 +99,7 @@ jobs:
# comment-tag: coverage-results # comment-tag: coverage-results
simulation: simulation:
name: deterministic simulation (${{ matrix.topology }}) name: deterministic simulation (${{ matrix.topology }}, ${{ matrix.delivery }})
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 60 timeout-minutes: 60
# Deliberately not on pull requests. The suite seeds from the clock, so it # Deliberately not on pull requests. The suite seeds from the clock, so it
@ -115,6 +115,7 @@ jobs:
# Keep the topology files below and shared ownership controls exhaustive # Keep the topology files below and shared ownership controls exhaustive
# when adding a new test file tagged dst. # when adding a new test file tagged dst.
topology: [convergence, cross_space] topology: [convergence, cross_space]
delivery: [full, delta]
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- uses: ./.github/actions/setup - uses: ./.github/actions/setup
@ -135,6 +136,7 @@ jobs:
DST_ROUNDS: 200 DST_ROUNDS: 200
DST_PROFILE: mixed DST_PROFILE: mixed
DST_GRAPH_WIDTH: 2 DST_GRAPH_WIDTH: 2
DST_DELIVERY: ${{ matrix.delivery }}
benchmarks: benchmarks:
name: run benchmarks name: run benchmarks

View file

@ -1,3 +1,17 @@
## Unreleased
- fix: Include the persisted local logical clock in resume checkpoints so
reconnecting within the same millisecond or after clock rollback settles.
- fix: Keep relay checkpoints tagged with their actual author and recover
incorrectly tagged stored progress without repeated transfers on reconnect.
- fix: Require `recordSyncCheckpoint` to receive an HLC authored by its
`otherNodeId`, throwing `ArgumentError` on a mismatch. Valid checkpoints
replace incorrectly tagged stored progress.
- fix: Reconsider released unique claims after local updates and upserts so
the author and peers receiving checkpoint deltas keep the same projection.
- test: Add checkpoint-based deterministic delivery alongside full-history
convergence, with replayable batches and separate coverage metrics.
## 0.0.9 ## 0.0.9
- fix: Refreshes node clocks across sync wrappers. - fix: Refreshes node clocks across sync wrappers.

View file

@ -21,12 +21,14 @@ not new user field updates emitted opportunistically during merge.
- Supported foreign keys are single-column child references to single-column - Supported foreign keys are single-column child references to single-column
parent references. Schemas with composite foreign keys must fail recorder parent references. Schemas with composite foreign keys must fail recorder
initialization instead of silently bypassing projection. initialization instead of silently bypassing projection.
- Merge input is causally complete: the sync protocol delivers each batch as a - Merge input is causally complete relative to the receiver: every merged FK
full causal snapshot of the sender, so every merged FK attempt's parent facts attempt's parent facts must already be committed at the receiver or accompany
precede or accompany the child fact. Input that violates this precondition the child fact in the same complete batch. A checkpoint-filtered delta can
(for example, a non-nullable FK attempt whose parent fact is absent and whose omit acknowledged facts; it need not repeat the sender's full history. Input
default target is missing) is outside the merge contract and fails the merge that violates this precondition (for example, a non-nullable FK attempt whose
transaction atomically against the database foreign key constraints. parent fact is absent and whose default target is missing) is outside the
merge contract and fails the merge transaction atomically against the database
foreign key constraints.
- Local `ON DELETE` actions are authored CRDT facts. `RESTRICT` and `NO ACTION` - Local `ON DELETE` actions are authored CRDT facts. `RESTRICT` and `NO ACTION`
reject the local delete while visible children exist; `SET NULL` and reject the local delete while visible children exist; `SET NULL` and
`SET DEFAULT` update child FK fields and their ordinary `CrdtDataField` HLCs; `SET DEFAULT` update child FK fields and their ordinary `CrdtDataField` HLCs;

View file

@ -52,8 +52,8 @@ membership relation and a sync protocol that iterates spaces.
2. **Smallest protocol delta.** The exchanged frames are today's, plus a 2. **Smallest protocol delta.** The exchanged frames are today's, plus a
per-cycle space-set exchange and a space tag on each frame. `collectNextBatch` per-cycle space-set exchange and a space tag on each frame. `collectNextBatch`
demultiplexes a cycle's combined batch, `collectPendingChanges` collects every demultiplexes a cycle's combined batch, `collectPendingChanges` collects every
active space in one pass, and the per-space merge plus `recordSyncCheckpoint` active space in one pass, and `mergeInboundBatch` applies each inbound space
run once per inbound space group. group while atomically persisting received progress for every author.
3. **Chunked streaming.** Outbound changes are collected in a single pass over 3. **Chunked streaming.** Outbound changes are collected in a single pass over
the active spaces and streamed in `syncBatchSize` chunks, so the wire payload the active spaces and streamed in `syncBatchSize` chunks, so the wire payload
stays bounded regardless of space count. The collection step itself stays bounded regardless of space count. The collection step itself
@ -128,7 +128,7 @@ They do not map one-to-one and neither replaces the other. One member may have
many nodes (one per replica/database install). The **server is a node** in every many nodes (one per replica/database install). The **server is a node** in every
space it syncs but is not a member row. A freshly invited member has **zero space it syncs but is not a member row. A freshly invited member has **zero
space-node rows** until their first device syncs. Nodes and space-node rows are space-node rows** until their first device syncs. Nodes and space-node rows are
created implicitly by sync (`getOrCreate`, `recordSyncCheckpoint`); membership created implicitly by space initialization and inbound merge; membership
rows are managed through `session.offlineSync.spaces` from application endpoints. rows are managed through `session.offlineSync.spaces` from application endpoints.
Removing the `nodes` list would break checkpointing and causal filtering — it is Removing the `nodes` list would break checkpointing and causal filtering — it is
the chain topology, not the access list. the chain topology, not the access list.
@ -316,10 +316,11 @@ The shared establishment exchanges the space set so a follower never acts before
it knows the agreed set. Both cadences handshake initial and newly added spaces it knows the agreed set. Both cadences handshake initial and newly added spaces
through the loop's own `SinceHlc` exchange. through the loop's own `SinceHlc` exchange.
`collectPendingChanges`, `mergeInboundBatch`, `recordSyncCheckpoint`, and the `collectPendingChanges`, `mergeInboundBatch`, and the
ownership-violation internals (`_streamInserts`/`Updates`/`Deletes`, ownership-violation internals (`_streamInserts`/`Updates`/`Deletes`,
`_fetchDomainRow`, the durable-violation path) are space-parameterized and merge `_fetchDomainRow`, the durable-violation path) are space-parameterized and merge
each inbound space group in its own `transactionForUser`/lock; any space's merge each inbound space group in its own `transactionForUser`/lock, including its
per-author checkpoints; any space's merge
failure (for example an ownership collision or unauthorized role write) records failure (for example an ownership collision or unauthorized role write) records
the durable violation and fails the session, preserving today's fail-fast the durable violation and fails the session, preserving today's fail-fast
semantics. semantics.

View file

@ -133,7 +133,11 @@ class OfflineSyncDatabase implements Database {
); );
} }
/// Records the latest acknowledged sync checkpoint for [otherNodeId]. /// Records acknowledged per-author progress for [otherNodeId].
///
/// [syncedHlc] must carry [otherNodeId] as its author. Throws [ArgumentError]
/// otherwise. A correctly tagged checkpoint replaces previously stored
/// progress tagged with another author; valid progress advances monotonically.
Future<void> recordSyncCheckpoint( Future<void> recordSyncCheckpoint(
UuidValue otherNodeId, UuidValue otherNodeId,
Hlc syncedHlc, { Hlc syncedHlc, {
@ -670,6 +674,7 @@ class OfflineSyncDatabase implements Database {
columns, columns,
tx, tx,
projectionUnchanged: plannedUpdates.projectionUnchanged, projectionUnchanged: plannedUpdates.projectionUnchanged,
projectionRows: plannedUpdates.projectionRows,
); );
return noReturn ? <T>[] : updatedRows; return noReturn ? <T>[] : updatedRows;
}, },
@ -707,6 +712,7 @@ class OfflineSyncDatabase implements Database {
columns, columns,
tx, tx,
projectionUnchanged: plannedUpdates.projectionUnchanged, projectionUnchanged: plannedUpdates.projectionUnchanged,
projectionRows: plannedUpdates.projectionRows,
); );
return updatedRow; return updatedRow;
}, },
@ -797,28 +803,42 @@ class OfflineSyncDatabase implements Database {
return _runTrackedWrite( return _runTrackedWrite(
transaction, transaction,
(tx) async { (tx) async {
final result = await _delegate.updateWhere<T>( final scopedWhere = (await _whereVisibleWithTombstone<T>(
columnValues: columnValues, where,
where: (await _whereVisibleWithTombstone<T>( null,
where, tx,
null, membershipWide: false,
tx, ))!;
membershipWide: false, final columns = columnValues.map((e) => e.column).toList();
))!, final plan = await _recorder.planLocalUpdateWhere<T>(
columns,
scopedWhere,
tx,
limit: limit, limit: limit,
offset: offset, offset: offset,
orderBy: orderBy, orderBy: orderBy,
orderByList: orderByList, orderByList: orderByList,
);
final selectedWhere = plan == null
? scopedWhere
: serializationManager.getTableForType(T)!.id.inSet(plan.rowIds);
final result = await _delegate.updateWhere<T>(
columnValues: columnValues,
where: selectedWhere,
limit: plan == null ? limit : null,
offset: plan == null ? offset : null,
orderBy: orderBy,
orderByList: orderByList,
transaction: tx, transaction: tx,
); );
final columns = columnValues.map((e) => e.column).toList();
_recorder.validateAuthoredRows(result, columns); _recorder.validateAuthoredRows(result, columns);
await _recorder.afterUpdate( await _recorder.afterUpdate(
result, result,
columns, columns,
tx, tx,
authoredColumnValues: true, authoredColumnValues: true,
projectionRows: plan?.projectionRows ?? const {},
); );
if (noReturn) return <T>[]; if (noReturn) return <T>[];
result.forEach(_stripSpaceId); result.forEach(_stripSpaceId);

View file

@ -195,8 +195,11 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
in maxIncomingHlcByNode.entries) { in maxIncomingHlcByNode.entries) {
final remoteNode = remoteNodes.spaceNodesByUuid[nodeId]; final remoteNode = remoteNodes.spaceNodesByUuid[nodeId];
if (remoteNode == null) continue; if (remoteNode == null) continue;
final previous = remoteNode.lastReceivedHlc;
final updatedSpaceNode = remoteNode.copyWith( final updatedSpaceNode = remoteNode.copyWith(
lastReceivedHlc: incomingHlc.maxBetween(remoteNode.lastReceivedHlc), lastReceivedHlc: incomingHlc.maxBetween(
previous?.nodeId == nodeId ? previous : null,
),
); );
if (updatedSpaceNode.lastReceivedHlc == remoteNode.lastReceivedHlc) { if (updatedSpaceNode.lastReceivedHlc == remoteNode.lastReceivedHlc) {
continue; continue;

View file

@ -340,12 +340,22 @@ class CrdtMutationRecorder {
Future<void> lockAndRefreshCurrentNodeHlc(Transaction transaction) => Future<void> lockAndRefreshCurrentNodeHlc(Transaction transaction) =>
_context.lockAndRefreshCurrentNodeHlc(transaction); _context.lockAndRefreshCurrentNodeHlc(transaction);
/// Records the latest acknowledged sync checkpoint for [otherNodeId]. /// Records per-author progress, rejecting an HLC not tagged with [otherNodeId].
///
/// Replaces mis-tagged stored progress; correctly tagged progress only advances.
Future<void> recordSyncCheckpoint( Future<void> recordSyncCheckpoint(
UuidValue userId, UuidValue userId,
UuidValue otherNodeId, UuidValue otherNodeId,
Hlc syncedHlc, Hlc syncedHlc,
) async { ) async {
if (syncedHlc.nodeId != otherNodeId) {
throw ArgumentError.value(
syncedHlc,
'syncedHlc',
'A per-author checkpoint must carry otherNodeId',
);
}
final space = await _context.spaceManager.getOrCreate(userId); final space = await _context.spaceManager.getOrCreate(userId);
await _db.transaction((transaction) async { await _db.transaction((transaction) async {
final node = await _context.findOrCreateNode(otherNodeId, transaction); final node = await _context.findOrCreateNode(otherNodeId, transaction);
@ -356,7 +366,9 @@ class CrdtMutationRecorder {
); );
final currentSyncHlc = spaceNode.lastReceivedHlc; final currentSyncHlc = spaceNode.lastReceivedHlc;
if (currentSyncHlc != null && currentSyncHlc >= syncedHlc) { if (currentSyncHlc != null &&
currentSyncHlc.nodeId == otherNodeId &&
currentSyncHlc >= syncedHlc) {
return; return;
} }
@ -574,21 +586,35 @@ class CrdtMutationRecorder {
} }
/// Plans FK/unique projection for rows that are about to be updated. /// Plans FK/unique projection for rows that are about to be updated.
Future<({List<T> rows, bool projectionUnchanged})> Future<({List<T> rows, bool projectionUnchanged, Set<MergeRowKey> projectionRows})>
planLocalUpdates<T extends TableRow>( planLocalUpdates<T extends TableRow>(
List<T> rows, List<T> rows,
List<Column>? columns, List<Column>? columns,
Transaction transaction, { Transaction transaction, {
bool restoring = false, bool restoring = false,
}) async { }) async {
if (rows.isEmpty) return (rows: rows, projectionUnchanged: true); if (rows.isEmpty) {
return (
rows: rows,
projectionUnchanged: true,
projectionRows: const <MergeRowKey>{},
);
}
final tableName = rows.first.table.tableName; final tableName = rows.first.table.tableName;
if (!_context.isCrdtTrackedTableName(tableName)) { if (!_context.isCrdtTrackedTableName(tableName)) {
return (rows: rows, projectionUnchanged: true); return (
rows: rows,
projectionUnchanged: true,
projectionRows: const <MergeRowKey>{},
);
} }
final columnNames = columns?.map((column) => column.columnName).toSet(); final columnNames = columns?.map((column) => column.columnName).toSet();
if (!_foreignKeyProjector.needsProjection(tableName, columnNames)) { if (!_foreignKeyProjector.needsProjection(tableName, columnNames)) {
return (rows: rows, projectionUnchanged: true); return (
rows: rows,
projectionUnchanged: true,
projectionRows: const <MergeRowKey>{},
);
} }
if (await _foreignKeyProjector.canLeaveLocalProjectionUnchanged( if (await _foreignKeyProjector.canLeaveLocalProjectionUnchanged(
@ -597,7 +623,11 @@ class CrdtMutationRecorder {
inserting: false, inserting: false,
columns: columns, columns: columns,
)) { )) {
return (rows: rows, projectionUnchanged: true); return (
rows: rows,
projectionUnchanged: true,
projectionRows: const <MergeRowKey>{},
);
} }
final authored = { final authored = {
@ -646,6 +676,7 @@ class CrdtMutationRecorder {
); );
return ( return (
projectionUnchanged: false, projectionUnchanged: false,
projectionRows: planned.domain.keys.toSet(),
rows: [ rows: [
for (final row in rows) for (final row in rows)
_withPlannedDomainValues( _withPlannedDomainValues(
@ -835,6 +866,8 @@ class CrdtMutationRecorder {
CrdtDataDeletedReason.userReinsert, CrdtDataDeletedReason.userReinsert,
transaction, transaction,
); );
// Hidden rows already released their old unique claims; seed projection
// from the restored rows to resolve their new values.
await _maybeProject(tableName, rowIds, null, transaction); await _maybeProject(tableName, rowIds, null, transaction);
// Only projected fields need a metadata row to hold their authored value. // Only projected fields need a metadata row to hold their authored value.
// All other fields now inherit the insertion timestamp implicitly. // All other fields now inherit the insertion timestamp implicitly.
@ -849,6 +882,52 @@ class CrdtMutationRecorder {
}); });
} }
/// Captures the old projection closure before a set-based physical update.
///
/// Updating a winning unique tuple can disconnect its released claimants.
/// Select targets while the write transaction holds the node lock. The
/// caller must update these identities instead of evaluating a limited
/// predicate again, which could select different rows without a total order.
Future<({Set<UuidValue> rowIds, Set<MergeRowKey> projectionRows})?>
planLocalUpdateWhere<T extends TableRow>(
List<Column> columns,
Expression where,
Transaction transaction, {
int? limit,
int? offset,
Column? orderBy,
List<Column>? orderByList,
}) async {
final table = _session.db.serializationManager.getTableForType(T)!;
if (!_foreignKeyProjector.needsProjection(
table.tableName,
columns.map((column) => column.columnName).toSet(),
)) {
return null;
}
final rows = await _session.db.find<T>(
where: where,
limit: limit,
offset: offset,
orderBy: orderBy,
orderByList: orderByList,
transaction: transaction,
);
final rowIds = {for (final row in rows) row.id as UuidValue};
if (rowIds.isEmpty) {
return (rowIds: rowIds, projectionRows: const <MergeRowKey>{});
}
final plan = await _foreignKeyProjector.project(
transaction,
seedTables: {table.tableName},
seedRows: {for (final rowId in rowIds) (table.tableName, rowId)},
materialize: false,
);
return (rowIds: rowIds, projectionRows: plan.domain.keys.toSet());
}
/// Records CRDT field metadata for updated rows. /// Records CRDT field metadata for updated rows.
/// ///
/// [authoredColumnValues] marks a write that reached the domain without a /// [authoredColumnValues] marks a write that reached the domain without a
@ -863,6 +942,7 @@ class CrdtMutationRecorder {
Transaction transaction, { Transaction transaction, {
bool projectionUnchanged = false, bool projectionUnchanged = false,
bool authoredColumnValues = false, bool authoredColumnValues = false,
Set<MergeRowKey> projectionRows = const {},
Map<MergeRowKey, Map<String, Object?>> domainBeforeUpsert = const {}, Map<MergeRowKey, Map<String, Object?>> domainBeforeUpsert = const {},
List<TableRow> upsertRows = const [], List<TableRow> upsertRows = const [],
}) async { }) async {
@ -957,6 +1037,7 @@ class CrdtMutationRecorder {
updatedColumnNames, updatedColumnNames,
transaction, transaction,
authoredOverlays: authoredOverlays, authoredOverlays: authoredOverlays,
projectionRows: {...projectionRows, ...domainBeforeUpsert.keys},
); );
}); });
} }
@ -967,12 +1048,15 @@ class CrdtMutationRecorder {
Set<String>? columnNames, Set<String>? columnNames,
Transaction transaction, { Transaction transaction, {
Map<MergeFieldKey, Object?> authoredOverlays = const {}, Map<MergeFieldKey, Object?> authoredOverlays = const {},
Set<MergeRowKey> projectionRows = const {},
}) async { }) async {
if (_foreignKeyProjector.needsProjection(tableName, columnNames)) { if (_foreignKeyProjector.needsProjection(tableName, columnNames)) {
await _foreignKeyProjector.project( await _foreignKeyProjector.project(
transaction, transaction,
seedTables: {tableName}, seedTables: {tableName},
seedRows: {for (final rowId in rowIds) (tableName, rowId)}, // A physical local write can remove the old tuple from the seed row.
// Retain its pre-write closure so released claimants are reconsidered.
seedRows: {...projectionRows, for (final rowId in rowIds) (tableName, rowId)},
authoredOverlays: authoredOverlays, authoredOverlays: authoredOverlays,
); );
} }

View file

@ -2,6 +2,7 @@ import 'dart:async';
import 'dart:convert'; import 'dart:convert';
import 'package:clock/clock.dart'; import 'package:clock/clock.dart';
import 'package:meta/meta.dart';
import 'package:serverpod_database/serverpod_database.dart'; import 'package:serverpod_database/serverpod_database.dart';
import 'package:uuid/uuid.dart'; import 'package:uuid/uuid.dart';
@ -192,18 +193,27 @@ class OfflineSyncEngine {
include: OfflineSyncSpaceNode.include(node: CrdtNode.include()), include: OfflineSyncSpaceNode.include(node: CrdtNode.include()),
); );
final checkpoints = <UuidValue, Hlc>{
// Cover committed logical ticks as well as wall time. A clock that has
// not advanced (or moved backward) must not make peers echo our facts.
localNodeId: Hlc.now(localNodeId).maxBetween(space.currentNode!.lastHlc),
};
for (final spaceNode in spaceNodes) {
final nodeId = spaceNode.node!.uuidNodeId;
final received = spaceNode.lastReceivedHlc;
// Older inbound bookkeeping could store a relayed author's clock under
// its direct peer. It proves no per-author progress for that peer.
final checkpoint = received?.nodeId == nodeId ? received! : Hlc.zero(nodeId);
checkpoints[nodeId] = checkpoint.maxBetween(checkpoints[nodeId]);
}
return OfflineSyncSinceHlc( return OfflineSyncSinceHlc(
uuidSpaceId: spaceId, uuidSpaceId: spaceId,
nodeCheckpoints: [ nodeCheckpoints: checkpoints.values.toList(),
// The local node is always included to avoid collecting its own changes.
Hlc.now(localNodeId),
for (final spaceNode in spaceNodes)
spaceNode.lastReceivedHlc ?? Hlc.zero(spaceNode.node!.uuidNodeId),
],
); );
} }
/// Merges a remote [mergeSet] and records the sync checkpoint for [otherNodeId]. /// Merges a remote [mergeSet] and atomically records its per-author progress.
/// ///
/// Inbound merge applies each remote change, then materializes foreign-key /// Inbound merge applies each remote change, then materializes foreign-key
/// projection into domain tables via [OfflineSyncDatabase.mergeChanges]. /// projection into domain tables via [OfflineSyncDatabase.mergeChanges].
@ -213,23 +223,16 @@ class OfflineSyncEngine {
/// ///
/// Returns the greatest HLC synced in the batch, or `null` if the batch is /// Returns the greatest HLC synced in the batch, or `null` if the batch is
/// empty. /// empty.
Future<Hlc?> _mergeInboundBatch( @visibleForTesting
Future<Hlc?> mergeInboundBatch(
DatabaseSession session, { DatabaseSession session, {
required UuidValue spaceId, required UuidValue spaceId,
required UuidValue otherNodeId,
required CrdtMergeSet mergeSet, required CrdtMergeSet mergeSet,
}) async { }) async {
if (mergeSet.isEmpty) return null; if (mergeSet.isEmpty) return null;
final maxSyncedHlc = mergeSet.maxHlc; final maxSyncedHlc = mergeSet.maxHlc;
final offlineSyncDb = _openOfflineSyncDatabase(session); final offlineSyncDb = _openOfflineSyncDatabase(session);
await offlineSyncDb.mergeChanges(mergeSet, spaceId: spaceId); await offlineSyncDb.mergeChanges(mergeSet, spaceId: spaceId);
if (maxSyncedHlc != null) {
await offlineSyncDb.recordSyncCheckpoint(
otherNodeId,
maxSyncedHlc,
userId: spaceId,
);
}
return maxSyncedHlc; return maxSyncedHlc;
} }
@ -434,10 +437,9 @@ class OfflineSyncEngine {
changes: entry.value, changes: entry.value,
); );
} }
final receivedHlc = await _mergeInboundBatch( final receivedHlc = await mergeInboundBatch(
session, session,
spaceId: spaceId, spaceId: spaceId,
otherNodeId: spaces.peerNodeId,
mergeSet: entry.value, mergeSet: entry.value,
); );
await _reportMerge(onMergeSuccess, spaces, spaceId, receivedHlc); await _reportMerge(onMergeSuccess, spaces, spaceId, receivedHlc);

View file

@ -197,11 +197,14 @@ class OfflineSyncSpaceState {
/// Records the peer's resume vector for [spaceId] from its [sinceHlc], /// Records the peer's resume vector for [spaceId] from its [sinceHlc],
/// completing the space's handshake from this peer's side. /// completing the space's handshake from this peer's side.
void recordPeerHandshake(UuidValue spaceId, OfflineSyncSinceHlc sinceHlc) { void recordPeerHandshake(UuidValue spaceId, OfflineSyncSinceHlc sinceHlc) {
_checkpointsBySpace[spaceId] = { _checkpointsBySpace[spaceId] = normalizeCheckpoints(sinceHlc.nodeCheckpoints);
for (final checkpoint in sinceHlc.nodeCheckpoints) checkpoint.nodeId: checkpoint,
};
} }
/// Normalizes a wire resume vector exactly as a peer handshake does.
static Map<UuidValue, Hlc> normalizeCheckpoints(Iterable<Hlc> checkpoints) => {
for (final checkpoint in checkpoints) checkpoint.nodeId: checkpoint,
};
/// Checkpoints for writable spaces whose handshake completed both ways, keyed /// Checkpoints for writable spaces whose handshake completed both ways, keyed
/// by space — the input to a pending-change collection pass. /// by space — the input to a pending-change collection pass.
Map<UuidValue, List<Hlc>> get sendableCheckpoints => { Map<UuidValue, List<Hlc>> get sendableCheckpoints => {

View file

@ -119,6 +119,7 @@ melos:
# seeds from the clock, so it searches fresh schedules on every run and # seeds from the clock, so it searches fresh schedules on every run and
# can fail for a defect unrelated to the change that triggered it. # can fail for a defect unrelated to the change that triggered it.
# #
# Select full-history or checkpoint delivery with DST_DELIVERY=full|delta.
# Widen the search with DST_SEEDS and DST_ROUNDS; a failure prints the # Widen the search with DST_SEEDS and DST_ROUNDS; a failure prints the
# command that replays it. Concurrency is pinned to one so the reported # command that replays it. Concurrency is pinned to one so the reported
# failures cannot interleave, which matters when reading a wide sweep. # failures cannot interleave, which matters when reading a wide sweep.

View file

@ -16,6 +16,7 @@
# Results live in .dst-soak: # Results live in .dst-soak:
# FAILURES.md one section per failing run, with its replay command # FAILURES.md one section per failing run, with its replay command
# RUNS.tsv every run attempted, so coverage is auditable # RUNS.tsv every run attempted, so coverage is auditable
# METRICS.jsonl coverage records retained for passing and failing runs
# runs/<id>.log full output, kept for failures only # runs/<id>.log full output, kept for failures only
# tree.txt, tree.diff the exact code state being hunted # tree.txt, tree.diff the exact code state being hunted
@ -50,8 +51,16 @@ mkdir -p "$OUT/runs"
git -C "$REPO_ROOT" diff HEAD > "$OUT/tree.diff" git -C "$REPO_ROOT" diff HEAD > "$OUT/tree.diff"
[ -f "$OUT/RUNS.tsv" ] || [ -f "$OUT/RUNS.tsv" ] ||
printf 'started\tid\tprofile\tseed_base\tseeds\trounds\twidth\ttarget\tresult\tseconds\n' \ printf 'started\tid\tprofile\tseed_base\tseeds\trounds\twidth\ttarget\tresult\tseconds\tdelivery\n' \
> "$OUT/RUNS.tsv" > "$OUT/RUNS.tsv"
# Upgrade existing ledgers without reinterpreting old runs: before delivery
# modes existed, every recorded run used full-history delivery.
if [ "$(head -n 1 "$OUT/RUNS.tsv" | awk -F'\t' '{print NF}')" -eq 10 ]; then
ledger_tmp=$(mktemp)
awk -F'\t' 'BEGIN { OFS = FS } { print $0, (NR == 1 ? "delivery" : "full") }' \
"$OUT/RUNS.tsv" > "$ledger_tmp" && mv "$ledger_tmp" "$OUT/RUNS.tsv"
fi
[ -f "$OUT/FAILURES.md" ] || { [ -f "$OUT/FAILURES.md" ] || {
echo "# Deterministic simulation failures" echo "# Deterministic simulation failures"
echo echo
@ -64,14 +73,16 @@ git -C "$REPO_ROOT" diff HEAD > "$OUT/tree.diff"
# how deep a schedule gets before the run ends. A single fixed shape would keep # how deep a schedule gets before the run ends. A single fixed shape would keep
# re-searching the same corner of the space all night. # re-searching the same corner of the space all night.
# #
# profile:seeds:rounds:width:target # profile:seeds:rounds:width:target:delivery
CONFIGS=( CONFIGS=(
"sparse:8:40:2:" "sparse:8:40:2::full"
"mixed:6:120:2:" "sparse:8:40:2::delta"
"mixed:4:200:2:test/dst/dst_cross_space_test.dart" "mixed:4:200:2:test/dst/dst_cross_space_test.dart:full"
"populated:3:200:3:" "mixed:4:200:2:test/dst/dst_cross_space_test.dart:delta"
"sparse:12:40:3:test/dst/dst_convergence_test.dart" "populated:3:200:3::full"
"mixed:8:80:2:" "populated:3:200:3::delta"
"mixed:6:120:2:test/dst/dst_convergence_test.dart:full"
"mixed:6:120:2:test/dst/dst_convergence_test.dart:delta"
) )
# Space-separated entries in the same shape, to steer or test the hunt. # Space-separated entries in the same shape, to steer or test the hunt.
[ -z "${DST_SOAK_CONFIGS:-}" ] || read -r -a CONFIGS <<< "$DST_SOAK_CONFIGS" [ -z "${DST_SOAK_CONFIGS:-}" ] || read -r -a CONFIGS <<< "$DST_SOAK_CONFIGS"
@ -87,24 +98,29 @@ index=0
suspicious=0 suspicious=0
while [ "$(date +%s)" -lt "$UNTIL" ]; do while [ "$(date +%s)" -lt "$UNTIL" ]; do
IFS=':' read -r profile seeds rounds width target <<< "${CONFIGS[index % ${#CONFIGS[@]}]}" IFS=':' read -r profile seeds rounds width target delivery <<< "${CONFIGS[index % ${#CONFIGS[@]}]}"
delivery=${delivery:-${DST_DELIVERY:-full}}
case "$delivery" in
full|delta) ;;
*) echo "invalid delivery mode: $delivery" >&2; exit 2 ;;
esac
index=$((index + 1)) index=$((index + 1))
# The index keeps two runs that start in the same second from sharing a log. # The index keeps two runs that start in the same second from sharing a log.
id="$(date +%Y%m%d-%H%M%S)-$index-$profile-$rounds" id="$(date +%Y%m%d-%H%M%S)-$index-$profile-$rounds-$delivery"
log="$OUT/runs/$id.log" log="$OUT/runs/$id.log"
seed_base=$cursor seed_base=$cursor
cursor=$((cursor + seeds)) cursor=$((cursor + seeds))
replay="DST_SEED_BASE=$seed_base DST_SEEDS=$seeds DST_ROUNDS=$rounds" replay="DST_SEED_BASE=$seed_base DST_SEEDS=$seeds DST_ROUNDS=$rounds"
replay="$replay DST_PROFILE=$profile DST_GRAPH_WIDTH=$width" replay="$replay DST_PROFILE=$profile DST_GRAPH_WIDTH=$width DST_DELIVERY=$delivery"
replay="$replay dart test -P dst --concurrency=1 ${target:+$target}" replay="$replay dart test -P dst --concurrency=1 ${target:+$target}"
# The command is written down before the run, so a kill -9 mid-run still # The command is written down before the run, so a kill -9 mid-run still
# leaves the seed behind. # leaves the seed behind.
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
"$(date -Is)" "$id" "$profile" "$seed_base" "$seeds" "$rounds" "$width" \ "$(date -Is)" "$id" "$profile" "$seed_base" "$seeds" "$rounds" "$width" \
"${target:-all}" "started" "" >> "$OUT/RUNS.tsv" "${target:-all}" "started" "" "$delivery" >> "$OUT/RUNS.tsv"
started=$(date +%s) started=$(date +%s)
( (
@ -114,6 +130,7 @@ while [ "$(date +%s)" -lt "$UNTIL" ]; do
DST_ROUNDS=$rounds \ DST_ROUNDS=$rounds \
DST_PROFILE=$profile \ DST_PROFILE=$profile \
DST_GRAPH_WIDTH=$width \ DST_GRAPH_WIDTH=$width \
DST_DELIVERY=$delivery \
timeout "$RUN_TIMEOUT" dart test -P dst --concurrency=1 \ timeout "$RUN_TIMEOUT" dart test -P dst --concurrency=1 \
--reporter=expanded ${target:+"$target"} --reporter=expanded ${target:+"$target"}
) > "$log" 2>&1 ) > "$log" 2>&1
@ -128,9 +145,12 @@ while [ "$(date +%s)" -lt "$UNTIL" ]; do
exit 4 exit 4
fi fi
# Preserve actual coverage before deleting bulky passing-run output.
sed -n "s/^.*DST_METRICS {/\{\"runId\":\"$id\",/p" "$log" >> "$OUT/METRICS.jsonl"
if [ $status -eq 0 ]; then if [ $status -eq 0 ]; then
# A passing run proves the seeds were searched; its output proves nothing, # Metrics above retain coverage evidence; keep bulky per-test output only
# and a night of them fills the disk. # for failures so an overnight hunt does not fill the disk.
rm -f "$log" rm -f "$log"
result="passed" result="passed"
suspicious=0 suspicious=0
@ -157,7 +177,7 @@ while [ "$(date +%s)" -lt "$UNTIL" ]; do
echo echo
# The suite prints its own single-seed replay line for each failure. It is # The suite prints its own single-seed replay line for each failure. It is
# narrower than the sweep command above, so it is kept when present. # narrower than the sweep command above, so it is kept when present.
sed -n 's/.*\(Replay: DST_SEED_BASE=[^ ]* DST_SEEDS=[^ ]* DST_ROUNDS=[^ ]* DST_PROFILE=[^ ]* DST_GRAPH_WIDTH=[0-9]*\).*/- \1/p' \ sed -n 's/.*\(Replay: DST_SEED_BASE=[^ ]* DST_SEEDS=[^ ]* DST_ROUNDS=[^ ]* DST_PROFILE=[^ ]* DST_GRAPH_WIDTH=[0-9]* DST_DELIVERY=[^ ]*\).*/- \1/p' \
"$log" | sort -u "$log" | sort -u
sed -n 's/^\s*\(test\/dst\/[a-z_]*\.dart:.*\)$/- \1/p' "$log" | sort -u | head -20 sed -n 's/^\s*\(test\/dst\/[a-z_]*\.dart:.*\)$/- \1/p' "$log" | sort -u | head -20
echo echo

View file

@ -2,7 +2,7 @@
These suites drive several replicas through randomized operations and These suites drive several replicas through randomized operations and
adversarial delivery, then check properties the engine must hold. A run is adversarial delivery, then check properties the engine must hold. A run is
described by its seed, rounds, workload profile, and graph width. Those inputs described by its seed, rounds, workload profile, graph width, and delivery mode. Those inputs
replay exactly within the same source/SDK revision. replay exactly within the same source/SDK revision.
They complement the `integration/` suites rather than replacing them. They complement the `integration/` suites rather than replacing them.
@ -32,6 +32,7 @@ directory are untagged and run with ordinary `dart test`.
| `DST_SEED_BASE` | unix seconds | First seed; successive seeds increment | | `DST_SEED_BASE` | unix seconds | First seed; successive seeds increment |
| `DST_PROFILE` | sparse | `sparse`, `populated`, or `mixed` (even seeds populated, odd sparse) | | `DST_PROFILE` | sparse | `sparse`, `populated`, or `mixed` (even seeds populated, odd sparse) |
| `DST_GRAPH_WIDTH` | 2 | Rows per table in each populated graph, minimum 2 | | `DST_GRAPH_WIDTH` | 2 | Rows per table in each populated graph, minimum 2 |
| `DST_DELIVERY` | full | `full` exports all retained facts; `delta` uses the receiver's committed checkpoint vector |
| `DST_DEBUG_TABLE` | unset | Print every merge touching this table, in delivery order | | `DST_DEBUG_TABLE` | unset | Print every merge touching this table, in delivery order |
A divergence is usually explained by which facts a replica had merged when it A divergence is usually explained by which facts a replica had merged when it
@ -47,7 +48,7 @@ which defects those fixed seeds reach.
Nothing is lost to reproducibility: a seed determines its simulation entirely Nothing is lost to reproducibility: a seed determines its simulation entirely
and a failure prints its replay command. Tests are named by position rather and a failure prints its replay command. Tests are named by position rather
than by seed, so the suite stays stable while the schedules underneath it vary. than by seed, so the suite stays stable while the schedules underneath it vary.
Pin the seed, rounds, profile, and graph width to re-run an exact sweep. Pin the seed, rounds, profile, graph width, and delivery mode to re-run an exact sweep.
The consequence to expect is that a run can fail for a defect unrelated to the The consequence to expect is that a run can fail for a defect unrelated to the
change that triggered it. That is the suite doing its job; take the seed from change that triggered it. That is the suite doing its job; take the seed from
@ -102,27 +103,39 @@ Reorders, delays, redelivers already-merged batches, and isolates incoming
delivery to selected replicas for a few rounds. Sources can still send during delivery to selected replicas for a few rounds. Sources can still send during
this receive isolation; this is not a bidirectional network partition. this receive isolation; this is not a bidirectional network partition.
Two moves are deliberately unavailable, both because the merge contract states `full` preserves the original merge convergence experiment: collection exports
that input arrives as a causally complete snapshot of the sender all retained facts, and a receiver's already-delivered keys decide whether another
(`docs/foreign-key-invariants.md`): complete export is worth scheduling. `delta` reads each receiver's production
`createSyncSinceHlc` vector under its seeded clock and passes it to the sender's
`collectPendingChanges`. Vectors are scoped by receiver, space, and author.
- **It never drops permanently.** Dropping manufactures failures outside the Causal completeness is relative to the receiver: prerequisites must already be
contract instead of finding real ones. committed there or accompany the incoming fact. Each delta is collected against
- **It never splits a collected batch.** An earlier version of this harness committed receiver progress, never a queued batch's anticipated progress. Pending
split batches at a random pivot to vary framing. That delivers, for example, a batches can overlap and arrive in any order without relying on one another.
delete without its insert; the engine ignores the orphaned change, the harness Collection and merge run sequentially; this does not simulate concurrent writes
records it as delivered, and the fact is lost - surfacing as a bogus during collection or the continuous sync driver's in-session send cursor.
convergence failure. Chunking in the real protocol sits *below* the merge
(`chunked()` emits frames that `collectNextBatch` reassembles until
`OfflineSyncEndOfBatch`), so the whole cycle is the causal unit and splitting here
models nothing real.
Delay, reorder, and redelivery are the honest moves - and redelivery is how Neither mode permanently drops facts or splits a collected batch. Wire chunking
idempotence gets probed. belongs below this boundary: the protocol reassembles chunks through
`OfflineSyncEndOfBatch` before merging. Delta replay selects a previously merged
batch from the same source/target/space edge. Captured payloads are serialized and
decoded afresh for each delivery, so later local writes or input mutation cannot
change the replay. Full mode retains its original complete-export resends.
When the schedule ends, quiescence drains every pending batch and collects only Quiescence clears receive isolation, drains pending batches, and stops scheduling
facts a receiver has not yet seen. It stops introducing deliberate duplicates, deliberate replays. In delta mode it requires empty checkpoint-based collections
so stable replicas cannot exhaust the convergence limit merely through resends. on every directed peer/space exchange. There is no full-history repair before
checking convergence or accepted authored facts. The existing export-to-empty
mirror check runs afterward and never repairs the simulated replicas.
```sh
DST_DELIVERY=delta DST_SEED_BASE=114 DST_SEEDS=1 DST_ROUNDS=200 DST_PROFILE=populated dart test -P dst
```
The modes need not author identical operations for an identical seed: delivery
changes visible state and subsequent generator choices. Fixed collection
regressions compare full and delta receivers of the same authored scenario.
## Layout ## Layout
@ -197,16 +210,34 @@ The populated profile creates every declared table and FK edge, closes the
person/company/town cycles, and drives unique conflicts, a real tuple exchange, person/company/town cycles, and drives unique conflicts, a real tuple exchange,
restore/redelete, FK retarget/detach, and a constrained refusal. These are actual restore/redelete, FK retarget/detach, and a constrained refusal. These are actual
ORM transactions, each checked immediately for structure, authored preservation, ORM transactions, each checked immediately for structure, authored preservation,
and causal monotonicity. Complete exports distribute the populated spaces before and causal monotonicity. The selected delivery mode distributes the populated
random scheduling starts. The sparse profile retains empty-world exploration. spaces before random scheduling starts; bootstrap naturally includes previously
unknown facts. The sparse profile retains empty-world exploration.
Every run emits one `DST_METRICS` JSON record, including failed runs. It separates Every run emits one `DST_METRICS` JSON record, including failed runs. Its
`topology` identifies `convergence` or `cross_space`. It separates
setup attempts from scheduled commits and records attempted, committed, rejected, setup attempts from scheduled commits and records attempted, committed, rejected,
skipped, unexpected failures, and committed transactions whose oracle failed. skipped, unexpected failures, and committed transactions whose oracle failed.
A run failing during setup reports zero scheduled activity. Paths retain table A run failing during setup reports zero scheduled activity. Paths retain table
and action; network observations include merge counts, duplicate batches, maximum and action; network observations include merge counts, duplicate batches, maximum
batch size, total delivered changes, and receive-isolation events. batch size, total delivered changes, and receive-isolation events.
Network observations also report checkpoint collections, collections with a
nonzero remote-author checkpoint, empty collections, partial batches, explicit
replays, and fresh/repeated delivered change keys. A partial batch is nonempty,
uses committed remote progress, and is smaller than the unchanged sender's full
export. Full exports in delta mode measure omissions only and are never queued.
Fresh means the key was not previously delivered to this receiver; it does not
claim the fact was absent from local authoring. A repeated key has the same
identity, not necessarily identical serialized insertion payload.
Counters carry `setup.`, `scheduled.`, or `drain.` prefixes according to when the
collection or delivery occurred. Exact replays do not count as new partial
batches. Delta stress runs (100+ rounds) require at least one scheduled partial
batch; setup and final draining cannot satisfy that gate. Shorter runs remain
smoke checks. The soak script retains `METRICS.jsonl` even for passing runs,
adding a `runId` that joins each record to the run's `id` in `RUNS.tsv`.
Coverage counts distinct field/tombstone events rather than repeated snapshot Coverage counts distinct field/tombstone events rather than repeated snapshot
appearances. FK edges and cycles are **authored graph** observations. Unique appearances. FK edges and cycles are **authored graph** observations. Unique
projection coverage reports columns, not a claim that every overlapping index projection coverage reports columns, not a claim that every overlapping index
@ -219,13 +250,12 @@ also require all declared authored FK edges and the mandatory semantic paths.
Setup commits alone cannot satisfy the scheduled-activity gate. Setup commits alone cannot satisfy the scheduled-activity gate.
CI alternates sparse and populated profiles across four consecutive seeds at CI alternates sparse and populated profiles across four consecutive seeds at
200 rounds in both topologies: 4,800 scheduled operation attempts, versus the 200 rounds in both topologies and both delivery modes: four separate jobs, each
previous 6,000 attempts spread across fifty shallow 20-round worlds. This keeps with 2,400 scheduled operation attempts and four ten-minute per-seed ceilings
comparable attempt volume while exploring ten times the history depth and within a sixty-minute job. A failure in one job does not cancel another.
connected graphs. Each topology runs in its own sixty-minute CI job with four All jobs also run the ownership-collision controls. The local `melos test-dst`
ten-minute per-seed ceilings; a failure in one job does not cancel the other. command runs both topologies for the selected `DST_DELIVERY` mode; run it once
Both jobs also run the ownership-collision controls. The local `melos test-dst` with `full` and once with `delta` to cover both modes.
command runs both topologies together, so it covers the combined CI matrix.
Known engine failures remain failures; CI is intentionally not a PR merge gate. Known engine failures remain failures; CI is intentionally not a PR merge gate.
```sh ```sh
@ -234,5 +264,8 @@ DST_SEED_BASE=62 DST_SEEDS=1 DST_ROUNDS=200 DST_PROFILE=populated DST_GRAPH_WIDT
``` ```
The oracle remains bounded: it does not independently arbitrate all unique The oracle remains bounded: it does not independently arbitrate all unique
winners or FK fixed points. Collector concurrency/checkpoint lifecycle, crash winners or FK fixed points. Concurrent collection, continuous-session send cursors, crash recovery, space
recovery, space grant/revoke, and transport framing remain outside this schedule. grant/revoke, and transport framing remain outside this schedule. Delta mode
exercises production resume-vector creation, peer-handshake vector normalization,
checkpoint filtering, and the inbound merge primitive that atomically persists
per-author received progress; it does not simulate the full sync driver.

View file

@ -1,3 +1,4 @@
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart'; import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
import 'package:test/test.dart'; import 'package:test/test.dart';
@ -56,6 +57,195 @@ void main() {
} }
}, },
); );
group('Given a delta receiver isolated while a parent and child are authored,', () {
final ids = DstIds(DstRandom(917));
final space = ids.next();
final clock = DstClock();
late DstReplica source;
late DstReplica target;
late DstAdversary adversary;
late City parent;
late List<String> initialCheckpoints;
setUpAll(() async {
source = await DstReplica.create(
name: 'source',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
target = await DstReplica.create(
name: 'target',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
initialCheckpoints = (await target.checkpoints(
space,
)).map((hlc) => hlc.toString()).toList();
adversary = DstAdversary(_DelayedNewestFirstSchedule(), [
source,
target,
], delivery: DstDeliveryMode.delta);
parent = City(id: ids.next(), name: 'parent');
});
group('when complete batches queue and the newest one arrives first,', () {
late List<String> queuedCheckpoints;
late String expected;
final observed = <String>[];
late CrdtMergeSet caughtUp;
setUpAll(() async {
Future<void> observe(DstReplica replica) async {
if (identical(replica, target)) {
observed.add((await DstSnapshot.capture(replica)).renderSpace(space));
}
}
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(source.session, parent, transaction: tx);
}),
);
await adversary.step(observe);
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await Town.db.insertRow(
source.session,
Town(id: ids.next(), name: 'child', cityId: parent.id),
transaction: tx,
);
}),
);
await adversary.step(observe);
queuedCheckpoints = (await target.checkpoints(
space,
)).map((hlc) => hlc.toString()).toList();
expected = (await DstSnapshot.capture(source)).renderSpace(space);
adversary.phase = DstNetworkPhase.drain;
await adversary.quiesce(observe);
caughtUp = await source.collect(
space,
checkpoints: await target.checkpoints(space),
);
});
test(
'then queuing advances no checkpoint and every delivered snapshot retains the child.',
() {
expect(queuedCheckpoints, initialCheckpoints);
expect(observed, hasLength(greaterThanOrEqualTo(3)));
expect(observed, everyElement(expected));
expect(caughtUp, isEmpty);
expect(adversary.metrics['scheduled.merges'], isNull);
expect(adversary.metrics['drain.merges'], greaterThanOrEqualTo(3));
},
);
});
});
group(
'Given a captured insertion batch and a delta adversary that requests replays,',
() {
final ids = DstIds(DstRandom(918));
final space = ids.next();
final clock = DstClock();
late DstReplica source;
late DstReplica target;
late City row;
late DstDelivery captured;
late CrdtMergeSet original;
late DstAdversary adversary;
setUpAll(() async {
source = await DstReplica.create(
name: 'source',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
target = await DstReplica.create(
name: 'target',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
row = City(id: ids.next(), name: 'original');
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(source.session, row, transaction: tx);
}),
);
original = await source.collect(space);
captured = DstDelivery(
source: source,
target: target,
spaceUuid: space,
changes: original,
);
adversary = DstAdversary(_RepeatedDeliverySchedule(), [
source,
target,
], delivery: DstDeliveryMode.delta)..phase = DstNetworkPhase.setup;
await adversary.quiesce((_) async {});
});
group('when the source changes and repeated partial deliveries drain,', () {
late String actual;
late String expected;
late String frozenName;
late String mutatedName;
late int partialBeforeDrain;
setUpAll(() async {
(original.single as CrdtMergeInsert).data = row.copyWith(
name: 'mutated input',
);
final decoded = captured.changes.single as CrdtMergeInsert
..data = row.copyWith(name: 'mutated decoded copy');
mutatedName = decoded.databaseColumns['name']! as String;
frozenName =
(captured.changes.single as CrdtMergeInsert).databaseColumns['name']!
as String;
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await City.db.updateRow(
source.session,
row.copyWith(name: 'changed'),
columns: (t) => [t.name],
transaction: tx,
);
}),
);
adversary.phase = DstNetworkPhase.scheduled;
await adversary.step((_) async {});
await adversary.step((_) async {});
await adversary.step((_) async {});
partialBeforeDrain = adversary.scheduledPartialBatches;
adversary.phase = DstNetworkPhase.drain;
await adversary.quiesce((_) async {});
actual = (await DstSnapshot.capture(target)).renderSpace(space);
expected = (await DstSnapshot.capture(source)).renderSpace(space);
});
test(
'then the captured payload is immutable and scheduled partial merges survive exact replays.',
() {
expect(mutatedName, 'mutated decoded copy');
expect(frozenName, 'original');
expect(actual, expected);
expect(partialBeforeDrain, greaterThan(0));
expect(adversary.scheduledPartialBatches, partialBeforeDrain);
expect(adversary.metrics['explicitReplays'], greaterThan(0));
expect(adversary.metrics['scheduled.emptyCollections'], greaterThan(0));
expect(adversary.metrics['setup.partialBatches'], isNull);
},
);
});
},
);
} }
/// A concrete network schedule: always collect the first replica, always /// A concrete network schedule: always collect the first replica, always
@ -73,3 +263,32 @@ class _RepeatedDeliverySchedule extends DstRandom {
@override @override
T pick<T>(List<T> items) => items.first; T pick<T>(List<T> items) => items.first;
} }
/// Isolate the target at each step, collect from the source, and drain newest
/// batches first. Every queued collection must use committed receiver progress.
class _DelayedNewestFirstSchedule extends DstRandom {
_DelayedNewestFirstSchedule() : super(917);
bool _selectingPartition = false;
@override
bool chance(double probability) {
if (probability == DstAdversary.receiveIsolationProbability) {
_selectingPartition = true;
return true;
}
return probability == DstAdversary.collectionProbability;
}
@override
int between(int min, int max) => max;
@override
T pick<T>(List<T> items) {
if (_selectingPartition) {
_selectingPartition = false;
return items.last;
}
return items.first is DstDelivery ? items.last : items.first;
}
}

View file

@ -32,11 +32,13 @@ void main() {
rounds: config.rounds, rounds: config.rounds,
profile: config.profile, profile: config.profile,
graphWidth: config.graphWidth, graphWidth: config.graphWidth,
delivery: config.delivery,
run: () => runDstSimulation( run: () => runDstSimulation(
seed: seed, seed: seed,
rounds: config.rounds, rounds: config.rounds,
profile: config.profile, profile: config.profile,
graphWidth: config.graphWidth, graphWidth: config.graphWidth,
delivery: config.delivery,
topology: DstTopology.singleSpace, topology: DstTopology.singleSpace,
), ),
); );

View file

@ -38,11 +38,13 @@ void main() {
rounds: config.rounds, rounds: config.rounds,
profile: config.profile, profile: config.profile,
graphWidth: config.graphWidth, graphWidth: config.graphWidth,
delivery: config.delivery,
run: () => runDstSimulation( run: () => runDstSimulation(
seed: seed, seed: seed,
rounds: config.rounds, rounds: config.rounds,
profile: config.profile, profile: config.profile,
graphWidth: config.graphWidth, graphWidth: config.graphWidth,
delivery: config.delivery,
topology: DstTopology.overlappingSpaces, topology: DstTopology.overlappingSpaces,
), ),
); );

View file

@ -0,0 +1,490 @@
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
import 'package:test/test.dart';
import '../integration/test_tools/client_session.dart';
import 'framework/dst_random.dart';
import 'framework/dst_snapshot.dart';
import 'framework/dst_world.dart';
void main() {
initTestClientSession(createSessionPerTest: false);
group('Given a city already acknowledged by a receiver,', () {
final ids = DstIds(DstRandom(420));
final clock = DstClock();
final space = ids.next();
late DstReplica source;
late DstReplica target;
late DstReplica reference;
late City city;
setUpAll(() async {
source = await _replica('source', [space], ids, clock);
target = await _replica('target', [space], ids, clock);
reference = await _replica('reference', [space], ids, clock);
city = City(id: ids.next(), name: 'initial');
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(source.session, city, transaction: tx);
}),
);
await target.merge(await source.collect(space), space);
await reference.merge(await source.collect(space), space);
});
group('when its name changes and later the city is deleted,', () {
late CrdtMergeSet update;
late CrdtMergeSet deletion;
late CrdtMergeSet caughtUp;
late String updatedTarget;
late String updatedReference;
late String deletedTarget;
late String deletedReference;
late String replayedTarget;
setUpAll(() async {
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await City.db.updateRow(
source.session,
city.copyWith(name: 'changed'),
columns: (t) => [t.name],
transaction: tx,
);
}),
);
update = await source.collect(
space,
checkpoints: await target.checkpoints(space),
);
await target.merge(update, space);
await reference.merge(await source.collect(space), space);
updatedTarget = (await DstSnapshot.capture(target)).renderSpace(space);
updatedReference = (await DstSnapshot.capture(reference)).renderSpace(space);
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await City.db.deleteRow(source.session, city, transaction: tx);
}),
);
deletion = await source.collect(
space,
checkpoints: await target.checkpoints(space),
);
await target.merge(deletion, space);
await reference.merge(await source.collect(space), space);
deletedTarget = (await DstSnapshot.capture(target)).renderSpace(space);
deletedReference = (await DstSnapshot.capture(reference)).renderSpace(space);
await target.merge(update, space);
await target.merge(deletion, space);
replayedTarget = (await DstSnapshot.capture(target)).renderSpace(space);
caughtUp = await source.collect(
space,
checkpoints: await target.checkpoints(space),
);
});
test('then the update batch omits the acknowledged insertion.', () {
expect(update, hasLength(1));
expect(update.single, isA<CrdtMergeUpdate>());
expect((update.single as CrdtMergeUpdate).value, 'changed');
expect(updatedTarget, updatedReference);
});
test('then the deletion batch contains only the new tombstone.', () {
expect(deletion, hasLength(1));
expect(deletion.single, isA<CrdtMergeDelete>());
expect(deletedTarget, deletedReference);
expect(deletedTarget, contains('HIDDEN'));
});
test('then replay changes neither the final state nor the empty catch-up.', () {
expect(replayedTarget, deletedTarget);
expect(caughtUp, isEmpty);
});
});
});
group('Given one author writing cities in two spaces,', () {
final ids = DstIds(DstRandom(421));
final clock = DstClock();
final firstSpace = ids.next();
final secondSpace = ids.next();
late DstReplica source;
late DstReplica target;
setUpAll(() async {
source = await _replica('source', [firstSpace, secondSpace], ids, clock);
target = await _replica('target', [firstSpace, secondSpace], ids, clock);
for (final space in [firstSpace, secondSpace]) {
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(
source.session,
City(id: ids.next(), name: space.toString()),
transaction: tx,
);
}),
);
}
});
group('when only the second space is acknowledged,', () {
late CrdtMergeSet firstPending;
late CrdtMergeSet secondPending;
setUpAll(() async {
await target.merge(await source.collect(secondSpace), secondSpace);
firstPending = await source.collect(
firstSpace,
checkpoints: await target.checkpoints(firstSpace),
);
secondPending = await source.collect(
secondSpace,
checkpoints: await target.checkpoints(secondSpace),
);
});
test('then the earlier first-space insertion is still pending.', () {
expect(firstPending, hasLength(1));
expect(firstPending.single.uuidSpaceId, firstSpace);
expect(secondPending, isEmpty);
});
});
});
group('Given a relay with a newer fact from one author,', () {
final ids = DstIds(DstRandom(422));
final clock = DstClock();
final space = ids.next();
late DstReplica older;
late DstReplica newer;
late DstReplica relay;
late DstReplica target;
late UuidValue olderRow;
setUpAll(() async {
older = await _replica('older', [space], ids, clock);
newer = await _replica('newer', [space], ids, clock);
relay = await _replica('relay', [space], ids, clock);
target = await _replica('target', [space], ids, clock);
olderRow = ids.next();
await older.withReplicaClock(
() => older.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(
older.session,
City(id: olderRow, name: 'older offline fact'),
transaction: tx,
);
}),
);
clock.advance(const Duration(seconds: 1));
await newer.withReplicaClock(
() => newer.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(
newer.session,
City(id: ids.next(), name: 'newer fact'),
transaction: tx,
);
}),
);
await relay.merge(await newer.collect(space), space);
await target.merge(await relay.collect(space), space);
});
group('when the older author reconnects through the relay,', () {
late CrdtMergeSet pending;
late String actual;
late String expected;
setUpAll(() async {
await relay.merge(await older.collect(space), space);
pending = await relay.collect(
space,
checkpoints: await target.checkpoints(space),
);
await target.merge(pending, space);
actual = (await DstSnapshot.capture(target)).renderSpace(space);
expected = (await DstSnapshot.capture(relay)).renderSpace(space);
});
test(
'then the older fact crosses both hops without resending the newer fact.',
() {
expect(pending, hasLength(1));
expect(pending.single.uuidRowId, olderRow);
expect(pending.single.uuidNodeId, older.nodeUuid);
expect(actual, expected);
},
);
});
});
group('Given a receiver that already holds a parent city,', () {
final ids = DstIds(DstRandom(423));
final clock = DstClock();
final space = ids.next();
late DstReplica source;
late DstReplica target;
late City parent;
setUpAll(() async {
source = await _replica('source', [space], ids, clock);
target = await _replica('target', [space], ids, clock);
parent = City(id: ids.next(), name: 'parent');
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(source.session, parent, transaction: tx);
}),
);
await target.merge(await source.collect(space), space);
});
group(
'when a child arrives alone followed by a new parent and child together,',
() {
late CrdtMergeSet childOnly;
late CrdtMergeSet together;
late DstSnapshot actual;
late DstSnapshot expected;
setUpAll(() async {
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await Town.db.insertRow(
source.session,
Town(id: ids.next(), name: 'first child', cityId: parent.id),
transaction: tx,
);
}),
);
childOnly = await source.collect(
space,
checkpoints: await target.checkpoints(space),
);
await target.merge(childOnly, space);
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
final nextParent = await City.db.insertRow(
source.session,
City(id: ids.next(), name: 'second parent'),
transaction: tx,
);
await Town.db.insertRow(
source.session,
Town(id: ids.next(), name: 'second child', cityId: nextParent.id),
transaction: tx,
);
}),
);
together = await source.collect(
space,
checkpoints: await target.checkpoints(space),
);
await target.merge(together, space);
actual = await DstSnapshot.capture(target);
expected = await DstSnapshot.capture(source);
});
test('then each partial batch retains exactly the needed new facts.', () {
expect(childOnly, hasLength(1));
expect(childOnly.single.tableName, 'town');
expect(together, hasLength(2));
expect(
together.map((change) => change.tableName),
unorderedEquals(['city', 'town']),
);
expect(actual.renderSpace(space), expected.renderSpace(space));
expect(DstOracle.invariants(actual), isEmpty);
});
},
);
});
group('Given two disconnected replicas claiming the same unique name,', () {
final ids = DstIds(DstRandom(424));
final clock = DstClock();
final space = ids.next();
late DstReplica source;
late DstReplica other;
late DstReplica delta;
late DstReplica full;
late Unique row;
setUpAll(() async {
source = await _replica('source', [space], ids, clock);
other = await _replica('other', [space], ids, clock);
delta = await _replica('delta', [space], ids, clock);
full = await _replica('full', [space], ids, clock);
row = Unique(id: ids.next(), name: 'claim');
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await Unique.db.insertRow(source.session, row, transaction: tx);
}),
);
clock.advance(const Duration(milliseconds: 1));
await other.withReplicaClock(
() => other.session.db.transactionForUser(space, (tx) async {
await Unique.db.insertRow(
other.session,
Unique(id: ids.next(), name: 'claim'),
transaction: tx,
);
}),
);
for (final receiver in [delta, full]) {
await receiver.merge(await source.collect(space), space);
await receiver.merge(await other.collect(space), space);
}
});
group('when one claim is deleted, restored, and deleted again through deltas,', () {
final batches = <CrdtMergeSet>[];
final actual = <String>[];
final expected = <String>[];
late DstSnapshot finalSnapshot;
setUpAll(() async {
// Each transition is a real local transaction with a complete collection
// boundary; only the delta receiver omits already acknowledged facts.
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await Unique.db.deleteRow(source.session, row, transaction: tx);
}),
);
Future<void> deliver() async {
final batch = await source.collect(
space,
checkpoints: await delta.checkpoints(space),
);
batches.add(batch);
await delta.merge(batch, space);
await full.merge(await source.collect(space), space);
actual.add((await DstSnapshot.capture(delta)).renderSpace(space));
expected.add((await DstSnapshot.capture(full)).renderSpace(space));
}
await deliver();
final hidden = (await DstTable.unique.model.find(
source.session,
includeHidden: true,
spaceUuid: space,
)).single;
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await source.session.db.insertRow(hidden, transaction: tx);
}),
);
await deliver();
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await Unique.db.deleteRow(source.session, row, transaction: tx);
}),
);
await deliver();
finalSnapshot = await DstSnapshot.capture(delta);
});
test('then each projection agrees with the full-history receiver.', () {
expect(batches, hasLength(3));
expect(batches.first.single, isA<CrdtMergeDelete>());
expect(batches.last.single, isA<CrdtMergeDelete>());
expect(actual, expected);
expect(finalSnapshot.tombstones['unique/${row.id}']!.clFlag, 4);
expect(DstOracle.invariants(finalSnapshot), isEmpty);
});
});
});
group('Given a batch that collides with a row owned by another space,', () {
final ids = DstIds(DstRandom(425));
final clock = DstClock();
final owner = ids.next();
final incoming = ids.next();
final collisionId = ids.next();
final innocentId = ids.next();
late DstReplica source;
late DstReplica target;
late List<String> before;
late CrdtMergeSet batch;
setUpAll(() async {
source = await _replica('source', [incoming], ids, clock);
target = await _replica('target', [owner, incoming], ids, clock);
await target.withReplicaClock(
() => target.session.db.transactionForUser(owner, (tx) async {
await City.db.insertRow(
target.session,
City(id: collisionId, name: 'owner'),
transaction: tx,
);
}),
);
await source.withReplicaClock(
() => source.session.db.transactionForUser(incoming, (tx) async {
await City.db.insertRow(
source.session,
City(id: innocentId, name: 'must roll back'),
transaction: tx,
);
await City.db.insertRow(
source.session,
City(id: collisionId, name: 'collision'),
transaction: tx,
);
}),
);
final checkpoints = await target.checkpoints(incoming);
before = checkpoints.map((hlc) => hlc.toString()).toList();
batch = await source.collect(incoming, checkpoints: checkpoints);
});
group('when the receiver rejects the complete batch,', () {
late Object failure;
late List<String> after;
late CrdtMergeSet retry;
late City? rolledBack;
setUpAll(() async {
failure = await target
.merge(batch, incoming)
.then<Object>(
(_) => 'unexpected success',
onError: (Object error) => error,
);
after = (await target.checkpoints(
incoming,
)).map((hlc) => hlc.toString()).toList();
retry = await source.collect(
incoming,
checkpoints: await target.checkpoints(incoming),
);
rolledBack = await City.db.findById(target.session, innocentId);
});
test(
'then no received progress or partial row survives and the entire batch remains pending.',
() {
expect(batch, hasLength(2));
expect(failure, isA<OfflineSyncIntegrityViolationException>());
expect(after, before);
expect(rolledBack, isNull);
expect(retry.map(dstChangeKey), batch.map(dstChangeKey));
},
);
});
});
}
Future<DstReplica> _replica(
String name,
List<UuidValue> spaces,
DstIds ids,
DstClock clock,
) => DstReplica.create(
name: name,
spaceUuids: spaces,
nodeUuid: ids.next(),
clock: clock.clock,
);

View file

@ -0,0 +1,297 @@
import 'package:serverpod_database/serverpod_database.dart'
show DatabaseUniqueViolationException;
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
import 'package:test/test.dart';
import '../integration/test_tools/client_session.dart';
import 'framework/dst_random.dart';
import 'framework/dst_snapshot.dart';
import 'framework/dst_world.dart';
void main() {
initTestClientSession(createSessionPerTest: false);
group('Given two synchronized UUID claims with one projected conflict,', () {
late DstIds ids;
late DstClock clock;
late UuidValue space;
late UuidValue contested;
late UuidValue replacement;
late DstReplica source;
late DstReplica peer;
late UniqueUuid winner;
late UniqueUuid loser;
late DstSnapshot before;
setUp(() async {
ids = DstIds(DstRandom(427));
clock = DstClock();
space = ids.next();
contested = ids.next();
replacement = ids.next();
source = await DstReplica.create(
name: 'source',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
peer = await DstReplica.create(
name: 'peer',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
winner = UniqueUuid(id: ids.next(), value: contested);
loser = UniqueUuid(id: ids.next(), value: contested);
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await UniqueUuid.db.insertRow(source.session, winner, transaction: tx);
}),
);
clock.advance(const Duration(milliseconds: 1));
await peer.withReplicaClock(
() => peer.session.db.transactionForUser(space, (tx) async {
await UniqueUuid.db.insertRow(peer.session, loser, transaction: tx);
}),
);
await source.merge(await peer.collect(space), space);
await peer.merge(await source.collect(space), space);
before = await DstSnapshot.capture(source);
});
group(
'when the winner locally changes to an uncontested UUID and sends only its delta,',
() {
late DstSnapshot local;
late DstSnapshot remote;
late CrdtMergeSet delta;
setUp(() async {
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await UniqueUuid.db.updateRow(
source.session,
winner.copyWith(value: replacement),
columns: (t) => [t.value],
transaction: tx,
);
}),
);
local = await DstSnapshot.capture(source);
delta = await source.collect(
space,
checkpoints: await peer.checkpoints(space),
);
await peer.merge(delta, space);
remote = await DstSnapshot.capture(peer);
});
test('then the old loser reclaims its authored UUID on both replicas.', () {
final key = ('unique_uuid', loser.id!, 'value');
expect(before.projections[key], isNotNull);
expect(delta, hasLength(1));
expect(delta.single, isA<CrdtMergeUpdate>());
expect(local.projections[key], isNull);
expect(remote.projections[key], isNull);
expect(local.renderSpace(space), remote.renderSpace(space));
});
},
);
group('when the winner changes in a batch update without returned rows,', () {
late DstSnapshot local;
late DstSnapshot remote;
late CrdtMergeSet delta;
setUp(() async {
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await UniqueUuid.db.update(
source.session,
[winner.copyWith(value: replacement)],
columns: (t) => [t.value],
noReturn: true,
transaction: tx,
);
}),
);
local = await DstSnapshot.capture(source);
delta = await source.collect(space, checkpoints: await peer.checkpoints(space));
await peer.merge(delta, space);
remote = await DstSnapshot.capture(peer);
});
test('then the released claimant reclaims its value before delta catch-up.', () {
final key = ('unique_uuid', loser.id!, 'value');
expect(before.projections[key], isNotNull);
expect(delta, hasLength(1));
expect(delta.single.uuidRowId, winner.id);
expect(local.projections[key], isNull);
expect(remote.projections[key], isNull);
expect(dstValue(local.authoredValue(key)), dstValue(before.authoredValue(key)));
expect(local.fieldHlc(key), before.fieldHlc(key));
expect(local.renderSpace(space), remote.renderSpace(space));
});
});
group('when a limited predicate update changes only the first claim,', () {
late DstSnapshot local;
late DstSnapshot remote;
late CrdtMergeSet delta;
setUp(() async {
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await UniqueUuid.db.updateWhere(
source.session,
columnValues: (t) => [t.value(replacement)],
where: (t) => t.id.inSet(<UuidValue>{winner.id!, loser.id!}),
orderBy: (t) => t.id,
limit: 1,
noReturn: true,
transaction: tx,
);
}),
);
local = await DstSnapshot.capture(source);
delta = await source.collect(space, checkpoints: await peer.checkpoints(space));
await peer.merge(delta, space);
remote = await DstSnapshot.capture(peer);
});
test('then the released claimant reclaims its value before delta catch-up.', () {
final key = ('unique_uuid', loser.id!, 'value');
expect(before.projections[key], isNotNull);
expect(delta, hasLength(1));
expect(delta.single.uuidRowId, winner.id);
expect(local.projections[key], isNull);
expect(remote.projections[key], isNull);
expect(dstValue(local.authoredValue(key)), dstValue(before.authoredValue(key)));
expect(local.fieldHlc(key), before.fieldHlc(key));
expect(local.renderSpace(space), remote.renderSpace(space));
});
});
group('when a predicate update skips the winner and selects the next claim,', () {
late List<UniqueUuid> updated;
late DstSnapshot local;
late DstSnapshot remote;
late CrdtMergeSet delta;
setUp(() async {
updated = await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
return UniqueUuid.db.updateWhere(
source.session,
columnValues: (t) => [t.value(replacement)],
where: (t) => t.id.inSet(<UuidValue>{winner.id!, loser.id!}),
orderBy: (t) => t.id,
offset: 1,
limit: 1,
transaction: tx,
);
}),
);
local = await DstSnapshot.capture(source);
delta = await source.collect(space, checkpoints: await peer.checkpoints(space));
await peer.merge(delta, space);
remote = await DstSnapshot.capture(peer);
});
test('then exactly the selected loser changes and is returned.', () {
final loserKey = ('unique_uuid', loser.id!, 'value');
final winnerKey = ('unique_uuid', winner.id!, 'value');
expect(before.projections[loserKey], isNotNull);
expect(updated, hasLength(1));
expect(updated.single.id, loser.id);
expect(updated.single.value, replacement);
expect(delta, hasLength(1));
expect(delta.single.uuidRowId, loser.id);
expect(local.projections[loserKey], isNull);
expect(dstValue(local.authoredValue(winnerKey)), dstValue(contested));
expect(local.fieldHlc(winnerKey), before.fieldHlc(winnerKey));
expect(local.renderSpace(space), remote.renderSpace(space));
});
});
group('when an upsert changes the winning claim,', () {
late DstSnapshot local;
late DstSnapshot remote;
late CrdtMergeSet delta;
setUp(() async {
await source.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await UniqueUuid.db.upsertRow(
source.session,
winner.copyWith(value: replacement),
conflictColumns: (t) => [t.id],
updateColumns: (t) => [t.value],
transaction: tx,
);
}),
);
local = await DstSnapshot.capture(source);
delta = await source.collect(space, checkpoints: await peer.checkpoints(space));
await peer.merge(delta, space);
remote = await DstSnapshot.capture(peer);
});
test('then the released claimant reclaims its value before delta catch-up.', () {
final key = ('unique_uuid', loser.id!, 'value');
expect(before.projections[key], isNotNull);
expect(delta, hasLength(1));
expect(delta.single.uuidRowId, winner.id);
expect(local.projections[key], isNull);
expect(remote.projections[key], isNull);
expect(dstValue(local.authoredValue(key)), dstValue(before.authoredValue(key)));
expect(local.fieldHlc(key), before.fieldHlc(key));
expect(local.renderSpace(space), remote.renderSpace(space));
});
});
group('when a predicate update fails its physical unique constraint,', () {
late Object failure;
late DstSnapshot after;
late List<String> checkpointBefore;
late List<String> checkpointAfter;
setUp(() async {
checkpointBefore = (await source.checkpoints(
space,
)).map((hlc) => hlc.toString()).toList();
failure = await source
.withReplicaClock(
() => source.session.db.transactionForUser(space, (tx) async {
await UniqueUuid.db.updateWhere(
source.session,
columnValues: (t) => [t.value(replacement)],
where: (t) => t.id.inSet(<UuidValue>{winner.id!, loser.id!}),
transaction: tx,
);
}),
)
.then<Object>(
(_) => 'unexpected success',
onError: (Object error) => error,
);
after = await DstSnapshot.capture(source);
checkpointAfter = (await source.checkpoints(
space,
)).map((hlc) => hlc.toString()).toList();
});
test(
'then pre-write closure capture changes no domain, authored metadata, or progress.',
() {
expect(before.projections[('unique_uuid', loser.id!, 'value')], isNotNull);
expect(failure, isA<DatabaseUniqueViolationException>());
expect(after.renderSpace(space), before.renderSpace(space));
expect(after.renderRawMetadata(), before.renderRawMetadata());
expect(checkpointAfter, checkpointBefore);
},
);
});
});
}

View file

@ -0,0 +1,84 @@
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
import 'package:test/test.dart';
import '../integration/test_tools/client_session.dart';
import 'framework/dst_random.dart';
import 'framework/dst_world.dart';
void main() {
initTestClientSession(createSessionPerTest: false);
group('Given two local writes committed within the same clock millisecond,', () {
final ids = DstIds(DstRandom(426));
final clock = DstClock();
final space = ids.next();
late DstReplica author;
late DstReplica peer;
late Hlc latestAuthored;
setUpAll(() async {
author = await DstReplica.create(
name: 'author',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
peer = await DstReplica.create(
name: 'peer',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
for (final name in ['first', 'second']) {
await author.withReplicaClock(
() => author.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(
author.session,
City(id: ids.next(), name: name),
transaction: tx,
);
}),
);
}
final changes = await author.collect(space);
latestAuthored = changes.maxHlc!;
await peer.merge(changes, space);
});
group(
'when the author reconnects before its wall clock advances and after it moves backward,',
() {
late Hlc sameMillisecond;
late Hlc backwardClock;
late CrdtMergeSet sameMillisecondEcho;
late CrdtMergeSet backwardClockEcho;
setUpAll(() async {
var checkpoints = await author.checkpoints(space);
sameMillisecond = checkpoints.singleWhere(
(hlc) => hlc.nodeId == author.nodeUuid,
);
sameMillisecondEcho = await peer.collect(space, checkpoints: checkpoints);
clock.advance(const Duration(seconds: -1));
checkpoints = await author.checkpoints(space);
backwardClock = checkpoints.singleWhere(
(hlc) => hlc.nodeId == author.nodeUuid,
);
backwardClockEcho = await peer.collect(space, checkpoints: checkpoints);
});
test('then both resume vectors cover the persisted logical counter.', () {
expect(latestAuthored.counter, greaterThan(0));
expect(sameMillisecond, greaterThanOrEqualTo(latestAuthored));
expect(backwardClock, greaterThanOrEqualTo(latestAuthored));
});
test("then neither resume causes the peer to echo the author's own facts.", () {
expect(sameMillisecondEcho, isEmpty);
expect(backwardClockEcho, isEmpty);
});
},
);
});
}

View file

@ -0,0 +1,204 @@
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
import 'package:test/test.dart';
import '../integration/test_tools/client_session.dart';
import 'framework/dst_random.dart';
import 'framework/dst_world.dart';
void main() {
initTestClientSession(createSessionPerTest: false);
group("Given a relay with its own older fact and another author's newer fact,", () {
late DstIds ids;
late DstClock clock;
late UuidValue space;
late DstReplica author;
late DstReplica relay;
late DstReplica receiver;
late CrdtMergeSet initial;
Future<void> insert(DstReplica replica, String name) => replica.withReplicaClock(
() => replica.session.db.transactionForUser(space, (tx) async {
await City.db.insertRow(
replica.session,
City(id: ids.next(), name: name),
transaction: tx,
);
}),
);
setUp(() async {
ids = DstIds(DstRandom(990));
clock = DstClock();
space = ids.next();
author = await DstReplica.create(
name: 'author',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
relay = await DstReplica.create(
name: 'relay',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
receiver = await DstReplica.create(
name: 'receiver',
spaceUuids: [space],
nodeUuid: ids.next(),
clock: clock.clock,
);
await insert(relay, 'relay fact');
clock.advance(const Duration(seconds: 1));
await insert(author, 'author fact');
await relay.merge(await author.collect(space), space);
initial = await relay.collect(space);
await receiver.merge(initial, space);
});
group('when the receiver reconnects after the production inbound merge,', () {
late List<Hlc> vector;
late CrdtMergeSet pending;
setUp(() async {
vector = await receiver.checkpoints(space);
pending = await relay.collect(space, checkpoints: vector);
});
test(
'then each author has its own checkpoint and the relay has nothing to resend.',
() {
expect(initial, hasLength(2));
expect(initial.maxHlc!.nodeId, author.nodeUuid);
expect(vector.map((hlc) => hlc.nodeId).toSet(), hasLength(3));
expect(vector, hasLength(3));
expect(
vector.singleWhere((hlc) => hlc.nodeId == relay.nodeUuid),
initial.singleWhere((change) => change.uuidNodeId == relay.nodeUuid).hlc,
);
expect(pending, isEmpty);
},
);
});
group('when a stale session echoes a receiver fact before another local write,', () {
late CrdtMergeSet echo;
late CrdtMergeSet pending;
late List<Hlc> vector;
setUp(() async {
final stale = await receiver.checkpoints(space);
clock.advance(const Duration(seconds: 1));
await insert(receiver, 'receiver fact one');
await relay.merge(
await receiver.collect(space, checkpoints: await relay.checkpoints(space)),
space,
);
echo = await relay.collect(space, checkpoints: stale);
await receiver.merge(echo, space);
clock.advance(const Duration(seconds: 1));
await insert(receiver, 'receiver fact two');
await relay.merge(
await receiver.collect(space, checkpoints: await relay.checkpoints(space)),
space,
);
vector = await receiver.checkpoints(space);
pending = await relay.collect(space, checkpoints: vector);
});
test(
'then reconnecting neither resends the relay fact nor echoes either local fact.',
() {
expect(echo, hasLength(1));
expect(echo.single.uuidNodeId, receiver.nodeUuid);
expect(vector, hasLength(3));
expect(vector.map((hlc) => hlc.nodeId).toSet(), hasLength(3));
expect(pending, isEmpty);
},
);
});
group("when the checkpoint API is given a relayed author under the peer's id,", () {
late Object failure;
late List<Hlc> before;
late List<Hlc> after;
setUp(() async {
before = await receiver.checkpoints(space);
failure = await receiver.session.db
.recordSyncCheckpoint(relay.nodeUuid, initial.maxHlc!, userId: space)
.then<Object>(
(_) => 'unexpected success',
onError: (Object error) => error,
);
after = await receiver.checkpoints(space);
});
test('then the invalid checkpoint is rejected without changing progress.', () {
expect(initial.maxHlc!.nodeId, author.nodeUuid);
expect(failure, isA<ArgumentError>());
expect(after, before);
});
});
group(
'when a stored relay checkpoint carries the wrong author from older bookkeeping,',
() {
late List<Hlc> beforeRepair;
late List<Hlc> afterRepair;
late CrdtMergeSet repair;
late CrdtMergeSet pending;
late Hlc? persisted;
setUp(() async {
final nodes = await OfflineSyncSpaceNode.db.find(
receiver.rawSession,
include: OfflineSyncSpaceNode.include(node: CrdtNode.include()),
);
final relayNode = nodes.singleWhere(
(node) => node.node!.uuidNodeId == relay.nodeUuid,
);
await OfflineSyncSpaceNode.db.updateRow(
receiver.rawSession,
relayNode.copyWith(lastReceivedHlc: initial.maxHlc),
columns: (t) => [t.lastReceivedHlc],
);
beforeRepair = await receiver.checkpoints(space);
repair = await relay.collect(space, checkpoints: beforeRepair);
await receiver.merge(repair, space);
afterRepair = await receiver.checkpoints(space);
persisted = (await OfflineSyncSpaceNode.db.findById(
receiver.rawSession,
relayNode.id!,
))!.lastReceivedHlc;
pending = await relay.collect(space, checkpoints: afterRepair);
});
test(
'then one safe retransmission repairs the persisted author tag and settles.',
() {
final relayHlc = initial
.singleWhere((change) => change.uuidNodeId == relay.nodeUuid)
.hlc;
expect(initial.maxHlc, greaterThan(relayHlc));
expect(beforeRepair, hasLength(3));
expect(
beforeRepair.singleWhere((hlc) => hlc.nodeId == relay.nodeUuid),
Hlc.zero(relay.nodeUuid),
);
expect(repair, hasLength(1));
expect(repair.single.uuidNodeId, relay.nodeUuid);
expect(persisted, relayHlc);
expect(
afterRepair.singleWhere((hlc) => hlc.nodeId == relay.nodeUuid),
relayHlc,
);
expect(pending, isEmpty);
},
);
},
);
});
}

View file

@ -36,13 +36,14 @@ void main() {
test( test(
'Given a property violation from seed 114 at 80 rounds, ' 'Given a property violation from seed 114 at 80 rounds, '
'when its failure is reported, ' 'when its failure is reported, '
'then the replay command preserves the seed, depth, profile, and graph width.', 'then the replay command preserves the seed, depth, profile, graph width, and delivery mode.',
() { () {
final failure = DstPropertyFailure( final failure = DstPropertyFailure(
seed: 114, seed: 114,
rounds: 80, rounds: 80,
profile: DstProfile.populated, profile: DstProfile.populated,
graphWidth: 3, graphWidth: 3,
delivery: DstDeliveryMode.delta,
violations: [(property: 'foreignKeyClosure', detail: 'orphan')], violations: [(property: 'foreignKeyClosure', detail: 'orphan')],
); );
@ -51,7 +52,7 @@ void main() {
expect( expect(
message, message,
'DST property failure (seed 114)\n' 'DST property failure (seed 114)\n'
'Replay: DST_SEED_BASE=114 DST_SEEDS=1 DST_ROUNDS=80 DST_PROFILE=populated DST_GRAPH_WIDTH=3 dart test -P dst\n' 'Replay: DST_SEED_BASE=114 DST_SEEDS=1 DST_ROUNDS=80 DST_PROFILE=populated DST_GRAPH_WIDTH=3 DST_DELIVERY=delta dart test -P dst\n'
'- foreignKeyClosure: orphan\n', '- foreignKeyClosure: orphan\n',
); );
}, },
@ -60,7 +61,7 @@ void main() {
test( test(
'Given an unexpected exception from seed 115 at 40 rounds, ' 'Given an unexpected exception from seed 115 at 40 rounds, '
'when its failure is reported, ' 'when its failure is reported, '
'then the replay command preserves the seed, depth, profile, and graph width.', 'then the replay command preserves the seed, depth, profile, graph width, and delivery mode.',
() async { () async {
final failure = await runWithSeedReported<Object>( final failure = await runWithSeedReported<Object>(
index: 1, index: 1,
@ -74,7 +75,7 @@ void main() {
expect( expect(
message, message,
'Bad state: Simulation 1 (seed 115) failed\n' 'Bad state: Simulation 1 (seed 115) failed\n'
'Replay: DST_SEED_BASE=115 DST_SEEDS=1 DST_ROUNDS=40 DST_PROFILE=sparse DST_GRAPH_WIDTH=2 dart test -P dst\n' 'Replay: DST_SEED_BASE=115 DST_SEEDS=1 DST_ROUNDS=40 DST_PROFILE=sparse DST_GRAPH_WIDTH=2 DST_DELIVERY=full dart test -P dst\n'
'Bad state: unexpected', 'Bad state: unexpected',
); );
}, },

Some files were not shown because too many files have changed in this diff Show more