fix: Correct checkpoint progress and add delta DST coverage (#158)
This commit is contained in:
parent
e42075cdd9
commit
f59a90bc97
24 changed files with 1829 additions and 163 deletions
4
.github/workflows/ci.yaml
vendored
4
.github/workflows/ci.yaml
vendored
|
|
@ -99,7 +99,7 @@ jobs:
|
||||||
# comment-tag: coverage-results
|
# comment-tag: coverage-results
|
||||||
|
|
||||||
simulation:
|
simulation:
|
||||||
name: deterministic simulation (${{ matrix.topology }})
|
name: deterministic simulation (${{ matrix.topology }}, ${{ matrix.delivery }})
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
timeout-minutes: 60
|
timeout-minutes: 60
|
||||||
# Deliberately not on pull requests. The suite seeds from the clock, so it
|
# Deliberately not on pull requests. The suite seeds from the clock, so it
|
||||||
|
|
@ -115,6 +115,7 @@ jobs:
|
||||||
# Keep the topology files below and shared ownership controls exhaustive
|
# Keep the topology files below and shared ownership controls exhaustive
|
||||||
# when adding a new test file tagged dst.
|
# when adding a new test file tagged dst.
|
||||||
topology: [convergence, cross_space]
|
topology: [convergence, cross_space]
|
||||||
|
delivery: [full, delta]
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
- uses: ./.github/actions/setup
|
- uses: ./.github/actions/setup
|
||||||
|
|
@ -135,6 +136,7 @@ jobs:
|
||||||
DST_ROUNDS: 200
|
DST_ROUNDS: 200
|
||||||
DST_PROFILE: mixed
|
DST_PROFILE: mixed
|
||||||
DST_GRAPH_WIDTH: 2
|
DST_GRAPH_WIDTH: 2
|
||||||
|
DST_DELIVERY: ${{ matrix.delivery }}
|
||||||
|
|
||||||
benchmarks:
|
benchmarks:
|
||||||
name: run benchmarks
|
name: run benchmarks
|
||||||
|
|
|
||||||
14
CHANGELOG.md
14
CHANGELOG.md
|
|
@ -1,3 +1,17 @@
|
||||||
|
## Unreleased
|
||||||
|
|
||||||
|
- fix: Include the persisted local logical clock in resume checkpoints so
|
||||||
|
reconnecting within the same millisecond or after clock rollback settles.
|
||||||
|
- fix: Keep relay checkpoints tagged with their actual author and recover
|
||||||
|
incorrectly tagged stored progress without repeated transfers on reconnect.
|
||||||
|
- fix: Require `recordSyncCheckpoint` to receive an HLC authored by its
|
||||||
|
`otherNodeId`, throwing `ArgumentError` on a mismatch. Valid checkpoints
|
||||||
|
replace incorrectly tagged stored progress.
|
||||||
|
- fix: Reconsider released unique claims after local updates and upserts so
|
||||||
|
the author and peers receiving checkpoint deltas keep the same projection.
|
||||||
|
- test: Add checkpoint-based deterministic delivery alongside full-history
|
||||||
|
convergence, with replayable batches and separate coverage metrics.
|
||||||
|
|
||||||
## 0.0.9
|
## 0.0.9
|
||||||
|
|
||||||
- fix: Refreshes node clocks across sync wrappers.
|
- fix: Refreshes node clocks across sync wrappers.
|
||||||
|
|
|
||||||
|
|
@ -21,12 +21,14 @@ not new user field updates emitted opportunistically during merge.
|
||||||
- Supported foreign keys are single-column child references to single-column
|
- Supported foreign keys are single-column child references to single-column
|
||||||
parent references. Schemas with composite foreign keys must fail recorder
|
parent references. Schemas with composite foreign keys must fail recorder
|
||||||
initialization instead of silently bypassing projection.
|
initialization instead of silently bypassing projection.
|
||||||
- Merge input is causally complete: the sync protocol delivers each batch as a
|
- Merge input is causally complete relative to the receiver: every merged FK
|
||||||
full causal snapshot of the sender, so every merged FK attempt's parent facts
|
attempt's parent facts must already be committed at the receiver or accompany
|
||||||
precede or accompany the child fact. Input that violates this precondition
|
the child fact in the same complete batch. A checkpoint-filtered delta can
|
||||||
(for example, a non-nullable FK attempt whose parent fact is absent and whose
|
omit acknowledged facts; it need not repeat the sender's full history. Input
|
||||||
default target is missing) is outside the merge contract and fails the merge
|
that violates this precondition (for example, a non-nullable FK attempt whose
|
||||||
transaction atomically against the database foreign key constraints.
|
parent fact is absent and whose default target is missing) is outside the
|
||||||
|
merge contract and fails the merge transaction atomically against the database
|
||||||
|
foreign key constraints.
|
||||||
- Local `ON DELETE` actions are authored CRDT facts. `RESTRICT` and `NO ACTION`
|
- Local `ON DELETE` actions are authored CRDT facts. `RESTRICT` and `NO ACTION`
|
||||||
reject the local delete while visible children exist; `SET NULL` and
|
reject the local delete while visible children exist; `SET NULL` and
|
||||||
`SET DEFAULT` update child FK fields and their ordinary `CrdtDataField` HLCs;
|
`SET DEFAULT` update child FK fields and their ordinary `CrdtDataField` HLCs;
|
||||||
|
|
|
||||||
|
|
@ -52,8 +52,8 @@ membership relation and a sync protocol that iterates spaces.
|
||||||
2. **Smallest protocol delta.** The exchanged frames are today's, plus a
|
2. **Smallest protocol delta.** The exchanged frames are today's, plus a
|
||||||
per-cycle space-set exchange and a space tag on each frame. `collectNextBatch`
|
per-cycle space-set exchange and a space tag on each frame. `collectNextBatch`
|
||||||
demultiplexes a cycle's combined batch, `collectPendingChanges` collects every
|
demultiplexes a cycle's combined batch, `collectPendingChanges` collects every
|
||||||
active space in one pass, and the per-space merge plus `recordSyncCheckpoint`
|
active space in one pass, and `mergeInboundBatch` applies each inbound space
|
||||||
run once per inbound space group.
|
group while atomically persisting received progress for every author.
|
||||||
3. **Chunked streaming.** Outbound changes are collected in a single pass over
|
3. **Chunked streaming.** Outbound changes are collected in a single pass over
|
||||||
the active spaces and streamed in `syncBatchSize` chunks, so the wire payload
|
the active spaces and streamed in `syncBatchSize` chunks, so the wire payload
|
||||||
stays bounded regardless of space count. The collection step itself
|
stays bounded regardless of space count. The collection step itself
|
||||||
|
|
@ -128,7 +128,7 @@ They do not map one-to-one and neither replaces the other. One member may have
|
||||||
many nodes (one per replica/database install). The **server is a node** in every
|
many nodes (one per replica/database install). The **server is a node** in every
|
||||||
space it syncs but is not a member row. A freshly invited member has **zero
|
space it syncs but is not a member row. A freshly invited member has **zero
|
||||||
space-node rows** until their first device syncs. Nodes and space-node rows are
|
space-node rows** until their first device syncs. Nodes and space-node rows are
|
||||||
created implicitly by sync (`getOrCreate`, `recordSyncCheckpoint`); membership
|
created implicitly by space initialization and inbound merge; membership
|
||||||
rows are managed through `session.offlineSync.spaces` from application endpoints.
|
rows are managed through `session.offlineSync.spaces` from application endpoints.
|
||||||
Removing the `nodes` list would break checkpointing and causal filtering — it is
|
Removing the `nodes` list would break checkpointing and causal filtering — it is
|
||||||
the chain topology, not the access list.
|
the chain topology, not the access list.
|
||||||
|
|
@ -316,10 +316,11 @@ The shared establishment exchanges the space set so a follower never acts before
|
||||||
it knows the agreed set. Both cadences handshake initial and newly added spaces
|
it knows the agreed set. Both cadences handshake initial and newly added spaces
|
||||||
through the loop's own `SinceHlc` exchange.
|
through the loop's own `SinceHlc` exchange.
|
||||||
|
|
||||||
`collectPendingChanges`, `mergeInboundBatch`, `recordSyncCheckpoint`, and the
|
`collectPendingChanges`, `mergeInboundBatch`, and the
|
||||||
ownership-violation internals (`_streamInserts`/`Updates`/`Deletes`,
|
ownership-violation internals (`_streamInserts`/`Updates`/`Deletes`,
|
||||||
`_fetchDomainRow`, the durable-violation path) are space-parameterized and merge
|
`_fetchDomainRow`, the durable-violation path) are space-parameterized and merge
|
||||||
each inbound space group in its own `transactionForUser`/lock; any space's merge
|
each inbound space group in its own `transactionForUser`/lock, including its
|
||||||
|
per-author checkpoints; any space's merge
|
||||||
failure (for example an ownership collision or unauthorized role write) records
|
failure (for example an ownership collision or unauthorized role write) records
|
||||||
the durable violation and fails the session, preserving today's fail-fast
|
the durable violation and fails the session, preserving today's fail-fast
|
||||||
semantics.
|
semantics.
|
||||||
|
|
|
||||||
|
|
@ -133,7 +133,11 @@ class OfflineSyncDatabase implements Database {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Records the latest acknowledged sync checkpoint for [otherNodeId].
|
/// Records acknowledged per-author progress for [otherNodeId].
|
||||||
|
///
|
||||||
|
/// [syncedHlc] must carry [otherNodeId] as its author. Throws [ArgumentError]
|
||||||
|
/// otherwise. A correctly tagged checkpoint replaces previously stored
|
||||||
|
/// progress tagged with another author; valid progress advances monotonically.
|
||||||
Future<void> recordSyncCheckpoint(
|
Future<void> recordSyncCheckpoint(
|
||||||
UuidValue otherNodeId,
|
UuidValue otherNodeId,
|
||||||
Hlc syncedHlc, {
|
Hlc syncedHlc, {
|
||||||
|
|
@ -670,6 +674,7 @@ class OfflineSyncDatabase implements Database {
|
||||||
columns,
|
columns,
|
||||||
tx,
|
tx,
|
||||||
projectionUnchanged: plannedUpdates.projectionUnchanged,
|
projectionUnchanged: plannedUpdates.projectionUnchanged,
|
||||||
|
projectionRows: plannedUpdates.projectionRows,
|
||||||
);
|
);
|
||||||
return noReturn ? <T>[] : updatedRows;
|
return noReturn ? <T>[] : updatedRows;
|
||||||
},
|
},
|
||||||
|
|
@ -707,6 +712,7 @@ class OfflineSyncDatabase implements Database {
|
||||||
columns,
|
columns,
|
||||||
tx,
|
tx,
|
||||||
projectionUnchanged: plannedUpdates.projectionUnchanged,
|
projectionUnchanged: plannedUpdates.projectionUnchanged,
|
||||||
|
projectionRows: plannedUpdates.projectionRows,
|
||||||
);
|
);
|
||||||
return updatedRow;
|
return updatedRow;
|
||||||
},
|
},
|
||||||
|
|
@ -797,28 +803,42 @@ class OfflineSyncDatabase implements Database {
|
||||||
return _runTrackedWrite(
|
return _runTrackedWrite(
|
||||||
transaction,
|
transaction,
|
||||||
(tx) async {
|
(tx) async {
|
||||||
final result = await _delegate.updateWhere<T>(
|
final scopedWhere = (await _whereVisibleWithTombstone<T>(
|
||||||
columnValues: columnValues,
|
where,
|
||||||
where: (await _whereVisibleWithTombstone<T>(
|
null,
|
||||||
where,
|
tx,
|
||||||
null,
|
membershipWide: false,
|
||||||
tx,
|
))!;
|
||||||
membershipWide: false,
|
final columns = columnValues.map((e) => e.column).toList();
|
||||||
))!,
|
final plan = await _recorder.planLocalUpdateWhere<T>(
|
||||||
|
columns,
|
||||||
|
scopedWhere,
|
||||||
|
tx,
|
||||||
limit: limit,
|
limit: limit,
|
||||||
offset: offset,
|
offset: offset,
|
||||||
orderBy: orderBy,
|
orderBy: orderBy,
|
||||||
orderByList: orderByList,
|
orderByList: orderByList,
|
||||||
|
);
|
||||||
|
final selectedWhere = plan == null
|
||||||
|
? scopedWhere
|
||||||
|
: serializationManager.getTableForType(T)!.id.inSet(plan.rowIds);
|
||||||
|
final result = await _delegate.updateWhere<T>(
|
||||||
|
columnValues: columnValues,
|
||||||
|
where: selectedWhere,
|
||||||
|
limit: plan == null ? limit : null,
|
||||||
|
offset: plan == null ? offset : null,
|
||||||
|
orderBy: orderBy,
|
||||||
|
orderByList: orderByList,
|
||||||
transaction: tx,
|
transaction: tx,
|
||||||
);
|
);
|
||||||
|
|
||||||
final columns = columnValues.map((e) => e.column).toList();
|
|
||||||
_recorder.validateAuthoredRows(result, columns);
|
_recorder.validateAuthoredRows(result, columns);
|
||||||
await _recorder.afterUpdate(
|
await _recorder.afterUpdate(
|
||||||
result,
|
result,
|
||||||
columns,
|
columns,
|
||||||
tx,
|
tx,
|
||||||
authoredColumnValues: true,
|
authoredColumnValues: true,
|
||||||
|
projectionRows: plan?.projectionRows ?? const {},
|
||||||
);
|
);
|
||||||
if (noReturn) return <T>[];
|
if (noReturn) return <T>[];
|
||||||
result.forEach(_stripSpaceId);
|
result.forEach(_stripSpaceId);
|
||||||
|
|
|
||||||
|
|
@ -195,8 +195,11 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
||||||
in maxIncomingHlcByNode.entries) {
|
in maxIncomingHlcByNode.entries) {
|
||||||
final remoteNode = remoteNodes.spaceNodesByUuid[nodeId];
|
final remoteNode = remoteNodes.spaceNodesByUuid[nodeId];
|
||||||
if (remoteNode == null) continue;
|
if (remoteNode == null) continue;
|
||||||
|
final previous = remoteNode.lastReceivedHlc;
|
||||||
final updatedSpaceNode = remoteNode.copyWith(
|
final updatedSpaceNode = remoteNode.copyWith(
|
||||||
lastReceivedHlc: incomingHlc.maxBetween(remoteNode.lastReceivedHlc),
|
lastReceivedHlc: incomingHlc.maxBetween(
|
||||||
|
previous?.nodeId == nodeId ? previous : null,
|
||||||
|
),
|
||||||
);
|
);
|
||||||
if (updatedSpaceNode.lastReceivedHlc == remoteNode.lastReceivedHlc) {
|
if (updatedSpaceNode.lastReceivedHlc == remoteNode.lastReceivedHlc) {
|
||||||
continue;
|
continue;
|
||||||
|
|
|
||||||
|
|
@ -340,12 +340,22 @@ class CrdtMutationRecorder {
|
||||||
Future<void> lockAndRefreshCurrentNodeHlc(Transaction transaction) =>
|
Future<void> lockAndRefreshCurrentNodeHlc(Transaction transaction) =>
|
||||||
_context.lockAndRefreshCurrentNodeHlc(transaction);
|
_context.lockAndRefreshCurrentNodeHlc(transaction);
|
||||||
|
|
||||||
/// Records the latest acknowledged sync checkpoint for [otherNodeId].
|
/// Records per-author progress, rejecting an HLC not tagged with [otherNodeId].
|
||||||
|
///
|
||||||
|
/// Replaces mis-tagged stored progress; correctly tagged progress only advances.
|
||||||
Future<void> recordSyncCheckpoint(
|
Future<void> recordSyncCheckpoint(
|
||||||
UuidValue userId,
|
UuidValue userId,
|
||||||
UuidValue otherNodeId,
|
UuidValue otherNodeId,
|
||||||
Hlc syncedHlc,
|
Hlc syncedHlc,
|
||||||
) async {
|
) async {
|
||||||
|
if (syncedHlc.nodeId != otherNodeId) {
|
||||||
|
throw ArgumentError.value(
|
||||||
|
syncedHlc,
|
||||||
|
'syncedHlc',
|
||||||
|
'A per-author checkpoint must carry otherNodeId',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
final space = await _context.spaceManager.getOrCreate(userId);
|
final space = await _context.spaceManager.getOrCreate(userId);
|
||||||
await _db.transaction((transaction) async {
|
await _db.transaction((transaction) async {
|
||||||
final node = await _context.findOrCreateNode(otherNodeId, transaction);
|
final node = await _context.findOrCreateNode(otherNodeId, transaction);
|
||||||
|
|
@ -356,7 +366,9 @@ class CrdtMutationRecorder {
|
||||||
);
|
);
|
||||||
|
|
||||||
final currentSyncHlc = spaceNode.lastReceivedHlc;
|
final currentSyncHlc = spaceNode.lastReceivedHlc;
|
||||||
if (currentSyncHlc != null && currentSyncHlc >= syncedHlc) {
|
if (currentSyncHlc != null &&
|
||||||
|
currentSyncHlc.nodeId == otherNodeId &&
|
||||||
|
currentSyncHlc >= syncedHlc) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -574,21 +586,35 @@ class CrdtMutationRecorder {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Plans FK/unique projection for rows that are about to be updated.
|
/// Plans FK/unique projection for rows that are about to be updated.
|
||||||
Future<({List<T> rows, bool projectionUnchanged})>
|
Future<({List<T> rows, bool projectionUnchanged, Set<MergeRowKey> projectionRows})>
|
||||||
planLocalUpdates<T extends TableRow>(
|
planLocalUpdates<T extends TableRow>(
|
||||||
List<T> rows,
|
List<T> rows,
|
||||||
List<Column>? columns,
|
List<Column>? columns,
|
||||||
Transaction transaction, {
|
Transaction transaction, {
|
||||||
bool restoring = false,
|
bool restoring = false,
|
||||||
}) async {
|
}) async {
|
||||||
if (rows.isEmpty) return (rows: rows, projectionUnchanged: true);
|
if (rows.isEmpty) {
|
||||||
|
return (
|
||||||
|
rows: rows,
|
||||||
|
projectionUnchanged: true,
|
||||||
|
projectionRows: const <MergeRowKey>{},
|
||||||
|
);
|
||||||
|
}
|
||||||
final tableName = rows.first.table.tableName;
|
final tableName = rows.first.table.tableName;
|
||||||
if (!_context.isCrdtTrackedTableName(tableName)) {
|
if (!_context.isCrdtTrackedTableName(tableName)) {
|
||||||
return (rows: rows, projectionUnchanged: true);
|
return (
|
||||||
|
rows: rows,
|
||||||
|
projectionUnchanged: true,
|
||||||
|
projectionRows: const <MergeRowKey>{},
|
||||||
|
);
|
||||||
}
|
}
|
||||||
final columnNames = columns?.map((column) => column.columnName).toSet();
|
final columnNames = columns?.map((column) => column.columnName).toSet();
|
||||||
if (!_foreignKeyProjector.needsProjection(tableName, columnNames)) {
|
if (!_foreignKeyProjector.needsProjection(tableName, columnNames)) {
|
||||||
return (rows: rows, projectionUnchanged: true);
|
return (
|
||||||
|
rows: rows,
|
||||||
|
projectionUnchanged: true,
|
||||||
|
projectionRows: const <MergeRowKey>{},
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (await _foreignKeyProjector.canLeaveLocalProjectionUnchanged(
|
if (await _foreignKeyProjector.canLeaveLocalProjectionUnchanged(
|
||||||
|
|
@ -597,7 +623,11 @@ class CrdtMutationRecorder {
|
||||||
inserting: false,
|
inserting: false,
|
||||||
columns: columns,
|
columns: columns,
|
||||||
)) {
|
)) {
|
||||||
return (rows: rows, projectionUnchanged: true);
|
return (
|
||||||
|
rows: rows,
|
||||||
|
projectionUnchanged: true,
|
||||||
|
projectionRows: const <MergeRowKey>{},
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
final authored = {
|
final authored = {
|
||||||
|
|
@ -646,6 +676,7 @@ class CrdtMutationRecorder {
|
||||||
);
|
);
|
||||||
return (
|
return (
|
||||||
projectionUnchanged: false,
|
projectionUnchanged: false,
|
||||||
|
projectionRows: planned.domain.keys.toSet(),
|
||||||
rows: [
|
rows: [
|
||||||
for (final row in rows)
|
for (final row in rows)
|
||||||
_withPlannedDomainValues(
|
_withPlannedDomainValues(
|
||||||
|
|
@ -835,6 +866,8 @@ class CrdtMutationRecorder {
|
||||||
CrdtDataDeletedReason.userReinsert,
|
CrdtDataDeletedReason.userReinsert,
|
||||||
transaction,
|
transaction,
|
||||||
);
|
);
|
||||||
|
// Hidden rows already released their old unique claims; seed projection
|
||||||
|
// from the restored rows to resolve their new values.
|
||||||
await _maybeProject(tableName, rowIds, null, transaction);
|
await _maybeProject(tableName, rowIds, null, transaction);
|
||||||
// Only projected fields need a metadata row to hold their authored value.
|
// Only projected fields need a metadata row to hold their authored value.
|
||||||
// All other fields now inherit the insertion timestamp implicitly.
|
// All other fields now inherit the insertion timestamp implicitly.
|
||||||
|
|
@ -849,6 +882,52 @@ class CrdtMutationRecorder {
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Captures the old projection closure before a set-based physical update.
|
||||||
|
///
|
||||||
|
/// Updating a winning unique tuple can disconnect its released claimants.
|
||||||
|
/// Select targets while the write transaction holds the node lock. The
|
||||||
|
/// caller must update these identities instead of evaluating a limited
|
||||||
|
/// predicate again, which could select different rows without a total order.
|
||||||
|
Future<({Set<UuidValue> rowIds, Set<MergeRowKey> projectionRows})?>
|
||||||
|
planLocalUpdateWhere<T extends TableRow>(
|
||||||
|
List<Column> columns,
|
||||||
|
Expression where,
|
||||||
|
Transaction transaction, {
|
||||||
|
int? limit,
|
||||||
|
int? offset,
|
||||||
|
Column? orderBy,
|
||||||
|
List<Column>? orderByList,
|
||||||
|
}) async {
|
||||||
|
final table = _session.db.serializationManager.getTableForType(T)!;
|
||||||
|
if (!_foreignKeyProjector.needsProjection(
|
||||||
|
table.tableName,
|
||||||
|
columns.map((column) => column.columnName).toSet(),
|
||||||
|
)) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
final rows = await _session.db.find<T>(
|
||||||
|
where: where,
|
||||||
|
limit: limit,
|
||||||
|
offset: offset,
|
||||||
|
orderBy: orderBy,
|
||||||
|
orderByList: orderByList,
|
||||||
|
transaction: transaction,
|
||||||
|
);
|
||||||
|
final rowIds = {for (final row in rows) row.id as UuidValue};
|
||||||
|
if (rowIds.isEmpty) {
|
||||||
|
return (rowIds: rowIds, projectionRows: const <MergeRowKey>{});
|
||||||
|
}
|
||||||
|
|
||||||
|
final plan = await _foreignKeyProjector.project(
|
||||||
|
transaction,
|
||||||
|
seedTables: {table.tableName},
|
||||||
|
seedRows: {for (final rowId in rowIds) (table.tableName, rowId)},
|
||||||
|
materialize: false,
|
||||||
|
);
|
||||||
|
return (rowIds: rowIds, projectionRows: plan.domain.keys.toSet());
|
||||||
|
}
|
||||||
|
|
||||||
/// Records CRDT field metadata for updated rows.
|
/// Records CRDT field metadata for updated rows.
|
||||||
///
|
///
|
||||||
/// [authoredColumnValues] marks a write that reached the domain without a
|
/// [authoredColumnValues] marks a write that reached the domain without a
|
||||||
|
|
@ -863,6 +942,7 @@ class CrdtMutationRecorder {
|
||||||
Transaction transaction, {
|
Transaction transaction, {
|
||||||
bool projectionUnchanged = false,
|
bool projectionUnchanged = false,
|
||||||
bool authoredColumnValues = false,
|
bool authoredColumnValues = false,
|
||||||
|
Set<MergeRowKey> projectionRows = const {},
|
||||||
Map<MergeRowKey, Map<String, Object?>> domainBeforeUpsert = const {},
|
Map<MergeRowKey, Map<String, Object?>> domainBeforeUpsert = const {},
|
||||||
List<TableRow> upsertRows = const [],
|
List<TableRow> upsertRows = const [],
|
||||||
}) async {
|
}) async {
|
||||||
|
|
@ -957,6 +1037,7 @@ class CrdtMutationRecorder {
|
||||||
updatedColumnNames,
|
updatedColumnNames,
|
||||||
transaction,
|
transaction,
|
||||||
authoredOverlays: authoredOverlays,
|
authoredOverlays: authoredOverlays,
|
||||||
|
projectionRows: {...projectionRows, ...domainBeforeUpsert.keys},
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
@ -967,12 +1048,15 @@ class CrdtMutationRecorder {
|
||||||
Set<String>? columnNames,
|
Set<String>? columnNames,
|
||||||
Transaction transaction, {
|
Transaction transaction, {
|
||||||
Map<MergeFieldKey, Object?> authoredOverlays = const {},
|
Map<MergeFieldKey, Object?> authoredOverlays = const {},
|
||||||
|
Set<MergeRowKey> projectionRows = const {},
|
||||||
}) async {
|
}) async {
|
||||||
if (_foreignKeyProjector.needsProjection(tableName, columnNames)) {
|
if (_foreignKeyProjector.needsProjection(tableName, columnNames)) {
|
||||||
await _foreignKeyProjector.project(
|
await _foreignKeyProjector.project(
|
||||||
transaction,
|
transaction,
|
||||||
seedTables: {tableName},
|
seedTables: {tableName},
|
||||||
seedRows: {for (final rowId in rowIds) (tableName, rowId)},
|
// A physical local write can remove the old tuple from the seed row.
|
||||||
|
// Retain its pre-write closure so released claimants are reconsidered.
|
||||||
|
seedRows: {...projectionRows, for (final rowId in rowIds) (tableName, rowId)},
|
||||||
authoredOverlays: authoredOverlays,
|
authoredOverlays: authoredOverlays,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -2,6 +2,7 @@ import 'dart:async';
|
||||||
import 'dart:convert';
|
import 'dart:convert';
|
||||||
|
|
||||||
import 'package:clock/clock.dart';
|
import 'package:clock/clock.dart';
|
||||||
|
import 'package:meta/meta.dart';
|
||||||
import 'package:serverpod_database/serverpod_database.dart';
|
import 'package:serverpod_database/serverpod_database.dart';
|
||||||
import 'package:uuid/uuid.dart';
|
import 'package:uuid/uuid.dart';
|
||||||
|
|
||||||
|
|
@ -192,18 +193,27 @@ class OfflineSyncEngine {
|
||||||
include: OfflineSyncSpaceNode.include(node: CrdtNode.include()),
|
include: OfflineSyncSpaceNode.include(node: CrdtNode.include()),
|
||||||
);
|
);
|
||||||
|
|
||||||
|
final checkpoints = <UuidValue, Hlc>{
|
||||||
|
// Cover committed logical ticks as well as wall time. A clock that has
|
||||||
|
// not advanced (or moved backward) must not make peers echo our facts.
|
||||||
|
localNodeId: Hlc.now(localNodeId).maxBetween(space.currentNode!.lastHlc),
|
||||||
|
};
|
||||||
|
for (final spaceNode in spaceNodes) {
|
||||||
|
final nodeId = spaceNode.node!.uuidNodeId;
|
||||||
|
final received = spaceNode.lastReceivedHlc;
|
||||||
|
// Older inbound bookkeeping could store a relayed author's clock under
|
||||||
|
// its direct peer. It proves no per-author progress for that peer.
|
||||||
|
final checkpoint = received?.nodeId == nodeId ? received! : Hlc.zero(nodeId);
|
||||||
|
checkpoints[nodeId] = checkpoint.maxBetween(checkpoints[nodeId]);
|
||||||
|
}
|
||||||
|
|
||||||
return OfflineSyncSinceHlc(
|
return OfflineSyncSinceHlc(
|
||||||
uuidSpaceId: spaceId,
|
uuidSpaceId: spaceId,
|
||||||
nodeCheckpoints: [
|
nodeCheckpoints: checkpoints.values.toList(),
|
||||||
// The local node is always included to avoid collecting its own changes.
|
|
||||||
Hlc.now(localNodeId),
|
|
||||||
for (final spaceNode in spaceNodes)
|
|
||||||
spaceNode.lastReceivedHlc ?? Hlc.zero(spaceNode.node!.uuidNodeId),
|
|
||||||
],
|
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Merges a remote [mergeSet] and records the sync checkpoint for [otherNodeId].
|
/// Merges a remote [mergeSet] and atomically records its per-author progress.
|
||||||
///
|
///
|
||||||
/// Inbound merge applies each remote change, then materializes foreign-key
|
/// Inbound merge applies each remote change, then materializes foreign-key
|
||||||
/// projection into domain tables via [OfflineSyncDatabase.mergeChanges].
|
/// projection into domain tables via [OfflineSyncDatabase.mergeChanges].
|
||||||
|
|
@ -213,23 +223,16 @@ class OfflineSyncEngine {
|
||||||
///
|
///
|
||||||
/// Returns the greatest HLC synced in the batch, or `null` if the batch is
|
/// Returns the greatest HLC synced in the batch, or `null` if the batch is
|
||||||
/// empty.
|
/// empty.
|
||||||
Future<Hlc?> _mergeInboundBatch(
|
@visibleForTesting
|
||||||
|
Future<Hlc?> mergeInboundBatch(
|
||||||
DatabaseSession session, {
|
DatabaseSession session, {
|
||||||
required UuidValue spaceId,
|
required UuidValue spaceId,
|
||||||
required UuidValue otherNodeId,
|
|
||||||
required CrdtMergeSet mergeSet,
|
required CrdtMergeSet mergeSet,
|
||||||
}) async {
|
}) async {
|
||||||
if (mergeSet.isEmpty) return null;
|
if (mergeSet.isEmpty) return null;
|
||||||
final maxSyncedHlc = mergeSet.maxHlc;
|
final maxSyncedHlc = mergeSet.maxHlc;
|
||||||
final offlineSyncDb = _openOfflineSyncDatabase(session);
|
final offlineSyncDb = _openOfflineSyncDatabase(session);
|
||||||
await offlineSyncDb.mergeChanges(mergeSet, spaceId: spaceId);
|
await offlineSyncDb.mergeChanges(mergeSet, spaceId: spaceId);
|
||||||
if (maxSyncedHlc != null) {
|
|
||||||
await offlineSyncDb.recordSyncCheckpoint(
|
|
||||||
otherNodeId,
|
|
||||||
maxSyncedHlc,
|
|
||||||
userId: spaceId,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
return maxSyncedHlc;
|
return maxSyncedHlc;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -434,10 +437,9 @@ class OfflineSyncEngine {
|
||||||
changes: entry.value,
|
changes: entry.value,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
final receivedHlc = await _mergeInboundBatch(
|
final receivedHlc = await mergeInboundBatch(
|
||||||
session,
|
session,
|
||||||
spaceId: spaceId,
|
spaceId: spaceId,
|
||||||
otherNodeId: spaces.peerNodeId,
|
|
||||||
mergeSet: entry.value,
|
mergeSet: entry.value,
|
||||||
);
|
);
|
||||||
await _reportMerge(onMergeSuccess, spaces, spaceId, receivedHlc);
|
await _reportMerge(onMergeSuccess, spaces, spaceId, receivedHlc);
|
||||||
|
|
|
||||||
|
|
@ -197,11 +197,14 @@ class OfflineSyncSpaceState {
|
||||||
/// Records the peer's resume vector for [spaceId] from its [sinceHlc],
|
/// Records the peer's resume vector for [spaceId] from its [sinceHlc],
|
||||||
/// completing the space's handshake from this peer's side.
|
/// completing the space's handshake from this peer's side.
|
||||||
void recordPeerHandshake(UuidValue spaceId, OfflineSyncSinceHlc sinceHlc) {
|
void recordPeerHandshake(UuidValue spaceId, OfflineSyncSinceHlc sinceHlc) {
|
||||||
_checkpointsBySpace[spaceId] = {
|
_checkpointsBySpace[spaceId] = normalizeCheckpoints(sinceHlc.nodeCheckpoints);
|
||||||
for (final checkpoint in sinceHlc.nodeCheckpoints) checkpoint.nodeId: checkpoint,
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Normalizes a wire resume vector exactly as a peer handshake does.
|
||||||
|
static Map<UuidValue, Hlc> normalizeCheckpoints(Iterable<Hlc> checkpoints) => {
|
||||||
|
for (final checkpoint in checkpoints) checkpoint.nodeId: checkpoint,
|
||||||
|
};
|
||||||
|
|
||||||
/// Checkpoints for writable spaces whose handshake completed both ways, keyed
|
/// Checkpoints for writable spaces whose handshake completed both ways, keyed
|
||||||
/// by space — the input to a pending-change collection pass.
|
/// by space — the input to a pending-change collection pass.
|
||||||
Map<UuidValue, List<Hlc>> get sendableCheckpoints => {
|
Map<UuidValue, List<Hlc>> get sendableCheckpoints => {
|
||||||
|
|
|
||||||
|
|
@ -119,6 +119,7 @@ melos:
|
||||||
# seeds from the clock, so it searches fresh schedules on every run and
|
# seeds from the clock, so it searches fresh schedules on every run and
|
||||||
# can fail for a defect unrelated to the change that triggered it.
|
# can fail for a defect unrelated to the change that triggered it.
|
||||||
#
|
#
|
||||||
|
# Select full-history or checkpoint delivery with DST_DELIVERY=full|delta.
|
||||||
# Widen the search with DST_SEEDS and DST_ROUNDS; a failure prints the
|
# Widen the search with DST_SEEDS and DST_ROUNDS; a failure prints the
|
||||||
# command that replays it. Concurrency is pinned to one so the reported
|
# command that replays it. Concurrency is pinned to one so the reported
|
||||||
# failures cannot interleave, which matters when reading a wide sweep.
|
# failures cannot interleave, which matters when reading a wide sweep.
|
||||||
|
|
|
||||||
|
|
@ -16,6 +16,7 @@
|
||||||
# Results live in .dst-soak:
|
# Results live in .dst-soak:
|
||||||
# FAILURES.md one section per failing run, with its replay command
|
# FAILURES.md one section per failing run, with its replay command
|
||||||
# RUNS.tsv every run attempted, so coverage is auditable
|
# RUNS.tsv every run attempted, so coverage is auditable
|
||||||
|
# METRICS.jsonl coverage records retained for passing and failing runs
|
||||||
# runs/<id>.log full output, kept for failures only
|
# runs/<id>.log full output, kept for failures only
|
||||||
# tree.txt, tree.diff the exact code state being hunted
|
# tree.txt, tree.diff the exact code state being hunted
|
||||||
|
|
||||||
|
|
@ -50,8 +51,16 @@ mkdir -p "$OUT/runs"
|
||||||
git -C "$REPO_ROOT" diff HEAD > "$OUT/tree.diff"
|
git -C "$REPO_ROOT" diff HEAD > "$OUT/tree.diff"
|
||||||
|
|
||||||
[ -f "$OUT/RUNS.tsv" ] ||
|
[ -f "$OUT/RUNS.tsv" ] ||
|
||||||
printf 'started\tid\tprofile\tseed_base\tseeds\trounds\twidth\ttarget\tresult\tseconds\n' \
|
printf 'started\tid\tprofile\tseed_base\tseeds\trounds\twidth\ttarget\tresult\tseconds\tdelivery\n' \
|
||||||
> "$OUT/RUNS.tsv"
|
> "$OUT/RUNS.tsv"
|
||||||
|
# Upgrade existing ledgers without reinterpreting old runs: before delivery
|
||||||
|
# modes existed, every recorded run used full-history delivery.
|
||||||
|
if [ "$(head -n 1 "$OUT/RUNS.tsv" | awk -F'\t' '{print NF}')" -eq 10 ]; then
|
||||||
|
ledger_tmp=$(mktemp)
|
||||||
|
awk -F'\t' 'BEGIN { OFS = FS } { print $0, (NR == 1 ? "delivery" : "full") }' \
|
||||||
|
"$OUT/RUNS.tsv" > "$ledger_tmp" && mv "$ledger_tmp" "$OUT/RUNS.tsv"
|
||||||
|
fi
|
||||||
|
|
||||||
[ -f "$OUT/FAILURES.md" ] || {
|
[ -f "$OUT/FAILURES.md" ] || {
|
||||||
echo "# Deterministic simulation failures"
|
echo "# Deterministic simulation failures"
|
||||||
echo
|
echo
|
||||||
|
|
@ -64,14 +73,16 @@ git -C "$REPO_ROOT" diff HEAD > "$OUT/tree.diff"
|
||||||
# how deep a schedule gets before the run ends. A single fixed shape would keep
|
# how deep a schedule gets before the run ends. A single fixed shape would keep
|
||||||
# re-searching the same corner of the space all night.
|
# re-searching the same corner of the space all night.
|
||||||
#
|
#
|
||||||
# profile:seeds:rounds:width:target
|
# profile:seeds:rounds:width:target:delivery
|
||||||
CONFIGS=(
|
CONFIGS=(
|
||||||
"sparse:8:40:2:"
|
"sparse:8:40:2::full"
|
||||||
"mixed:6:120:2:"
|
"sparse:8:40:2::delta"
|
||||||
"mixed:4:200:2:test/dst/dst_cross_space_test.dart"
|
"mixed:4:200:2:test/dst/dst_cross_space_test.dart:full"
|
||||||
"populated:3:200:3:"
|
"mixed:4:200:2:test/dst/dst_cross_space_test.dart:delta"
|
||||||
"sparse:12:40:3:test/dst/dst_convergence_test.dart"
|
"populated:3:200:3::full"
|
||||||
"mixed:8:80:2:"
|
"populated:3:200:3::delta"
|
||||||
|
"mixed:6:120:2:test/dst/dst_convergence_test.dart:full"
|
||||||
|
"mixed:6:120:2:test/dst/dst_convergence_test.dart:delta"
|
||||||
)
|
)
|
||||||
# Space-separated entries in the same shape, to steer or test the hunt.
|
# Space-separated entries in the same shape, to steer or test the hunt.
|
||||||
[ -z "${DST_SOAK_CONFIGS:-}" ] || read -r -a CONFIGS <<< "$DST_SOAK_CONFIGS"
|
[ -z "${DST_SOAK_CONFIGS:-}" ] || read -r -a CONFIGS <<< "$DST_SOAK_CONFIGS"
|
||||||
|
|
@ -87,24 +98,29 @@ index=0
|
||||||
suspicious=0
|
suspicious=0
|
||||||
|
|
||||||
while [ "$(date +%s)" -lt "$UNTIL" ]; do
|
while [ "$(date +%s)" -lt "$UNTIL" ]; do
|
||||||
IFS=':' read -r profile seeds rounds width target <<< "${CONFIGS[index % ${#CONFIGS[@]}]}"
|
IFS=':' read -r profile seeds rounds width target delivery <<< "${CONFIGS[index % ${#CONFIGS[@]}]}"
|
||||||
|
delivery=${delivery:-${DST_DELIVERY:-full}}
|
||||||
|
case "$delivery" in
|
||||||
|
full|delta) ;;
|
||||||
|
*) echo "invalid delivery mode: $delivery" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
index=$((index + 1))
|
index=$((index + 1))
|
||||||
|
|
||||||
# The index keeps two runs that start in the same second from sharing a log.
|
# The index keeps two runs that start in the same second from sharing a log.
|
||||||
id="$(date +%Y%m%d-%H%M%S)-$index-$profile-$rounds"
|
id="$(date +%Y%m%d-%H%M%S)-$index-$profile-$rounds-$delivery"
|
||||||
log="$OUT/runs/$id.log"
|
log="$OUT/runs/$id.log"
|
||||||
seed_base=$cursor
|
seed_base=$cursor
|
||||||
cursor=$((cursor + seeds))
|
cursor=$((cursor + seeds))
|
||||||
|
|
||||||
replay="DST_SEED_BASE=$seed_base DST_SEEDS=$seeds DST_ROUNDS=$rounds"
|
replay="DST_SEED_BASE=$seed_base DST_SEEDS=$seeds DST_ROUNDS=$rounds"
|
||||||
replay="$replay DST_PROFILE=$profile DST_GRAPH_WIDTH=$width"
|
replay="$replay DST_PROFILE=$profile DST_GRAPH_WIDTH=$width DST_DELIVERY=$delivery"
|
||||||
replay="$replay dart test -P dst --concurrency=1 ${target:+$target}"
|
replay="$replay dart test -P dst --concurrency=1 ${target:+$target}"
|
||||||
|
|
||||||
# The command is written down before the run, so a kill -9 mid-run still
|
# The command is written down before the run, so a kill -9 mid-run still
|
||||||
# leaves the seed behind.
|
# leaves the seed behind.
|
||||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
|
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
|
||||||
"$(date -Is)" "$id" "$profile" "$seed_base" "$seeds" "$rounds" "$width" \
|
"$(date -Is)" "$id" "$profile" "$seed_base" "$seeds" "$rounds" "$width" \
|
||||||
"${target:-all}" "started" "" >> "$OUT/RUNS.tsv"
|
"${target:-all}" "started" "" "$delivery" >> "$OUT/RUNS.tsv"
|
||||||
|
|
||||||
started=$(date +%s)
|
started=$(date +%s)
|
||||||
(
|
(
|
||||||
|
|
@ -114,6 +130,7 @@ while [ "$(date +%s)" -lt "$UNTIL" ]; do
|
||||||
DST_ROUNDS=$rounds \
|
DST_ROUNDS=$rounds \
|
||||||
DST_PROFILE=$profile \
|
DST_PROFILE=$profile \
|
||||||
DST_GRAPH_WIDTH=$width \
|
DST_GRAPH_WIDTH=$width \
|
||||||
|
DST_DELIVERY=$delivery \
|
||||||
timeout "$RUN_TIMEOUT" dart test -P dst --concurrency=1 \
|
timeout "$RUN_TIMEOUT" dart test -P dst --concurrency=1 \
|
||||||
--reporter=expanded ${target:+"$target"}
|
--reporter=expanded ${target:+"$target"}
|
||||||
) > "$log" 2>&1
|
) > "$log" 2>&1
|
||||||
|
|
@ -128,9 +145,12 @@ while [ "$(date +%s)" -lt "$UNTIL" ]; do
|
||||||
exit 4
|
exit 4
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Preserve actual coverage before deleting bulky passing-run output.
|
||||||
|
sed -n "s/^.*DST_METRICS {/\{\"runId\":\"$id\",/p" "$log" >> "$OUT/METRICS.jsonl"
|
||||||
|
|
||||||
if [ $status -eq 0 ]; then
|
if [ $status -eq 0 ]; then
|
||||||
# A passing run proves the seeds were searched; its output proves nothing,
|
# Metrics above retain coverage evidence; keep bulky per-test output only
|
||||||
# and a night of them fills the disk.
|
# for failures so an overnight hunt does not fill the disk.
|
||||||
rm -f "$log"
|
rm -f "$log"
|
||||||
result="passed"
|
result="passed"
|
||||||
suspicious=0
|
suspicious=0
|
||||||
|
|
@ -157,7 +177,7 @@ while [ "$(date +%s)" -lt "$UNTIL" ]; do
|
||||||
echo
|
echo
|
||||||
# The suite prints its own single-seed replay line for each failure. It is
|
# The suite prints its own single-seed replay line for each failure. It is
|
||||||
# narrower than the sweep command above, so it is kept when present.
|
# narrower than the sweep command above, so it is kept when present.
|
||||||
sed -n 's/.*\(Replay: DST_SEED_BASE=[^ ]* DST_SEEDS=[^ ]* DST_ROUNDS=[^ ]* DST_PROFILE=[^ ]* DST_GRAPH_WIDTH=[0-9]*\).*/- \1/p' \
|
sed -n 's/.*\(Replay: DST_SEED_BASE=[^ ]* DST_SEEDS=[^ ]* DST_ROUNDS=[^ ]* DST_PROFILE=[^ ]* DST_GRAPH_WIDTH=[0-9]* DST_DELIVERY=[^ ]*\).*/- \1/p' \
|
||||||
"$log" | sort -u
|
"$log" | sort -u
|
||||||
sed -n 's/^\s*\(test\/dst\/[a-z_]*\.dart:.*\)$/- \1/p' "$log" | sort -u | head -20
|
sed -n 's/^\s*\(test\/dst\/[a-z_]*\.dart:.*\)$/- \1/p' "$log" | sort -u | head -20
|
||||||
echo
|
echo
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
|
|
||||||
These suites drive several replicas through randomized operations and
|
These suites drive several replicas through randomized operations and
|
||||||
adversarial delivery, then check properties the engine must hold. A run is
|
adversarial delivery, then check properties the engine must hold. A run is
|
||||||
described by its seed, rounds, workload profile, and graph width. Those inputs
|
described by its seed, rounds, workload profile, graph width, and delivery mode. Those inputs
|
||||||
replay exactly within the same source/SDK revision.
|
replay exactly within the same source/SDK revision.
|
||||||
|
|
||||||
They complement the `integration/` suites rather than replacing them.
|
They complement the `integration/` suites rather than replacing them.
|
||||||
|
|
@ -32,6 +32,7 @@ directory are untagged and run with ordinary `dart test`.
|
||||||
| `DST_SEED_BASE` | unix seconds | First seed; successive seeds increment |
|
| `DST_SEED_BASE` | unix seconds | First seed; successive seeds increment |
|
||||||
| `DST_PROFILE` | sparse | `sparse`, `populated`, or `mixed` (even seeds populated, odd sparse) |
|
| `DST_PROFILE` | sparse | `sparse`, `populated`, or `mixed` (even seeds populated, odd sparse) |
|
||||||
| `DST_GRAPH_WIDTH` | 2 | Rows per table in each populated graph, minimum 2 |
|
| `DST_GRAPH_WIDTH` | 2 | Rows per table in each populated graph, minimum 2 |
|
||||||
|
| `DST_DELIVERY` | full | `full` exports all retained facts; `delta` uses the receiver's committed checkpoint vector |
|
||||||
| `DST_DEBUG_TABLE` | unset | Print every merge touching this table, in delivery order |
|
| `DST_DEBUG_TABLE` | unset | Print every merge touching this table, in delivery order |
|
||||||
|
|
||||||
A divergence is usually explained by which facts a replica had merged when it
|
A divergence is usually explained by which facts a replica had merged when it
|
||||||
|
|
@ -47,7 +48,7 @@ which defects those fixed seeds reach.
|
||||||
Nothing is lost to reproducibility: a seed determines its simulation entirely
|
Nothing is lost to reproducibility: a seed determines its simulation entirely
|
||||||
and a failure prints its replay command. Tests are named by position rather
|
and a failure prints its replay command. Tests are named by position rather
|
||||||
than by seed, so the suite stays stable while the schedules underneath it vary.
|
than by seed, so the suite stays stable while the schedules underneath it vary.
|
||||||
Pin the seed, rounds, profile, and graph width to re-run an exact sweep.
|
Pin the seed, rounds, profile, graph width, and delivery mode to re-run an exact sweep.
|
||||||
|
|
||||||
The consequence to expect is that a run can fail for a defect unrelated to the
|
The consequence to expect is that a run can fail for a defect unrelated to the
|
||||||
change that triggered it. That is the suite doing its job; take the seed from
|
change that triggered it. That is the suite doing its job; take the seed from
|
||||||
|
|
@ -102,27 +103,39 @@ Reorders, delays, redelivers already-merged batches, and isolates incoming
|
||||||
delivery to selected replicas for a few rounds. Sources can still send during
|
delivery to selected replicas for a few rounds. Sources can still send during
|
||||||
this receive isolation; this is not a bidirectional network partition.
|
this receive isolation; this is not a bidirectional network partition.
|
||||||
|
|
||||||
Two moves are deliberately unavailable, both because the merge contract states
|
`full` preserves the original merge convergence experiment: collection exports
|
||||||
that input arrives as a causally complete snapshot of the sender
|
all retained facts, and a receiver's already-delivered keys decide whether another
|
||||||
(`docs/foreign-key-invariants.md`):
|
complete export is worth scheduling. `delta` reads each receiver's production
|
||||||
|
`createSyncSinceHlc` vector under its seeded clock and passes it to the sender's
|
||||||
|
`collectPendingChanges`. Vectors are scoped by receiver, space, and author.
|
||||||
|
|
||||||
- **It never drops permanently.** Dropping manufactures failures outside the
|
Causal completeness is relative to the receiver: prerequisites must already be
|
||||||
contract instead of finding real ones.
|
committed there or accompany the incoming fact. Each delta is collected against
|
||||||
- **It never splits a collected batch.** An earlier version of this harness
|
committed receiver progress, never a queued batch's anticipated progress. Pending
|
||||||
split batches at a random pivot to vary framing. That delivers, for example, a
|
batches can overlap and arrive in any order without relying on one another.
|
||||||
delete without its insert; the engine ignores the orphaned change, the harness
|
Collection and merge run sequentially; this does not simulate concurrent writes
|
||||||
records it as delivered, and the fact is lost - surfacing as a bogus
|
during collection or the continuous sync driver's in-session send cursor.
|
||||||
convergence failure. Chunking in the real protocol sits *below* the merge
|
|
||||||
(`chunked()` emits frames that `collectNextBatch` reassembles until
|
|
||||||
`OfflineSyncEndOfBatch`), so the whole cycle is the causal unit and splitting here
|
|
||||||
models nothing real.
|
|
||||||
|
|
||||||
Delay, reorder, and redelivery are the honest moves - and redelivery is how
|
Neither mode permanently drops facts or splits a collected batch. Wire chunking
|
||||||
idempotence gets probed.
|
belongs below this boundary: the protocol reassembles chunks through
|
||||||
|
`OfflineSyncEndOfBatch` before merging. Delta replay selects a previously merged
|
||||||
|
batch from the same source/target/space edge. Captured payloads are serialized and
|
||||||
|
decoded afresh for each delivery, so later local writes or input mutation cannot
|
||||||
|
change the replay. Full mode retains its original complete-export resends.
|
||||||
|
|
||||||
When the schedule ends, quiescence drains every pending batch and collects only
|
Quiescence clears receive isolation, drains pending batches, and stops scheduling
|
||||||
facts a receiver has not yet seen. It stops introducing deliberate duplicates,
|
deliberate replays. In delta mode it requires empty checkpoint-based collections
|
||||||
so stable replicas cannot exhaust the convergence limit merely through resends.
|
on every directed peer/space exchange. There is no full-history repair before
|
||||||
|
checking convergence or accepted authored facts. The existing export-to-empty
|
||||||
|
mirror check runs afterward and never repairs the simulated replicas.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
DST_DELIVERY=delta DST_SEED_BASE=114 DST_SEEDS=1 DST_ROUNDS=200 DST_PROFILE=populated dart test -P dst
|
||||||
|
```
|
||||||
|
|
||||||
|
The modes need not author identical operations for an identical seed: delivery
|
||||||
|
changes visible state and subsequent generator choices. Fixed collection
|
||||||
|
regressions compare full and delta receivers of the same authored scenario.
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
|
|
@ -197,16 +210,34 @@ The populated profile creates every declared table and FK edge, closes the
|
||||||
person/company/town cycles, and drives unique conflicts, a real tuple exchange,
|
person/company/town cycles, and drives unique conflicts, a real tuple exchange,
|
||||||
restore/redelete, FK retarget/detach, and a constrained refusal. These are actual
|
restore/redelete, FK retarget/detach, and a constrained refusal. These are actual
|
||||||
ORM transactions, each checked immediately for structure, authored preservation,
|
ORM transactions, each checked immediately for structure, authored preservation,
|
||||||
and causal monotonicity. Complete exports distribute the populated spaces before
|
and causal monotonicity. The selected delivery mode distributes the populated
|
||||||
random scheduling starts. The sparse profile retains empty-world exploration.
|
spaces before random scheduling starts; bootstrap naturally includes previously
|
||||||
|
unknown facts. The sparse profile retains empty-world exploration.
|
||||||
|
|
||||||
Every run emits one `DST_METRICS` JSON record, including failed runs. It separates
|
Every run emits one `DST_METRICS` JSON record, including failed runs. Its
|
||||||
|
`topology` identifies `convergence` or `cross_space`. It separates
|
||||||
setup attempts from scheduled commits and records attempted, committed, rejected,
|
setup attempts from scheduled commits and records attempted, committed, rejected,
|
||||||
skipped, unexpected failures, and committed transactions whose oracle failed.
|
skipped, unexpected failures, and committed transactions whose oracle failed.
|
||||||
A run failing during setup reports zero scheduled activity. Paths retain table
|
A run failing during setup reports zero scheduled activity. Paths retain table
|
||||||
and action; network observations include merge counts, duplicate batches, maximum
|
and action; network observations include merge counts, duplicate batches, maximum
|
||||||
batch size, total delivered changes, and receive-isolation events.
|
batch size, total delivered changes, and receive-isolation events.
|
||||||
|
|
||||||
|
Network observations also report checkpoint collections, collections with a
|
||||||
|
nonzero remote-author checkpoint, empty collections, partial batches, explicit
|
||||||
|
replays, and fresh/repeated delivered change keys. A partial batch is nonempty,
|
||||||
|
uses committed remote progress, and is smaller than the unchanged sender's full
|
||||||
|
export. Full exports in delta mode measure omissions only and are never queued.
|
||||||
|
Fresh means the key was not previously delivered to this receiver; it does not
|
||||||
|
claim the fact was absent from local authoring. A repeated key has the same
|
||||||
|
identity, not necessarily identical serialized insertion payload.
|
||||||
|
|
||||||
|
Counters carry `setup.`, `scheduled.`, or `drain.` prefixes according to when the
|
||||||
|
collection or delivery occurred. Exact replays do not count as new partial
|
||||||
|
batches. Delta stress runs (100+ rounds) require at least one scheduled partial
|
||||||
|
batch; setup and final draining cannot satisfy that gate. Shorter runs remain
|
||||||
|
smoke checks. The soak script retains `METRICS.jsonl` even for passing runs,
|
||||||
|
adding a `runId` that joins each record to the run's `id` in `RUNS.tsv`.
|
||||||
|
|
||||||
Coverage counts distinct field/tombstone events rather than repeated snapshot
|
Coverage counts distinct field/tombstone events rather than repeated snapshot
|
||||||
appearances. FK edges and cycles are **authored graph** observations. Unique
|
appearances. FK edges and cycles are **authored graph** observations. Unique
|
||||||
projection coverage reports columns, not a claim that every overlapping index
|
projection coverage reports columns, not a claim that every overlapping index
|
||||||
|
|
@ -219,13 +250,12 @@ also require all declared authored FK edges and the mandatory semantic paths.
|
||||||
Setup commits alone cannot satisfy the scheduled-activity gate.
|
Setup commits alone cannot satisfy the scheduled-activity gate.
|
||||||
|
|
||||||
CI alternates sparse and populated profiles across four consecutive seeds at
|
CI alternates sparse and populated profiles across four consecutive seeds at
|
||||||
200 rounds in both topologies: 4,800 scheduled operation attempts, versus the
|
200 rounds in both topologies and both delivery modes: four separate jobs, each
|
||||||
previous 6,000 attempts spread across fifty shallow 20-round worlds. This keeps
|
with 2,400 scheduled operation attempts and four ten-minute per-seed ceilings
|
||||||
comparable attempt volume while exploring ten times the history depth and
|
within a sixty-minute job. A failure in one job does not cancel another.
|
||||||
connected graphs. Each topology runs in its own sixty-minute CI job with four
|
All jobs also run the ownership-collision controls. The local `melos test-dst`
|
||||||
ten-minute per-seed ceilings; a failure in one job does not cancel the other.
|
command runs both topologies for the selected `DST_DELIVERY` mode; run it once
|
||||||
Both jobs also run the ownership-collision controls. The local `melos test-dst`
|
with `full` and once with `delta` to cover both modes.
|
||||||
command runs both topologies together, so it covers the combined CI matrix.
|
|
||||||
Known engine failures remain failures; CI is intentionally not a PR merge gate.
|
Known engine failures remain failures; CI is intentionally not a PR merge gate.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|
@ -234,5 +264,8 @@ DST_SEED_BASE=62 DST_SEEDS=1 DST_ROUNDS=200 DST_PROFILE=populated DST_GRAPH_WIDT
|
||||||
```
|
```
|
||||||
|
|
||||||
The oracle remains bounded: it does not independently arbitrate all unique
|
The oracle remains bounded: it does not independently arbitrate all unique
|
||||||
winners or FK fixed points. Collector concurrency/checkpoint lifecycle, crash
|
winners or FK fixed points. Concurrent collection, continuous-session send cursors, crash recovery, space
|
||||||
recovery, space grant/revoke, and transport framing remain outside this schedule.
|
grant/revoke, and transport framing remain outside this schedule. Delta mode
|
||||||
|
exercises production resume-vector creation, peer-handshake vector normalization,
|
||||||
|
checkpoint filtering, and the inbound merge primitive that atomically persists
|
||||||
|
per-author received progress; it does not simulate the full sync driver.
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,4 @@
|
||||||
|
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
|
||||||
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
|
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
|
||||||
import 'package:test/test.dart';
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
|
|
@ -56,6 +57,195 @@ void main() {
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
|
group('Given a delta receiver isolated while a parent and child are authored,', () {
|
||||||
|
final ids = DstIds(DstRandom(917));
|
||||||
|
final space = ids.next();
|
||||||
|
final clock = DstClock();
|
||||||
|
late DstReplica source;
|
||||||
|
late DstReplica target;
|
||||||
|
late DstAdversary adversary;
|
||||||
|
late City parent;
|
||||||
|
late List<String> initialCheckpoints;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
source = await DstReplica.create(
|
||||||
|
name: 'source',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
target = await DstReplica.create(
|
||||||
|
name: 'target',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
initialCheckpoints = (await target.checkpoints(
|
||||||
|
space,
|
||||||
|
)).map((hlc) => hlc.toString()).toList();
|
||||||
|
adversary = DstAdversary(_DelayedNewestFirstSchedule(), [
|
||||||
|
source,
|
||||||
|
target,
|
||||||
|
], delivery: DstDeliveryMode.delta);
|
||||||
|
parent = City(id: ids.next(), name: 'parent');
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when complete batches queue and the newest one arrives first,', () {
|
||||||
|
late List<String> queuedCheckpoints;
|
||||||
|
late String expected;
|
||||||
|
final observed = <String>[];
|
||||||
|
late CrdtMergeSet caughtUp;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
Future<void> observe(DstReplica replica) async {
|
||||||
|
if (identical(replica, target)) {
|
||||||
|
observed.add((await DstSnapshot.capture(replica)).renderSpace(space));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(source.session, parent, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await adversary.step(observe);
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await Town.db.insertRow(
|
||||||
|
source.session,
|
||||||
|
Town(id: ids.next(), name: 'child', cityId: parent.id),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await adversary.step(observe);
|
||||||
|
queuedCheckpoints = (await target.checkpoints(
|
||||||
|
space,
|
||||||
|
)).map((hlc) => hlc.toString()).toList();
|
||||||
|
expected = (await DstSnapshot.capture(source)).renderSpace(space);
|
||||||
|
adversary.phase = DstNetworkPhase.drain;
|
||||||
|
await adversary.quiesce(observe);
|
||||||
|
caughtUp = await source.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await target.checkpoints(space),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
'then queuing advances no checkpoint and every delivered snapshot retains the child.',
|
||||||
|
() {
|
||||||
|
expect(queuedCheckpoints, initialCheckpoints);
|
||||||
|
expect(observed, hasLength(greaterThanOrEqualTo(3)));
|
||||||
|
expect(observed, everyElement(expected));
|
||||||
|
expect(caughtUp, isEmpty);
|
||||||
|
expect(adversary.metrics['scheduled.merges'], isNull);
|
||||||
|
expect(adversary.metrics['drain.merges'], greaterThanOrEqualTo(3));
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group(
|
||||||
|
'Given a captured insertion batch and a delta adversary that requests replays,',
|
||||||
|
() {
|
||||||
|
final ids = DstIds(DstRandom(918));
|
||||||
|
final space = ids.next();
|
||||||
|
final clock = DstClock();
|
||||||
|
late DstReplica source;
|
||||||
|
late DstReplica target;
|
||||||
|
late City row;
|
||||||
|
late DstDelivery captured;
|
||||||
|
late CrdtMergeSet original;
|
||||||
|
late DstAdversary adversary;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
source = await DstReplica.create(
|
||||||
|
name: 'source',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
target = await DstReplica.create(
|
||||||
|
name: 'target',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
row = City(id: ids.next(), name: 'original');
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(source.session, row, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
original = await source.collect(space);
|
||||||
|
captured = DstDelivery(
|
||||||
|
source: source,
|
||||||
|
target: target,
|
||||||
|
spaceUuid: space,
|
||||||
|
changes: original,
|
||||||
|
);
|
||||||
|
adversary = DstAdversary(_RepeatedDeliverySchedule(), [
|
||||||
|
source,
|
||||||
|
target,
|
||||||
|
], delivery: DstDeliveryMode.delta)..phase = DstNetworkPhase.setup;
|
||||||
|
await adversary.quiesce((_) async {});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when the source changes and repeated partial deliveries drain,', () {
|
||||||
|
late String actual;
|
||||||
|
late String expected;
|
||||||
|
late String frozenName;
|
||||||
|
late String mutatedName;
|
||||||
|
late int partialBeforeDrain;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
(original.single as CrdtMergeInsert).data = row.copyWith(
|
||||||
|
name: 'mutated input',
|
||||||
|
);
|
||||||
|
final decoded = captured.changes.single as CrdtMergeInsert
|
||||||
|
..data = row.copyWith(name: 'mutated decoded copy');
|
||||||
|
mutatedName = decoded.databaseColumns['name']! as String;
|
||||||
|
frozenName =
|
||||||
|
(captured.changes.single as CrdtMergeInsert).databaseColumns['name']!
|
||||||
|
as String;
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.updateRow(
|
||||||
|
source.session,
|
||||||
|
row.copyWith(name: 'changed'),
|
||||||
|
columns: (t) => [t.name],
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
adversary.phase = DstNetworkPhase.scheduled;
|
||||||
|
await adversary.step((_) async {});
|
||||||
|
await adversary.step((_) async {});
|
||||||
|
await adversary.step((_) async {});
|
||||||
|
partialBeforeDrain = adversary.scheduledPartialBatches;
|
||||||
|
adversary.phase = DstNetworkPhase.drain;
|
||||||
|
await adversary.quiesce((_) async {});
|
||||||
|
actual = (await DstSnapshot.capture(target)).renderSpace(space);
|
||||||
|
expected = (await DstSnapshot.capture(source)).renderSpace(space);
|
||||||
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
'then the captured payload is immutable and scheduled partial merges survive exact replays.',
|
||||||
|
() {
|
||||||
|
expect(mutatedName, 'mutated decoded copy');
|
||||||
|
expect(frozenName, 'original');
|
||||||
|
expect(actual, expected);
|
||||||
|
expect(partialBeforeDrain, greaterThan(0));
|
||||||
|
expect(adversary.scheduledPartialBatches, partialBeforeDrain);
|
||||||
|
expect(adversary.metrics['explicitReplays'], greaterThan(0));
|
||||||
|
expect(adversary.metrics['scheduled.emptyCollections'], greaterThan(0));
|
||||||
|
expect(adversary.metrics['setup.partialBatches'], isNull);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A concrete network schedule: always collect the first replica, always
|
/// A concrete network schedule: always collect the first replica, always
|
||||||
|
|
@ -73,3 +263,32 @@ class _RepeatedDeliverySchedule extends DstRandom {
|
||||||
@override
|
@override
|
||||||
T pick<T>(List<T> items) => items.first;
|
T pick<T>(List<T> items) => items.first;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Isolate the target at each step, collect from the source, and drain newest
|
||||||
|
/// batches first. Every queued collection must use committed receiver progress.
|
||||||
|
class _DelayedNewestFirstSchedule extends DstRandom {
|
||||||
|
_DelayedNewestFirstSchedule() : super(917);
|
||||||
|
|
||||||
|
bool _selectingPartition = false;
|
||||||
|
|
||||||
|
@override
|
||||||
|
bool chance(double probability) {
|
||||||
|
if (probability == DstAdversary.receiveIsolationProbability) {
|
||||||
|
_selectingPartition = true;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return probability == DstAdversary.collectionProbability;
|
||||||
|
}
|
||||||
|
|
||||||
|
@override
|
||||||
|
int between(int min, int max) => max;
|
||||||
|
|
||||||
|
@override
|
||||||
|
T pick<T>(List<T> items) {
|
||||||
|
if (_selectingPartition) {
|
||||||
|
_selectingPartition = false;
|
||||||
|
return items.last;
|
||||||
|
}
|
||||||
|
return items.first is DstDelivery ? items.last : items.first;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -32,11 +32,13 @@ void main() {
|
||||||
rounds: config.rounds,
|
rounds: config.rounds,
|
||||||
profile: config.profile,
|
profile: config.profile,
|
||||||
graphWidth: config.graphWidth,
|
graphWidth: config.graphWidth,
|
||||||
|
delivery: config.delivery,
|
||||||
run: () => runDstSimulation(
|
run: () => runDstSimulation(
|
||||||
seed: seed,
|
seed: seed,
|
||||||
rounds: config.rounds,
|
rounds: config.rounds,
|
||||||
profile: config.profile,
|
profile: config.profile,
|
||||||
graphWidth: config.graphWidth,
|
graphWidth: config.graphWidth,
|
||||||
|
delivery: config.delivery,
|
||||||
topology: DstTopology.singleSpace,
|
topology: DstTopology.singleSpace,
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
|
|
|
||||||
|
|
@ -38,11 +38,13 @@ void main() {
|
||||||
rounds: config.rounds,
|
rounds: config.rounds,
|
||||||
profile: config.profile,
|
profile: config.profile,
|
||||||
graphWidth: config.graphWidth,
|
graphWidth: config.graphWidth,
|
||||||
|
delivery: config.delivery,
|
||||||
run: () => runDstSimulation(
|
run: () => runDstSimulation(
|
||||||
seed: seed,
|
seed: seed,
|
||||||
rounds: config.rounds,
|
rounds: config.rounds,
|
||||||
profile: config.profile,
|
profile: config.profile,
|
||||||
graphWidth: config.graphWidth,
|
graphWidth: config.graphWidth,
|
||||||
|
delivery: config.delivery,
|
||||||
topology: DstTopology.overlappingSpaces,
|
topology: DstTopology.overlappingSpaces,
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,490 @@
|
||||||
|
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
|
||||||
|
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
|
||||||
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
|
import '../integration/test_tools/client_session.dart';
|
||||||
|
import 'framework/dst_random.dart';
|
||||||
|
import 'framework/dst_snapshot.dart';
|
||||||
|
import 'framework/dst_world.dart';
|
||||||
|
|
||||||
|
void main() {
|
||||||
|
initTestClientSession(createSessionPerTest: false);
|
||||||
|
|
||||||
|
group('Given a city already acknowledged by a receiver,', () {
|
||||||
|
final ids = DstIds(DstRandom(420));
|
||||||
|
final clock = DstClock();
|
||||||
|
final space = ids.next();
|
||||||
|
late DstReplica source;
|
||||||
|
late DstReplica target;
|
||||||
|
late DstReplica reference;
|
||||||
|
late City city;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
source = await _replica('source', [space], ids, clock);
|
||||||
|
target = await _replica('target', [space], ids, clock);
|
||||||
|
reference = await _replica('reference', [space], ids, clock);
|
||||||
|
city = City(id: ids.next(), name: 'initial');
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(source.session, city, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await target.merge(await source.collect(space), space);
|
||||||
|
await reference.merge(await source.collect(space), space);
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when its name changes and later the city is deleted,', () {
|
||||||
|
late CrdtMergeSet update;
|
||||||
|
late CrdtMergeSet deletion;
|
||||||
|
late CrdtMergeSet caughtUp;
|
||||||
|
late String updatedTarget;
|
||||||
|
late String updatedReference;
|
||||||
|
late String deletedTarget;
|
||||||
|
late String deletedReference;
|
||||||
|
late String replayedTarget;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.updateRow(
|
||||||
|
source.session,
|
||||||
|
city.copyWith(name: 'changed'),
|
||||||
|
columns: (t) => [t.name],
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
update = await source.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await target.checkpoints(space),
|
||||||
|
);
|
||||||
|
await target.merge(update, space);
|
||||||
|
await reference.merge(await source.collect(space), space);
|
||||||
|
updatedTarget = (await DstSnapshot.capture(target)).renderSpace(space);
|
||||||
|
updatedReference = (await DstSnapshot.capture(reference)).renderSpace(space);
|
||||||
|
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.deleteRow(source.session, city, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
deletion = await source.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await target.checkpoints(space),
|
||||||
|
);
|
||||||
|
await target.merge(deletion, space);
|
||||||
|
await reference.merge(await source.collect(space), space);
|
||||||
|
deletedTarget = (await DstSnapshot.capture(target)).renderSpace(space);
|
||||||
|
deletedReference = (await DstSnapshot.capture(reference)).renderSpace(space);
|
||||||
|
await target.merge(update, space);
|
||||||
|
await target.merge(deletion, space);
|
||||||
|
replayedTarget = (await DstSnapshot.capture(target)).renderSpace(space);
|
||||||
|
caughtUp = await source.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await target.checkpoints(space),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then the update batch omits the acknowledged insertion.', () {
|
||||||
|
expect(update, hasLength(1));
|
||||||
|
expect(update.single, isA<CrdtMergeUpdate>());
|
||||||
|
expect((update.single as CrdtMergeUpdate).value, 'changed');
|
||||||
|
expect(updatedTarget, updatedReference);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then the deletion batch contains only the new tombstone.', () {
|
||||||
|
expect(deletion, hasLength(1));
|
||||||
|
expect(deletion.single, isA<CrdtMergeDelete>());
|
||||||
|
expect(deletedTarget, deletedReference);
|
||||||
|
expect(deletedTarget, contains('HIDDEN'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then replay changes neither the final state nor the empty catch-up.', () {
|
||||||
|
expect(replayedTarget, deletedTarget);
|
||||||
|
expect(caughtUp, isEmpty);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('Given one author writing cities in two spaces,', () {
|
||||||
|
final ids = DstIds(DstRandom(421));
|
||||||
|
final clock = DstClock();
|
||||||
|
final firstSpace = ids.next();
|
||||||
|
final secondSpace = ids.next();
|
||||||
|
late DstReplica source;
|
||||||
|
late DstReplica target;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
source = await _replica('source', [firstSpace, secondSpace], ids, clock);
|
||||||
|
target = await _replica('target', [firstSpace, secondSpace], ids, clock);
|
||||||
|
for (final space in [firstSpace, secondSpace]) {
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(
|
||||||
|
source.session,
|
||||||
|
City(id: ids.next(), name: space.toString()),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when only the second space is acknowledged,', () {
|
||||||
|
late CrdtMergeSet firstPending;
|
||||||
|
late CrdtMergeSet secondPending;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
await target.merge(await source.collect(secondSpace), secondSpace);
|
||||||
|
firstPending = await source.collect(
|
||||||
|
firstSpace,
|
||||||
|
checkpoints: await target.checkpoints(firstSpace),
|
||||||
|
);
|
||||||
|
secondPending = await source.collect(
|
||||||
|
secondSpace,
|
||||||
|
checkpoints: await target.checkpoints(secondSpace),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then the earlier first-space insertion is still pending.', () {
|
||||||
|
expect(firstPending, hasLength(1));
|
||||||
|
expect(firstPending.single.uuidSpaceId, firstSpace);
|
||||||
|
expect(secondPending, isEmpty);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('Given a relay with a newer fact from one author,', () {
|
||||||
|
final ids = DstIds(DstRandom(422));
|
||||||
|
final clock = DstClock();
|
||||||
|
final space = ids.next();
|
||||||
|
late DstReplica older;
|
||||||
|
late DstReplica newer;
|
||||||
|
late DstReplica relay;
|
||||||
|
late DstReplica target;
|
||||||
|
late UuidValue olderRow;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
older = await _replica('older', [space], ids, clock);
|
||||||
|
newer = await _replica('newer', [space], ids, clock);
|
||||||
|
relay = await _replica('relay', [space], ids, clock);
|
||||||
|
target = await _replica('target', [space], ids, clock);
|
||||||
|
olderRow = ids.next();
|
||||||
|
await older.withReplicaClock(
|
||||||
|
() => older.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(
|
||||||
|
older.session,
|
||||||
|
City(id: olderRow, name: 'older offline fact'),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
clock.advance(const Duration(seconds: 1));
|
||||||
|
await newer.withReplicaClock(
|
||||||
|
() => newer.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(
|
||||||
|
newer.session,
|
||||||
|
City(id: ids.next(), name: 'newer fact'),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await relay.merge(await newer.collect(space), space);
|
||||||
|
await target.merge(await relay.collect(space), space);
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when the older author reconnects through the relay,', () {
|
||||||
|
late CrdtMergeSet pending;
|
||||||
|
late String actual;
|
||||||
|
late String expected;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
await relay.merge(await older.collect(space), space);
|
||||||
|
pending = await relay.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await target.checkpoints(space),
|
||||||
|
);
|
||||||
|
await target.merge(pending, space);
|
||||||
|
actual = (await DstSnapshot.capture(target)).renderSpace(space);
|
||||||
|
expected = (await DstSnapshot.capture(relay)).renderSpace(space);
|
||||||
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
'then the older fact crosses both hops without resending the newer fact.',
|
||||||
|
() {
|
||||||
|
expect(pending, hasLength(1));
|
||||||
|
expect(pending.single.uuidRowId, olderRow);
|
||||||
|
expect(pending.single.uuidNodeId, older.nodeUuid);
|
||||||
|
expect(actual, expected);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('Given a receiver that already holds a parent city,', () {
|
||||||
|
final ids = DstIds(DstRandom(423));
|
||||||
|
final clock = DstClock();
|
||||||
|
final space = ids.next();
|
||||||
|
late DstReplica source;
|
||||||
|
late DstReplica target;
|
||||||
|
late City parent;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
source = await _replica('source', [space], ids, clock);
|
||||||
|
target = await _replica('target', [space], ids, clock);
|
||||||
|
parent = City(id: ids.next(), name: 'parent');
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(source.session, parent, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await target.merge(await source.collect(space), space);
|
||||||
|
});
|
||||||
|
|
||||||
|
group(
|
||||||
|
'when a child arrives alone followed by a new parent and child together,',
|
||||||
|
() {
|
||||||
|
late CrdtMergeSet childOnly;
|
||||||
|
late CrdtMergeSet together;
|
||||||
|
late DstSnapshot actual;
|
||||||
|
late DstSnapshot expected;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await Town.db.insertRow(
|
||||||
|
source.session,
|
||||||
|
Town(id: ids.next(), name: 'first child', cityId: parent.id),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
childOnly = await source.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await target.checkpoints(space),
|
||||||
|
);
|
||||||
|
await target.merge(childOnly, space);
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
final nextParent = await City.db.insertRow(
|
||||||
|
source.session,
|
||||||
|
City(id: ids.next(), name: 'second parent'),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
await Town.db.insertRow(
|
||||||
|
source.session,
|
||||||
|
Town(id: ids.next(), name: 'second child', cityId: nextParent.id),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
together = await source.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await target.checkpoints(space),
|
||||||
|
);
|
||||||
|
await target.merge(together, space);
|
||||||
|
actual = await DstSnapshot.capture(target);
|
||||||
|
expected = await DstSnapshot.capture(source);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then each partial batch retains exactly the needed new facts.', () {
|
||||||
|
expect(childOnly, hasLength(1));
|
||||||
|
expect(childOnly.single.tableName, 'town');
|
||||||
|
expect(together, hasLength(2));
|
||||||
|
expect(
|
||||||
|
together.map((change) => change.tableName),
|
||||||
|
unorderedEquals(['city', 'town']),
|
||||||
|
);
|
||||||
|
expect(actual.renderSpace(space), expected.renderSpace(space));
|
||||||
|
expect(DstOracle.invariants(actual), isEmpty);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
group('Given two disconnected replicas claiming the same unique name,', () {
|
||||||
|
final ids = DstIds(DstRandom(424));
|
||||||
|
final clock = DstClock();
|
||||||
|
final space = ids.next();
|
||||||
|
late DstReplica source;
|
||||||
|
late DstReplica other;
|
||||||
|
late DstReplica delta;
|
||||||
|
late DstReplica full;
|
||||||
|
late Unique row;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
source = await _replica('source', [space], ids, clock);
|
||||||
|
other = await _replica('other', [space], ids, clock);
|
||||||
|
delta = await _replica('delta', [space], ids, clock);
|
||||||
|
full = await _replica('full', [space], ids, clock);
|
||||||
|
row = Unique(id: ids.next(), name: 'claim');
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await Unique.db.insertRow(source.session, row, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
clock.advance(const Duration(milliseconds: 1));
|
||||||
|
await other.withReplicaClock(
|
||||||
|
() => other.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await Unique.db.insertRow(
|
||||||
|
other.session,
|
||||||
|
Unique(id: ids.next(), name: 'claim'),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
for (final receiver in [delta, full]) {
|
||||||
|
await receiver.merge(await source.collect(space), space);
|
||||||
|
await receiver.merge(await other.collect(space), space);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when one claim is deleted, restored, and deleted again through deltas,', () {
|
||||||
|
final batches = <CrdtMergeSet>[];
|
||||||
|
final actual = <String>[];
|
||||||
|
final expected = <String>[];
|
||||||
|
late DstSnapshot finalSnapshot;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
// Each transition is a real local transaction with a complete collection
|
||||||
|
// boundary; only the delta receiver omits already acknowledged facts.
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await Unique.db.deleteRow(source.session, row, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
Future<void> deliver() async {
|
||||||
|
final batch = await source.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await delta.checkpoints(space),
|
||||||
|
);
|
||||||
|
batches.add(batch);
|
||||||
|
await delta.merge(batch, space);
|
||||||
|
await full.merge(await source.collect(space), space);
|
||||||
|
actual.add((await DstSnapshot.capture(delta)).renderSpace(space));
|
||||||
|
expected.add((await DstSnapshot.capture(full)).renderSpace(space));
|
||||||
|
}
|
||||||
|
|
||||||
|
await deliver();
|
||||||
|
final hidden = (await DstTable.unique.model.find(
|
||||||
|
source.session,
|
||||||
|
includeHidden: true,
|
||||||
|
spaceUuid: space,
|
||||||
|
)).single;
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await source.session.db.insertRow(hidden, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await deliver();
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await Unique.db.deleteRow(source.session, row, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await deliver();
|
||||||
|
finalSnapshot = await DstSnapshot.capture(delta);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then each projection agrees with the full-history receiver.', () {
|
||||||
|
expect(batches, hasLength(3));
|
||||||
|
expect(batches.first.single, isA<CrdtMergeDelete>());
|
||||||
|
expect(batches.last.single, isA<CrdtMergeDelete>());
|
||||||
|
expect(actual, expected);
|
||||||
|
expect(finalSnapshot.tombstones['unique/${row.id}']!.clFlag, 4);
|
||||||
|
expect(DstOracle.invariants(finalSnapshot), isEmpty);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('Given a batch that collides with a row owned by another space,', () {
|
||||||
|
final ids = DstIds(DstRandom(425));
|
||||||
|
final clock = DstClock();
|
||||||
|
final owner = ids.next();
|
||||||
|
final incoming = ids.next();
|
||||||
|
final collisionId = ids.next();
|
||||||
|
final innocentId = ids.next();
|
||||||
|
late DstReplica source;
|
||||||
|
late DstReplica target;
|
||||||
|
late List<String> before;
|
||||||
|
late CrdtMergeSet batch;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
source = await _replica('source', [incoming], ids, clock);
|
||||||
|
target = await _replica('target', [owner, incoming], ids, clock);
|
||||||
|
await target.withReplicaClock(
|
||||||
|
() => target.session.db.transactionForUser(owner, (tx) async {
|
||||||
|
await City.db.insertRow(
|
||||||
|
target.session,
|
||||||
|
City(id: collisionId, name: 'owner'),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(incoming, (tx) async {
|
||||||
|
await City.db.insertRow(
|
||||||
|
source.session,
|
||||||
|
City(id: innocentId, name: 'must roll back'),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
await City.db.insertRow(
|
||||||
|
source.session,
|
||||||
|
City(id: collisionId, name: 'collision'),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
final checkpoints = await target.checkpoints(incoming);
|
||||||
|
before = checkpoints.map((hlc) => hlc.toString()).toList();
|
||||||
|
batch = await source.collect(incoming, checkpoints: checkpoints);
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when the receiver rejects the complete batch,', () {
|
||||||
|
late Object failure;
|
||||||
|
late List<String> after;
|
||||||
|
late CrdtMergeSet retry;
|
||||||
|
late City? rolledBack;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
failure = await target
|
||||||
|
.merge(batch, incoming)
|
||||||
|
.then<Object>(
|
||||||
|
(_) => 'unexpected success',
|
||||||
|
onError: (Object error) => error,
|
||||||
|
);
|
||||||
|
after = (await target.checkpoints(
|
||||||
|
incoming,
|
||||||
|
)).map((hlc) => hlc.toString()).toList();
|
||||||
|
retry = await source.collect(
|
||||||
|
incoming,
|
||||||
|
checkpoints: await target.checkpoints(incoming),
|
||||||
|
);
|
||||||
|
rolledBack = await City.db.findById(target.session, innocentId);
|
||||||
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
'then no received progress or partial row survives and the entire batch remains pending.',
|
||||||
|
() {
|
||||||
|
expect(batch, hasLength(2));
|
||||||
|
expect(failure, isA<OfflineSyncIntegrityViolationException>());
|
||||||
|
expect(after, before);
|
||||||
|
expect(rolledBack, isNull);
|
||||||
|
expect(retry.map(dstChangeKey), batch.map(dstChangeKey));
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
Future<DstReplica> _replica(
|
||||||
|
String name,
|
||||||
|
List<UuidValue> spaces,
|
||||||
|
DstIds ids,
|
||||||
|
DstClock clock,
|
||||||
|
) => DstReplica.create(
|
||||||
|
name: name,
|
||||||
|
spaceUuids: spaces,
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
|
@ -0,0 +1,297 @@
|
||||||
|
import 'package:serverpod_database/serverpod_database.dart'
|
||||||
|
show DatabaseUniqueViolationException;
|
||||||
|
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
|
||||||
|
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
|
||||||
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
|
import '../integration/test_tools/client_session.dart';
|
||||||
|
import 'framework/dst_random.dart';
|
||||||
|
import 'framework/dst_snapshot.dart';
|
||||||
|
import 'framework/dst_world.dart';
|
||||||
|
|
||||||
|
void main() {
|
||||||
|
initTestClientSession(createSessionPerTest: false);
|
||||||
|
|
||||||
|
group('Given two synchronized UUID claims with one projected conflict,', () {
|
||||||
|
late DstIds ids;
|
||||||
|
late DstClock clock;
|
||||||
|
late UuidValue space;
|
||||||
|
late UuidValue contested;
|
||||||
|
late UuidValue replacement;
|
||||||
|
late DstReplica source;
|
||||||
|
late DstReplica peer;
|
||||||
|
late UniqueUuid winner;
|
||||||
|
late UniqueUuid loser;
|
||||||
|
late DstSnapshot before;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
ids = DstIds(DstRandom(427));
|
||||||
|
clock = DstClock();
|
||||||
|
space = ids.next();
|
||||||
|
contested = ids.next();
|
||||||
|
replacement = ids.next();
|
||||||
|
source = await DstReplica.create(
|
||||||
|
name: 'source',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
peer = await DstReplica.create(
|
||||||
|
name: 'peer',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
winner = UniqueUuid(id: ids.next(), value: contested);
|
||||||
|
loser = UniqueUuid(id: ids.next(), value: contested);
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await UniqueUuid.db.insertRow(source.session, winner, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
clock.advance(const Duration(milliseconds: 1));
|
||||||
|
await peer.withReplicaClock(
|
||||||
|
() => peer.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await UniqueUuid.db.insertRow(peer.session, loser, transaction: tx);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
await source.merge(await peer.collect(space), space);
|
||||||
|
await peer.merge(await source.collect(space), space);
|
||||||
|
before = await DstSnapshot.capture(source);
|
||||||
|
});
|
||||||
|
|
||||||
|
group(
|
||||||
|
'when the winner locally changes to an uncontested UUID and sends only its delta,',
|
||||||
|
() {
|
||||||
|
late DstSnapshot local;
|
||||||
|
late DstSnapshot remote;
|
||||||
|
late CrdtMergeSet delta;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await UniqueUuid.db.updateRow(
|
||||||
|
source.session,
|
||||||
|
winner.copyWith(value: replacement),
|
||||||
|
columns: (t) => [t.value],
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
local = await DstSnapshot.capture(source);
|
||||||
|
delta = await source.collect(
|
||||||
|
space,
|
||||||
|
checkpoints: await peer.checkpoints(space),
|
||||||
|
);
|
||||||
|
await peer.merge(delta, space);
|
||||||
|
remote = await DstSnapshot.capture(peer);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then the old loser reclaims its authored UUID on both replicas.', () {
|
||||||
|
final key = ('unique_uuid', loser.id!, 'value');
|
||||||
|
expect(before.projections[key], isNotNull);
|
||||||
|
expect(delta, hasLength(1));
|
||||||
|
expect(delta.single, isA<CrdtMergeUpdate>());
|
||||||
|
expect(local.projections[key], isNull);
|
||||||
|
expect(remote.projections[key], isNull);
|
||||||
|
expect(local.renderSpace(space), remote.renderSpace(space));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
group('when the winner changes in a batch update without returned rows,', () {
|
||||||
|
late DstSnapshot local;
|
||||||
|
late DstSnapshot remote;
|
||||||
|
late CrdtMergeSet delta;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await UniqueUuid.db.update(
|
||||||
|
source.session,
|
||||||
|
[winner.copyWith(value: replacement)],
|
||||||
|
columns: (t) => [t.value],
|
||||||
|
noReturn: true,
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
local = await DstSnapshot.capture(source);
|
||||||
|
delta = await source.collect(space, checkpoints: await peer.checkpoints(space));
|
||||||
|
await peer.merge(delta, space);
|
||||||
|
remote = await DstSnapshot.capture(peer);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then the released claimant reclaims its value before delta catch-up.', () {
|
||||||
|
final key = ('unique_uuid', loser.id!, 'value');
|
||||||
|
expect(before.projections[key], isNotNull);
|
||||||
|
expect(delta, hasLength(1));
|
||||||
|
expect(delta.single.uuidRowId, winner.id);
|
||||||
|
expect(local.projections[key], isNull);
|
||||||
|
expect(remote.projections[key], isNull);
|
||||||
|
expect(dstValue(local.authoredValue(key)), dstValue(before.authoredValue(key)));
|
||||||
|
expect(local.fieldHlc(key), before.fieldHlc(key));
|
||||||
|
expect(local.renderSpace(space), remote.renderSpace(space));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when a limited predicate update changes only the first claim,', () {
|
||||||
|
late DstSnapshot local;
|
||||||
|
late DstSnapshot remote;
|
||||||
|
late CrdtMergeSet delta;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await UniqueUuid.db.updateWhere(
|
||||||
|
source.session,
|
||||||
|
columnValues: (t) => [t.value(replacement)],
|
||||||
|
where: (t) => t.id.inSet(<UuidValue>{winner.id!, loser.id!}),
|
||||||
|
orderBy: (t) => t.id,
|
||||||
|
limit: 1,
|
||||||
|
noReturn: true,
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
local = await DstSnapshot.capture(source);
|
||||||
|
delta = await source.collect(space, checkpoints: await peer.checkpoints(space));
|
||||||
|
await peer.merge(delta, space);
|
||||||
|
remote = await DstSnapshot.capture(peer);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then the released claimant reclaims its value before delta catch-up.', () {
|
||||||
|
final key = ('unique_uuid', loser.id!, 'value');
|
||||||
|
expect(before.projections[key], isNotNull);
|
||||||
|
expect(delta, hasLength(1));
|
||||||
|
expect(delta.single.uuidRowId, winner.id);
|
||||||
|
expect(local.projections[key], isNull);
|
||||||
|
expect(remote.projections[key], isNull);
|
||||||
|
expect(dstValue(local.authoredValue(key)), dstValue(before.authoredValue(key)));
|
||||||
|
expect(local.fieldHlc(key), before.fieldHlc(key));
|
||||||
|
expect(local.renderSpace(space), remote.renderSpace(space));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when a predicate update skips the winner and selects the next claim,', () {
|
||||||
|
late List<UniqueUuid> updated;
|
||||||
|
late DstSnapshot local;
|
||||||
|
late DstSnapshot remote;
|
||||||
|
late CrdtMergeSet delta;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
updated = await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
return UniqueUuid.db.updateWhere(
|
||||||
|
source.session,
|
||||||
|
columnValues: (t) => [t.value(replacement)],
|
||||||
|
where: (t) => t.id.inSet(<UuidValue>{winner.id!, loser.id!}),
|
||||||
|
orderBy: (t) => t.id,
|
||||||
|
offset: 1,
|
||||||
|
limit: 1,
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
local = await DstSnapshot.capture(source);
|
||||||
|
delta = await source.collect(space, checkpoints: await peer.checkpoints(space));
|
||||||
|
await peer.merge(delta, space);
|
||||||
|
remote = await DstSnapshot.capture(peer);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then exactly the selected loser changes and is returned.', () {
|
||||||
|
final loserKey = ('unique_uuid', loser.id!, 'value');
|
||||||
|
final winnerKey = ('unique_uuid', winner.id!, 'value');
|
||||||
|
expect(before.projections[loserKey], isNotNull);
|
||||||
|
expect(updated, hasLength(1));
|
||||||
|
expect(updated.single.id, loser.id);
|
||||||
|
expect(updated.single.value, replacement);
|
||||||
|
expect(delta, hasLength(1));
|
||||||
|
expect(delta.single.uuidRowId, loser.id);
|
||||||
|
expect(local.projections[loserKey], isNull);
|
||||||
|
expect(dstValue(local.authoredValue(winnerKey)), dstValue(contested));
|
||||||
|
expect(local.fieldHlc(winnerKey), before.fieldHlc(winnerKey));
|
||||||
|
expect(local.renderSpace(space), remote.renderSpace(space));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when an upsert changes the winning claim,', () {
|
||||||
|
late DstSnapshot local;
|
||||||
|
late DstSnapshot remote;
|
||||||
|
late CrdtMergeSet delta;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
await source.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await UniqueUuid.db.upsertRow(
|
||||||
|
source.session,
|
||||||
|
winner.copyWith(value: replacement),
|
||||||
|
conflictColumns: (t) => [t.id],
|
||||||
|
updateColumns: (t) => [t.value],
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
local = await DstSnapshot.capture(source);
|
||||||
|
delta = await source.collect(space, checkpoints: await peer.checkpoints(space));
|
||||||
|
await peer.merge(delta, space);
|
||||||
|
remote = await DstSnapshot.capture(peer);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then the released claimant reclaims its value before delta catch-up.', () {
|
||||||
|
final key = ('unique_uuid', loser.id!, 'value');
|
||||||
|
expect(before.projections[key], isNotNull);
|
||||||
|
expect(delta, hasLength(1));
|
||||||
|
expect(delta.single.uuidRowId, winner.id);
|
||||||
|
expect(local.projections[key], isNull);
|
||||||
|
expect(remote.projections[key], isNull);
|
||||||
|
expect(dstValue(local.authoredValue(key)), dstValue(before.authoredValue(key)));
|
||||||
|
expect(local.fieldHlc(key), before.fieldHlc(key));
|
||||||
|
expect(local.renderSpace(space), remote.renderSpace(space));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when a predicate update fails its physical unique constraint,', () {
|
||||||
|
late Object failure;
|
||||||
|
late DstSnapshot after;
|
||||||
|
late List<String> checkpointBefore;
|
||||||
|
late List<String> checkpointAfter;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
checkpointBefore = (await source.checkpoints(
|
||||||
|
space,
|
||||||
|
)).map((hlc) => hlc.toString()).toList();
|
||||||
|
failure = await source
|
||||||
|
.withReplicaClock(
|
||||||
|
() => source.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await UniqueUuid.db.updateWhere(
|
||||||
|
source.session,
|
||||||
|
columnValues: (t) => [t.value(replacement)],
|
||||||
|
where: (t) => t.id.inSet(<UuidValue>{winner.id!, loser.id!}),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.then<Object>(
|
||||||
|
(_) => 'unexpected success',
|
||||||
|
onError: (Object error) => error,
|
||||||
|
);
|
||||||
|
after = await DstSnapshot.capture(source);
|
||||||
|
checkpointAfter = (await source.checkpoints(
|
||||||
|
space,
|
||||||
|
)).map((hlc) => hlc.toString()).toList();
|
||||||
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
'then pre-write closure capture changes no domain, authored metadata, or progress.',
|
||||||
|
() {
|
||||||
|
expect(before.projections[('unique_uuid', loser.id!, 'value')], isNotNull);
|
||||||
|
expect(failure, isA<DatabaseUniqueViolationException>());
|
||||||
|
expect(after.renderSpace(space), before.renderSpace(space));
|
||||||
|
expect(after.renderRawMetadata(), before.renderRawMetadata());
|
||||||
|
expect(checkpointAfter, checkpointBefore);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,84 @@
|
||||||
|
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
|
||||||
|
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
|
||||||
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
|
import '../integration/test_tools/client_session.dart';
|
||||||
|
import 'framework/dst_random.dart';
|
||||||
|
import 'framework/dst_world.dart';
|
||||||
|
|
||||||
|
void main() {
|
||||||
|
initTestClientSession(createSessionPerTest: false);
|
||||||
|
|
||||||
|
group('Given two local writes committed within the same clock millisecond,', () {
|
||||||
|
final ids = DstIds(DstRandom(426));
|
||||||
|
final clock = DstClock();
|
||||||
|
final space = ids.next();
|
||||||
|
late DstReplica author;
|
||||||
|
late DstReplica peer;
|
||||||
|
late Hlc latestAuthored;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
author = await DstReplica.create(
|
||||||
|
name: 'author',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
peer = await DstReplica.create(
|
||||||
|
name: 'peer',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
for (final name in ['first', 'second']) {
|
||||||
|
await author.withReplicaClock(
|
||||||
|
() => author.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(
|
||||||
|
author.session,
|
||||||
|
City(id: ids.next(), name: name),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
final changes = await author.collect(space);
|
||||||
|
latestAuthored = changes.maxHlc!;
|
||||||
|
await peer.merge(changes, space);
|
||||||
|
});
|
||||||
|
|
||||||
|
group(
|
||||||
|
'when the author reconnects before its wall clock advances and after it moves backward,',
|
||||||
|
() {
|
||||||
|
late Hlc sameMillisecond;
|
||||||
|
late Hlc backwardClock;
|
||||||
|
late CrdtMergeSet sameMillisecondEcho;
|
||||||
|
late CrdtMergeSet backwardClockEcho;
|
||||||
|
|
||||||
|
setUpAll(() async {
|
||||||
|
var checkpoints = await author.checkpoints(space);
|
||||||
|
sameMillisecond = checkpoints.singleWhere(
|
||||||
|
(hlc) => hlc.nodeId == author.nodeUuid,
|
||||||
|
);
|
||||||
|
sameMillisecondEcho = await peer.collect(space, checkpoints: checkpoints);
|
||||||
|
clock.advance(const Duration(seconds: -1));
|
||||||
|
checkpoints = await author.checkpoints(space);
|
||||||
|
backwardClock = checkpoints.singleWhere(
|
||||||
|
(hlc) => hlc.nodeId == author.nodeUuid,
|
||||||
|
);
|
||||||
|
backwardClockEcho = await peer.collect(space, checkpoints: checkpoints);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then both resume vectors cover the persisted logical counter.', () {
|
||||||
|
expect(latestAuthored.counter, greaterThan(0));
|
||||||
|
expect(sameMillisecond, greaterThanOrEqualTo(latestAuthored));
|
||||||
|
expect(backwardClock, greaterThanOrEqualTo(latestAuthored));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("then neither resume causes the peer to echo the author's own facts.", () {
|
||||||
|
expect(sameMillisecondEcho, isEmpty);
|
||||||
|
expect(backwardClockEcho, isEmpty);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,204 @@
|
||||||
|
import 'package:serverpod_offline_sync_server/serverpod_offline_sync_server.dart';
|
||||||
|
import 'package:serverpod_offline_sync_test_client/serverpod_offline_sync_test_client.dart';
|
||||||
|
import 'package:test/test.dart';
|
||||||
|
|
||||||
|
import '../integration/test_tools/client_session.dart';
|
||||||
|
import 'framework/dst_random.dart';
|
||||||
|
import 'framework/dst_world.dart';
|
||||||
|
|
||||||
|
void main() {
|
||||||
|
initTestClientSession(createSessionPerTest: false);
|
||||||
|
|
||||||
|
group("Given a relay with its own older fact and another author's newer fact,", () {
|
||||||
|
late DstIds ids;
|
||||||
|
late DstClock clock;
|
||||||
|
late UuidValue space;
|
||||||
|
late DstReplica author;
|
||||||
|
late DstReplica relay;
|
||||||
|
late DstReplica receiver;
|
||||||
|
late CrdtMergeSet initial;
|
||||||
|
|
||||||
|
Future<void> insert(DstReplica replica, String name) => replica.withReplicaClock(
|
||||||
|
() => replica.session.db.transactionForUser(space, (tx) async {
|
||||||
|
await City.db.insertRow(
|
||||||
|
replica.session,
|
||||||
|
City(id: ids.next(), name: name),
|
||||||
|
transaction: tx,
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
ids = DstIds(DstRandom(990));
|
||||||
|
clock = DstClock();
|
||||||
|
space = ids.next();
|
||||||
|
author = await DstReplica.create(
|
||||||
|
name: 'author',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
relay = await DstReplica.create(
|
||||||
|
name: 'relay',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
receiver = await DstReplica.create(
|
||||||
|
name: 'receiver',
|
||||||
|
spaceUuids: [space],
|
||||||
|
nodeUuid: ids.next(),
|
||||||
|
clock: clock.clock,
|
||||||
|
);
|
||||||
|
await insert(relay, 'relay fact');
|
||||||
|
clock.advance(const Duration(seconds: 1));
|
||||||
|
await insert(author, 'author fact');
|
||||||
|
await relay.merge(await author.collect(space), space);
|
||||||
|
initial = await relay.collect(space);
|
||||||
|
await receiver.merge(initial, space);
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when the receiver reconnects after the production inbound merge,', () {
|
||||||
|
late List<Hlc> vector;
|
||||||
|
late CrdtMergeSet pending;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
vector = await receiver.checkpoints(space);
|
||||||
|
pending = await relay.collect(space, checkpoints: vector);
|
||||||
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
'then each author has its own checkpoint and the relay has nothing to resend.',
|
||||||
|
() {
|
||||||
|
expect(initial, hasLength(2));
|
||||||
|
expect(initial.maxHlc!.nodeId, author.nodeUuid);
|
||||||
|
expect(vector.map((hlc) => hlc.nodeId).toSet(), hasLength(3));
|
||||||
|
expect(vector, hasLength(3));
|
||||||
|
expect(
|
||||||
|
vector.singleWhere((hlc) => hlc.nodeId == relay.nodeUuid),
|
||||||
|
initial.singleWhere((change) => change.uuidNodeId == relay.nodeUuid).hlc,
|
||||||
|
);
|
||||||
|
expect(pending, isEmpty);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
group('when a stale session echoes a receiver fact before another local write,', () {
|
||||||
|
late CrdtMergeSet echo;
|
||||||
|
late CrdtMergeSet pending;
|
||||||
|
late List<Hlc> vector;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
final stale = await receiver.checkpoints(space);
|
||||||
|
clock.advance(const Duration(seconds: 1));
|
||||||
|
await insert(receiver, 'receiver fact one');
|
||||||
|
await relay.merge(
|
||||||
|
await receiver.collect(space, checkpoints: await relay.checkpoints(space)),
|
||||||
|
space,
|
||||||
|
);
|
||||||
|
echo = await relay.collect(space, checkpoints: stale);
|
||||||
|
await receiver.merge(echo, space);
|
||||||
|
clock.advance(const Duration(seconds: 1));
|
||||||
|
await insert(receiver, 'receiver fact two');
|
||||||
|
await relay.merge(
|
||||||
|
await receiver.collect(space, checkpoints: await relay.checkpoints(space)),
|
||||||
|
space,
|
||||||
|
);
|
||||||
|
vector = await receiver.checkpoints(space);
|
||||||
|
pending = await relay.collect(space, checkpoints: vector);
|
||||||
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
'then reconnecting neither resends the relay fact nor echoes either local fact.',
|
||||||
|
() {
|
||||||
|
expect(echo, hasLength(1));
|
||||||
|
expect(echo.single.uuidNodeId, receiver.nodeUuid);
|
||||||
|
expect(vector, hasLength(3));
|
||||||
|
expect(vector.map((hlc) => hlc.nodeId).toSet(), hasLength(3));
|
||||||
|
expect(pending, isEmpty);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
group("when the checkpoint API is given a relayed author under the peer's id,", () {
|
||||||
|
late Object failure;
|
||||||
|
late List<Hlc> before;
|
||||||
|
late List<Hlc> after;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
before = await receiver.checkpoints(space);
|
||||||
|
failure = await receiver.session.db
|
||||||
|
.recordSyncCheckpoint(relay.nodeUuid, initial.maxHlc!, userId: space)
|
||||||
|
.then<Object>(
|
||||||
|
(_) => 'unexpected success',
|
||||||
|
onError: (Object error) => error,
|
||||||
|
);
|
||||||
|
after = await receiver.checkpoints(space);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('then the invalid checkpoint is rejected without changing progress.', () {
|
||||||
|
expect(initial.maxHlc!.nodeId, author.nodeUuid);
|
||||||
|
expect(failure, isA<ArgumentError>());
|
||||||
|
expect(after, before);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
group(
|
||||||
|
'when a stored relay checkpoint carries the wrong author from older bookkeeping,',
|
||||||
|
() {
|
||||||
|
late List<Hlc> beforeRepair;
|
||||||
|
late List<Hlc> afterRepair;
|
||||||
|
late CrdtMergeSet repair;
|
||||||
|
late CrdtMergeSet pending;
|
||||||
|
late Hlc? persisted;
|
||||||
|
|
||||||
|
setUp(() async {
|
||||||
|
final nodes = await OfflineSyncSpaceNode.db.find(
|
||||||
|
receiver.rawSession,
|
||||||
|
include: OfflineSyncSpaceNode.include(node: CrdtNode.include()),
|
||||||
|
);
|
||||||
|
final relayNode = nodes.singleWhere(
|
||||||
|
(node) => node.node!.uuidNodeId == relay.nodeUuid,
|
||||||
|
);
|
||||||
|
await OfflineSyncSpaceNode.db.updateRow(
|
||||||
|
receiver.rawSession,
|
||||||
|
relayNode.copyWith(lastReceivedHlc: initial.maxHlc),
|
||||||
|
columns: (t) => [t.lastReceivedHlc],
|
||||||
|
);
|
||||||
|
beforeRepair = await receiver.checkpoints(space);
|
||||||
|
repair = await relay.collect(space, checkpoints: beforeRepair);
|
||||||
|
await receiver.merge(repair, space);
|
||||||
|
afterRepair = await receiver.checkpoints(space);
|
||||||
|
persisted = (await OfflineSyncSpaceNode.db.findById(
|
||||||
|
receiver.rawSession,
|
||||||
|
relayNode.id!,
|
||||||
|
))!.lastReceivedHlc;
|
||||||
|
pending = await relay.collect(space, checkpoints: afterRepair);
|
||||||
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
'then one safe retransmission repairs the persisted author tag and settles.',
|
||||||
|
() {
|
||||||
|
final relayHlc = initial
|
||||||
|
.singleWhere((change) => change.uuidNodeId == relay.nodeUuid)
|
||||||
|
.hlc;
|
||||||
|
expect(initial.maxHlc, greaterThan(relayHlc));
|
||||||
|
expect(beforeRepair, hasLength(3));
|
||||||
|
expect(
|
||||||
|
beforeRepair.singleWhere((hlc) => hlc.nodeId == relay.nodeUuid),
|
||||||
|
Hlc.zero(relay.nodeUuid),
|
||||||
|
);
|
||||||
|
expect(repair, hasLength(1));
|
||||||
|
expect(repair.single.uuidNodeId, relay.nodeUuid);
|
||||||
|
expect(persisted, relayHlc);
|
||||||
|
expect(
|
||||||
|
afterRepair.singleWhere((hlc) => hlc.nodeId == relay.nodeUuid),
|
||||||
|
relayHlc,
|
||||||
|
);
|
||||||
|
expect(pending, isEmpty);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
@ -36,13 +36,14 @@ void main() {
|
||||||
test(
|
test(
|
||||||
'Given a property violation from seed 114 at 80 rounds, '
|
'Given a property violation from seed 114 at 80 rounds, '
|
||||||
'when its failure is reported, '
|
'when its failure is reported, '
|
||||||
'then the replay command preserves the seed, depth, profile, and graph width.',
|
'then the replay command preserves the seed, depth, profile, graph width, and delivery mode.',
|
||||||
() {
|
() {
|
||||||
final failure = DstPropertyFailure(
|
final failure = DstPropertyFailure(
|
||||||
seed: 114,
|
seed: 114,
|
||||||
rounds: 80,
|
rounds: 80,
|
||||||
profile: DstProfile.populated,
|
profile: DstProfile.populated,
|
||||||
graphWidth: 3,
|
graphWidth: 3,
|
||||||
|
delivery: DstDeliveryMode.delta,
|
||||||
violations: [(property: 'foreignKeyClosure', detail: 'orphan')],
|
violations: [(property: 'foreignKeyClosure', detail: 'orphan')],
|
||||||
);
|
);
|
||||||
|
|
||||||
|
|
@ -51,7 +52,7 @@ void main() {
|
||||||
expect(
|
expect(
|
||||||
message,
|
message,
|
||||||
'DST property failure (seed 114)\n'
|
'DST property failure (seed 114)\n'
|
||||||
'Replay: DST_SEED_BASE=114 DST_SEEDS=1 DST_ROUNDS=80 DST_PROFILE=populated DST_GRAPH_WIDTH=3 dart test -P dst\n'
|
'Replay: DST_SEED_BASE=114 DST_SEEDS=1 DST_ROUNDS=80 DST_PROFILE=populated DST_GRAPH_WIDTH=3 DST_DELIVERY=delta dart test -P dst\n'
|
||||||
'- foreignKeyClosure: orphan\n',
|
'- foreignKeyClosure: orphan\n',
|
||||||
);
|
);
|
||||||
},
|
},
|
||||||
|
|
@ -60,7 +61,7 @@ void main() {
|
||||||
test(
|
test(
|
||||||
'Given an unexpected exception from seed 115 at 40 rounds, '
|
'Given an unexpected exception from seed 115 at 40 rounds, '
|
||||||
'when its failure is reported, '
|
'when its failure is reported, '
|
||||||
'then the replay command preserves the seed, depth, profile, and graph width.',
|
'then the replay command preserves the seed, depth, profile, graph width, and delivery mode.',
|
||||||
() async {
|
() async {
|
||||||
final failure = await runWithSeedReported<Object>(
|
final failure = await runWithSeedReported<Object>(
|
||||||
index: 1,
|
index: 1,
|
||||||
|
|
@ -74,7 +75,7 @@ void main() {
|
||||||
expect(
|
expect(
|
||||||
message,
|
message,
|
||||||
'Bad state: Simulation 1 (seed 115) failed\n'
|
'Bad state: Simulation 1 (seed 115) failed\n'
|
||||||
'Replay: DST_SEED_BASE=115 DST_SEEDS=1 DST_ROUNDS=40 DST_PROFILE=sparse DST_GRAPH_WIDTH=2 dart test -P dst\n'
|
'Replay: DST_SEED_BASE=115 DST_SEEDS=1 DST_ROUNDS=40 DST_PROFILE=sparse DST_GRAPH_WIDTH=2 DST_DELIVERY=full dart test -P dst\n'
|
||||||
'Bad state: unexpected',
|
'Bad state: unexpected',
|
||||||
);
|
);
|
||||||
},
|
},
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue