fix: Rebuild foreign key and unique projection from authored facts (#115)
This commit is contained in:
Marcelo Mendonça Soares
GitHub
parent
c78af247fe
commit
519a8efca0
83 changed files with 49203 additions and 3053 deletions
|
|
@ -31,13 +31,6 @@ not new user field updates emitted opportunistically during merge.
|
|||
reject the local delete while visible children exist; `SET NULL` and
|
||||
`SET DEFAULT` update child FK fields and their ordinary `CrdtDataField` HLCs;
|
||||
`CASCADE` records user-delete tombstones for the visible cascade descendants.
|
||||
- Keep the user/attempted FK value as the CRDT field value for authored
|
||||
operations.
|
||||
- During merge projection, materialize a separate visible FK value only when FK
|
||||
repair requires it. `CrdtDataForeignKey.attemptedValue` is the durable FK
|
||||
field-value payload for FK columns because `CrdtDataField` stores only HLC
|
||||
metadata. `visibleValue`, `hasOverride`, and `overrideReason` are projection
|
||||
metadata, not business conflict classes.
|
||||
- Do not create ordinary user field updates for `SET NULL` or `SET DEFAULT`
|
||||
repairs during merge. Doing so makes repair HLCs depend on merge timing. This
|
||||
does not apply to locally initiated `SET NULL` or `SET DEFAULT` actions,
|
||||
|
|
@ -91,63 +84,14 @@ would cascade-delete `B`, but `B` has a visible `RESTRICT` child `C`, then `A`,
|
|||
`B`, and `C` remain visible. The result must not depend on whether cascade or
|
||||
restrict edges are evaluated first internally.
|
||||
|
||||
## Projection Metadata
|
||||
## Combined Projection Boundary
|
||||
|
||||
`CrdtDataForeignKey` should represent the FK field payload and materialized FK
|
||||
projection for one `CrdtDataField`:
|
||||
|
||||
- `attemptedValue`: the current user-attempted FK value carried by the CRDT
|
||||
field. It may be null for nullable FKs. This is durable FK field-value storage
|
||||
and is part of the CRDT facts for FK columns.
|
||||
- `visibleValue`: the value currently materialized into the domain row when an
|
||||
override is active (`overrideReason` is non-null).
|
||||
- `overrideReason`: the authoritative stored indicator that an override is
|
||||
active, and the cause. Non-null means an override is active; null means no
|
||||
override (the domain row carries `attemptedValue`). Values: `setNull` — the
|
||||
parent was hidden/deleted and the column was set null by the FK action;
|
||||
`setDefault` — the column was set to a default FK value; `missingParent` —
|
||||
the parent is hidden or missing and cannot be repaired, so the row is hidden
|
||||
by projection. The derived `hasOverride` bool (`overrideReason != null`) is
|
||||
available as a convenience getter.
|
||||
|
||||
`visibleValue` is meaningful only when `overrideReason` is non-null. When
|
||||
`overrideReason` is null the domain row carries `attemptedValue` directly; when
|
||||
`overrideReason` is non-null and `visibleValue` is null it means "materialize
|
||||
null".
|
||||
|
||||
The resolver may use `visibleValue` and `overrideReason` for efficient
|
||||
materialization and write deduplication. It must not use those projection fields
|
||||
as the authority for FK conflict decisions; the authority is the deterministic
|
||||
resolver over merged CRDT row, field, tombstone, and FK attempt facts.
|
||||
|
||||
A local full-row update (no column narrowing) that writes an FK value equal to
|
||||
the materialized projected value is a passthrough of the repair, not an authored
|
||||
FK change: the existing override and attempted value are preserved and the FK
|
||||
field HLC does not advance. Writing a different FK value, or updating the FK
|
||||
column with narrowed columns (even when the written value equals the projected
|
||||
value), authors the change as an ordinary user field fact: the FK field HLC
|
||||
advances, `attemptedValue` becomes the new value, and `overrideReason` becomes
|
||||
null.
|
||||
|
||||
## Merge Pipeline
|
||||
|
||||
1. Merge CRDT row, field, and tombstone facts by their existing HLC/CLFlag
|
||||
rules.
|
||||
2. Identify whether inserts, updates, deletes, FK columns, parent reference
|
||||
columns, or existing FK projection records can affect FK state. Skip FK
|
||||
projection when the merge touches none of them.
|
||||
3. For FK-affecting work, expand to the FK closure in both directions: parents
|
||||
needed by children and children affected by parent visibility. A full graph
|
||||
recomputation may be used instead of the minimal closure.
|
||||
4. Compute effective row visibility and visible FK values to a fixed point.
|
||||
5. Apply only the materialized differences:
|
||||
- update domain FK columns when projection differs from attempted value;
|
||||
- update `CrdtDataForeignKey` projection metadata;
|
||||
- update system tombstones only for deterministic derived visibility where
|
||||
that is part of the chosen storage model.
|
||||
6. Re-run unique conflict projection after FK projection if FK repairs can
|
||||
change unique columns, or define the opposite stage order and prove it with
|
||||
the full recomputation oracle.
|
||||
The attempted-value schema, hidden-row lifecycle, exact FK-before-unique
|
||||
pipeline, atomic materialization rules, and rebuild behavior are defined once in
|
||||
[Rebuildable FK and unique projection model](projection-model.md).
|
||||
This document adds only the FK requirement: compute effective visibility and an
|
||||
FK-safe candidate to the affected-closure fixed point without treating derived
|
||||
unique output as an FK fact.
|
||||
|
||||
## Invariants
|
||||
|
||||
|
|
@ -235,18 +179,9 @@ null.
|
|||
|
||||
### Cross-Cutting
|
||||
|
||||
13. Given an FK repair changes a unique-indexed column, when unique projection
|
||||
and FK projection are recomputed, then the documented stage order prevents
|
||||
visible unique and FK violations.
|
||||
|
||||
14. Given the same remote operations arrive in one batch or several smaller
|
||||
batches, when all operations have merged, then visible rows, visible FK
|
||||
values, FK projection metadata, unique projection, and tombstones are the
|
||||
same.
|
||||
|
||||
15. Given a merge touches only a non-indexed, non-FK field that is not a parent
|
||||
reference column, then FK projection is skipped and the visible FK graph is
|
||||
unchanged.
|
||||
The combined FK/unique, hidden-row restoration, batching, and unrelated-update
|
||||
cases are centralized under the
|
||||
[projection model verification gates](projection-model.md#verification-gates).
|
||||
|
||||
## What These Tests Prove
|
||||
|
||||
|
|
|
|||
326
docs/projection-model.md
Normal file
326
docs/projection-model.md
Normal file
|
|
@ -0,0 +1,326 @@
|
|||
# Rebuildable FK and unique projection model
|
||||
|
||||
Given the same authored CRDT facts and synchronized schema, every replica must
|
||||
rebuild the same domain rows and projection metadata, including hidden rows.
|
||||
The result must not depend on arrival order, batching, local history, or
|
||||
bootstrap from another replica.
|
||||
|
||||
Related documents:
|
||||
|
||||
- [Foreign-key projection invariants](foreign-key-invariants.md)
|
||||
- [Unique-constraint invariants](unique-constraint-invariants.md)
|
||||
- [Outbound collection consistency](outbound-collection-consistency.md)
|
||||
|
||||
## Decisions
|
||||
|
||||
- Domain columns are materialized projection whenever FK or unique processing
|
||||
changes an authored value. The authored value must remain available.
|
||||
- `CrdtDataAttemptedValue` is the single, generic attempted-value authority for
|
||||
FK, unique, and fields governed by both. Projectors never keep independent
|
||||
attempted values.
|
||||
- FK-safe candidates are ephemeral planner output and are never persisted;
|
||||
`CrdtDataAttemptedValue` also carries a generic diagnostic
|
||||
`projectionReason`.
|
||||
- Hidden rows participate in FK and release projection. They may regain an
|
||||
FK-safe candidate, but are always released before unique conflict grouping
|
||||
and never participate in winner selection.
|
||||
- FK projection runs before unique projection. Unique projection consumes
|
||||
FK-safe candidates and effective visibility, and produces final domain values.
|
||||
- Projection never emits an authored field update or advances its HLC.
|
||||
- Physical FK and unique constraints remain correctness backstops.
|
||||
- Outbound sync sends authored values, never projected or temporary values.
|
||||
- Persisted schema types are validated eagerly at initialization. A migration
|
||||
hook may automate conversion later, but is not required for safety.
|
||||
|
||||
## Storage ownership
|
||||
|
||||
### Authored facts
|
||||
|
||||
`CrdtDataField` owns the field HLC. Its authored value is read as:
|
||||
|
||||
```text
|
||||
attemptedValue(field) =
|
||||
CrdtDataAttemptedValue.value, if that row exists
|
||||
domainRow[field], otherwise
|
||||
```
|
||||
|
||||
`CrdtDataAttemptedValue` is a sparse one-to-one dependent of
|
||||
`CrdtDataField`:
|
||||
|
||||
```yaml
|
||||
# Backlink added to CrdtDataField:
|
||||
attemptedValue: CrdtDataAttemptedValue?, relation(optional, name=field_attempted_value)
|
||||
|
||||
class: CrdtDataAttemptedValue
|
||||
table: crdt_data_attempted_value
|
||||
database: all
|
||||
fields:
|
||||
fieldId: int, unique
|
||||
field: CrdtDataField?, relation(name=field_attempted_value, field=fieldId, onDelete=Cascade)
|
||||
value: dynamic, serializationDataType=jsonb
|
||||
projectionReason: CrdtProjectionReason
|
||||
```
|
||||
|
||||
Use Serverpod's existing dynamic `{className, data}` representation, as it
|
||||
preserves `String`, `UuidValue`, null, and other supported Serverpod values
|
||||
independently of the database dialect.
|
||||
|
||||
Create or update the row whenever the final domain value differs from the
|
||||
authored value. Remove it only when the domain value again equals the authored
|
||||
value and no projector still needs an override. At that point the domain column
|
||||
itself safely carries the authored value. No changes to the `CrdtDataField`
|
||||
existence policy, which is whenever its HLC is newer than the row HLC.
|
||||
|
||||
### Diagnostic projection reason
|
||||
|
||||
`projectionReason` replaces the FK-specific override enum and can describe any
|
||||
projector, for example:
|
||||
|
||||
```yaml
|
||||
enum: CrdtProjectionReason
|
||||
serialized: byIndex
|
||||
values:
|
||||
- foreignKeySetNull
|
||||
- foreignKeySetDefault
|
||||
- foreignKeyMissingParent
|
||||
- uniqueConflict
|
||||
- hiddenUniqueRelease
|
||||
```
|
||||
|
||||
It is diagnostic-only derived state. Projection decisions must never read it;
|
||||
the planner recomputes from authored facts and schema, then rewrites the reason
|
||||
with the final domain value. Because it is singular, it records the terminal
|
||||
stage that selected that value. Earlier causes remain reconstructible rather
|
||||
than becoming combinatorial enum variants.
|
||||
|
||||
The row therefore deliberately contains data with two lifetimes:
|
||||
|
||||
- `value` is a durable authored fact and must survive rebuilds; and
|
||||
- `projectionReason` is disposable and may be recomputed in place.
|
||||
|
||||
Code that rebuilds diagnostics must update the reason without deleting the row.
|
||||
|
||||
Examples:
|
||||
|
||||
| State | Attempted row | Reason | Domain value |
|
||||
| --- | --- | --- | --- |
|
||||
| No projection | absent | absent | authored value |
|
||||
| FK `SET NULL` repair | authored UUID | `foreignKeySetNull` | `null` |
|
||||
| Unique loser | authored value | `uniqueConflict` | released value |
|
||||
| Hidden unique row | authored value | `hiddenUniqueRelease` | released value |
|
||||
| FK candidate then unique release | authored UUID | `uniqueConflict` | released value |
|
||||
|
||||
## Exact projection order
|
||||
|
||||
Local writes and inbound merges use one fact-plus-projection planner.
|
||||
|
||||
1. **Merge or author facts.** Resolve row, field, tombstone, and HLC facts.
|
||||
Capture every affected authored field value before changing its materialized
|
||||
domain value. Projection does not alter these facts.
|
||||
2. **Compute visibility and FK candidates.** Starting from authored FK values,
|
||||
resolve cascade, restrict/no-action, set-null, and set-default to the FK
|
||||
fixed point over visible and hidden rows. Produce effective visibility and
|
||||
an FK-safe candidate for every FK field.
|
||||
3. **Release hidden rows, then resolve visible claims.** First assign every
|
||||
hidden row a deterministic conflict-free value, even when it is the only row
|
||||
with that claim. Hidden rows do not join conflict groups or winner selection.
|
||||
Then build conflict groups from visible rows only. For an FK field, key the
|
||||
lookup by its FK-safe candidate; for other fields, key it by its authored
|
||||
value from `CrdtDataAttemptedValue` when present, otherwise the domain
|
||||
column. Never key it by a previously released materialized value. Claim
|
||||
ordering still uses canonical authored field HLCs. The deterministic visible
|
||||
winner materializes the canonical claim and every visible loser receives a
|
||||
deterministic conflict-free value.
|
||||
4. **Build one final plan.** The plan contains final visibility, final domain
|
||||
values, and required `CrdtDataAttemptedValue` rows with diagnostic reasons.
|
||||
Unique output does not feed back into FK visibility: it changes
|
||||
materialization, not the authored FK or FK-safe candidate. A full
|
||||
recomputation is the oracle for any incremental affected-closure algorithm.
|
||||
5. **Materialize atomically.** Park changing unique tuples at deterministic
|
||||
constraint-safe temporary values when immediate indexes require it, then
|
||||
write final domain values and metadata in the same transaction. Temporary
|
||||
values are never facts or outbound changes.
|
||||
|
||||
This order resolves a hidden unique FK without an FK-side exception. If its
|
||||
parent becomes visible, the FK stage may restore the authored UUID as its
|
||||
candidate; the unique stage then releases it because the child is hidden. When
|
||||
the child is restored, it enters visible winner selection and either reclaims
|
||||
the UUID or receives the supported loser value. For a unique FK, that release
|
||||
must be FK-safe; under the current policy the FK is nullable and releases to
|
||||
`null`.
|
||||
|
||||
Local insert, update, upsert, delete, and tombstone restoration need this plan
|
||||
before the underlying write can violate an immediate unique constraint. A
|
||||
full-row passthrough of an existing projected value remains non-authored;
|
||||
explicit/narrowed field writes author a new attempted value and field HLC.
|
||||
|
||||
## Outbound and rebuild behavior
|
||||
|
||||
Outbound collection substitutes `CrdtDataAttemptedValue.value` whenever it
|
||||
exists; otherwise it reads the domain value. It never exports an FK candidate,
|
||||
unique loser value, or temporary parking value as an authored update.
|
||||
|
||||
An empty replica merging a complete scope export must reproduce normalized
|
||||
domain rows, hidden state, authored values/HLCs, and derived projection. A
|
||||
rebuild may rewrite `projectionReason`, but it must preserve
|
||||
`CrdtDataAttemptedValue.value`: while the row exists, that is the only copy of
|
||||
the authored fact.
|
||||
|
||||
## Type and migration safety
|
||||
|
||||
### Persisted type identity
|
||||
|
||||
Add at least these fields to `CrdtSchemaColumn`:
|
||||
|
||||
```yaml
|
||||
columnType: ColumnType
|
||||
dartType: String
|
||||
isNullable: bool
|
||||
```
|
||||
|
||||
All three are required. Serverpod's stable `ColumnType` enum may be persisted
|
||||
directly and records database storage; `dartType` records dynamic value
|
||||
semantics. For example, `bigint` may represent `int`, `Duration`, or a by-index
|
||||
enum, while one Dart value can use different storage such as `json` or `jsonb`.
|
||||
Relevant parameters, nullability, FK definitions, unique indexes, and release
|
||||
policy also belong in the persisted schema identity and sync compatibility hash.
|
||||
|
||||
### Dynamic validation
|
||||
|
||||
For normal engine writes and inbound sync, let Serverpod deserialize
|
||||
`CrdtDataAttemptedValue.value`, then compare its canonical serialized class name
|
||||
with the class expected by `dartType`. Do not compare
|
||||
`runtimeType.toString()` and do not manually query inside the JSON envelope.
|
||||
Null is validated from the persisted schema because it has no concrete runtime
|
||||
class.
|
||||
|
||||
Top-level `List` does not validate its element type. Any supported generic or
|
||||
custom type therefore needs recursive validation plus deterministic,
|
||||
cross-dialect unique equality. Initialization must reject a schema containing a
|
||||
releasable type whose validation or release semantics the projector does not
|
||||
support; this is a schema check and does not scan persisted attempted values.
|
||||
|
||||
For now, synchronized unique projection must reject any `ColumnType.json` or
|
||||
`ColumnType.jsonb` domain column participating in a unique claim. Neither
|
||||
nullable JSON as a releasable candidate nor non-nullable JSON as a fixed
|
||||
component is supported: sync semantics do not yet define canonical,
|
||||
cross-dialect JSON equality. This restriction applies to the domain column's
|
||||
schema type, not to the JSONB envelope used by `CrdtDataAttemptedValue.value` to
|
||||
store an otherwise supported dynamic value.
|
||||
|
||||
### Initialization algorithm
|
||||
|
||||
Call `await crdtDatabase.initialize()` after application migrations and before
|
||||
serving requests or starting sync. Reconciliation must plan before mutating:
|
||||
|
||||
1. Load the persisted CRDT registry and target `TableDefinition`s.
|
||||
2. Compare names, persisted type identity, indexes, and relations.
|
||||
3. Classify additions, drops, possible renames, and type/policy changes.
|
||||
4. If types match, trust validated engine writes and do not scan values.
|
||||
5. For each changed type, perform only an existence check for attempted rows;
|
||||
never deserialize them during initialization.
|
||||
6. If attempted rows exist, fail without changing registry or projection state.
|
||||
The initialization exception must identify the table, column, stored and
|
||||
target type identities, and instruct the developer to convert the domain and
|
||||
attempted values and then attest completion in `CrdtSchemaColumn`.
|
||||
7. Also fail without mutation when a rename or drop/add is ambiguous.
|
||||
8. Otherwise update the registry, clear obsolete derived caches, rebuild, and
|
||||
publish the in-memory schema.
|
||||
|
||||
Initialization is therefore bounded by schema metadata and existence queries;
|
||||
it never adds a data-size-dependent attempted-value scan to application startup.
|
||||
|
||||
### Direct migrations without a hook
|
||||
|
||||
For a non-destructive type change with attempted rows, the initialization
|
||||
exception instructs the application migration to perform these operations in
|
||||
one transaction and in this order:
|
||||
|
||||
1. Convert the domain column without destroying authored meaning.
|
||||
2. Convert every affected `crdt_data_attempted_value.value` envelope using
|
||||
Serverpod's serialization format.
|
||||
3. As the final step, update the corresponding `CrdtSchemaColumn` type and
|
||||
policy identity to the target values reported by the exception.
|
||||
|
||||
The final registry update is the migration's explicit attestation that all
|
||||
dependent data was converted. If conversion fails before that step, the old
|
||||
identity remains and initialization fails again safely. If the developer
|
||||
converts the data but omits the attestation, the same actionable exception is
|
||||
raised on the next start. Once the stored identity matches, initialization
|
||||
trusts the migration and does not rescan attempted values.
|
||||
|
||||
This is an intentional trust boundary. Updating `CrdtSchemaColumn` without
|
||||
correctly converting the data can cause a later deserialization failure;
|
||||
proving otherwise would require the costly startup scan this model rejects.
|
||||
Manual migrations already carry equivalent responsibility for domain data and
|
||||
constraints.
|
||||
|
||||
For a rename with field metadata, the migration updates the existing
|
||||
`CrdtSchemaColumn.name` so its identity survives. For a real drop, it
|
||||
explicitly removes the old schema column and cascading field metadata. An
|
||||
ambiguous drop/add with metadata fails before reconciliation mutates anything.
|
||||
|
||||
The future migration hook tracked by
|
||||
[issue #87](https://github.com/marcelomendoncasoares/serverpod_offline_sync/issues/87)
|
||||
can abstract the conversion and final attestation from application developers,
|
||||
but it follows the same protocol and does not make startup scan existing rows.
|
||||
New invalid peer input, external SQL after initialization, a live schema change,
|
||||
or a falsely attested manual migration can still fail at runtime as an atomic
|
||||
integrity or deserialization error. Unattested migration changes fail at
|
||||
initialization with direct repair instructions.
|
||||
|
||||
## Rejected alternatives
|
||||
|
||||
- **Ignoring hidden-row materialization:** hidden values remain observable,
|
||||
exported, and indexed, so freezing them makes projection history-dependent.
|
||||
Unique conflict grouping is nevertheless intentionally visible-only after
|
||||
hidden rows have been unconditionally released.
|
||||
- **A hidden unique-FK guard:** it preserves stale materialization and merely
|
||||
moves failure to a later claim or restoration.
|
||||
- **Extending `releaseOnDelete`:** it misses remote/cascade hiding and later
|
||||
authored changes.
|
||||
- **Partial indexes excluding hidden rows:** visibility is stored outside the
|
||||
domain table; denormalizing it creates another engine-owned source of truth.
|
||||
- **Removing physical unique indexes:** this hides projection defects instead
|
||||
of preventing divergence.
|
||||
- **Independent FK and unique attempted stores:** a field governed by both
|
||||
would have competing authored authorities.
|
||||
- **A separate FK projection table:** its candidate is recomputable planner
|
||||
output and its only useful persisted field was a diagnostic reason. A generic
|
||||
reason on `CrdtDataAttemptedValue` avoids another relation and table.
|
||||
- **Untagged JSON or a custom codec:** Serverpod dynamic already preserves the
|
||||
wire type through `className`.
|
||||
- **One table per Dart type:** it multiplies schema and migration machinery for
|
||||
values always addressed by field identity.
|
||||
- **Scanning attempted values after a type change:** it proves more about a
|
||||
manual migration, but makes application initialization proportional to real
|
||||
data volume. The migration's final registry update is the explicit attestation
|
||||
instead.
|
||||
|
||||
## Verification gates
|
||||
|
||||
Deterministic tests must cover:
|
||||
|
||||
- hidden plain and FK unique claims, including lone hidden claimants;
|
||||
- later authored writes while hidden and deterministic restoration;
|
||||
- a visible loser reclaiming its canonical attempted value after the previous
|
||||
winner becomes hidden, without treating hidden rows as conflict candidates;
|
||||
- bootstrap/export convergence for visible and hidden rows;
|
||||
- all local write paths before immediate constraint checks;
|
||||
- swaps, composite indexes, overlapping indexes, and safe two-phase writes;
|
||||
- incremental projection equivalence to full recomputation;
|
||||
- batch partition and replay convergence, plus proof that unrelated non-FK
|
||||
updates do not run FK projection;
|
||||
- matching schema startup without value scans;
|
||||
- initialization rejection of nullable and non-nullable JSON-backed unique
|
||||
candidates;
|
||||
- changed types with no attempted rows, and actionable failure without scanning
|
||||
or mutation when attempted rows exist;
|
||||
- transactional conversion with `CrdtSchemaColumn` attested last, including
|
||||
repeated failure when the attestation is omitted;
|
||||
- normal write and sync rejection of malformed or incorrectly typed dynamic
|
||||
values;
|
||||
- rename and drop migrations;
|
||||
- sync rejection for incompatible type or projection schemas; and
|
||||
- rebuild and mutation validation proving canonical-claim substitution and
|
||||
unconditional hidden-row release are required.
|
||||
|
|
@ -50,7 +50,7 @@ also reject server-authoritative reference-table relations even when they use
|
|||
reference them.
|
||||
- **Package metadata table:** CRDT infrastructure tables owned by this package,
|
||||
especially `crdt_scopes`, `crdt_nodes`, `crdt_data_rows`,
|
||||
`crdt_data_fields`, `crdt_data_tombstone`, and FK projection tables.
|
||||
`crdt_data_fields`, `crdt_data_tombstone`, and `crdt_data_attempted_value`.
|
||||
|
||||
`crdt_scopes` is not a synced domain table. The `scopeId -> crdt_scopes.id`
|
||||
relation is a special package-owned purge boundary, not an application-level
|
||||
|
|
|
|||
|
|
@ -2,15 +2,15 @@
|
|||
|
||||
## Summary
|
||||
|
||||
Concurrent inserts or updates can make multiple CRDT rows claim the same unique
|
||||
value. The current merge model resolves this without hiding rows: every row
|
||||
remains visible, and the losing unique claims are materialized with
|
||||
Concurrent inserts or updates can make multiple visible CRDT rows claim the same
|
||||
unique value. The current merge model resolves this without hiding rows because
|
||||
of the unique conflict itself: losing unique claims are materialized with
|
||||
deterministic conflict-free visible values.
|
||||
|
||||
This is the `flag` policy. It is intentionally simple:
|
||||
|
||||
- all rows remain CRDT facts;
|
||||
- all rows remain visible domain rows;
|
||||
- all rows involved in a unique conflict remain visible domain rows;
|
||||
- one deterministic winner keeps the claimed unique value;
|
||||
- each loser keeps its own row identity and receives a conflict-free visible
|
||||
unique value;
|
||||
|
|
@ -22,7 +22,13 @@ implementation.
|
|||
|
||||
## Current Policy: Flag
|
||||
|
||||
For each unique index conflict group:
|
||||
Candidate construction, hidden-row release, canonical attempted-value lookup,
|
||||
and restoration are defined by
|
||||
[Rebuildable FK and unique projection model](projection-model.md#exact-projection-order).
|
||||
The `flag` policy operates only on the visible conflict groups supplied by that
|
||||
shared planner.
|
||||
|
||||
For each conflict group among visible rows:
|
||||
|
||||
1. Compute each row's unique claim timestamp from the HLC of the unique-indexed
|
||||
field or fields. For an insert, use the insert HLC and any incoming field HLCs
|
||||
|
|
@ -30,20 +36,25 @@ For each unique index conflict group:
|
|||
2. Pick the winner deterministically by oldest unique-claim HLC, using the same
|
||||
full HLC ordering as the rest of CRDT merge. This includes node id as the HLC
|
||||
tie-break. Row id is only a final fallback for identical HLCs.
|
||||
3. Leave the winner's unique value unchanged.
|
||||
3. Materialize the winner's canonical claim. This lets a previously released
|
||||
visible loser reclaim its authored value after the prior winner is hidden or
|
||||
changes its claim.
|
||||
4. Rewrite each loser's visible releasable unique-indexed value to a
|
||||
deterministic conflict-free value:
|
||||
- nullable unique values may be released to `null`;
|
||||
- nullable values of a supported candidate type may be released to `null`;
|
||||
- text values receive a stable `__conflict__<rowId>` suffix;
|
||||
- UUID values receive a deterministic synthetic UUID.
|
||||
5. Keep all rows visible. Application code can decide whether and how to surface
|
||||
the changed visible value as a business conflict.
|
||||
5. Keep every conflict-group row visible. Application code can decide whether
|
||||
and how to surface the changed visible value as a business conflict.
|
||||
|
||||
For a composite unique index, fixed partition columns such as `scopeId` are not
|
||||
released, and not every remaining indexed column has to be releasable. The index
|
||||
is supported when at least one non-`scopeId` column can be released
|
||||
deterministically. Stable discriminator columns may remain unchanged; changing
|
||||
the releasable column or columns is enough to make the unique tuple conflict-free.
|
||||
Supported candidate types are defined centrally under
|
||||
[Dynamic validation](projection-model.md#dynamic-validation);
|
||||
nullability alone does not make a type supported.
|
||||
|
||||
For the global foreign-key-only unique indexes that Serverpod emits for synced
|
||||
relations, every indexed FK column must be nullable under the current `flag`
|
||||
|
|
@ -84,7 +95,7 @@ unique facts -> conflict components -> winner + losers + released loser values
|
|||
Future policies then interpret that canonical result:
|
||||
|
||||
```text
|
||||
flag = show all rows; losers show released unique values
|
||||
flag = show all conflict-component rows; losers show released unique values
|
||||
overwrite = use flag winner; alias/hide losers in the visible projection
|
||||
merge = overwrite projection + deterministic field folding into winner
|
||||
```
|
||||
|
|
@ -152,8 +163,9 @@ For now, the `flag` policy gives a safer and smaller invariant:
|
|||
unique projection changes values, not row identity or row visibility
|
||||
```
|
||||
|
||||
Foreign key repair can then be developed independently as a projection over row
|
||||
visibility and FK values, without also needing row aliasing and field folding.
|
||||
The `flag` winner/loser policy therefore remains independent of FK action
|
||||
semantics even though the shared planner feeds FK-safe candidates into unique
|
||||
resolution.
|
||||
|
||||
## Invariants
|
||||
|
||||
|
|
|
|||
|
|
@ -111,13 +111,13 @@ class DemoSnapshot {
|
|||
DemoSnapshot({
|
||||
required this.catalog,
|
||||
required this.rows,
|
||||
required this.foreignKeys,
|
||||
required this.attemptedValues,
|
||||
required this.tombstones,
|
||||
});
|
||||
|
||||
final RelationshipCatalog catalog;
|
||||
final List<RowView> rows;
|
||||
final List<offline.CrdtDataForeignKey> foreignKeys;
|
||||
final List<offline.CrdtDataAttemptedValue> attemptedValues;
|
||||
final List<offline.CrdtDataDeleted> tombstones;
|
||||
|
||||
/// Loads the visible rows and the full set (visible + CRDT-hidden, via
|
||||
|
|
@ -147,7 +147,7 @@ class DemoSnapshot {
|
|||
return DemoSnapshot(
|
||||
catalog: catalog,
|
||||
rows: rows,
|
||||
foreignKeys: await offline.CrdtDataForeignKey.db.find(crdt),
|
||||
attemptedValues: await offline.CrdtDataAttemptedValue.db.find(crdt),
|
||||
tombstones: await offline.CrdtDataDeleted.db.find(crdt),
|
||||
);
|
||||
}
|
||||
|
|
@ -173,7 +173,7 @@ class DemoSnapshot {
|
|||
return DemoSnapshot(
|
||||
catalog: catalog,
|
||||
rows: rows,
|
||||
foreignKeys: const [],
|
||||
attemptedValues: const [],
|
||||
tombstones: const [],
|
||||
);
|
||||
}
|
||||
|
|
@ -263,21 +263,20 @@ class DemoSnapshot {
|
|||
|
||||
final nodes = <TreeNode<DemoTreeItem>>[
|
||||
...forest,
|
||||
if (foreignKeys.isNotEmpty || tombstones.isNotEmpty)
|
||||
if (attemptedValues.isNotEmpty || tombstones.isNotEmpty)
|
||||
TreeItem(
|
||||
data: DemoTreeItem.group(
|
||||
'CRDT metadata',
|
||||
'${foreignKeys.length} FK projections · ${tombstones.length} tombstones',
|
||||
'${attemptedValues.length} attempted values · ${tombstones.length} tombstones',
|
||||
),
|
||||
expanded: false,
|
||||
children: [
|
||||
for (final fk in foreignKeys)
|
||||
for (final attempted in attemptedValues)
|
||||
TreeItem(
|
||||
data: DemoTreeItem.metadata(
|
||||
'field ${fk.fieldId}',
|
||||
'attempted ${shortId(fk.attemptedValue)} '
|
||||
'visible ${shortId(fk.visibleValue)} '
|
||||
'reason ${fk.overrideReason?.name ?? 'none'}',
|
||||
'field ${attempted.fieldId}',
|
||||
'value ${attempted.value} '
|
||||
'reason ${attempted.projectionReason.name}',
|
||||
),
|
||||
),
|
||||
for (final tombstone in tombstones)
|
||||
|
|
|
|||
|
|
@ -27,6 +27,10 @@ dependencies:
|
|||
dev_dependencies:
|
||||
flutter_lints: ^6.0.0
|
||||
|
||||
dependency_overrides:
|
||||
serverpod_offline_sync:
|
||||
path: ../../packages/serverpod_offline_sync
|
||||
|
||||
# dependency_overrides:
|
||||
# meta: 1.18.2
|
||||
# serverpod_auth_core_client:
|
||||
|
|
|
|||
|
|
@ -52,16 +52,6 @@ extension CrdtDataDeletedExtension on CrdtDataDeleted {
|
|||
: toHlcForNode(node!.uuidNodeId);
|
||||
}
|
||||
|
||||
/// Extensions for the [CrdtDataForeignKey] class.
|
||||
extension CrdtDataForeignKeyExtension on CrdtDataForeignKey {
|
||||
/// Whether an active projection override exists for this FK field.
|
||||
///
|
||||
/// Derived from [CrdtDataForeignKey.overrideReason]: an override is active if
|
||||
/// and only if [overrideReason] is non-null. This is the authoritative test
|
||||
/// for whether the domain row's FK value differs from the attempted value.
|
||||
bool get hasOverride => overrideReason != null;
|
||||
}
|
||||
|
||||
/// Extensions for the [CrdtDataDeletedReason] enum.
|
||||
extension CrdtDataDeletedReasonExtension on CrdtDataDeletedReason {
|
||||
/// Whether this tombstone reason is synced across replicas. All reasons that
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import 'package:uuid/uuid.dart';
|
|||
import '../database/database.dart';
|
||||
import '../database/merge_utils/database_helpers.dart';
|
||||
import '../database/recorder.dart';
|
||||
import '../database/unique_index_utils.dart';
|
||||
import '../generated/protocol.dart';
|
||||
import '../hlc/hlc.dart';
|
||||
import '../managers/scope.dart';
|
||||
|
|
@ -135,7 +136,7 @@ class CrdtSync {
|
|||
/// column payloads are resolved incrementally as each change is yielded.
|
||||
///
|
||||
/// Foreign-key columns with an active projection override are sent with their
|
||||
/// durable [CrdtDataForeignKey.attemptedValue], not the visible value stored
|
||||
/// durable [CrdtDataAttemptedValue.value], not the visible value stored
|
||||
/// in the domain table. Peers need the attempted fact to converge; local FK
|
||||
/// projection materializes only the safe visible value into domain tables.
|
||||
///
|
||||
|
|
@ -571,7 +572,7 @@ class CrdtSync {
|
|||
),
|
||||
);
|
||||
|
||||
final foreignKeyAttemptFieldsByRowId = await _loadProjectedForeignKeyAttemptFields(
|
||||
final attemptedValueFieldsByRowId = await _loadAttemptedValueFields(
|
||||
session,
|
||||
rows,
|
||||
);
|
||||
|
|
@ -593,7 +594,7 @@ class CrdtSync {
|
|||
row.uuidRowId,
|
||||
table,
|
||||
dartName,
|
||||
foreignKeyAttemptFieldsByRowId[row.id!],
|
||||
attemptedValueFieldsByRowId[row.id!],
|
||||
scopeId,
|
||||
ownerCache,
|
||||
);
|
||||
|
|
@ -649,7 +650,7 @@ class CrdtSync {
|
|||
row: CrdtDataRow.include(tbl: CrdtSchemaTable.include()),
|
||||
column: CrdtSchemaColumn.include(),
|
||||
node: CrdtNode.include(),
|
||||
foreignKey: CrdtDataForeignKey.include(),
|
||||
attemptedValue: CrdtDataAttemptedValue.include(),
|
||||
),
|
||||
);
|
||||
|
||||
|
|
@ -670,7 +671,7 @@ class CrdtSync {
|
|||
tableName,
|
||||
field.row!.uuidRowId,
|
||||
columnName,
|
||||
field.foreignKey,
|
||||
field.attemptedValue,
|
||||
scopeId,
|
||||
ownerCache,
|
||||
);
|
||||
|
|
@ -811,8 +812,8 @@ class CrdtSync {
|
|||
|
||||
/// Loads a domain row for outbound insert sync.
|
||||
///
|
||||
/// Reads the materialized row from the domain table, then swaps any FK columns
|
||||
/// with an active override back to [CrdtDataForeignKey.attemptedValue] before
|
||||
/// Reads the materialized row from the domain table, then swaps any columns
|
||||
/// with an active [CrdtDataAttemptedValue] back to the authored value before
|
||||
/// deserializing. The domain table and projected columns are only known at
|
||||
/// runtime, so a generated repository cannot express this query. Keeping one
|
||||
/// targeted SQL projection also avoids fetching and serializing unrelated
|
||||
|
|
@ -824,7 +825,7 @@ class CrdtSync {
|
|||
UuidValue rowId,
|
||||
Table table,
|
||||
String dartName,
|
||||
List<CrdtDataField>? foreignKeyAttemptFields,
|
||||
List<CrdtDataField>? attemptedValueFields,
|
||||
int scopeId,
|
||||
DomainRowOwnerCache ownerCache,
|
||||
) async {
|
||||
|
|
@ -848,7 +849,7 @@ class CrdtSync {
|
|||
final columnMap = result.first.toColumnMap()
|
||||
// Domain columns hold visible/materialized FK values; restore attempted
|
||||
// values for override columns before building the outbound merge payload.
|
||||
..applyProjectedForeignKeyAttempts(foreignKeyAttemptFields)
|
||||
..applyAuthoredAttemptedValues(attemptedValueFields)
|
||||
// scopeId is local ownership metadata; it is never emitted on the wire.
|
||||
..remove('scopeId');
|
||||
|
||||
|
|
@ -861,21 +862,20 @@ class CrdtSync {
|
|||
|
||||
/// Resolves a column value for outbound update sync.
|
||||
///
|
||||
/// When [projection] has an active override, returns the attempted value
|
||||
/// from [CrdtDataForeignKey.attemptedValue] instead of the materialized
|
||||
/// domain column value. Non-FK columns and FK columns without an override
|
||||
/// are read directly from the domain table. Both the table and column are
|
||||
/// When [attempted] is present, returns its authored value instead of the
|
||||
/// materialized domain column value. Columns without an attempted row are
|
||||
/// read directly from the domain table. Both the table and column are
|
||||
/// runtime schema values, so this cannot use a statically typed repository.
|
||||
Future<({bool exists, int? ownerScopeId, dynamic value})> _fetchOwnedColumnValue(
|
||||
DatabaseSession session,
|
||||
String tableName,
|
||||
UuidValue rowId,
|
||||
String columnName,
|
||||
CrdtDataForeignKey? projection,
|
||||
CrdtDataAttemptedValue? attempted,
|
||||
int scopeId,
|
||||
DomainRowOwnerCache ownerCache,
|
||||
) async {
|
||||
if (projection != null && projection.hasOverride) {
|
||||
if (attempted != null) {
|
||||
final owner = await _readDomainRowOwner(session, tableName, rowId, ownerCache);
|
||||
if (!owner.exists || owner.scopeId != scopeId) {
|
||||
return (exists: owner.exists, ownerScopeId: owner.scopeId, value: null);
|
||||
|
|
@ -883,7 +883,7 @@ class CrdtSync {
|
|||
return (
|
||||
exists: true,
|
||||
ownerScopeId: scopeId,
|
||||
value: _decodeColumnValue(tableName, columnName, projection.attemptedValue),
|
||||
value: canonicalDomainValue(attempted.value),
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -997,12 +997,12 @@ class CrdtSync {
|
|||
return scope?.uuidScopeId;
|
||||
}
|
||||
|
||||
/// Loads FK projection metadata for outbound insert sync.
|
||||
/// Loads attempted-value metadata for outbound insert sync.
|
||||
///
|
||||
/// Returns only fields whose [CrdtDataForeignKey.overrideReason] is non-null,
|
||||
/// keyed by CRDT row id. These are the columns whose domain-table value
|
||||
/// differs from the durable attempted FK fact.
|
||||
Future<Map<int, List<CrdtDataField>>> _loadProjectedForeignKeyAttemptFields(
|
||||
/// Returns fields that currently have a [CrdtDataAttemptedValue] row, keyed
|
||||
/// by CRDT row id. These are the columns whose domain-table value differs
|
||||
/// from the durable authored fact.
|
||||
Future<Map<int, List<CrdtDataField>>> _loadAttemptedValueFields(
|
||||
DatabaseSession session,
|
||||
List<CrdtDataRow> rows,
|
||||
) async {
|
||||
|
|
@ -1011,10 +1011,10 @@ class CrdtSync {
|
|||
|
||||
final fields = await CrdtDataField.db.find(
|
||||
session,
|
||||
where: (t) => t.rowId.inSet(rowIds) & t.foreignKey.overrideReason.notEquals(null),
|
||||
where: (t) => t.rowId.inSet(rowIds) & t.attemptedValue.id.notEquals(null),
|
||||
include: CrdtDataField.include(
|
||||
column: CrdtSchemaColumn.include(),
|
||||
foreignKey: CrdtDataForeignKey.include(),
|
||||
attemptedValue: CrdtDataAttemptedValue.include(),
|
||||
),
|
||||
);
|
||||
|
||||
|
|
@ -1065,11 +1065,13 @@ class CrdtSync {
|
|||
.map((table) {
|
||||
final definition = tableDefinitionsByName[table.tableName];
|
||||
final columns = [
|
||||
for (final column in definition?.columns ?? const <ColumnDefinition>[])
|
||||
if (column.name != 'scopeId') column.name,
|
||||
if (definition == null)
|
||||
for (final column in table.columns)
|
||||
if (column.columnName != 'scopeId') column.columnName,
|
||||
if (definition != null)
|
||||
for (final column in definition.columns)
|
||||
if (column.name != 'scopeId')
|
||||
_canonicalColumnIdentity(definition, column)
|
||||
else
|
||||
for (final column in table.columns)
|
||||
if (column.columnName != 'scopeId') column.columnName,
|
||||
]..sort();
|
||||
final foreignKeys = _canonicalForeignKeys(definition);
|
||||
final uniqueIndexes = _canonicalUniqueIndexes(definition);
|
||||
|
|
@ -1081,6 +1083,15 @@ class CrdtSync {
|
|||
.join(';');
|
||||
}
|
||||
|
||||
static String _canonicalColumnIdentity(
|
||||
TableDefinition table,
|
||||
ColumnDefinition column,
|
||||
) {
|
||||
final releaseKind = crdtUniqueConflictReleaseKindForColumn(table, column);
|
||||
return '${column.name}:${column.columnType.name}:${column.dartType}:'
|
||||
'${column.isNullable}:${releaseKind?.name ?? '-'}';
|
||||
}
|
||||
|
||||
static List<String> _canonicalForeignKeys(TableDefinition? definition) {
|
||||
if (definition == null) return const [];
|
||||
final entries = <String>[
|
||||
|
|
@ -1145,19 +1156,18 @@ Expression _afterAnyScopeCheckpointFilter(
|
|||
}
|
||||
|
||||
extension on Map<String, dynamic> {
|
||||
/// Replaces materialized FK column values with attempted values for sync.
|
||||
/// Replaces materialized column values with authored values for sync.
|
||||
///
|
||||
/// After local FK projection (for example `SET NULL` or `SET DEFAULT`), the
|
||||
/// domain table stores the safe visible value while
|
||||
/// [CrdtDataForeignKey.attemptedValue] preserves what was actually tried.
|
||||
/// After local projection, the domain table stores the safe visible value
|
||||
/// while [CrdtDataAttemptedValue.value] preserves what was actually tried.
|
||||
/// Outbound sync must send the attempted value so peers can apply their own
|
||||
/// projection from the same fact.
|
||||
void applyProjectedForeignKeyAttempts(List<CrdtDataField>? foreignKeyAttemptFields) {
|
||||
if (foreignKeyAttemptFields == null) return;
|
||||
for (final field in foreignKeyAttemptFields) {
|
||||
final projection = field.foreignKey;
|
||||
if (projection == null || !projection.hasOverride) continue;
|
||||
this[field.column!.name] = projection.attemptedValue;
|
||||
void applyAuthoredAttemptedValues(List<CrdtDataField>? attemptedValueFields) {
|
||||
if (attemptedValueFields == null) return;
|
||||
for (final field in attemptedValueFields) {
|
||||
final attempted = field.attemptedValue;
|
||||
if (attempted == null) continue;
|
||||
this[field.column!.name] = canonicalDomainValue(attempted.value);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,18 @@
|
|||
### Authored field value retained while the domain column holds a projection.
|
||||
###
|
||||
### Sparse one-to-one dependent of CrdtDataField. Present only while the
|
||||
### materialized domain value differs from the authored value. `value` is a
|
||||
### durable authored fact; `projectionReason` is disposable diagnostic state.
|
||||
class: CrdtDataAttemptedValue
|
||||
table: crdt_data_attempted_value
|
||||
database: all
|
||||
fields:
|
||||
### The field whose authored value this row preserves.
|
||||
fieldId: int, unique
|
||||
field: CrdtDataField?, relation(name=field_attempted_value, field=fieldId, onDelete=Cascade)
|
||||
|
||||
### The authored value, using Serverpod's dynamic `{className, data}` envelope.
|
||||
value: dynamic, serializationDataType=jsonb
|
||||
|
||||
### Terminal projector that selected the current domain value.
|
||||
projectionReason: CrdtProjectionReason
|
||||
|
|
@ -16,9 +16,9 @@ fields:
|
|||
### The node that updated the field.
|
||||
node: CrdtNode?, relation
|
||||
|
||||
### The foreign key information for the field, if this field holds a
|
||||
### reference to another row.
|
||||
foreignKey: CrdtDataForeignKey?, relation(optional, name=field_foreign_key)
|
||||
### Authored value retained while a projector materializes a different domain
|
||||
### value for this field.
|
||||
attemptedValue: CrdtDataAttemptedValue?, relation(optional, name=field_attempted_value)
|
||||
|
||||
indexes:
|
||||
crdt_data_fields_row_column_idx:
|
||||
|
|
|
|||
|
|
@ -1,40 +0,0 @@
|
|||
### CRDT field foreign key value and projection table.
|
||||
###
|
||||
### This table stores the user/attempted foreign key value for a CRDT field and
|
||||
### the materialized foreign key projection for that value. CrdtDataField stores
|
||||
### only HLC metadata, so FK columns store their durable attempted value here.
|
||||
###
|
||||
### Repairs must be recomputable from merged CRDT row, field, and tombstone
|
||||
### facts plus this attempted FK value. The projection columns record what was
|
||||
### materialized into the domain row; they do not decide business semantics or
|
||||
### create user field updates during merge.
|
||||
class: CrdtDataForeignKey
|
||||
table: crdt_data_foreign_key
|
||||
database: all
|
||||
fields:
|
||||
### The field that the foreign key belongs to.
|
||||
field: CrdtDataField?, relation(name=field_foreign_key, field=fieldId, onDelete=Cascade)
|
||||
fieldId: int
|
||||
|
||||
### The user-attempted foreign key value that the CRDT field currently carries.
|
||||
###
|
||||
### This may be null for nullable foreign keys. This is durable FK field-value
|
||||
### storage for CRDT columns whose CrdtDataField row stores only HLC metadata.
|
||||
attemptedValue: UuidValue?, required
|
||||
|
||||
### The foreign key value that is currently visible in the actual data row
|
||||
### when an override is active (overrideReason is non-null).
|
||||
visibleValue: UuidValue?
|
||||
|
||||
### The authoritative indicator and reason for an active projection override.
|
||||
###
|
||||
### This is the single source of truth for whether an override is active:
|
||||
### non-null means an override is active; null means no override. It also
|
||||
### names the cause of the override. The FK resolver recomputes it from merged
|
||||
### row, field, and tombstone facts.
|
||||
overrideReason: CrdtForeignKeyOverrideReason?
|
||||
|
||||
indexes:
|
||||
crdt_data_foreign_key_field_idx:
|
||||
fields: fieldId
|
||||
unique: true
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
### Why a foreign key projection override is active.
|
||||
enum: CrdtForeignKeyOverrideReason
|
||||
serialized: byIndex
|
||||
values:
|
||||
### The attempted parent target is hidden or missing and the FK projection
|
||||
### set the column to null.
|
||||
- setNull
|
||||
### The attempted parent target is hidden or missing and the FK projection
|
||||
### set the column to the default value.
|
||||
- setDefault
|
||||
### The attempted parent target is hidden or missing and the FK projection
|
||||
### cannot repair the value (restrict, no action, cascade, or the column is
|
||||
### non-nullable), so the row is hidden by projection while the attempt is
|
||||
### preserved.
|
||||
- missingParent
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
### Why a domain column currently differs from its authored value.
|
||||
###
|
||||
### Diagnostic-only derived state. Projection decisions never read this enum;
|
||||
### the planner recomputes from authored facts and schema, then rewrites the
|
||||
### reason with the final domain value. Because it is singular, it records the
|
||||
### terminal stage that selected that value.
|
||||
enum: CrdtProjectionReason
|
||||
serialized: byIndex
|
||||
values:
|
||||
### The attempted parent target is hidden or missing and FK projection set
|
||||
### the column to null.
|
||||
- foreignKeySetNull
|
||||
### The attempted parent target is hidden or missing and FK projection set
|
||||
### the column to the default value.
|
||||
- foreignKeySetDefault
|
||||
### The attempted parent target is hidden or missing and FK projection cannot
|
||||
### repair the value, so the row is hidden while the attempt is preserved.
|
||||
- foreignKeyMissingParent
|
||||
### A visible unique-index loser received a deterministic conflict-free value.
|
||||
- uniqueConflict
|
||||
### A hidden row was released from a unique index so visible rows can claim it.
|
||||
- hiddenUniqueRelease
|
||||
|
|
@ -333,6 +333,7 @@ class CrdtDatabase implements Database {
|
|||
for (final row in prepared.rows)
|
||||
if (!deletedRowIds.contains(row.id)) row,
|
||||
];
|
||||
final plannedInsert = await _recorder.planLocalInserts(rowsToInsert, tx);
|
||||
|
||||
// Tracked rows can only skip RETURNING when the prepared rows are
|
||||
// exactly what gets inserted: every id is caller-provided and a
|
||||
|
|
@ -340,17 +341,21 @@ class CrdtDatabase implements Database {
|
|||
final skipReturn =
|
||||
noReturn && !ignoreConflicts && !rows.any((row) => row.id == null);
|
||||
|
||||
final result = rowsToInsert.isEmpty
|
||||
final result = plannedInsert.rows.isEmpty
|
||||
? <T>[]
|
||||
: await _delegate.insert<T>(
|
||||
rowsToInsert,
|
||||
plannedInsert.rows,
|
||||
transaction: tx,
|
||||
ignoreConflicts: ignoreConflicts,
|
||||
noReturn: skipReturn,
|
||||
);
|
||||
|
||||
final insertedRows = skipReturn ? rowsToInsert : result;
|
||||
await _recorder.afterInsert<T>(insertedRows, tx);
|
||||
final insertedRows = skipReturn ? plannedInsert.rows : result;
|
||||
await _recorder.afterInsert<T>(
|
||||
insertedRows,
|
||||
tx,
|
||||
attempts: plannedInsert.attempts,
|
||||
);
|
||||
|
||||
final reinsertedRows = await _reinsertTombstonedRows(
|
||||
prepared.rows,
|
||||
|
|
@ -405,6 +410,13 @@ class CrdtDatabase implements Database {
|
|||
transaction,
|
||||
(tx) async {
|
||||
final prepared = _prepareRowsForInsert(rows, tx);
|
||||
if (rows.isNotEmpty) {
|
||||
final rowIds = {
|
||||
for (final row in rows)
|
||||
if (row.id case final UuidValue rowId) rowId,
|
||||
};
|
||||
await _recorder.projectCurrent(rows.first.table.tableName, rowIds, tx);
|
||||
}
|
||||
|
||||
// CRDT metadata needs the affected rows even when the public call uses
|
||||
// noReturn, because inserted rows may have database-generated ids.
|
||||
|
|
@ -482,8 +494,13 @@ class CrdtDatabase implements Database {
|
|||
];
|
||||
if (rowsToReinsert.isEmpty) return [];
|
||||
|
||||
final reinsertedRows = await _delegate.update<T>(
|
||||
final plannedReinserts = await _recorder.planLocalUpdates(
|
||||
rowsToReinsert,
|
||||
null,
|
||||
transaction,
|
||||
);
|
||||
final reinsertedRows = await _delegate.update<T>(
|
||||
plannedReinserts,
|
||||
transaction: transaction,
|
||||
);
|
||||
await _recorder.afterReinsert(reinsertedRows, transaction);
|
||||
|
|
@ -585,8 +602,9 @@ class CrdtDatabase implements Database {
|
|||
_delegate,
|
||||
transaction,
|
||||
(tx) async {
|
||||
final plannedUpdates = await _recorder.planLocalUpdates(rows, columns, tx);
|
||||
final updatedRows = [
|
||||
for (final row in rows)
|
||||
for (final row in plannedUpdates)
|
||||
await _updateRowWithoutRecording(
|
||||
row,
|
||||
stripScopeId: _shouldStripReturnedScopeId(row, tx),
|
||||
|
|
@ -612,9 +630,10 @@ class CrdtDatabase implements Database {
|
|||
_delegate,
|
||||
transaction,
|
||||
(tx) async {
|
||||
final plannedUpdates = await _recorder.planLocalUpdates([row], columns, tx);
|
||||
final stripScopeId = _shouldStripReturnedScopeId(row, tx);
|
||||
final updatedRow = await _updateRowWithoutRecording(
|
||||
row,
|
||||
plannedUpdates.single,
|
||||
stripScopeId: stripScopeId,
|
||||
transaction: tx,
|
||||
columns: columns,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,68 @@
|
|||
part of 'recorder.dart';
|
||||
|
||||
/// Field metadata for merged inserts, waiting to be written as one batch.
|
||||
///
|
||||
/// Every query behind [CrdtForeignKeyProjector.recordInsertAttempts] already
|
||||
/// takes a set of row ids, so a batch of inserts into one table costs the same
|
||||
/// as one insert. Rows are grouped by node as well as table because the merge
|
||||
/// cache stores each field with the node that authored it.
|
||||
class _PendingInsertAttempts {
|
||||
final Map<(String, int), ({Set<UuidValue> rowIds, CrdtNode node})> _byGroup = {};
|
||||
final Map<String, ProjectionAttemptsByField> _attemptsByTable = {};
|
||||
final Set<MergeRowKey> _rows = {};
|
||||
|
||||
bool get isEmpty => _byGroup.isEmpty;
|
||||
|
||||
/// Whether metadata for [rowKey] is still unwritten.
|
||||
bool holds(MergeRowKey rowKey) => _rows.contains(rowKey);
|
||||
|
||||
void add({
|
||||
required String tableName,
|
||||
required UuidValue rowId,
|
||||
required CrdtNode node,
|
||||
required ProjectionAttemptsByField attempts,
|
||||
}) {
|
||||
_byGroup
|
||||
.putIfAbsent(
|
||||
(tableName, node.id!),
|
||||
() => (rowIds: <UuidValue>{}, node: node),
|
||||
)
|
||||
.rowIds
|
||||
.add(rowId);
|
||||
_rows.add((tableName, rowId));
|
||||
if (attempts.isEmpty) return;
|
||||
_attemptsByTable
|
||||
.putIfAbsent(tableName, () => <MergeFieldKey, ProjectionAttempt>{})
|
||||
.addAll(attempts);
|
||||
}
|
||||
|
||||
/// Empties the collector and returns what it held.
|
||||
List<
|
||||
({
|
||||
String tableName,
|
||||
Set<UuidValue> rowIds,
|
||||
CrdtNode node,
|
||||
ProjectionAttemptsByField attempts,
|
||||
})
|
||||
>
|
||||
take() {
|
||||
final groups = [
|
||||
for (final MapEntry(key: key, value: group) in _byGroup.entries)
|
||||
(
|
||||
tableName: key.$1,
|
||||
rowIds: group.rowIds,
|
||||
node: group.node,
|
||||
attempts:
|
||||
_attemptsByTable[key.$1] ?? const <MergeFieldKey, ProjectionAttempt>{},
|
||||
),
|
||||
];
|
||||
_byGroup.clear();
|
||||
_attemptsByTable.clear();
|
||||
_rows.clear();
|
||||
return groups;
|
||||
}
|
||||
}
|
||||
|
||||
/// Adds merge-specific behavior to [CrdtMutationRecorder].
|
||||
extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
||||
/// Locks the current user row so merges can serialize with other work.
|
||||
|
|
@ -24,8 +87,13 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
if (mergeSet.isEmpty) return;
|
||||
|
||||
final operations = mergeSet.causallyOrderedChanges;
|
||||
final shouldProjectForeignKeys = _foreignKeyProjector
|
||||
.mergeOperationsMayAffectForeignKeys(operations);
|
||||
// Nothing in this batch touches a foreign key or a unique index, so no
|
||||
// projection decision can change: skip both passes instead of loading
|
||||
// state for tables that cannot produce a candidate or a conflict.
|
||||
final mayAffectProjection = _foreignKeyProjector.mergeOperationsMayAffectProjection(
|
||||
operations,
|
||||
);
|
||||
final authoredOverlays = <MergeFieldKey, Object?>{};
|
||||
final currentUser = _context.effectiveScopeFor(transaction);
|
||||
final remoteNodes = await _findOrCreateNodesForMerge(
|
||||
currentUser.id!,
|
||||
|
|
@ -35,10 +103,49 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
final nodesByUuid = remoteNodes.nodesByUuid;
|
||||
final context = await _loadMergeContext(mergeSet, transaction);
|
||||
|
||||
// Deletes carry no authored overlay but still change visibility, so the
|
||||
// seed is every touched row, not just the ones with overlays.
|
||||
final touchedTables = <String>{};
|
||||
final touchedRows = <MergeRowKey>{};
|
||||
for (final operation in operations) {
|
||||
if (!_context.isCrdtTrackedTableName(operation.tableName)) continue;
|
||||
touchedTables.add(operation.tableName);
|
||||
touchedRows.add((operation.tableName, operation.uuidRowId));
|
||||
}
|
||||
|
||||
final batch = mayAffectProjection
|
||||
? await _planBatchInserts(
|
||||
operations,
|
||||
nodesByUuid,
|
||||
context,
|
||||
touchedTables,
|
||||
touchedRows,
|
||||
transaction,
|
||||
)
|
||||
: (plan: null, pending: const <PendingProjectionRow>[]);
|
||||
// Every merged insert has to know whether the rows its foreign keys name
|
||||
// are there, which is one query per edge unless the batch resolves them
|
||||
// together and then keeps the answers current as it writes.
|
||||
final presence = batch.pending.isEmpty
|
||||
? null
|
||||
: await _foreignKeyProjector.resolvePresenceForInserts(
|
||||
batch.pending,
|
||||
transaction,
|
||||
);
|
||||
|
||||
// Field metadata for merged inserts is recorded per table and node rather
|
||||
// than per row: every query behind it already takes a set of row ids.
|
||||
final pendingAttempts = _PendingInsertAttempts();
|
||||
|
||||
for (final operation in operations) {
|
||||
if (!_context.isCrdtTrackedTableName(operation.tableName)) {
|
||||
continue;
|
||||
}
|
||||
// A later operation on a row resolves its field metadata through the
|
||||
// merge cache, so anything still pending for that row is written first.
|
||||
if (pendingAttempts.holds((operation.tableName, operation.uuidRowId))) {
|
||||
await _flushInsertAttempts(pendingAttempts, context, transaction);
|
||||
}
|
||||
|
||||
switch (operation) {
|
||||
case final CrdtMergeInsert insert:
|
||||
|
|
@ -46,6 +153,10 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
insert,
|
||||
nodesByUuid,
|
||||
context,
|
||||
authoredOverlays,
|
||||
batch.plan,
|
||||
pendingAttempts,
|
||||
presence,
|
||||
transaction,
|
||||
);
|
||||
case final CrdtMergeUpdate update:
|
||||
|
|
@ -53,6 +164,7 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
update,
|
||||
nodesByUuid,
|
||||
context,
|
||||
authoredOverlays,
|
||||
transaction,
|
||||
);
|
||||
case final CrdtMergeDelete delete:
|
||||
|
|
@ -62,11 +174,28 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
context,
|
||||
transaction,
|
||||
);
|
||||
// A delete can hide the row, and a resurrecting one can bring it
|
||||
// back; either way what was known about it no longer holds.
|
||||
presence?.forget((delete.tableName, delete.uuidRowId));
|
||||
}
|
||||
}
|
||||
|
||||
if (shouldProjectForeignKeys) {
|
||||
await _foreignKeyProjector.project(transaction);
|
||||
await _flushInsertAttempts(pendingAttempts, context, transaction);
|
||||
|
||||
if (mayAffectProjection) {
|
||||
await _foreignKeyProjector.project(
|
||||
transaction,
|
||||
authoredOverlays: authoredOverlays,
|
||||
seedTables: touchedTables,
|
||||
seedRows: touchedRows,
|
||||
);
|
||||
} else {
|
||||
// The pass is also what writes an update's value, so its authored
|
||||
// values still have to land when it is skipped.
|
||||
await _foreignKeyProjector.writeAuthoredValues(
|
||||
authoredOverlays,
|
||||
transaction,
|
||||
);
|
||||
}
|
||||
await _updateHlcFromIncomingOperations(
|
||||
operations,
|
||||
|
|
@ -260,10 +389,70 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
);
|
||||
}
|
||||
|
||||
/// Plans every insert in the batch in one projection pass.
|
||||
///
|
||||
/// The physical write of each insert has to be constraint safe before it
|
||||
/// runs, but that plan does not have to be recomputed per insert: one pass
|
||||
/// over all of them resolves the batch's claims against each other and
|
||||
/// against stored rows. The end-of-batch pass remains authoritative for what
|
||||
/// is finally materialized.
|
||||
Future<({ProjectionPlan? plan, List<PendingProjectionRow> pending})>
|
||||
_planBatchInserts(
|
||||
List<CrdtMergeChange> operations,
|
||||
Map<UuidValue, CrdtNode> remoteNodes,
|
||||
MergeContext context,
|
||||
Set<String> seedTables,
|
||||
Set<MergeRowKey> seedRows,
|
||||
Transaction transaction,
|
||||
) async {
|
||||
final pending = <PendingProjectionRow>[];
|
||||
final seenRowKeys = <MergeRowKey>{};
|
||||
|
||||
for (final operation in operations) {
|
||||
// An update or a delete only seeds the pass. Operations are causally
|
||||
// ordered, so an update newer than its insert is applied after it, and
|
||||
// the end-of-batch pass is what resolves the value it claims.
|
||||
if (operation is! CrdtMergeInsert) continue;
|
||||
if (!_context.isCrdtTrackedTableName(operation.tableName)) continue;
|
||||
|
||||
// Only rows this merge will create need a pre-write plan, and only the
|
||||
// first insert for a row id takes that path.
|
||||
final rowKey = (operation.tableName, operation.uuidRowId);
|
||||
if (context.rows[rowKey] != null) continue;
|
||||
if (!seenRowKeys.add(rowKey)) continue;
|
||||
pending.add((
|
||||
tableName: operation.tableName,
|
||||
rowId: operation.uuidRowId,
|
||||
authoredValues: _sanitizeMergeRowData(
|
||||
operation.tableName,
|
||||
operation.databaseColumns,
|
||||
),
|
||||
rowHlc: operation.hlc,
|
||||
node: _requireRemoteNode(remoteNodes, operation.uuidNodeId),
|
||||
hidden: false,
|
||||
));
|
||||
}
|
||||
if (pending.isEmpty) return (plan: null, pending: pending);
|
||||
|
||||
return (
|
||||
plan: await _foreignKeyProjector.project(
|
||||
transaction,
|
||||
pendingInserts: pending,
|
||||
seedTables: seedTables,
|
||||
seedRows: seedRows,
|
||||
),
|
||||
pending: pending,
|
||||
);
|
||||
}
|
||||
|
||||
Future<void> _applyMergeInsert(
|
||||
CrdtMergeInsert insert,
|
||||
Map<UuidValue, CrdtNode> remoteNodes,
|
||||
MergeContext context,
|
||||
Map<MergeFieldKey, Object?> authoredOverlays,
|
||||
ProjectionPlan? batchPlan,
|
||||
_PendingInsertAttempts pendingAttempts,
|
||||
CrdtForeignKeyPresenceCache? presence,
|
||||
Transaction transaction,
|
||||
) async {
|
||||
final rowKey = (insert.tableName, insert.uuidRowId);
|
||||
|
|
@ -275,6 +464,10 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
return;
|
||||
}
|
||||
|
||||
// Set inside the savepoint and read after it, so a rolled back insert
|
||||
// records nothing.
|
||||
ProjectionAttemptsByField? pendingWrites;
|
||||
|
||||
final data = _sanitizeMergeRowData(insert.tableName, insert.databaseColumns);
|
||||
|
||||
if (currentRow == null) {
|
||||
|
|
@ -290,34 +483,21 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
_db,
|
||||
transaction,
|
||||
(savepoint) async {
|
||||
final resolvedInsert = await _uniqueResolver.resolveForIncomingInsert(
|
||||
final planned = await _planMergeInsertValues(
|
||||
insert,
|
||||
data,
|
||||
context,
|
||||
batchPlan,
|
||||
presence,
|
||||
savepoint,
|
||||
);
|
||||
|
||||
final safeInsert = await _foreignKeyProjector.safeIncomingData(
|
||||
insert.tableName,
|
||||
insert.uuidRowId,
|
||||
resolvedInsert.data,
|
||||
savepoint,
|
||||
);
|
||||
|
||||
final insertedRow = await _applyMergeInsertForMissingRow(
|
||||
insert,
|
||||
remoteNode,
|
||||
incomingHlc,
|
||||
safeInsert.data,
|
||||
planned.domain,
|
||||
savepoint,
|
||||
);
|
||||
await _foreignKeyProjector.recordInsertAttempts(
|
||||
insert.tableName,
|
||||
{insert.uuidRowId},
|
||||
savepoint,
|
||||
safeInsert.attempts,
|
||||
mergeCache: (fields: context.fields, node: remoteNode),
|
||||
);
|
||||
pendingWrites = planned.attempts;
|
||||
return insertedRow;
|
||||
},
|
||||
);
|
||||
|
|
@ -325,6 +505,15 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
// same-scope recovery), so same-merge field updates resolve ownership
|
||||
// from the cache instead of re-reading the domain row.
|
||||
context.domainOwners[rowKey] = (exists: true, scopeId: mergingScopeId);
|
||||
pendingAttempts.add(
|
||||
tableName: insert.tableName,
|
||||
rowId: insert.uuidRowId,
|
||||
node: remoteNode,
|
||||
attempts: pendingWrites ?? const {},
|
||||
);
|
||||
// Written and visible, so the children arriving after it in this batch
|
||||
// can point at it without asking the database again.
|
||||
presence?.markVisible(rowKey);
|
||||
} on _ForeignScopeRowCollision catch (collision) {
|
||||
await _throwOwnershipCollision(
|
||||
operation: CrdtSyncViolationOperation.mergeInsert,
|
||||
|
|
@ -346,8 +535,89 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
incomingHlc,
|
||||
data,
|
||||
context,
|
||||
authoredOverlays,
|
||||
transaction,
|
||||
);
|
||||
// This one can resurrect a tombstoned row, so drop what was known.
|
||||
presence?.forget(rowKey);
|
||||
}
|
||||
}
|
||||
|
||||
/// The domain values a merged insert can physically write, paired with the
|
||||
/// authored values projection displaced.
|
||||
///
|
||||
/// The batch plan already resolved this row's claims against the rest of the
|
||||
/// batch. Foreign key safety is folded in after it, so where safety forces a
|
||||
/// null it is also the terminal cause for that column.
|
||||
Future<({Map<String, Object?> domain, ProjectionAttemptsByField attempts})>
|
||||
_planMergeInsertValues(
|
||||
CrdtMergeInsert insert,
|
||||
Map<String, Object?> data,
|
||||
ProjectionPlan? batchPlan,
|
||||
CrdtForeignKeyPresenceCache? presence,
|
||||
Transaction transaction,
|
||||
) async {
|
||||
final fkSafe = await _foreignKeyProjector.safeIncomingData(
|
||||
insert.tableName,
|
||||
insert.uuidRowId,
|
||||
data,
|
||||
transaction,
|
||||
presence: presence,
|
||||
);
|
||||
final forcedNullColumns = {
|
||||
for (final MapEntry(key: columnName, value: value) in fkSafe.data.entries)
|
||||
if (value == null) columnName,
|
||||
};
|
||||
final plannedDomain = {
|
||||
...data,
|
||||
...?batchPlan?.domain[(insert.tableName, insert.uuidRowId)],
|
||||
for (final columnName in forcedNullColumns) columnName: null,
|
||||
};
|
||||
|
||||
final attempts = <MergeFieldKey, ProjectionAttempt>{};
|
||||
for (final MapEntry(key: columnName, value: authored) in data.entries) {
|
||||
if (projectionValuesEqual(plannedDomain[columnName], authored)) continue;
|
||||
|
||||
final fieldKey = (insert.tableName, insert.uuidRowId, columnName);
|
||||
final planReason = batchPlan?.reasons[fieldKey];
|
||||
final foreignKeyReason = fkSafe.attempts[fieldKey]?.reason;
|
||||
final reason = forcedNullColumns.contains(columnName)
|
||||
? foreignKeyReason ?? planReason
|
||||
: planReason ?? foreignKeyReason;
|
||||
if (reason == null) {
|
||||
throw StateError(
|
||||
'Projection changed ${insert.tableName}.$columnName on row '
|
||||
'${insert.uuidRowId} without recording why.',
|
||||
);
|
||||
}
|
||||
attempts[fieldKey] = (value: authored, reason: reason);
|
||||
}
|
||||
|
||||
return (domain: plannedDomain, attempts: attempts);
|
||||
}
|
||||
|
||||
/// Writes the field metadata collected for merged inserts, one call per
|
||||
/// table and node.
|
||||
///
|
||||
/// Grouped by node because the merge cache pairs every field it stores with
|
||||
/// the node that authored it, which is how the cached field resolves its own
|
||||
/// clock.
|
||||
Future<void> _flushInsertAttempts(
|
||||
_PendingInsertAttempts pending,
|
||||
MergeContext context,
|
||||
Transaction transaction,
|
||||
) async {
|
||||
if (pending.isEmpty) return;
|
||||
|
||||
final groups = pending.take();
|
||||
for (final group in groups) {
|
||||
await _foreignKeyProjector.recordInsertAttempts(
|
||||
group.tableName,
|
||||
group.rowIds,
|
||||
transaction,
|
||||
group.attempts,
|
||||
mergeCache: (fields: context.fields, node: group.node),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -355,6 +625,7 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
CrdtMergeUpdate update,
|
||||
Map<UuidValue, CrdtNode> remoteNodes,
|
||||
MergeContext context,
|
||||
Map<MergeFieldKey, Object?> authoredOverlays,
|
||||
Transaction transaction,
|
||||
) async {
|
||||
final rowKey = (update.tableName, update.uuidRowId);
|
||||
|
|
@ -387,31 +658,8 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
|
||||
if (!shouldApply) return;
|
||||
|
||||
final resolvedUpdates = await _uniqueResolver.resolveForIncomingUpdate(
|
||||
update,
|
||||
row,
|
||||
context,
|
||||
transaction,
|
||||
);
|
||||
final safeUpdate = await _foreignKeyProjector.safeIncomingData(
|
||||
update.tableName,
|
||||
update.uuidRowId,
|
||||
resolvedUpdates,
|
||||
transaction,
|
||||
);
|
||||
await _context.updateDomainRows(
|
||||
update.tableName,
|
||||
{update.uuidRowId},
|
||||
safeUpdate.data,
|
||||
transaction,
|
||||
);
|
||||
await _foreignKeyProjector.recordAttemptsForRows(
|
||||
update.tableName,
|
||||
{update.uuidRowId},
|
||||
resolvedUpdates.keys.toSet(),
|
||||
transaction,
|
||||
attemptedValues: safeUpdate.attempts,
|
||||
);
|
||||
authoredOverlays[(update.tableName, update.uuidRowId, update.columnName)] =
|
||||
update.value;
|
||||
}
|
||||
|
||||
Future<void> _applyMergeDelete(
|
||||
|
|
@ -514,7 +762,18 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
CrdtDataField? currentField,
|
||||
Transaction transaction,
|
||||
) async {
|
||||
if (currentField == null) {
|
||||
// Projection can create field metadata for a row this merge context loaded
|
||||
// before the row existed, so an unseen field may still be persisted. Adopt
|
||||
// it instead of inserting a second row onto the row/column unique index.
|
||||
final existingField =
|
||||
currentField ??
|
||||
await CrdtDataField.db.findFirstRow(
|
||||
_session,
|
||||
where: (t) => t.rowId.equals(row.id) & t.columnId.equals(column.id),
|
||||
transaction: transaction,
|
||||
);
|
||||
|
||||
if (existingField == null) {
|
||||
final insertedField = await CrdtDataField.db.insertRow(
|
||||
_session,
|
||||
CrdtDataField(
|
||||
|
|
@ -534,7 +793,7 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
);
|
||||
}
|
||||
|
||||
final updatedField = currentField.copyWith(
|
||||
final updatedField = existingField.copyWith(
|
||||
row: row,
|
||||
column: column,
|
||||
nodeId: remoteNode.id,
|
||||
|
|
@ -608,23 +867,13 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
Map<String, Object?> data,
|
||||
Transaction transaction,
|
||||
) async {
|
||||
final insertedCrdtRow = await _upsertMergeRow(
|
||||
insert.tableName,
|
||||
insert.uuidRowId,
|
||||
remoteNode,
|
||||
incomingHlc,
|
||||
null,
|
||||
transaction,
|
||||
);
|
||||
|
||||
final tableRow = _requireMergeInsertTableRow(insert);
|
||||
final patchedRow = _db.serializationManager.patchTableRow(tableRow, data);
|
||||
final patchedRow = _withPlannedDomainValues(tableRow, data);
|
||||
final mergingScopeId = _context.hlcManagerFor(transaction).normalizedScopeId;
|
||||
final scopedRow = patchedRow.copyWithScopeId(mergingScopeId);
|
||||
try {
|
||||
// Insert first: the domain primary key resolves creation races
|
||||
// atomically. The inner savepoint keeps a violation from poisoning the
|
||||
// surrounding transaction so the recovery update below can still run.
|
||||
// Domain insert first. A CRDT tracker row for this id, written before
|
||||
// the domain row, made SQLite reject a later null unique-FK insert.
|
||||
await DatabaseUtil.runInTransactionOrSavepoint(
|
||||
_db,
|
||||
transaction,
|
||||
|
|
@ -656,7 +905,14 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
);
|
||||
}
|
||||
|
||||
return insertedCrdtRow;
|
||||
return _upsertMergeRow(
|
||||
insert.tableName,
|
||||
insert.uuidRowId,
|
||||
remoteNode,
|
||||
incomingHlc,
|
||||
null,
|
||||
transaction,
|
||||
);
|
||||
}
|
||||
|
||||
Future<void> _applyMergeInsertForExistingRow(
|
||||
|
|
@ -666,6 +922,7 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
Hlc incomingHlc,
|
||||
Map<String, Object?> data,
|
||||
MergeContext context,
|
||||
Map<MergeFieldKey, Object?> authoredOverlays,
|
||||
Transaction transaction,
|
||||
) async {
|
||||
if (!await _domainRowOwnedByEffectiveScope(
|
||||
|
|
@ -682,7 +939,6 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
}
|
||||
|
||||
final (_, columnsByName) = _context.schema[insert.tableName]!;
|
||||
final updatedValues = <String, Object?>{};
|
||||
for (final MapEntry(key: columnName, value: schemaColumn)
|
||||
in columnsByName.entries) {
|
||||
if (columnName == 'id') continue;
|
||||
|
|
@ -700,40 +956,8 @@ extension CrdtMergeRecorderExtension on CrdtMutationRecorder {
|
|||
);
|
||||
|
||||
if (!shouldApply) continue;
|
||||
updatedValues[columnName] = data[columnName];
|
||||
}
|
||||
|
||||
var updatesToApply = updatedValues;
|
||||
if (updatesToApply.isNotEmpty) {
|
||||
updatesToApply = await _uniqueResolver.resolveForIncomingUpdates(
|
||||
tableName: insert.tableName,
|
||||
row: currentRow,
|
||||
updates: updatesToApply,
|
||||
context: context,
|
||||
transaction: transaction,
|
||||
);
|
||||
}
|
||||
|
||||
if (updatesToApply.isNotEmpty) {
|
||||
final safeUpdate = await _foreignKeyProjector.safeIncomingData(
|
||||
insert.tableName,
|
||||
insert.uuidRowId,
|
||||
updatesToApply,
|
||||
transaction,
|
||||
);
|
||||
await _context.updateDomainRows(
|
||||
insert.tableName,
|
||||
{insert.uuidRowId},
|
||||
safeUpdate.data,
|
||||
transaction,
|
||||
);
|
||||
await _foreignKeyProjector.recordAttemptsForRows(
|
||||
insert.tableName,
|
||||
{insert.uuidRowId},
|
||||
updatesToApply.keys.toSet(),
|
||||
transaction,
|
||||
attemptedValues: safeUpdate.attempts,
|
||||
);
|
||||
authoredOverlays[(insert.tableName, insert.uuidRowId, columnName)] =
|
||||
data[columnName];
|
||||
}
|
||||
|
||||
final rowKey = (insert.tableName, insert.uuidRowId);
|
||||
|
|
@ -941,16 +1165,3 @@ class _ForeignScopeRowCollision implements Exception {
|
|||
|
||||
final int? owningScopeId;
|
||||
}
|
||||
|
||||
extension on DatabaseSerializationManager {
|
||||
/// Patches a [TableRow] with the given data.
|
||||
T patchTableRow<T extends TableRow>(T row, Map<String, Object?> data) {
|
||||
final rowJson = Map<String, dynamic>.from(row.toJson() as Map);
|
||||
for (final column in row.table.crdtSyncableColumns) {
|
||||
if (data.containsKey(column.columnName)) {
|
||||
rowJson[column.fieldName] = data[column.columnName];
|
||||
}
|
||||
}
|
||||
return deserialize<T>(rowJson);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,11 @@
|
|||
import 'dart:typed_data';
|
||||
|
||||
import 'package:meta/meta.dart';
|
||||
import 'package:serverpod_database/serverpod_database.dart';
|
||||
import 'package:serverpod_serialization/serverpod_serialization.dart';
|
||||
|
||||
import 'types.dart';
|
||||
|
||||
/// Extensions on [String] used for SQL construction.
|
||||
@internal
|
||||
extension SqlStringExtension on String {
|
||||
|
|
@ -46,12 +50,53 @@ extension UuidValueExtension on Object? {
|
|||
UuidValue? toUuidValue() {
|
||||
final value = this;
|
||||
if (value == null) return null;
|
||||
if (value is UuidValue) return value;
|
||||
if (value is String) return UuidValue.withValidation(value);
|
||||
return UuidValueJsonExtension.fromJson(value);
|
||||
}
|
||||
}
|
||||
|
||||
/// Orders row keys by table name, then row id, for deterministic planning.
|
||||
@internal
|
||||
int compareMergeRowKeys(MergeRowKey left, MergeRowKey right) {
|
||||
final tableComparison = left.$1.compareTo(right.$1);
|
||||
if (tableComparison != 0) return tableComparison;
|
||||
return left.$2.uuid.compareTo(right.$2.uuid);
|
||||
}
|
||||
|
||||
/// Canonicalizes a domain/attempted value so UUID blobs round-trip as
|
||||
/// [UuidValue] rather than as SQLite byte lists.
|
||||
///
|
||||
/// Only values already shaped like a UUID are converted. Anything else is
|
||||
/// returned unchanged, so an ordinary string or number never pays for a thrown
|
||||
/// [FormatException] on the projection and outbound paths.
|
||||
@internal
|
||||
Object? canonicalDomainValue(Object? value) {
|
||||
if (value is UuidValue) return value;
|
||||
if (value is Uint8List) {
|
||||
return value.length == 16 ? UuidValue.fromByteList(value) : value;
|
||||
}
|
||||
if (value is String) {
|
||||
return _looksLikeUuid(value) ? UuidValue.withValidation(value) : value;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/// Whether [value] has the 8-4-4-4-12 shape of a canonical UUID string.
|
||||
bool _looksLikeUuid(String value) {
|
||||
if (value.length != 36) return false;
|
||||
for (var index = 0; index < 36; index++) {
|
||||
final code = value.codeUnitAt(index);
|
||||
if (index == 8 || index == 13 || index == 18 || index == 23) {
|
||||
if (code != 0x2d) return false;
|
||||
continue;
|
||||
}
|
||||
final isDigit = code >= 0x30 && code <= 0x39;
|
||||
final isLower = code >= 0x61 && code <= 0x66;
|
||||
final isUpper = code >= 0x41 && code <= 0x46;
|
||||
if (!isDigit && !isLower && !isUpper) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/// Extensions on [UuidValue?] used for UUID comparison.
|
||||
@internal
|
||||
extension UuidValueComparisonExtension on UuidValue? {
|
||||
|
|
|
|||
|
|
@ -132,6 +132,29 @@ class CrdtForeignKeyGraph {
|
|||
);
|
||||
}
|
||||
|
||||
/// Tables joined to [seedTables] by any chain of foreign keys.
|
||||
///
|
||||
/// Projection resolves cascade and repair to a fixed point over the foreign
|
||||
/// key graph, so a row can only be affected by a change in a table its own
|
||||
/// table can reach. Unique conflicts are resolved within one table and never
|
||||
/// cross this boundary, so loading the seeds' connected components is enough
|
||||
/// to reproduce a full-graph pass.
|
||||
Set<String> connectedTables(Iterable<String> seedTables) {
|
||||
final reached = <String>{};
|
||||
final queue = [...seedTables];
|
||||
while (queue.isNotEmpty) {
|
||||
final tableName = queue.removeLast();
|
||||
if (!reached.add(tableName)) continue;
|
||||
for (final edge in edgesByChildTable[tableName] ?? const <ForeignKeyEdge>[]) {
|
||||
queue.add(edge.parentTableName);
|
||||
}
|
||||
for (final edge in edgesByParentTable[tableName] ?? const <ForeignKeyEdge>[]) {
|
||||
queue.add(edge.childTableName);
|
||||
}
|
||||
}
|
||||
return reached;
|
||||
}
|
||||
|
||||
/// The child column names of foreign keys outgoing from [tableName].
|
||||
///
|
||||
/// When [columnNames] is provided, only edges whose child column is in the set
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,5 +1,5 @@
|
|||
import 'package:meta/meta.dart';
|
||||
import 'package:serverpod_database/serverpod_database.dart';
|
||||
import 'package:serverpod_database/serverpod_database.dart' hide Protocol;
|
||||
import 'package:serverpod_serialization/serverpod_serialization.dart';
|
||||
|
||||
import '../../crdt/extensions.dart';
|
||||
|
|
@ -361,56 +361,189 @@ class CrdtRecorderContext {
|
|||
);
|
||||
}
|
||||
|
||||
/// Row ids in [tableName] whose [columnName] holds one of [values].
|
||||
///
|
||||
/// Unlike [findVisibleReferencingRowIds] this keeps hidden rows: projection
|
||||
/// has to see a hidden child before it can decide whether the child comes
|
||||
/// back with its parent.
|
||||
Future<Set<UuidValue>> findDomainRowIdsWhereColumnIn({
|
||||
required String tableName,
|
||||
required String columnName,
|
||||
required Set<Object?> values,
|
||||
required Transaction transaction,
|
||||
}) => findDomainRowIdsWhereColumnsIn(
|
||||
tableName: tableName,
|
||||
valuesByColumn: {columnName: values},
|
||||
transaction: transaction,
|
||||
);
|
||||
|
||||
/// Row ids of [tableName] owed one of [values] on one of [columnNames].
|
||||
///
|
||||
/// A row that does not hold its authored value keeps it in a sparse attempted
|
||||
/// value, so this is how a pass finds the rows waiting for a claim or a
|
||||
/// parent it is about to free. The comparison literal is built by the same
|
||||
/// encoders the write path uses — the dynamic envelope from the protocol, then
|
||||
/// the structured-column encoder — which is what keeps it dialect correct:
|
||||
/// `jsonb(...)` against SQLite's binary JSONB, a jsonb literal on Postgres.
|
||||
Future<Set<UuidValue>> findRowIdsHoldingAttemptedValues({
|
||||
required String tableName,
|
||||
required Set<String> columnNames,
|
||||
required Set<Object?> values,
|
||||
required Transaction transaction,
|
||||
}) async {
|
||||
if (columnNames.isEmpty || values.isEmpty) return const {};
|
||||
|
||||
final (tableId, columns) = schema[tableName]!;
|
||||
final columnIds = {
|
||||
for (final columnName in columnNames) ?columns[columnName]?.id,
|
||||
};
|
||||
if (columnIds.isEmpty) return const {};
|
||||
|
||||
final encodedValues = {
|
||||
for (final value in values)
|
||||
if (value != null)
|
||||
ValueEncoder.instance.encodeColumnValue(
|
||||
CrdtDataAttemptedValue.t.value,
|
||||
Protocol().dynamicFieldToJson(value),
|
||||
),
|
||||
};
|
||||
if (encodedValues.isEmpty) return const {};
|
||||
|
||||
final scopeId = hlcManagerFor(transaction).normalizedScopeId;
|
||||
final result = await database.unsafeQuery(
|
||||
'''
|
||||
SELECT DISTINCT r."uuidRowId"
|
||||
FROM "crdt_data_attempted_value" a
|
||||
JOIN "crdt_data_fields" f ON f."id" = a."fieldId"
|
||||
JOIN "crdt_data_rows" r ON r."id" = f."rowId"
|
||||
WHERE r."scopeId" = $scopeId
|
||||
AND r."tblId" = $tableId
|
||||
AND f."columnId" IN (${columnIds.join(', ')})
|
||||
AND a."value" IN (${encodedValues.join(', ')})
|
||||
''',
|
||||
transaction: transaction,
|
||||
);
|
||||
|
||||
return _rowIds(result);
|
||||
}
|
||||
|
||||
/// Row ids in [tableName] whose columns hold one of the listed values.
|
||||
///
|
||||
/// A composite unique index is matched column by column, so the result is a
|
||||
/// superset of the exact tuples. Loading a few extra rows is what a whole
|
||||
/// table load did anyway, and the query stays a plain indexed lookup.
|
||||
Future<Set<UuidValue>> findDomainRowIdsWhereColumnsIn({
|
||||
required String tableName,
|
||||
required Map<String, Set<Object?>> valuesByColumn,
|
||||
required Transaction transaction,
|
||||
}) async {
|
||||
if (valuesByColumn.isEmpty) return const {};
|
||||
if (valuesByColumn.values.any((values) => values.isEmpty)) return const {};
|
||||
|
||||
final predicates = valuesByColumn.entries
|
||||
.map(
|
||||
(entry) =>
|
||||
'"${entry.key.escapeIdentifier()}" IN (${entry.value.sqlLiteralList()})',
|
||||
)
|
||||
.join(') AND (');
|
||||
|
||||
final result = await database.unsafeQuery(
|
||||
'''
|
||||
SELECT "id"
|
||||
FROM "${tableName.escapeIdentifier()}"
|
||||
WHERE ($predicates)
|
||||
''',
|
||||
transaction: transaction,
|
||||
);
|
||||
|
||||
return _rowIds(result);
|
||||
}
|
||||
|
||||
Future<Set<UuidValue>> findVisibleDomainRowIdsWhere({
|
||||
required String tableName,
|
||||
required List<String> predicates,
|
||||
required Transaction transaction,
|
||||
}) async {
|
||||
final (tableId, _) = schema[tableName]!;
|
||||
final scopeId = hlcManagerFor(transaction).normalizedScopeId;
|
||||
final result = await database.unsafeQuery(
|
||||
'''
|
||||
SELECT d."id"
|
||||
FROM "${tableName.escapeIdentifier()}" d
|
||||
LEFT JOIN "crdt_data_rows" r
|
||||
ON r."scopeId" = $scopeId AND r."tblId" = $tableId AND r."uuidRowId" = d."id"
|
||||
${_visibilityJoin(tableName, transaction)}
|
||||
WHERE (${predicates.join(') AND (')})
|
||||
AND (r."id" IS NULL OR r."visibility" <= $crdtRowLastVisibleVisibilityIndex)
|
||||
AND ($_rowVisible)
|
||||
''',
|
||||
transaction: transaction,
|
||||
);
|
||||
return {
|
||||
for (final row in result) UuidValueJsonExtension.fromJson(row.first),
|
||||
};
|
||||
return _rowIds(result);
|
||||
}
|
||||
|
||||
/// The `FROM` and `JOIN` pairing domain rows of [tableName] with their CRDT
|
||||
/// metadata, as the aliases `d` and `r`.
|
||||
///
|
||||
/// An untracked row has no metadata, so `r` is null for it; [_rowVisible]
|
||||
/// reads that pair as one boolean.
|
||||
String _visibilityJoin(String tableName, Transaction transaction) {
|
||||
final (tableId, _) = schema[tableName]!;
|
||||
final scopeId = hlcManagerFor(transaction).normalizedScopeId;
|
||||
return '''
|
||||
FROM "${tableName.escapeIdentifier()}" d
|
||||
LEFT JOIN "crdt_data_rows" r
|
||||
ON r."scopeId" = $scopeId AND r."tblId" = $tableId AND r."uuidRowId" = d."id"''';
|
||||
}
|
||||
|
||||
/// Whether the row joined by [_visibilityJoin] is visible in this scope.
|
||||
static final _rowVisible =
|
||||
'r."id" IS NULL OR r."visibility" <= $crdtRowLastVisibleVisibilityIndex';
|
||||
|
||||
/// Presence of the [value] target in [parentTableName].
|
||||
///
|
||||
/// A foreign key can only reference a unique column, so at most one row
|
||||
/// answers; a value with no row at all is [ForeignKeyTargetPresence.absent].
|
||||
Future<ForeignKeyTargetPresence> lookupForeignKeyTargetPresence({
|
||||
required String parentTableName,
|
||||
required String parentColumn,
|
||||
required UuidValue value,
|
||||
required Transaction transaction,
|
||||
}) async {
|
||||
final (tableId, _) = schema[parentTableName]!;
|
||||
final presences = await lookupForeignKeyTargetPresences(
|
||||
parentTableName: parentTableName,
|
||||
parentColumn: parentColumn,
|
||||
values: {value},
|
||||
transaction: transaction,
|
||||
);
|
||||
return presences[value] ?? ForeignKeyTargetPresence.absent;
|
||||
}
|
||||
|
||||
/// Presence of every [values] target in [parentTableName], by value.
|
||||
///
|
||||
/// Resolving a whole merge batch at once is what keeps the single-value form
|
||||
/// off the insert path, where it asked once per foreign key of every row.
|
||||
/// Values with no row are absent from the result rather than reported.
|
||||
Future<Map<UuidValue, ForeignKeyTargetPresence>> lookupForeignKeyTargetPresences({
|
||||
required String parentTableName,
|
||||
required String parentColumn,
|
||||
required Set<UuidValue> values,
|
||||
required Transaction transaction,
|
||||
}) async {
|
||||
if (values.isEmpty) return const {};
|
||||
|
||||
final scopeId = hlcManagerFor(transaction).normalizedScopeId;
|
||||
final result = await database.unsafeQuery(
|
||||
'''
|
||||
SELECT CASE
|
||||
WHEN r."id" IS NULL OR r."visibility" <= $crdtRowLastVisibleVisibilityIndex THEN 1
|
||||
ELSE 0
|
||||
END AS visible
|
||||
FROM "${parentTableName.escapeIdentifier()}" d
|
||||
LEFT JOIN "crdt_data_rows" r
|
||||
ON r."scopeId" = $scopeId AND r."tblId" = $tableId AND r."uuidRowId" = d."id"
|
||||
SELECT d."${parentColumn.escapeIdentifier()}",
|
||||
CASE WHEN $_rowVisible THEN 1 ELSE 0 END AS visible
|
||||
${_visibilityJoin(parentTableName, transaction)}
|
||||
WHERE (${domainColumnPredicate('scopeId', scopeId)})
|
||||
AND (${domainColumnPredicate(parentColumn, value)})
|
||||
LIMIT 1
|
||||
AND (d."${parentColumn.escapeIdentifier()}" IN (${values.sqlLiteralList()}))
|
||||
''',
|
||||
transaction: transaction,
|
||||
);
|
||||
if (result.isEmpty) return ForeignKeyTargetPresence.absent;
|
||||
return (result.first[0] as num) == 1
|
||||
? ForeignKeyTargetPresence.visible
|
||||
: ForeignKeyTargetPresence.hidden;
|
||||
|
||||
return {
|
||||
for (final row in result)
|
||||
UuidValueJsonExtension.fromJson(row.first): (row[1] as num) == 1
|
||||
? ForeignKeyTargetPresence.visible
|
||||
: ForeignKeyTargetPresence.hidden,
|
||||
};
|
||||
}
|
||||
|
||||
Future<void> updateDomainRows(
|
||||
|
|
@ -435,6 +568,11 @@ WHERE "id" IN (${rowIds.sqlLiteralList()})
|
|||
);
|
||||
}
|
||||
|
||||
/// The `id` column of every row of a result whose first column is a row id.
|
||||
Set<UuidValue> _rowIds(DatabaseResult result) => {
|
||||
for (final row in result) UuidValueJsonExtension.fromJson(row.first),
|
||||
};
|
||||
|
||||
Future<Map<UuidValue, Map<String, Object?>>> readDomainColumnValues(
|
||||
String tableName,
|
||||
Set<UuidValue> rowIds,
|
||||
|
|
|
|||
|
|
@ -41,3 +41,26 @@ typedef MergeNodes = ({
|
|||
Map<UuidValue, CrdtNode> nodesByUuid,
|
||||
Map<UuidValue, CrdtScopeNode> scopeNodesByUuid,
|
||||
});
|
||||
|
||||
/// A row that will be written after projection, included in unique/FK planning.
|
||||
@internal
|
||||
typedef PendingProjectionRow = ({
|
||||
String tableName,
|
||||
UuidValue rowId,
|
||||
Map<String, Object?> authoredValues,
|
||||
Hlc rowHlc,
|
||||
CrdtNode node,
|
||||
bool hidden,
|
||||
});
|
||||
|
||||
/// The outcome of one projection pass.
|
||||
///
|
||||
/// `domain` holds the final materialized value per row and column; `reasons`
|
||||
/// holds the terminal projector that selected a value that differs from the
|
||||
/// authored one, so callers that persist attempted values after their own
|
||||
/// write record the real cause instead of guessing.
|
||||
@internal
|
||||
typedef ProjectionPlan = ({
|
||||
Map<MergeRowKey, Map<String, Object?>> domain,
|
||||
Map<MergeFieldKey, CrdtProjectionReason> reasons,
|
||||
});
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue