383 lines
16 KiB
Markdown
383 lines
16 KiB
Markdown
# Serverpod Offline Sync
|
||
|
||
[](https://github.com/marcelomendoncasoares)
|
||
[](https://pub.dev/packages/serverpod_offline_sync)
|
||
[](https://pub.dev/packages/serverpod_offline_sync)
|
||
[](https://github.com/marcelomendoncasoares/serverpod_offline_sync/blob/main/LICENSE)
|
||
[](https://github.com/marcelomendoncasoares/serverpod_offline_sync/actions/workflows/ci.yaml)
|
||
|
||
A plugin to build offline-first applications with Serverpod. Develop your app
|
||
as a normal offline app and use this non-invasive plugin to automatically
|
||
handle the sync process using CRDT (Conflict-free Replicated Data Type). Works
|
||
out-of-the-box with Serverpod's existing APIs.
|
||
|
||
- [Serverpod Offline Sync](#serverpod-offline-sync)
|
||
- [Overview](#overview)
|
||
- [Features](#features)
|
||
- [Getting started](#getting-started)
|
||
- [1. Add the package to the project](#1-add-the-package-to-the-project)
|
||
- [2. Prepare the synced tables](#2-prepare-the-synced-tables)
|
||
- [3. Configure the sync engine on the server](#3-configure-the-sync-engine-on-the-server)
|
||
- [4. Configure the sync engine on the client](#4-configure-the-sync-engine-on-the-client)
|
||
- [5. Wire the sync call on the client](#5-wire-the-sync-call-on-the-client)
|
||
- [Usage](#usage)
|
||
- [Scopes and sharing](#scopes-and-sharing)
|
||
- [Data modeling limitations](#data-modeling-limitations)
|
||
- [How it works](#how-it-works)
|
||
- [Performance](#performance)
|
||
- [Example](#example)
|
||
- [Additional information](#additional-information)
|
||
- [License](#license)
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
**Serverpod Offline Sync** turns an ordinary [Serverpod](https://serverpod.dev)
|
||
backend into an offline-first, collaborative system, without changing how you
|
||
write your app. It is a delta-CRDT engine that runs **the same code on the
|
||
client and server databases**, transparently and with minimal configuration.
|
||
|
||
It sits nearly alone in the space of offline-first solutions, because most
|
||
others either use a non-relational database, require a new data API, or don't
|
||
preserve foreign-key/unique invariants while merging data. This package is built
|
||
on top of Serverpod, leveraging its ORM, server, and client to expose these
|
||
capabilities with almost nothing new to learn: you build a normal offline
|
||
Serverpod app (generated models, ordinary CRUD) and the plugin handles
|
||
replication automatically.
|
||
|
||
The sync engine is a **black box that "just works"**: changes are tracked
|
||
atomically with each operation and you never intervene in conflict resolution.
|
||
Conflicts that need a human decision **bubble up as visible state** for the end
|
||
user to act on (e.g. a delete reverted by a concurrent constraint reappears, so
|
||
the user can simply delete it again). Migrations work normally, and deleted rows
|
||
never leave their tables — they are transparently hidden from the user.
|
||
|
||
> The CRDT layer was inspired by the excellent work on
|
||
> [Synql](https://github.com/coast-team/synql)
|
||
> ([Ignat et al., DAIS 2024](https://inria.hal.science/hal-04969158v4/document)).
|
||
> This package goes beyond Synql by covering the full set of foreign-key
|
||
> `onDelete` actions, extending unique-constraint handling to deeply relational
|
||
> schemas, and shipping a production-ready implementation.
|
||
|
||
## Features
|
||
|
||
- **Offline-first, no new API.** Read and write generated models with ordinary
|
||
Serverpod CRUD against a local database; the plugin replicates for you.
|
||
- **Same code on both ends.** Identical logic runs on the client's SQLite and
|
||
the server's Postgres, so there are no asymmetric merge results to reconcile.
|
||
- **Automatic conflict resolution.** Field-level merges ordered by hybrid
|
||
logical clocks, with a monotone causal-length tombstone governing row
|
||
existence so add/delete/restore can never oscillate.
|
||
- **Relational integrity preserved.** All foreign-key `onDelete` actions
|
||
(restrict / cascade / set null / set default) and unique constraints are
|
||
honored after every merge, as if the merged data had existed in a single
|
||
database at the time of each operation.
|
||
- **Conflicts surface to the user.** Resolutions that require a human decision
|
||
become visible state instead of silent data loss.
|
||
- **Scopes and sharing.** Data is isolated per scope; each user has a personal
|
||
scope and can be a member of any number of read-only or read-write shared
|
||
scopes.
|
||
- **One-shot and continuous sync.** The same stream-based protocol powers both
|
||
pull-to-refresh and near-real-time streaming.
|
||
- **Minimal overhead.** Metadata is normalized and never copies row data,
|
||
keeping the storage and performance penalty small (see [Performance](#performance)).
|
||
|
||
## Getting started
|
||
|
||
Using the package on a project takes just five steps. After this setup, you
|
||
develop the app with normal repository calls against the wrapped `session`.
|
||
|
||
> [!NOTE]
|
||
> This package is still in development and is not yet ready for production use.
|
||
> Although the package is functional and feature-complete, it will still pass
|
||
> through some refactors and breaking changes for a better integration on real
|
||
> Serverpod projects.
|
||
|
||
### 1. Add the package to the project
|
||
|
||
Add the packages to the server and client `pubspec.yaml` files.
|
||
|
||
```yaml
|
||
# your_project_client/pubspec.yaml
|
||
dependencies:
|
||
serverpod_offline_sync_client: 0.0.1
|
||
```
|
||
|
||
```yaml
|
||
# your_project_server/pubspec.yaml
|
||
dependencies:
|
||
serverpod_offline_sync_server: 0.0.1
|
||
```
|
||
|
||
> [!NOTE]
|
||
> Version `0.0.1` requires Serverpod `4.0.0-beta.0`.
|
||
|
||
After adding the dependencies, list the `serverpod_offline_sync` module on the
|
||
`generator.yaml` file:
|
||
|
||
```yaml
|
||
modules:
|
||
serverpod_offline_sync:
|
||
nickname: offline_sync
|
||
```
|
||
|
||
The next call to `serverpod generate` and `serverpod create-migration` will
|
||
include the `offline_sync` module in the project. Note that the client tables
|
||
will only be present on the client's migration if at least one model on the
|
||
project is defined with `database: all` or `database: client`.
|
||
|
||
### 2. Prepare the synced tables
|
||
|
||
Each model you want to sync must be defined with `database: all` (so it exists
|
||
on both the client and the server), have a UUID primary key, and declare a
|
||
`scopeId` field. You mostly never set `scopeId` nor see its value on reads; it
|
||
is maintained by the CRDT sync layer.
|
||
|
||
```yaml
|
||
class: Person
|
||
table: person
|
||
database: all
|
||
fields:
|
||
id: UuidValue?, defaultPersist=random_v7
|
||
### Owner scope of this row. Maintained by the CRDT sync layer.
|
||
### The user never sets this field nor sees its value on reads.
|
||
scopeId: int?, relation(parent=crdt_scopes, onDelete=Cascade)
|
||
name: String
|
||
```
|
||
|
||
### 3. Configure the sync engine on the server
|
||
|
||
On the server, the `databaseInterceptor` is what ensures that all `session.db`
|
||
objects are database instances that tracks operations for sync.
|
||
|
||
```dart
|
||
final pod = Serverpod(
|
||
args,
|
||
Protocol(),
|
||
Endpoints(),
|
||
// Add the database interceptor.
|
||
databaseInterceptor: crdtDatabaseInterceptor,
|
||
);
|
||
|
||
// Initialize the sync engine before pod.start() is called.
|
||
pod.initializeCrdtSync(
|
||
syncTables: [Person.t, Book.t, Author.t],
|
||
);
|
||
```
|
||
|
||
Don't forget to also configure the authentication following the regular
|
||
Serverpod auth setup. Authentication is required because the sync needs a
|
||
logged in user to synchronize data.
|
||
|
||
### 4. Configure the sync engine on the client
|
||
|
||
On the client, the regular `DatabaseSession` must be replaced by a wrapped
|
||
version that includes the sync engine.
|
||
|
||
```dart
|
||
final session = CrdtDatabaseSession.wraps(
|
||
await client.createSession(databasePath, isDebugMode: kDebugMode),
|
||
syncTables: [Person.t, Book.t, Author.t], // Must be the same as on the server.
|
||
persistentUserId: persistentUserId,
|
||
);
|
||
|
||
// Initialize the sync engine.
|
||
await session.db.initialize();
|
||
```
|
||
|
||
Then, store the `session` instance in the service locator.
|
||
|
||
> [!NOTE]
|
||
> The list of synced tables must be the same on the server and the client.
|
||
> Otherwise, all sync attempts will fail with a schema mismatch error.
|
||
|
||
### 5. Wire the sync call on the client
|
||
|
||
Be sure to authenticate the client before calling the sync methods. Otherwise,
|
||
the sync operation will fail with a `Unauthorized` error.
|
||
|
||
```dart
|
||
// Push local changes and merge remote ones once (i.e. pull-to-refresh).
|
||
await client.crdt.syncOnce(session);
|
||
|
||
// Or stream changes both ways until cancelled for a near-real-time sync.
|
||
final syncSession = client.crdt.syncContinuously(session);
|
||
```
|
||
|
||
## Usage
|
||
|
||
Once wired up, use normal generated-model CRUD against the `session` instance,
|
||
like `Person.db.insertRow(session, person)`, `Person.db.find(session)`, etc.
|
||
The sync layer tracks every operation atomically; you never touch conflict
|
||
resolution.
|
||
|
||
### Scopes and sharing
|
||
|
||
Data is isolated per **scope**. Each user has their own implicit personal scope
|
||
and can be a member of any number of shared scopes, with read-only or read-write
|
||
access.
|
||
|
||
- **On the server**, wrap database operations in `session.db.transactionForUser`
|
||
to target the correct scope.
|
||
- **On the client**, set a `persistentUserId` to target the personal scope by
|
||
default, or use `transactionForUser` to target a shared scope.
|
||
|
||
Shared scopes are created and managed **on the server only** through the
|
||
`session.crdt.scopes` API.
|
||
|
||
```dart
|
||
// Create a shared scope granting read-write access to a single user.
|
||
final scopeId = await session.crdt.scopes.createFor(userUuid);
|
||
|
||
// Create a shared scope with multiple members.
|
||
final scopeId = await session.crdt.scopes.create(
|
||
grants: {
|
||
userUuid: CrdtScopeRole.readWrite,
|
||
anotherUserUuid: CrdtScopeRole.readOnly,
|
||
},
|
||
// It is also possible to pass a transaction object to all operations.
|
||
transaction: transaction,
|
||
)
|
||
|
||
// Add or update a member's role (readOnly or readWrite).
|
||
await session.crdt.scopes.grant(
|
||
scope: scopeId,
|
||
user: bobUuid,
|
||
role: CrdtScopeRole.readWrite,
|
||
);
|
||
|
||
// Remove a member. No-op if they are not already a member.
|
||
await session.crdt.scopes.revoke(scope: scopeId, user: bobUuid);
|
||
```
|
||
|
||
If a streaming sync is ongoing, clients learn of new scopes and role changes on
|
||
the next sync cycle and apply them immediately. There is no need to restart the
|
||
sync session for changes to take effect.
|
||
|
||
> The package does not expose a management endpoint, since this is a domain
|
||
> responsibility. If desired, create your own endpoints where you can enforce
|
||
> an authorization policy (invitations, team ownership, etc.) and call the
|
||
> `session.crdt.scopes` API accordingly.
|
||
|
||
### Data modeling limitations
|
||
|
||
Because of the nature of merge conflicts, the package imposes some data-modeling
|
||
limitations, enforced at runtime during initialization. Most are fundamental to
|
||
the design and can never be lifted.
|
||
|
||
- Every synced table must:
|
||
- Exist on both the client and the server with `database: all`.
|
||
- Have a UUID primary key.
|
||
- Declare the field `scopeId: int?, relation(parent=crdt_scopes, onDelete=Cascade)`.
|
||
- Unique indexes must include `scopeId` together with the target columns.
|
||
- Global unique indexes are unsupported, except for FK-only indexes.
|
||
- Unique indexes are only supported with at least one
|
||
`String`/`UuidValue`/nullable column.
|
||
- All 1:1 relations must have the foreign-key column nullable (`optional`
|
||
relation).
|
||
- The only allowed non-synced-to-synced relation is `scopeId -> crdt_scopes.id`.
|
||
- All foreign-key relations must be `deferrable` (pending [serverpod#5338](https://github.com/serverpod/serverpod/issues/5338)).
|
||
|
||
Respecting these limitations, all other database invariants — including
|
||
foreign-key actions — are preserved, with the exception of check constraints,
|
||
which Serverpod does not support either.
|
||
|
||
## How it works
|
||
|
||
- **Architecture.** The engine is a transparent layer between the application
|
||
and its database. Its models and database implementation live in the shared
|
||
package; the client package only binds the generated module caller to the
|
||
shared sync transport. Because identical code runs on the client's SQLite and
|
||
the server's Postgres, there are no asymmetric merge results to reconcile.
|
||
Migrations are generated symmetrically for both ends. The engine respects the
|
||
three mathematical properties a CRDT must hold to be consistent: idempotence,
|
||
associativity, and commutativity.
|
||
- **Metadata model.** Three metadata tables track change history *without copying
|
||
row data*: `crdt_data_rows` (insert + visibility), `crdt_data_fields` (update),
|
||
and `crdt_data_tombstone` (delete/restore). Hybrid logical clocks order
|
||
field-level values and break ties. A monotone causal-length tombstone governs
|
||
row existence, so add/delete/restore advance generations forward and can never
|
||
oscillate.
|
||
- **Visibility as a pure function.** Replicas never coordinate. Each merge
|
||
applies remote facts monotonically, then derives the visible database —
|
||
tombstone state, unique-conflict winners, and foreign-key repair — as a
|
||
deterministic projection computed to a fixed point.
|
||
- **Ownership & scopes.** A *scope* is the unit of replication and ownership; a
|
||
*user* is an authentication identity. Each row is globally identified by
|
||
`(table, rowId)` and owned by exactly one scope, so merging is always one chain
|
||
per row. Scopes can be purged for GDPR / account erasure via a
|
||
database-enforced cascade.
|
||
- **Sync protocol.** A session handshakes once to validate that both ends share
|
||
the same table schema. Per cycle, the server sends the authoritative scope set,
|
||
and every scope the user can access is synced (transferred all at once in
|
||
chunks, merged one at a time). The same stream-based implementation powers both
|
||
one-shot and continuous sync.
|
||
|
||
For a deeper dive, see the design docs in [`docs/`](docs):
|
||
[row ownership](docs/row-ownership.md),
|
||
[foreign-key invariants](docs/foreign-key-invariants.md),
|
||
[unique-constraint invariants](docs/unique-constraint-invariants.md), and
|
||
[sync/non-sync relations](docs/sync-non-sync-relations.md).
|
||
|
||
## Performance
|
||
|
||
Since all operations are tracked, a CRDT layer can weigh heavily on storage and
|
||
speed. This package was built to keep that penalty minimal. The benchmark below
|
||
uses 1,000 rows to measure the impact against a baseline of the same operations
|
||
without the CRDT layer, both on SQLite (a fresh report is posted as a comment
|
||
on every pull request):
|
||
|
||
```
|
||
📊 SELECT performance impact:
|
||
Time: 12.62 ms --> 12.67 ms (48.50 μs)
|
||
CRDT overhead: 0.38% slower
|
||
|
||
📊 INSERT performance impact:
|
||
Delay per insert: 231.04 μs
|
||
|
||
📊 UPDATE performance impact:
|
||
Delay per update: 1.88 ms
|
||
|
||
📊 DELETE performance impact:
|
||
Delay per delete: 408.75 μs
|
||
|
||
💽 Storage impact (base footprint removed):
|
||
Storage overhead range: 62.07% --> 300.00%
|
||
```
|
||
|
||
Despite the high *percentage* slowdown on insert/update/delete, the absolute
|
||
impact is mostly in the microsecond range — barely noticeable in practice, and
|
||
far below the network delay those operations would incur through an online
|
||
endpoint. Storage tops out at ~3× the base size, an acceptable trade for
|
||
offline-first speed and availability.
|
||
|
||
## Example
|
||
|
||
The bundled desktop demo shows the model directly: two independent replicas of
|
||
one account side by side, plus the server's merged truth. Drive each replica's
|
||
sync independently and watch them converge. It visualizes hidden / tombstoned
|
||
rows and every foreign-key `onDelete` action, with guided scenarios for the
|
||
restrict / cascade / set-null / set-default / unique-conflict cases.
|
||
|
||
Run it by starting the test server — the app launches alongside it:
|
||
|
||
```sh
|
||
cd test/serverpod_offline_sync_test_server
|
||
serverpod start
|
||
```
|
||
|
||
See [`example/offline_sync_demo`](example/offline_sync_demo) for details; the
|
||
entire integration lives in ~60 lines in
|
||
[`lib/src/offline_replica.dart`](example/offline_sync_demo/lib/src/offline_replica.dart).
|
||
|
||
## Additional information
|
||
|
||
The package is tested extensively following Serverpod's testing principles. The
|
||
vast majority of the tests live in the `serverpod_offline_sync_test_server`
|
||
package, whose server is shared with the `example/offline_sync_demo` project.
|
||
|
||
## License
|
||
|
||
This package is distributed under the BSD 3-Clause License. See the LICENSE
|
||
file for more information.
|