serverpod-offline-sync/README.md

# Serverpod Offline Sync

[![GitHub](https://img.shields.io/badge/GitHub-marcelomendoncasoares-181717.svg?style=flat&logo=github)](https://github.com/marcelomendoncasoares)
[![Pub Package](https://img.shields.io/pub/v/serverpod_offline_sync.svg)](https://pub.dev/packages/serverpod_offline_sync)
[![Dart SDK Version](https://badgen.net/pub/sdk-version/serverpod_offline_sync)](https://pub.dev/packages/serverpod_offline_sync)
[![License: BSD 3-Clause](https://img.shields.io/badge/license-BSD_3--Clause-yellow.svg)](https://github.com/marcelomendoncasoares/serverpod_offline_sync/blob/main/LICENSE)
[![CI](https://github.com/marcelomendoncasoares/serverpod_offline_sync/actions/workflows/ci.yaml/badge.svg?branch=main)](https://github.com/marcelomendoncasoares/serverpod_offline_sync/actions/workflows/ci.yaml)

A plugin to build offline-first applications with Serverpod. Develop your app
as a normal offline app and use this non-invasive plugin to automatically
handle the sync process using CRDT (Conflict-free Replicated Data Type). Works
out-of-the-box with Serverpod's existing APIs.

- [Serverpod Offline Sync](#serverpod-offline-sync)
  - [Overview](#overview)
  - [Features](#features)
  - [Getting started](#getting-started)
    - [1. Add the package to the project](#1-add-the-package-to-the-project)
    - [2. Prepare the synced tables](#2-prepare-the-synced-tables)
    - [3. Configure the sync engine on the client](#3-configure-the-sync-engine-on-the-client)
    - [4. Wire the sync call on the client](#4-wire-the-sync-call-on-the-client)
  - [Usage](#usage)
    - [Spaces and sharing](#spaces-and-sharing)
    - [Data modeling limitations](#data-modeling-limitations)
      - [Reserved values](#reserved-values)
  - [How it works](#how-it-works)
  - [Performance](#performance)
  - [Example](#example)
  - [Additional information](#additional-information)
  - [License](#license)

---

## Overview

**Serverpod Offline Sync** turns an ordinary [Serverpod](https://serverpod.dev)
backend into an offline-first, collaborative system, without changing how you
write your app. It is a delta-CRDT engine that runs **the same code on the
client and server databases**, transparently and with minimal configuration.

It sits nearly alone in the space of offline-first solutions, because most
others either use a non-relational database, require a new data API, or don't
preserve foreign-key/unique invariants while merging data. This package is built
on top of Serverpod, leveraging its ORM, server, and client to expose these
capabilities with almost nothing new to learn: you build a normal offline
Serverpod app (generated models, ordinary CRUD) and the plugin handles
replication automatically.

The sync engine is a **black box that "just works"**: changes are tracked
atomically with each operation and you never intervene in conflict resolution.
Conflicts that need a human decision **bubble up as visible state** for the end
user to act on (e.g. a delete reverted by a concurrent constraint reappears, so
the user can simply delete it again). Migrations work normally, and deleted rows
never leave their tables — they are transparently hidden from the user.

> The CRDT layer was inspired by the excellent work on
> [Synql](https://github.com/coast-team/synql)
> ([Ignat et al., DAIS 2024](https://inria.hal.science/hal-04969158v4/document)).
> This package goes beyond Synql by covering the full set of foreign-key
> `onDelete` actions, extending unique-constraint handling to deeply relational
> schemas, and shipping a production-ready implementation.

## Features

- **Offline-first, no new API.** Read and write generated models with ordinary
  Serverpod CRUD against a local database; the plugin replicates for you.
- **Same code on both ends.** Identical logic runs on the client's SQLite and
  the server's Postgres, so there are no asymmetric merge results to reconcile.
- **Automatic conflict resolution.** Field-level merges ordered by hybrid
  logical clocks, with a monotone causal-length tombstone governing row
  existence so add/delete/restore can never oscillate.
- **Relational integrity preserved.** All foreign-key `onDelete` actions
  (restrict / cascade / set null / set default) and unique constraints are
  honored after every merge, as if the merged data had existed in a single
  database at the time of each operation.
- **Conflicts surface to the user.** Resolutions that require a human decision
  become visible state instead of silent data loss.
- **Spaces and sharing.** Data is isolated per space; each user has a personal
  space and can be a member of any number of read-only or read-write shared
  spaces.
- **One-shot and continuous sync.** The same stream-based protocol powers both
  pull-to-refresh and near-real-time streaming.
- **Minimal overhead.** Metadata is normalized and never copies row data,
  keeping the storage and performance penalty small (see [Performance](#performance)).

## Getting started

Using the package on a project takes just five steps. After this setup, you
develop the app with normal repository calls against the wrapped `session`.

> [!NOTE]
> This package is still in development and is not yet ready for production use.
> Although the package is functional and feature-complete, it will still pass
> through some refactors and breaking changes for a better integration on real
> Serverpod projects.

### 1. Add the package to the project

Add the packages to the server and client `pubspec.yaml` files.

```yaml
# your_project_client/pubspec.yaml
dependencies:
  serverpod_offline_sync_client: 0.0.10
```

```yaml
# your_project_server/pubspec.yaml
dependencies:
  serverpod_offline_sync_server: 0.0.10
```

> [!NOTE]
> Version `0.0.10` requires Serverpod `>=4.0.4 <4.1.0`.

After adding the dependencies, list the `serverpod_offline_sync` module on the
`generator.yaml` file and enable the experimental `database: sync` option:

```yaml
modules:
  serverpod_offline_sync:
    nickname: offline_sync

experimental_features:
  databaseSync: true
```

The next call to `serverpod generate` and `serverpod create-migration` will
include the `offline_sync` module in the project. Note that the client tables
will only be present on the client's migration if at least one model on the
project is defined with `database: sync`, `database: all`, or `database: client`.

### 2. Prepare the synced tables

Each model you want to sync must be defined with `database: sync`. That generates
the table on both the client and the server, uses a UUID primary key, and
injects the `spaceId` ownership field. You mostly never set `spaceId` nor see
its value on reads; it is maintained by the CRDT sync layer.

```yaml
class: Person
table: person
database: sync
fields:
  name: String
```

The `serverpod generate` command will emit the `syncTables` list on the server
and the client. It will also wire the sync engine on the generated `Serverpod`
class, so no extra configuration is needed on the server.

### 3. Configure the sync engine on the client

The generated client opens a CRDT-wrapped session for the same tables:

```dart
final session = await client.createSyncSession(
  databasePath,
  isDebugMode: kDebugMode,
  persistentUserId: persistentUserId,
);
```

Then, store the `session` instance in the service locator.

### 4. Wire the sync call on the client

Be sure to authenticate the client before calling the sync methods. Otherwise,
the sync operation will fail with a `Unauthorized` error.

```dart
// Push local changes and merge remote ones once (i.e. pull-to-refresh).
await client.offlineSync.syncOnce(session);

// Or stream changes both ways until cancelled for a near-real-time sync.
final syncSession = client.offlineSync.syncContinuously(session);
```

When disposing a client replica, cancel its sync subscription before closing
the local database:

```dart
await syncSession.cancel();
await session.close();
```

## Usage

Once wired up, use normal generated-model CRUD against the `session` instance,
like `Person.db.insertRow(session, person)`, `Person.db.find(session)`, etc.
The sync layer tracks every operation atomically; you never touch conflict
resolution.

### Spaces and sharing

Data is isolated per **space**. Each user has their own implicit personal space
and can be a member of any number of shared spaces, with read-only or read-write
access.

- **On the server**, wrap database operations in `session.db.transactionForUser`
  to target the correct space.
- **On the client**, set a `persistentUserId` to target the personal space by
  default, or use `transactionForUser` to target a shared space.

Shared spaces are created and managed **on the server only** through the
`session.offlineSync.spaces` API.

```dart
// Create a shared space granting read-write access to a single user.
final spaceId = await session.offlineSync.spaces.createFor(userUuid);

// Create a shared space with multiple members.
final spaceId = await session.offlineSync.spaces.create(
  grants: {
    userUuid: OfflineSyncSpaceRole.readWrite,
    anotherUserUuid: OfflineSyncSpaceRole.readOnly,
  },
  // It is also possible to pass a transaction object to all operations.
  transaction: transaction,
)

// Add or update a member's role (readOnly or readWrite).
await session.offlineSync.spaces.grant(
  space: spaceId,
  user: bobUuid,
  role: OfflineSyncSpaceRole.readWrite,
);

// Remove a member. No-op if they are not already a member.
await session.offlineSync.spaces.revoke(space: spaceId, user: bobUuid);
```

If a streaming sync is ongoing, clients learn of new spaces and role changes on
the next sync cycle and apply them immediately. There is no need to restart the
sync session for changes to take effect.

> The package does not expose a management endpoint, since this is a domain
> responsibility. If desired, create your own endpoints where you can enforce
> an authorization policy (invitations, team ownership, etc.) and call the
> `session.offlineSync.spaces` API accordingly.

### Data modeling limitations

Because of the nature of merge conflicts, the package imposes some data-modeling
limitations. With `database: sync`, Serverpod reports most of them at generate
time. The rest are enforced at runtime during initialization. Most are
fundamental to the design and can never be lifted.

- Every synced table must be defined with `database: sync`.
  - Will exist on both the client and the server, like `database: all`.
  - The type of the `id` will be `UuidValue` with `defaultPersist=random_v7`.
  - A `spaceId` field will be added as a cascade relation to `offline_sync_spaces`.
  - Both `id` and `spaceId` can be manually declared for more control.
- Unique indexes must include `spaceId` together with the target columns.
- Global unique indexes are unsupported, except for FK-only indexes.
- Unique indexes are only supported with at least one
  `String`/`UuidValue`/nullable column.
- All 1:1 relations must have the foreign-key column nullable (`optional`
  relation).
- The only allowed non-synced-to-synced relation is `spaceId -> offline_sync_spaces.id`.
- Non-nullable foreign-key relations must be declared as `deferred`.
- Foreign-key action `onDelete=Restrict` must be replaced by `onDelete=NoAction`.

Respecting these limitations, all other database invariants — including
foreign-key actions — are preserved, with the exception of check constraints,
which Serverpod does not support either.

#### Reserved values

Besides the schema restrictions, some values are reserved by the sync layer and
cannot be authored by the user. Such values are used by the engine to resolve
unique conflicts and are not expected to represent a restriction to real
production usage. Trying to author a reserved value will immediately throw an
`OfflineSyncReservedValueException` exception to prevent later conflicts.

- Unique text values cannot be authored with a suffix matching
  `__conflict__<UUID>`, `__hidden__<UUID>`, or `__park__<UUID>`. Non-unique
  text fields are unrestricted.
- Non-nullable unique UUID columns that are not foreign keys reserve version 8
  for generated alternatives. Primary keys, foreign keys, nullable UUID columns
  and non-unique UUID columns are unrestricted.

## How it works

- **Architecture.** The engine is a transparent layer between the application
  and its database. Its models and database implementation live in the shared
  package; the client package only binds the generated module caller to the
  shared sync transport. Because identical code runs on the client's SQLite and
  the server's Postgres, there are no asymmetric merge results to reconcile.
  Migrations are generated symmetrically for both ends. The engine respects the
  three mathematical properties a CRDT must hold to be consistent: idempotence,
  associativity, and commutativity.
- **Metadata model.** Four metadata tables track change history and relational
  projection state: `crdt_data_rows` (insert + visibility), `crdt_data_fields`
  (update), `crdt_data_foreign_key` (attempted and materialized foreign-key
  values), and `crdt_data_tombstone` (delete/restore). Hybrid logical clocks
  order field-level values and break ties. A monotone causal-length tombstone
  governs row existence, so add/delete/restore advance generations forward and
  can never oscillate.
- **Visibility as a pure function.** Replicas never coordinate. Each merge
  applies remote facts monotonically, then derives the visible database —
  tombstone state, unique-conflict winners, and foreign-key repair — as a
  deterministic projection computed to a fixed point.
- **Ownership & spaces.** A *space* is the unit of replication and ownership; a
  *user* is an authentication identity. Each row is globally identified by
  `(table, rowId)` and owned by exactly one space, so merging is always one chain
  per row. Spaces can be purged for GDPR / account erasure via a
  database-enforced cascade.
- **Sync protocol.** A session handshakes once to validate that both ends share
  the same table schema. Per cycle, the server sends the authoritative space set,
  and every space the user can access is synced (transferred all at once in
  chunks, merged one at a time). The same stream-based implementation powers both
  one-shot and continuous sync.

For a deeper dive, see the design docs in [`docs/`](docs):
[row ownership](docs/row-ownership.md),
[foreign-key invariants](docs/foreign-key-invariants.md),
[unique-constraint invariants](docs/unique-constraint-invariants.md), and
[sync/non-sync relations](docs/sync-non-sync-relations.md).

## Performance

Since all operations are tracked, a CRDT layer can weigh heavily on storage and
speed. This package was built to keep that penalty minimal. The benchmark below
uses 1,000 rows to measure the impact against a baseline of the same operations
without the CRDT layer, both on SQLite (a fresh report is posted as a comment
on every pull request):

```
📊 SELECT performance impact:
  Time: 12.62 ms --> 12.67 ms (48.50 μs)
  CRDT overhead: 0.38% slower

📊 INSERT performance impact:
  Delay per insert: 231.04 μs

📊 UPDATE performance impact:
  Delay per update: 1.88 ms

📊 DELETE performance impact:
  Delay per delete: 408.75 μs

💽 Storage impact (base footprint removed):
  Storage overhead range: 62.07% --> 300.00%
```

Despite the high *percentage* slowdown on insert/update/delete, the absolute
impact is mostly in the microsecond range — barely noticeable in practice, and
far below the network delay those operations would incur through an online
endpoint. Storage tops out at ~3× the base size, an acceptable trade for
offline-first speed and availability.

## Example

The bundled desktop demo shows the model directly: two independent replicas of
one account side by side, plus the server's merged truth. Drive each replica's
sync independently and watch them converge. It visualizes hidden / tombstoned
rows and every foreign-key `onDelete` action, with guided scenarios for the
restrict / cascade / set-null / set-default / unique-conflict cases.

Run it by starting the test server — the app launches alongside it:

```sh
cd test/serverpod_offline_sync_test_server
serverpod start
```

See [`example/offline_sync_demo`](example/offline_sync_demo) for details; the
entire integration lives in ~60 lines in
[`lib/src/offline_replica.dart`](example/offline_sync_demo/lib/src/offline_replica.dart).

## Additional information

The package is tested extensively following Serverpod's testing principles. The
vast majority of the tests live in the `serverpod_offline_sync_test_server`
package, whose server is shared with the `example/offline_sync_demo` project.

## License

This package is distributed under the BSD 3-Clause License. See the LICENSE
file for more information.