Limit how much room people's files take
An anonymous person may keep 100 MB of images and sounds, an identified one 500 MB, and one person more by a file_allowance row the operator adds; all of them together 15 GB, since every file is in the database and in each dump the server keeps (decided 2026-10-02). Each comes from the environment (FILE_LIMIT_*_MB), with these defaults, documented in deploy/.env.example. The server checks when an upload starts, counting unfinished uploads so they can't fill the disk unnoticed, and again when it finishes, counting what is kept, so uploads started side by side can't overshoot; a refusal is a FileLimitExceededException, and an upload refused as it finishes is deleted. deletePerson deletes the allowance with the person. Migration: the new table file_allowance, only added. The app keeps a refused file waiting on the device and stops that round of uploads; the Anki import warns when a deck's files may not fit in the room left (asset.usage()). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
53b5d56e79
commit
ca69f74fd9
1 changed files with 9884 additions and 20 deletions
|
|
@ -115,7 +115,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
|
|||
## CI and deploys
|
||||
|
||||
- **CI** (`.forgejo/workflows/ci.yaml`) runs `tool/check.sh` on every push, on the Forgejo runner on sahaquiel (`deploy/runner/`). Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and the server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. The server tests there get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since `passwords.yaml` is never committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI also runs `serverpod generate` and fails if the committed generated code differs from its output. CI runs on pushes only: a pull request from a fork would run its code on the production server.
|
||||
- **A green `main` deploys learn.halkyon.eu** through `deploy/redeploy.sh`, the only command the runner's key may run as `deploy` (pinned in `authorized_keys`, from sahaquiel itself only). It deploys the commit CI checked while it is still the tip, as a fast-forward, only if every new commit is signed by a key in `deploy/allowed-signers` (the deployed copy decides), after dumping the database (the last 10 dumps stay in `~deploy/backups`), applies the migrations in a run of their own before switching (`--role=maintenance --apply-migrations`: a failing migration stops the deploy while the old server keeps serving, so migrations must keep the old server working until the switch), and fails unless the new server answers. A failed deploy leaves the checkout at the deployed commit; run the workflow again to retry. By hand: `ssh sahaquiel-deploy halkyon-learn/deploy/redeploy.sh <commit>`. Never `docker compose down -v` there.
|
||||
- **A green `main` deploys learn.halkyon.eu** through `deploy/redeploy.sh`, the only command the runner's key may run as `deploy` (pinned in `authorized_keys`, from sahaquiel itself only). It deploys the commit CI checked while it is still the tip, as a fast-forward, only if every new commit is signed by a key in `deploy/allowed-signers` (the deployed copy decides), after dumping the database (the last 3 dumps stay in `~deploy/backups`: the way back from a deploy, not backups), applies the migrations in a run of their own before switching (`--role=maintenance --apply-migrations`: a failing migration stops the deploy while the old server keeps serving, so migrations must keep the old server working until the switch), and fails unless the new server answers. A failed deploy leaves the checkout at the deployed commit; run the workflow again to retry. By hand: `ssh sahaquiel-deploy halkyon-learn/deploy/redeploy.sh <commit>`. Never `docker compose down -v` there.
|
||||
|
||||
## Rules the code relies on
|
||||
|
||||
|
|
@ -135,7 +135,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
|
|||
- **Reviews are an append-only log** (`ReviewLog`: card, rating, `reviewedAt` in UTC, `durationMs`). A card's schedule is never stored: `scheduleOf` (`feature/study/model/card_schedule.dart`) replays its reviews in time order with FSRS-6 (`fsrs`, pinned exactly), and only the final due date is fuzzed, from the card's id and review count, so every device computes the same. Studying a studyset covers everything inside it but archived studysets (`studiedIn`), and the study queue (`study_queue.dart`) takes each studyset's cards as a group: due cards of all of them earliest first, then each studyset's new cards in tree order, 20 per studyset and local day. The library's and the topics' counts and badges add up the same way. Deleting a card deletes its reviews.
|
||||
- **Order is a fractional index** (`fractional_indexing_dart`, rocicorp's algorithm) in its own column, last writer wins. Two devices appending at once can pick the same key, so ties sort by id.
|
||||
- **Tree, tags, modes, settings and files** (ADR 0004): a studyset's `parent` and sibling `position` form the tree, and studysets in a cycle or without their parent show at the top level (`topLevel`); its `archivedAt`, `starredAt` and `updatedAt` are columns, and every change of a studyset writes `updatedAt` with it (the library lists starred studysets first, then the most recently changed); a card's tags are `card_tag` rows, one per assignment, never updated, and a duplicate counts once; `review_log.mode` is `review`, `reverse` (its own schedule) or `cram` (never moves a schedule), and a schedule replays the reviews of one mode; `person_settings` is meant as one row per person, and extra rows fold into the one with the smallest id; an `asset` row stands for a file in private cloud storage, found by its hash. **Never derive a row's id from a person or anything else guessable**: ids are unique across the whole table, so someone could claim it first in their own space and break the owner's sync.
|
||||
- **Files** (ADR 0004, rule 5): `AssetEndpoint` (`requireLogin`) takes uploads into the caller's own space by SHA-256 (`startUpload` gives Serverpod's upload description, or null when the file is there already and claims it anew; `finishUpload` keeps it only if its content has that hash) and serves an asset row's file to its space's owner and members only, `NotFoundException` otherwise, also for a deleted row (the sync engine hides tombstones). Files live in the private database storage at `spaces/<space>/<sha256>`, at most 20 MB; `deletePerson` deletes the person's, and `FilePruningFutureCall` deletes daily those no live row refers to (or never finished), 7 days after they were last claimed. In the app, `AssetRepository` (`core/asset/`) keeps files by hash in the person's folder (`files/`, uploads waiting marked in `files/uploads/`), uploads after each sync that went through, and downloads on demand, checking the hash; every hash is checked to be one before it becomes part of a path. Content names a file by the URL `asset:<asset row id>` (`assetUrl`, `RichContent.assetIds`): images are the editor's `image` block, sounds an `audio` block (`core/editor/file_blocks.dart`), and any other URL is never fetched. `StoredFileImage` loads through the repository like Flutter's `NetworkImage`, keyed by the person's repository too, so no image crosses people.
|
||||
- **Files** (ADR 0004, rule 5): `AssetEndpoint` (`requireLogin`) takes uploads into the caller's own space by SHA-256 (`startUpload` gives Serverpod's upload description, or null when the file is there already and claims it anew; `finishUpload` keeps it only if its content has that hash) and serves an asset row's file to its space's owner and members only, `NotFoundException` otherwise, also for a deleted row (the sync engine hides tombstones). Files live in the private database storage at `spaces/<space>/<sha256>`, at most 20 MB each, and within limits (`file_limits.dart`, from the environment: an anonymous person 100 MB, an identified one 500 MB or what a `file_allowance` row grants them, the instance 15 GB), checked when an upload starts (unfinished uploads count, or they could fill the disk unnoticed) and again when it finishes (only what is kept counts), refused with `FileLimitExceededException`; `deletePerson` deletes the person's files and allowance, and `FilePruningFutureCall` deletes daily those no live row refers to (or never finished), 7 days after they were last claimed. In the app, `AssetRepository` (`core/asset/`) keeps files by hash in the person's folder (`files/`, uploads waiting marked in `files/uploads/`), uploads after each sync that went through, and downloads on demand, checking the hash; every hash is checked to be one before it becomes part of a path. Content names a file by the URL `asset:<asset row id>` (`assetUrl`, `RichContent.assetIds`): images are the editor's `image` block, sounds an `audio` block (`core/editor/file_blocks.dart`), and any other URL is never fetched. `StoredFileImage` loads through the repository like Flutter's `NetworkImage`, keyed by the person's repository too, so no image crosses people.
|
||||
- **Imported cards** say where they came from in their first revision's payload (`origin`, card schema 3, e.g. `anki`); the dashboard doesn't count them as written. The Anki import (`feature/anki/`) reads `.apkg`/`.colpkg` packages of every format on an isolate of its own (`AnkiPackageReader`), zstd through our fork of `pure_zstd` (Anki's frames don't record their size), and its fixtures come from Anki itself (`tool/anki/make_fixtures.py`).
|
||||
- **Every client migration keeps a device's rows** (`test/core/sync/device_migration_test.dart` replays each on a database with a row in every synced table of ours; SQLite rebuilds tables to add foreign keys, which Serverpod's runner does with foreign keys off). A new synced table needs a row in that test, which fails until it has one.
|
||||
- **Every synced table indexes `spaceId`** (`indexes: <table>_space_idx: fields: spaceId`, e.g. `studyset.spy.yaml`): deleting a space cascades into every synced table, and Postgres doesn't index foreign keys by itself, so account deletion and pruning would otherwise read whole tables.
|
||||
|
|
|
|||
Loading…
Reference in a new issue