halkyon-days/halkyon
Watch
1
0
Fork
You've already forked halkyon
 0
Code Issues Pull requests Projects Releases 40 Packages Wiki Activity Actions

build(train): Train's server and BRouter deployable beside the instance (ADR 0030, D46)

Browse source 
deploy/compose.train.yaml adds Train's server at train.halkyon.eu, a
member of Learn's register, with its database halkyon_train in the
instance's PostgreSQL, and BRouter beside it alone (v1.7.10, checked by
SHA-256, no log), its segments of the operator's region kept current
monthly. Nothing changes until deploy/.env names the compose file:
redeploy.sh and backup.sh take Train in only then, and the operator's
steps are in sidecars/README.md. restic now forgets snapshots grouped by
host alone, so adding Train's dump leaves no group behind for good.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-29 00:05:39 +01:00
commit a8f7c26e69
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
13 changed files with 483 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

2
apps/train/AGENTS.md
View file

@ -3,7 +3,7 @@
What holds for every app and package of this repository is in the [root AGENTS.md](../../AGENTS.md); it applies here in full, and Learn's patterns (`apps/learn/AGENTS.md`) wherever Train needs the same thing.
- **A shell on the shared packages**, Todo's copied and renamed (decided 2026-10-28, ADR 0030, D12; what Train is to do: ADR 0030): signing in, the instances, the account and the look work, and since step 5a (28 Oct 2026) a person's workouts are kept and synced with `serverpod_offline_sync`, every row sealed (ADR 0030, D7, D14, D15), and on Android a workout is recorded (5b: ADR 0030, D13, D16, D18 to D21), and since 5c home lists them and each has its page with its map (D22 to D24). Packages: `halkyon_train_server`, `halkyon_train_client` (generated, never edited), `halkyon_train_flutter`; app ID `app.halkyon.train` on every platform; no web build. Licences as Learn's (server SSPL, client and app AGPL; `LICENSE.md`).
- **The server** includes `halkyon_base` (ADR 0014) and configures it in `server.dart` (`trainBase()`, `lib/src/train_base.dart`): its basics say it serves `train` (never change it once apps use it), sign-in mails name Halkyon Train, the source is this repository, `minimumAppBuild` 1, and `deleteTrainData` (`lib/src/train_data.dart`) as its `deletePersonData` hook, which deletes the person's sync spaces and so, by the sync engine's cascades, every row of theirs (`test/integration/workout/person_deletion_test.dart`). Authentication through the module's builders: a password, SimpleLogin on the register, email codes proving an address, server-side sessions; no anonymous accounts (ADR 0020, D25), nor GitHub (D36). **Accounts** (ADR 0016): Train holds its own register when it runs alone, and is a member of the instance's register (Learn's server on learn.halkyon.eu) where the operator sets `ACCOUNT_REGISTER_URL` (`trainBase(accounts: AccountsMode.fromEnvironment())`) with the instance's shared `INSTANCE_SECRET` (`instanceSecret`); then sign-in, handles and profiles are the register's, the email settings are the register's alone, and an identified person's deletion goes to the register first. `addInstanceRoutes` serves the calls between the services on its web server, and the server doesn't start without the secret where it needs it. `schedulePruning` (`lib/src/pruning.dart`) keeps the module's pruning scheduled, one entry each however often the server starts (and cancels the one under the pruning's name before ADR 0020, D25): the anonymous accounts left from before daily, sign-up attempts and codes hourly, as the register the deletions its members haven't confirmed hourly, as a member the people the register no longer knows daily (each a no-op in the other mode). Session logs are never persisted, in any run mode. **The Loop's firmware relay** (ADR 0030, D10, D40, `lib/src/firmware/`): the `firmware` endpoint asks Polar's firmware service in the person's place and hands on the file, kept in memory a day per hardware and version (`FirmwareRelay`); on by default, off where the operator sets `LOOP_FIRMWARE_RELAY=off`, the app then offering the direct way alone; tested on a fake Polar (`test/integration/firmware/`). **Routes** (ADR 0030, D11, D44, `lib/src/routing/`): the `routing` endpoint plans a route through the instance's BRouter (`BROUTER_URL`; none: `offered` is false and plans are refused) for the people signed in, at most 50 waypoints and 30 plans a person a minute, nothing kept or logged; BRouter's GeoJSON read in `brouter.dart` (points, elevations, climbs above 3 m, turn hints, stretches by surface), tested on its format (`test/unit/routing/`) and on a fake BRouter (`test/integration/routing/`). Train's running profile is `apps/train/brouter/halkyon-run.brf`, which the sidecar's profiles hold beside BRouter's own. Migrations (2026-10-28): an initial one with Serverpod's tables, its auth modules' and the module's as they stand (Todo's grew one by one with the module; Train starts where they ended), then `workouts`, purely additive: the sync engine's tables and Train's first three, then `profile`, `days` (Train's `day`, `day_series` and `night`) and `plans` (`plan`, D42), each purely additive. **Synced models** (`database: sync`, the module `offline_sync`, `experimental_features: databaseSync` in `config/generator.yaml`): `content_key` (the person's key sealed to the account), `workout` and `workout_series` (a workout's series in chunks of at most ten minutes, deleted with it), `profile` (the body's figures, one row a person), `day`, `day_series` (a day's 24/7 heart rate in chunks of three hours, deleted with it) and `night` (one row a person and date each, ADR 0030, D37) and `plan` (an interval plan, D42), each a sealed `payload` with its `keyId` and `nonce`, nothing else readable. Tests (`test/integration/train_server_test.dart`) call the module's endpoints through the server's own dispatch (`halkyon_base.<endpoint>`), since the generated test tools expose the project's endpoints alone.
- **The server** includes `halkyon_base` (ADR 0014) and configures it in `server.dart` (`trainBase()`, `lib/src/train_base.dart`): its basics say it serves `train` (never change it once apps use it), sign-in mails name Halkyon Train, the source is this repository, `minimumAppBuild` 1, and `deleteTrainData` (`lib/src/train_data.dart`) as its `deletePersonData` hook, which deletes the person's sync spaces and so, by the sync engine's cascades, every row of theirs (`test/integration/workout/person_deletion_test.dart`). Authentication through the module's builders: a password, SimpleLogin on the register, email codes proving an address, server-side sessions; no anonymous accounts (ADR 0020, D25), nor GitHub (D36). **Accounts** (ADR 0016): Train holds its own register when it runs alone, and is a member of the instance's register (Learn's server on learn.halkyon.eu) where the operator sets `ACCOUNT_REGISTER_URL` (`trainBase(accounts: AccountsMode.fromEnvironment())`) with the instance's shared `INSTANCE_SECRET` (`instanceSecret`); then sign-in, handles and profiles are the register's, the email settings are the register's alone, and an identified person's deletion goes to the register first. `addInstanceRoutes` serves the calls between the services on its web server, and the server doesn't start without the secret where it needs it. `schedulePruning` (`lib/src/pruning.dart`) keeps the module's pruning scheduled, one entry each however often the server starts (and cancels the one under the pruning's name before ADR 0020, D25): the anonymous accounts left from before daily, sign-up attempts and codes hourly, as the register the deletions its members haven't confirmed hourly, as a member the people the register no longer knows daily (each a no-op in the other mode). Session logs are never persisted, in any run mode. **The Loop's firmware relay** (ADR 0030, D10, D40, `lib/src/firmware/`): the `firmware` endpoint asks Polar's firmware service in the person's place and hands on the file, kept in memory a day per hardware and version (`FirmwareRelay`); on by default, off where the operator sets `LOOP_FIRMWARE_RELAY=off`, the app then offering the direct way alone; tested on a fake Polar (`test/integration/firmware/`). **Routes** (ADR 0030, D11, D44, `lib/src/routing/`): the `routing` endpoint plans a route through the instance's BRouter (`BROUTER_URL`; none: `offered` is false and plans are refused) for the people signed in, at most 50 waypoints and 30 plans a person a minute, nothing kept or logged; BRouter's GeoJSON read in `brouter.dart` (points, elevations, climbs above 3 m, turn hints, stretches by surface), tested on its format (`test/unit/routing/`) and on a fake BRouter (`test/integration/routing/`). Train's running profile is `apps/train/brouter/halkyon-run.brf`, which the sidecar's profiles hold beside BRouter's own. **Deployed** (D46) beside learn.halkyon.eu's server once `deploy/.env` names `deploy/compose.train.yaml`: at train.halkyon.eu (`config/production.yaml`, the image `Dockerfile` as Learn's), a member of Learn's register, its database `halkyon_train` in the instance's PostgreSQL, BRouter's image `deploy/brouter/` (its segments kept current by `segments.sh`); `redeploy.sh` and `backup.sh` take Train's server and database in only where it is switched on, and the operator's steps are `deploy/sidecars/README.md`, *Train*. Migrations (2026-10-28): an initial one with Serverpod's tables, its auth modules' and the module's as they stand (Todo's grew one by one with the module; Train starts where they ended), then `workouts`, purely additive: the sync engine's tables and Train's first three, then `profile`, `days` (Train's `day`, `day_series` and `night`) and `plans` (`plan`, D42), each purely additive. **Synced models** (`database: sync`, the module `offline_sync`, `experimental_features: databaseSync` in `config/generator.yaml`): `content_key` (the person's key sealed to the account), `workout` and `workout_series` (a workout's series in chunks of at most ten minutes, deleted with it), `profile` (the body's figures, one row a person), `day`, `day_series` (a day's 24/7 heart rate in chunks of three hours, deleted with it) and `night` (one row a person and date each, ADR 0030, D37) and `plan` (an interval plan, D42), each a sealed `payload` with its `keyId` and `nonce`, nothing else readable. Tests (`test/integration/train_server_test.dart`) call the module's endpoints through the server's own dispatch (`halkyon_base.<endpoint>`), since the generated test tools expose the project's endpoints alone.
- **The app** is halkyon_app's `InstanceGate` and `SessionGate` with Train's frame (`trainFrame`: "Train" under "Halkyon", its icon, `train.example.org`, its own sentence) and `ServerpodInstanceChecker(app: 'train')`; `TrainDependencies` (`AppDependencies<TrainPerson>`) on one instance, `TrainPerson` per person: their own sync database in their folder (`sync.sqlite`, halkyon_app's `PersonFolders`, one person's at a time), halkyon_app's `SyncService` behind `AppCompatibility` (the module's basics' `minimumAppBuild`, against the build `package_info_plus` reads), their key (`TrainKeys`, `lib/src/core/privacy/`: one key per person, IDs `train1.<ms>.<nonce>`, the oldest-made writing, kept in `content_key` and opened only in Train's form, ADR 0030, D15) and their workouts (`WorkoutRepository`, `lib/src/feature/workout/`: halkyon_app's `PayloadEnvelope` seals a workout for its row and a series chunk for its row and its workout; payloads are versioned JSON, a series' columns delta-coded, gzipped before sealing; a deletion blanks every payload first); the gateways' tests run on a sync database opened without a server, the repository's on memory gateways and real libsodium; a person's app whose home lists their workouts (`WorkoutsCubit`, with `TrainPerson.deletions`, halkyon_app's `DelayedDeletions`) and opens each one's page (`WorkoutCubit`: `analyse` in `workout_analysis.dart` reads its series into the track's pieces, splits, elevation and charts, drawn by `workout_chart.dart`; the map with flutter_map on the instance's source, `MapSourceRepository` asking `map.source`, its tiles from `TrainPerson.mapTiles`, vector_map_tiles in the app and none in widget tests), and the account sheet: halkyon_app's `AccountTiles` (whose account on which instance, Switch account, Sign out, Delete account, which asks first) in Train's words, and halkyon_app's About (`showAppAbout`: the source of what the instance runs, from `TrainDependencies.sourceCodeUrl`, and the licence list). Texts through gen-l10n in British English (the template), German (informal du), an empty `en_US` and `en_GB` (`test/l10n/arb_files_test.dart` keeps them in step). The look follows the system's theme mode and language until Train has settings. Countries in where an instance keeps its data are named by halkyon_app's CLDR names (`AssetCountryNames`), its language picked by halkyon_app's `resolveAppLocale`. Widget tests start Train's whole tree with halkyon_app's fakes (`test/app/train_app_test.dart`).
- **Recording** (ADR 0030, D13, D16, D18 to D21, D26 to D28) is Android's alone for now (iOS waits for a Mac, D25) (`TrainPerson.recording` null elsewhere): `RecorderService` (Kotlin, `android/app/src/main/kotlin/app/halkyon/train/`), a foreground service of the types `location` and `health` started from the open app, takes fixes from LocationManager's GPS (no Play services), steps from the step detector the heart rate from the strap and a row from the rower the person chose (ADR 0030, D26 to D28: `NotifyLink.kt` on Android's own Bluetooth API, no package, `HeartRateMeasurement.kt` parsing 0x2A37 and `RowerData.kt` FTMS's Rower Data 0x2AD1; `DeviceScanner.kt` finds them for the sheets over `…/recorder/straps` and `…/recorder/rowers`; each remembered in the person's `strap.json` and `rower.json`, `DeviceStore`), decides auto-pause (`AutoPause.kt`) and appends each sample as a line to the person's `recording.txt` (format in `recording_file.dart`), flushed every 5 s; `MainActivity` asks the permissions (never background location) and bridges `app.halkyon.train/recorder` (methods) and `…/recorder/lines` (the file so far, then each line). The app (`lib/src/feature/recording/`) follows those lines with `RecordingBloc`, works out what the screen and the notification show (`liveOf`, the saved totals' functions), and saves a recording as a workout and its series (`recordedWorkout`) or throws it away; one cut off, or stopped and not saved, is found as the person's app opens. Tests: the Dart side with `test/support/fake_recorder_platform.dart`; the Kotlin auto-pause and the strap's and the rower's parsers on the JVM, `JAVA_HOME=<Android Studio's jbr> ./gradlew :app:testDebugUnitTest` in `android/` after a first `flutter build apk` (which writes `gradlew`), not part of `tool/check.dart`, as CI's runner has no Android SDK.
- **Health Connect** (ADR 0030, D30, D31) is Android's (`TrainPerson.health` null elsewhere): `HealthConnectBridge.kt` on AndroidX's `connect-client` 1.1.0 over `app.halkyon.train/health` (its records' mapping in `HealthRecords.kt`, tested on the JVM; the rationale's links from Health Connect in the manifest, `…/health/rationale` to the app), `HealthPlatform` in Dart (`lib/src/feature/health/`), `HealthConnect` per person: with writing on, every workout written as `healthRecordsOf` makes it (client IDs from the workout's ID, a version counted up as it changes, a fingerprint deciding whether it did), written again or deleted as the workouts change and as the app opens, what was written kept in the person's `health_connect.json` (`HealthStore`, never synced); with reading on, the sessions of the apps the person turned on brought in as workouts through the changes token (`importedWorkout` in `health_import.dart`, the workout's `origin` in its payload; a double of one Train has brings at most its heart rate, `isDouble`; one deleted in Train is remembered so it stays out; a track only when asked for on its page, `bringInTrack`), as the app opens and comes to the front (`HealthConnectListener`, which opens the rationale too); the screen from the account sheet (`HealthScreen`, `HealthCubit`). Tests on `test/support/fake_health_platform.dart`. Train needs Android 8 (API 26) for it.

60
apps/train/halkyon_train_server/Dockerfile Normal file
View file

@ -0,0 +1,60 @@
# Train's server image (ADR 0030, D46). Built from the repository root (the build
# context), because the workspace's root pubspec and lockfile live there:
#
# docker build -f apps/train/halkyon_train_server/Dockerfile .
#
# Learn's server image (apps/learn/halkyon_learn_server/Dockerfile) for Train: the
# same Dart, the same Ubuntu runtime and start command; no git, no repositories
# folder, no data of its own.
# ── Build stage: compile the server to a native executable ────────────────────────
FROM dart:3.13.4 AS build
WORKDIR /app
COPY pubspec.yaml pubspec.lock ./
COPY tool/workspace_only.sh tool/
COPY apps/train/halkyon_train_server apps/train/halkyon_train_server
COPY packages/halkyon_base_server packages/halkyon_base_server
COPY packages/halkyon_base_client packages/halkyon_base_client
COPY packages/halkyon_core packages/halkyon_core
COPY packages/halkyon_llm packages/halkyon_llm
COPY packages/halkyon_oidc_server packages/halkyon_oidc_server
COPY packages/halkyon_oidc_client packages/halkyon_oidc_client
# The workspace root with only the server and the modules it builds on, as Learn's
# image has it (the reasons there).
RUN sh tool/workspace_only.sh apps/train/halkyon_train_server packages/halkyon_base_server packages/halkyon_base_client packages/halkyon_core packages/halkyon_llm packages/halkyon_oidc_server packages/halkyon_oidc_client
RUN dart pub get
WORKDIR /app/apps/train/halkyon_train_server
RUN dart build cli --target bin/main.dart --output build
RUN mv build/bundle/bin/main build/bundle/bin/server
RUN mkdir -p config migrations
# ── Runtime stage ─────────────────────────────────────────────────────────────────
# Ubuntu 24.04 as Learn's; CA certificates for outgoing TLS (Polar's firmware
# service, D40).
FROM ubuntu:24.04
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
ENV runmode=production
ENV serverid=default
ENV logging=normal
ENV role=monolith
COPY --from=build /app/apps/train/halkyon_train_server/build/bundle/ ./
COPY --from=build /app/apps/train/halkyon_train_server/config/ config/
COPY --from=build /app/apps/train/halkyon_train_server/migrations/ migrations/
COPY --from=build /app/apps/train/halkyon_train_server/lib/src/generated/protocol.yaml lib/src/generated/protocol.yaml
# No root rights needed: it listens on 8080 and 8082 and writes nothing.
RUN useradd --system --no-create-home --shell /usr/sbin/nologin app
USER app
# The API server, and the web server the register tells of accounts gone (ADR
# 0016), reached on the instance's own network alone.
EXPOSE 8080 8082
ENTRYPOINT ["/bin/sh", "-c", "exec ./bin/server --mode=$runmode --server-id=$serverid --logging=$logging --role=$role \"$@\"", "--"]
CMD []

28
apps/train/halkyon_train_server/config/production.yaml
View file

@ -1,27 +1,20 @@
# This is the configuration file for your production environment.
# Typically, you will want to route the traffic through a load balancer
# which adds SSL security through https. If you use Serverpod's standard
# Terraform scripts to deploy your server, all you need to change in
# this file is the examplepod.com domain name.
# Train's server on learn.halkyon.eu's server, at train.halkyon.eu (ADR 0030, D46),
# behind the instance's Caddy (deploy/compose.train.yaml): its API and its web server
# on the one domain, as Learn's; its database in the instance's PostgreSQL.
# Configuration for the main API server.
apiServer:
port: 8080
publicHost: api.examplepod.com
publicHost: train.halkyon.eu
publicPort: 443
publicScheme: https
# Configuration for the Insights server.
insightsServer:
port: 8081
publicHost: insights.examplepod.com
publicPort: 443
publicScheme: https
# No insightsServer, as Learn's: without the block Serverpod doesn't start Insights.
# Configuration for the web server.
webServer:
port: 8082
publicHost: app.examplepod.com
publicHost: train.halkyon.eu
publicPort: 443
publicScheme: https
@ -30,11 +23,12 @@ webServer:
# If you are connecting on a public IP (e.g. on AWS or Google Cloud Run), you
# connect on the public IP of the database e.g. database.examplepod.com.
database:
host: database.private-production.examplepod.com
host: postgres
port: 5432
name: serverpod
user: postgres
requireSsl: true
name: halkyon_train
user: halkyon_train
# On the instance's own network, never leaving the server.
requireSsl: false
#isUnixSocket: true # defaults to false
#maxConnectionCount: 10 # Defaults to 10, a negative or null value removes the limit

24
deploy/.env.example
View file

@ -37,13 +37,35 @@ DOMAIN=http://localhost
# Sidecars switched on beside the instance (sidecars/README.md), each by its compose file
# after the instance's own: compose.jellyfin.yaml, a Jellyfin people watch and listen to
# learn from. Unset: the instance alone.
# learn from; compose.languagetool.yaml, spelling and grammar checked; compose.train.yaml,
# Halkyon Train's server with its BRouter. Unset: the instance alone.
#COMPOSE_FILE=compose.yaml:compose.jellyfin.yaml
# With the Jellyfin sidecar: the domain it answers at, which the instance then suggests
# to its people (the app's Jellyfin sign-in fills it in).
#JELLYFIN_DOMAIN=jellyfin.example.org
# With Halkyon Train beside the instance (compose.train.yaml, sidecars/README.md,
# Train): the domain its apps reach it at, its database's password, its own session
# pepper (never change it once people sign in), INSTANCE_SECRET above set, and the
# region of BRouter's segments, `west,south,east,north` in whole degrees, multiples of
# 5 (empty: Europe's, -25,35,40,70, about 3.3 GB). Generate each secret:
# openssl rand -hex 32. Its server is told of accounts gone without INSTANCE_MEMBERS;
# setting it, name Train's too (`http://train-server:8082/`).
#TRAIN_DOMAIN=train.example.org
#TRAIN_POSTGRES_PASSWORD=
#TRAIN_SESSION_KEY_HASH_PEPPER=
#BROUTER_REGION=
# The Loop's firmware through the instance (Train's ADR 0030, D40): `off` to stop
# relaying it; empty: on.
#LOOP_FIRMWARE_RELAY=
# Train's map (ADR 0030, D22): a MapLibre style over HTTPS, with its name, credit and
# page; empty: OpenFreeMap's.
#MAP_STYLE_URL=
#MAP_SOURCE_NAME=
#MAP_ATTRIBUTION=
#MAP_ABOUT_URL=
# Postgres password, shared by the postgres and server services.
# Generate: openssl rand -base64 32
POSTGRES_PASSWORD=

23
deploy/backup.sh
View file

@ -4,7 +4,8 @@
# how it was set up and how to restore.
#
# Everything people keep is in the database, their files too (Serverpod's database
# cloud storage), but for the releases of published studysets, which are git
# cloud storage), and in Train's where .env switches Train on (compose.train.yaml,
# Halkyon Train's ADR 0030, D46), but for the releases of published studysets, which are git
# repositories in the server's volume: a dump and a bundle of each repository cover
# the instance; the secrets in deploy/.env are kept in Proton Pass instead. restic stores the dump encrypted in a Hetzner Storage Box
# and keeps 24 hourly, 7 daily, 4 weekly and 3 monthly snapshots (decided 2026-10-03).
@ -35,6 +36,11 @@ cd "$(dirname "$0")"
# the last hour and stores it only once; restic compresses what it stores.
mkdir -p -m 700 "$HOME/backups"
dump="$HOME/backups/latest.dump"
train_dump="$HOME/backups/latest-train.dump"
train=
if docker compose config --services | grep -qx train-server; then
train=yes
fi
(
umask 077
# Never during a deploy (redeploy.sh holds this lock): its migrations would wait
@ -42,8 +48,14 @@ dump="$HOME/backups/latest.dump"
exec 9>"$HOME/.redeploy.lock"
flock 9
docker compose exec -T postgres pg_dump -U halkyon_learn -Fc -Z 0 halkyon_learn >"$dump.partial"
if [ -n "$train" ]; then
docker compose exec -T postgres pg_dump -U halkyon_learn -Fc -Z 0 halkyon_train >"$train_dump.partial"
fi
)
mv "$dump.partial" "$dump"
if [ -n "$train" ]; then
mv "$train_dump.partial" "$train_dump"
fi
# Each repository as a git bundle, after the dump: a release the dump lacks is never
# numbered again (the server numbers past the highest tag), while one the bundles
@ -67,8 +79,15 @@ releases="$HOME/backups/repositories.tar"
)
mv "$releases.partial" "$releases"
if [ -n "$train" ]; then
restic backup --quiet "$dump" "$train_dump" "$releases"
else
restic backup --quiet "$dump" "$releases"
restic forget --quiet --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 3
fi
# By host alone: restic groups by the paths backed up too, so a path added (Train's
# dump) would leave the snapshots before it a group of their own, never forgotten, and
# a deleted account in them for good.
restic forget --quiet --group-by host --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 3
# Once a day, in the run at 03:00: free the space of forgotten snapshots, and read the
# repository's structure; on Sundays also a tenth of the stored data, chosen at random,

15
deploy/backups.md
View file

@ -196,6 +196,21 @@ the way back from a restore that went wrong.
docker compose up -d
```
Train's database, where `.env` switches Train on (`compose.train.yaml`, Halkyon Train's
ADR 0030, D46), is in the same snapshots as `latest-train.dump`, and is restored the
same way, its server stopped meanwhile (`docker compose stop train-server`), from the
same snapshot as Learn's, so that the register's accounts and Train's people agree:
```sh
restic dump <snapshot> /home/deploy/backups/latest-train.dump >~/backups/restore-train.dump
docker compose exec -T postgres \
pg_restore -U halkyon_learn -d postgres --clean --create <~/backups/restore-train.dump
```
`pg_restore --create` makes the database anew owned as it was, by `halkyon_train`,
whose role is the cluster's and so survives the restore (on a new server, make it first:
`sidecars/README.md`, *Train*, step 2).
To check a backup without touching the instance, restore it into a database of its own
next to the real one, look, and drop it:

32
deploy/brouter/Dockerfile Normal file
View file

@ -0,0 +1,32 @@
# BRouter for Train's routes (ADR 0030, D11, D44, D46): its release, checked against
# GitHub's SHA-256 of it, on Eclipse Temurin's Java 17 as BRouter's own image, with
# its profiles and Train's running profile. Built from the repository root:
#
# docker build -f deploy/brouter/Dockerfile .
#
# The same image refreshes the segments (segments.sh), so it carries curl.
FROM eclipse-temurin:17-jre-noble
# BRouter v1.7.10, released 17 Jul 2026.
ADD --checksum=sha256:023fec3ba997758e8cd7ab9e1bae52e962af3f00b57683e3de86b84ffad01532 \
https://github.com/abrensch/brouter/releases/download/v1.7.10/brouter-1.7.10.zip /tmp/brouter.zip
RUN apt-get update \
&& apt-get install -y --no-install-recommends unzip curl ca-certificates \
&& rm -rf /var/lib/apt/lists/* \
&& unzip -q /tmp/brouter.zip -d /tmp \
&& mkdir -p /app \
&& mv /tmp/brouter-1.7.10/brouter-1.7.10-all.jar /app/brouter.jar \
&& mv /tmp/brouter-1.7.10/profiles2 /profiles2 \
&& rm -rf /tmp/brouter.zip /tmp/brouter-1.7.10
COPY apps/train/brouter/halkyon-run.brf /profiles2/halkyon-run.brf
COPY --chmod=755 deploy/brouter/segments.sh /app/segments.sh
WORKDIR /app
EXPOSE 17777
# Segments in /segments4 (a volume), profiles in the image, no custom profiles; four
# routes at a time, each stopped after a minute. -Xmx as BRouter's script raised: the
# memory a route takes grows with its length, and Europe's long routes need more than
# its 128 MB.
CMD ["java", "-Xmx384M", "-Xms128M", "-Xmn16M", "-DmaxRunningTime=60", "-DuseRFCMimeType=false", "-DusePOSTRequests=false", \
"-cp", "/app/brouter.jar", "btools.server.RouteServer", "/segments4", "/profiles2", "/tmp", "17777", "4", "0.0.0.0"]

58
deploy/brouter/segments.sh Executable file
View file

@ -0,0 +1,58 @@
#!/bin/sh
# Keeps BRouter's segments of the operator's region current (ADR 0030, D11, D44):
# every 5°×5° segment of BRouter's project whose south-west corner lies in
# BROUTER_REGION (`west,south,east,north` in whole degrees, multiples of 5; Europe's
# `-25,35,40,70` by default), downloaded where the project has a newer one than the
# volume holds, then again every BROUTER_REFRESH_DAYS days (30). Each is written beside
# its old self and moved over it, so BRouter never reads half a file. Segments of
# the sea, which the project doesn't have, are skipped.
set -eu
base=https://brouter.de/brouter/segments4
region=${BROUTER_REGION:--25,35,40,70}
days=${BROUTER_REFRESH_DAYS:-30}
cd /segments4
IFS=, read -r west south east north <<END
$region
END
name() {
lon=$1 lat=$2
if [ "$lon" -lt 0 ]; then ew="W$((-lon))"; else ew="E$lon"; fi
if [ "$lat" -lt 0 ]; then ns="S$((-lat))"; else ns="N$lat"; fi
echo "${ew}_${ns}.rd5"
}
refresh() {
listing=$(curl -fsS "$base/") || { echo "The segments' listing didn't come." >&2; return 0; }
fetched=0 kept=0
lon=$west
while [ "$lon" -le "$east" ]; do
lat=$south
while [ "$lat" -le "$north" ]; do
file=$(name "$lon" "$lat")
if echo "$listing" | grep -q "href=\"$file\""; then
# Only where the project's is newer than ours (If-Modified-Since): a
# current one comes back empty.
if [ -e "$file" ]; then set -- -z "$file"; else set --; fi
if curl -fsS -R --retry 3 "$@" -o "$file.part" "$base/$file" &&
[ -s "$file.part" ]; then
mv "$file.part" "$file"
fetched=$((fetched + 1))
else
rm -f "$file.part"
kept=$((kept + 1))
fi
fi
lat=$((lat + 5))
done
lon=$((lon + 5))
done
echo "Segments: $fetched fetched, $kept current, $(du -sh . | cut -f1) in all."
}
while :; do
refresh
sleep $((days * 86400))
done

114
deploy/compose.train.yaml Normal file
View file

@ -0,0 +1,114 @@
# Halkyon Train's server beside the instance (ADR 0030, D46), with the BRouter that
# plans its routes (D11, D44). Switched on in .env, with Train's domain and secrets:
#
# COMPOSE_FILE=compose.yaml:compose.train.yaml
# TRAIN_DOMAIN=train.halkyon.eu
#
# Train is a member of the instance's register, which Learn's server holds (ADR 0016):
# one account a person, signing in through the register, the shared INSTANCE_SECRET
# on every call between them, which reach each other on the instance's own networks
# alone. Its database is `halkyon_train` in the instance's PostgreSQL, under a role of
# its own that can't connect to Learn's. BRouter has no site and no sign-in: only
# Train's server reaches it, on a network of their own, and asks it nothing that is
# logged. Its segments (Europe's, about 3.3 GB) are in ./brouter/segments, which the
# refresher keeps current monthly, owned by a user ID of their own, which no person,
# no other container and not the CI runner uses. Setting it up: sidecars/README.md.
services:
train-server:
build:
context: ..
dockerfile: apps/train/halkyon_train_server/Dockerfile
image: halkyon-train-server
restart: unless-stopped
# As Learn's: migrations at every start (redeploy.sh has applied them before).
command: ["--apply-migrations"]
environment:
SERVERPOD_DATABASE_PASSWORD: ${TRAIN_POSTGRES_PASSWORD:?set TRAIN_POSTGRES_PASSWORD in deploy/.env}
SERVERPOD_PASSWORD_serverSideSessionKeyHashPepper: ${TRAIN_SESSION_KEY_HASH_PEPPER:?set TRAIN_SESSION_KEY_HASH_PEPPER in deploy/.env}
# The register (ADR 0016): Learn's web server, on the instance's own network,
# and the secret every call between them carries.
ACCOUNT_REGISTER_URL: http://server:8082/
SERVERPOD_PASSWORD_instanceSecret: ${INSTANCE_SECRET:?set INSTANCE_SECRET in deploy/.env}
# Caddy, as for Learn's server (compose.yaml).
TRUSTED_PROXIES: 172.30.80.2 fd4a:6c6b:7e80::2
SOURCE_URL: ${SOURCE_URL:-}
DATA_LOCATION: ${DATA_LOCATION:-}
BACKUP_LOCATION: ${BACKUP_LOCATION:-}
# Routes (D44), from the BRouter below.
BROUTER_URL: http://brouter:17777/
# The Loop's firmware through the instance (D40): on unless `off`.
LOOP_FIRMWARE_RELAY: ${LOOP_FIRMWARE_RELAY:-}
# The map source (D22); empty: OpenFreeMap.
MAP_STYLE_URL: ${MAP_STYLE_URL:-}
MAP_SOURCE_NAME: ${MAP_SOURCE_NAME:-}
MAP_ATTRIBUTION: ${MAP_ATTRIBUTION:-}
MAP_ABOUT_URL: ${MAP_ABOUT_URL:-}
networks:
- default
- edge
- routing
depends_on:
postgres:
condition: service_healthy
logging:
driver: journald
brouter:
build:
context: ..
dockerfile: deploy/brouter/Dockerfile
image: halkyon-brouter
restart: unless-stopped
user: "61177:61177"
read_only: true
tmpfs:
- /tmp:size=16m,uid=61177,gid=61177
volumes:
- ./brouter/segments:/segments4:ro
# Beside Train's server alone: no site, no published port, never beside Caddy or
# Postgres.
networks:
- routing
mem_limit: 640m
# No log at all: BRouter writes each route's waypoints to its output, and where
# people run must never be kept. To look into a failure, switch to journald for
# a while.
logging:
driver: none
brouter-segments:
image: halkyon-brouter
restart: unless-stopped
user: "61177:61177"
read_only: true
command: ["/app/segments.sh"]
environment:
# `west,south,east,north` of the segments' south-west corners; Europe's.
BROUTER_REGION: ${BROUTER_REGION:--25,35,40,70}
volumes:
- ./brouter/segments:/segments4
# Out to brouter.de alone; on a network of its own, beside nothing.
networks:
- downloads
mem_limit: 64m
logging:
driver: journald
caddy:
environment:
TRAIN_DOMAIN: ${TRAIN_DOMAIN:?set TRAIN_DOMAIN in deploy/.env}
volumes:
- ./sidecars/train.caddy:/etc/caddy/sidecars/train.caddy:ro
server:
environment:
# Train's web server, told when an account goes (ADR 0016); set
# INSTANCE_MEMBERS in .env to name more members, Train's among them.
INSTANCE_MEMBERS: ${INSTANCE_MEMBERS:-http://train-server:8082/}
networks:
# Train's server and BRouter, without a way out.
routing:
internal: true
downloads:

53
deploy/redeploy.sh
View file

@ -10,7 +10,8 @@
# - as a fast-forward of the deployed commit,
# - if every new commit is signed by a key in deploy/allowed-signers, as the deployed
# checkout lists them (so no commit can add its own key),
# dumps the database first (the last 3 dumps stay in ~/backups), applies the migrations
# dumps the database first (the last 3 dumps stay in ~/backups; Train's too where .env
# switches Train on, compose.train.yaml), applies the migrations
# before switching to the new server, starts it once more after a migration (see below),
# and fails unless that server answers afterwards. A failure leaves the checkout at the deployed commit, so
# running the CI job again retries. By hand, with your own key, which passes the commit as CI does:
@ -59,15 +60,30 @@ main() {
git merge -q --ff-only "$commit"
trap back_out EXIT
cd deploy
# Train's server, where .env switches it on (compose.train.yaml): built, dumped,
# migrated and waited for as Learn's.
train=
if docker compose config --services | grep -qx train-server; then
train=yes
fi
# Build while the running server keeps serving; then dump, migrate, switch.
docker compose build -q server dictionaries
dump
if [ -n "$train" ]; then
docker compose build -q train-server brouter
fi
dump halkyon_learn
if [ -n "$train" ]; then
dump halkyon_train
fi
# Migrations as a run of their own, before the switch: Serverpod's maintenance role
# applies them and exits with 0 only if that worked. Each migration is one
# transaction, so a failing one stops the deploy here and the running server keeps
# serving the schema it knows. (Until the switch, the old server serves the new
# schema; migrations that only add, as they should, keep that harmless.)
docker compose run --rm --no-deps -e role=maintenance server --apply-migrations
if [ -n "$train" ]; then
docker compose run --rm --no-deps -e role=maintenance train-server --apply-migrations
fi
switched=yes
docker compose up -d --remove-orphans
# Caddy reads its Caddyfile when it starts, and `up` doesn't restart it for a
@ -79,7 +95,10 @@ main() {
if ! git diff --quiet "$deployed" "$commit" -- Caddyfile 'sidecars/*.caddy'; then
docker compose up -d --force-recreate --no-deps caddy
fi
wait_until_answering
wait_until_answering server
if [ -n "$train" ]; then
wait_until_answering train-server
fi
# After a migration the new server is started once more. The sync engine
# (serverpod_offline_sync v0.0.10) reconciles its registry of synced tables as a
# server first uses the database; when a migration changed a synced table, that
@ -90,42 +109,50 @@ main() {
# finds the registry as the first start left it.
if ! git diff --quiet "$deployed" "$commit" -- ../apps/learn/halkyon_learn_server/migrations; then
docker compose restart server
wait_until_answering
wait_until_answering server
fi
# Train's server too: it syncs through the same engine.
if [ -n "$train" ] &&
! git diff --quiet "$deployed" "$commit" -- ../apps/train/halkyon_train_server/migrations; then
docker compose restart train-server
wait_until_answering train-server
fi
trap - EXIT
docker image prune -f >/dev/null
echo "Deployed $commit."
}
# Dumps the database, as it is before the new server applies its migrations.
# Dumps the database $1 (halkyon_learn, halkyon_train), as it is before the new server
# applies its migrations; as the instance's superuser, halkyon_learn.
dump() {
mkdir -p "$HOME/backups"
file="$HOME/backups/halkyon_learn-$(git rev-parse --short "$deployed")-$(date -u +%Y%m%dT%H%M%SZ).dump"
file="$HOME/backups/$1-$(git rev-parse --short "$deployed")-$(date -u +%Y%m%dT%H%M%SZ).dump"
(
umask 077
docker compose exec -T postgres pg_dump -U halkyon_learn -Fc halkyon_learn >"$file"
docker compose exec -T postgres pg_dump -U halkyon_learn -Fc "$1" >"$file"
)
echo "Database dumped to $file ($(du -h "$file" | cut -f1))."
# Only the newest 3 (decided 2026-10-02): they are the deploys' way back, not backups;
# those go off-site. Every file people keep is in each of them, so fewer leave room.
# (ls is safe here: the names are ours, letters, digits and dashes.)
# shellcheck disable=SC2012
ls -1t "$HOME"/backups/halkyon_learn-*.dump | tail -n +4 | xargs -r rm --
ls -1t "$HOME"/backups/"$1"-*.dump | tail -n +4 | xargs -r rm --
}
# Waits up to a minute for the new server to answer inside the stack's network.
# Waits up to a minute for the new server $1 (server, train-server) to answer inside the
# stack's network.
wait_until_answering() {
tries=0
while [ $tries -lt 30 ]; do
if docker compose exec -T caddy wget -q -O /dev/null http://server:8080/ 2>/dev/null; then
echo 'The server answers.'
if docker compose exec -T caddy wget -q -O /dev/null "http://$1:8080/" 2>/dev/null; then
echo "The $1 answers."
return 0
fi
tries=$((tries + 1))
sleep 2
done
docker compose logs --tail 40 server >&2
fail "the server doesn't answer a minute after the switch"
docker compose logs --tail 40 "$1" >&2
fail "the $1 doesn't answer a minute after the switch"
}
# A failed deploy puts the checkout back at the deployed commit, so running the CI job

74
deploy/sidecars/README.md
View file

@ -19,6 +19,7 @@ site changes.
|---|---|---|
| Jellyfin, which people watch and listen to learn from | `compose.jellyfin.yaml` | `jellyfin.caddy` |
| LanguageTool, which checks spelling and grammar | `compose.languagetool.yaml` | none |
| Halkyon Train's server, with BRouter for its routes | `compose.train.yaml` | `train.caddy` |
Both together: `COMPOSE_FILE=compose.yaml:compose.jellyfin.yaml:compose.languagetool.yaml`.
@ -151,3 +152,76 @@ Steps, in order; ✱ marks a step as root.
Switching it off: the compose file out of `COMPOSE_FILE`, then `docker compose up -d
--remove-orphans`; the app's Writing settings then say the instance checks nothing, and
`deploy/languagetool/` stays until removed by hand.
## Train
Halkyon Train's server beside the instance (Halkyon Train's ADR 0030, D46), at its own
domain, a member of the register Learn's server holds (ADR 0016), its database
`halkyon_train` in the instance's PostgreSQL, and BRouter planning its routes (D11, D44)
from the segments of the operator's region (Europe's by default, about 3.3 GB on
28 Oct 2026), refreshed monthly. `redeploy.sh` builds, dumps, migrates and waits for it
as for Learn's server once `.env` names its compose file, and `backup.sh` backs its
database up beside Learn's. The map stays OpenFreeMap's until the instance serves its
own tiles (D22, D23).
Steps, in order; ✱ marks a step as root.
1. **Room on the disk** (the operator, on the instance's server): `df -h ~deploy`.
BRouter's segments take about 3.3 GB for Europe, and a refresh writes each one beside
the old before replacing it; Train's dumps go to `~/backups` with Learn's.
2. **Its database** (the operator, on the instance's server, as `deploy`, in
`~/halkyon-learn/deploy`): a role of Train's own, with the password that goes into
`.env` as `TRAIN_POSTGRES_PASSWORD` (`openssl rand -hex 32`), and its database:
```sh
docker compose exec -T postgres psql -U halkyon_learn -d postgres -v ON_ERROR_STOP=1 \
-v password="<TRAIN_POSTGRES_PASSWORD>" <<'SQL'
CREATE ROLE halkyon_train LOGIN PASSWORD :'password';
CREATE DATABASE halkyon_train OWNER halkyon_train;
REVOKE CONNECT ON DATABASE halkyon_train FROM PUBLIC;
REVOKE CONNECT ON DATABASE halkyon_learn FROM PUBLIC;
SQL
```
`CREATE ROLE` makes the role Train's server signs in as, which may log in and is
nothing more (no superuser, no creating roles or databases); `CREATE DATABASE … OWNER`
gives it its own database, where its migrations make the tables. The two `REVOKE`s
take away the right every role has by default to connect to a database: only its
owner (and the instance's superuser, `halkyon_learn`, which Learn's server signs in
as) connects to `halkyon_train`, and `halkyon_train` no longer connects to Learn's.
Nothing of Learn's changes. (Learn's server, signing in as the superuser, could still
read Train's; giving it a role of its own is a step of its own.)
3. ✱ **BRouter's user and folder** (the operator, as a user with sudo): the user and
group ID 61177 are BRouter's alone, as the compose file runs it; the folder its
segments go to is theirs.
```sh
sudo groupadd --system --gid 61177 brouter
sudo useradd --system --uid 61177 --gid 61177 --no-create-home --shell /usr/sbin/nologin brouter
sudo mkdir -p ~deploy/halkyon-learn/deploy/brouter/segments
sudo chown -R brouter:brouter ~deploy/halkyon-learn/deploy/brouter/segments
```
4. **DNS** (the operator, at the domain's DNS): Train's domain, `train.halkyon.eu`,
pointing at the instance's server (`A`, and `AAAA` if it has an IPv6 address).
5. **Switch it on** (the operator, as `deploy`): in `deploy/.env`, `compose.train.yaml`
added to `COMPOSE_FILE`, and Train's settings (`.env.example`, *With Halkyon Train*):
```sh
COMPOSE_FILE=compose.yaml:compose.languagetool.yaml:compose.train.yaml
TRAIN_DOMAIN=train.halkyon.eu
TRAIN_POSTGRES_PASSWORD=<as in step 2>
TRAIN_SESSION_KEY_HASH_PEPPER=<openssl rand -hex 32>
```
`INSTANCE_SECRET` must be set (Learn's register takes no member without it). Where
`INSTANCE_MEMBERS` is set, add `http://train-server:8082/` to it; unset, the compose
file names Train.
6. **Start it** (the operator, as `deploy`): `docker compose up -d --build`. Caddy and
Learn's server are recreated with it (Train's site, and the register's member); the
instance is away for seconds. Train's server applies its migrations as it starts.
The segments come in the first hour or so (`docker compose logs brouter-segments`
says when); until then BRouter finds no route.
7. **Check it**: `https://train.halkyon.eu/` answers; in Train, signed in with a Learn
account, a route plans between two points of the region.

10
deploy/sidecars/train.caddy Normal file
View file

@ -0,0 +1,10 @@
# Halkyon Train's site (compose.train.yaml, ADR 0030, D46): its API server alone. Its
# web server (8082) takes only the register's calls between the instance's services
# (`/halkyon-base/…`, ADR 0016), on the instance's own network, never through here.
{$TRAIN_DOMAIN} {
encode zstd gzip
# Serverpod's streaming methods use WebSockets; reverse_proxy upgrades them as-is.
# The visitor's address comes last in X-Forwarded-For, as for Learn's server.
reverse_proxy train-server:8080
}

24
docs/adr/0030-halkyon-train.md
View file

@ -1,6 +1,6 @@
# ADR 0030: Halkyon Train, what it does and where its data comes from
- **Status:** decided 27–28 Oct 2026, by the developer (D1–D45 below). The shell (D12)
- **Status:** decided 27–28 Oct 2026, by the developer (D1–D46 below). The shell (D12)
is built; recording (D13–D28) is built on Android, iOS waiting for a Mac; Health
Connect (D29–D31) is built; the Loop's setup and keeping (7a, D34–D36) and its
daily data (7b, D37, D38), its sessions (7c, D39) and its firmware (D10, D40) are
@ -881,6 +881,28 @@ Strava. What was found, 27 Oct:
(to CoMaps or anything, with waypoints and turn hints) and come in as GPX shared to
Train or opened in it, snapped to a profile or kept as drawn (over out alone, and
over neither yet).
- **D46 Train's server on the instance** (recommended, the developer's; 28 Oct): it
joins learn.halkyon.eu's server, as BRouter needs a server of Train's to ask it
(D11, D22), at **train.halkyon.eu** (over a path on learn.halkyon.eu, which would
carry Learn's name for good); its data in the instance's PostgreSQL as a database of
its own, `halkyon_train`, under a role that can't connect to Learn's (over a second
PostgreSQL, twice the memory and a second backup); a **member of the instance's
register** on Learn's server (ADR 0016: one account a person across the operator's
services, over a register of its own). It is a compose file of its own,
`deploy/compose.train.yaml`, switched on in `.env` as the sidecars are, so nothing
changes on the instance until the operator does (the steps in
`deploy/sidecars/README.md`, *Train*); `redeploy.sh` then builds, dumps, migrates
and waits for it as for Learn's server, and `backup.sh` backs its database up in the
same snapshots. BRouter (v1.7.10, its release checked by SHA-256) runs beside Train's
server alone, on a network without a way out, with no log (it writes the waypoints),
its segments refreshed monthly by a container of its own; Europe's took 3.3 GB on
28 Oct. Whether the server's disk has room is the developer's to check before
switching it on. The map stays OpenFreeMap's: the PMTiles sidecar needs a style for
Protomaps' schema, a step of its own (about 40 GB for Europe, estimated from the
planet's 120 GB). Learn's server signs in to PostgreSQL as its superuser, so it could
read Train's database; a role of its own for it is left for later. restic now forgets
snapshots grouped by host alone, so that the snapshots before Train's dump joined
them aren't kept for good as a group of their own.
## Consequences