feat(deploy): sidecars beside the instance, Jellyfin the first, behind an edge network

An instance may now run services beside itself, each a compose file of
its own switched on in deploy/.env (COMPOSE_FILE=compose.yaml:compose.
jellyfin.yaml), so the deploy runs them with the instance. The first is
Jellyfin, which people watch and listen to learn from: the instance then
suggests it (JELLYFIN_URL from JELLYFIN_DOMAIN), and Caddy serves its
site, refusing every change to a user from outside, so a passwordless
demo account stays everyone's; its admin signs in through an SSH tunnel.

The instance's compose gets a second network, `edge`, for Caddy (at a
fixed address, which sidecars trust as their proxy) and the server;
Postgres stays on the internal one. Caddy loses its needless way to
Postgres, and a sidecar never gains one. A sidecar publishes no port,
as the CI runner may reach 127.0.0.1, and runs as a user ID of its own.
Caddy imports the sidecars' sites from sidecars/, mounted by their
compose files (none without), and redeploy.sh recreates it when one
changes. sidecars/README.md has the steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

.gitignore

@@ -24,6 +24,8 @@ pubspec_overrides.yaml
 .scloud/
 # Deployment secrets (see deploy/.env.example)
 deploy/.env
+# A sidecar's files on the server, such as Jellyfin's configuration and media
+deploy/jellyfin/
 # Dictionary files an instance loads: made by halkyon_learn_curator, not kept in git.
 deploy/dictionaries/*.jsonl
 deploy/dictionaries/*.jsonl.gz

AGENTS.md

@@ -126,6 +126,7 @@ A spaced-repetition learning app: studysets of flashcards, reviewed on an FSRS s
 
 - **CI** (`.forgejo/workflows/ci.yaml`) runs `tool/check.sh` on every push, on the Forgejo runner on learn.halkyon.eu's server (`deploy/runner/`), except for a commit that passed before: the job marks passed commits in the runner's home (`checked/`, kept 90 days), so a branch's commit fast-forwarded into `main` goes straight to its deploy. Jobs run directly on the host as the unprivileged user `runner`, never in containers: Docker access would be root on the production server. The runner's systemd unit keeps jobs away from production (no `/home`, no Docker socket, Docker's container networks unreachable, memory and CPU capped); keep it that way when changing it. `runner` has user ID 61000 and must never share one with a container's processes (PostgreSQL and the server run as 999, not remapped): the same user ID can read their environment and files. The job's first step (*Sandbox*) fails if any of this leaks. Flutter is pinned in the workflow (`FLUTTER_VERSION`): move it together with the local one. The server tests there get throwaway passwords from the environment (`SERVERPOD_…`, new for every job), since `passwords.yaml` is never committed; without them they exit with code 1 and no message (Serverpod's own template does the same). CI also runs `serverpod generate` and fails if the committed generated code differs from its output. So never run `dart format` over the server's `lib/` or `test/` as a whole: it reformats the generated code (`lib/src/generated/`, `serverpod_test_tools.dart`); format the files you wrote. CI runs on pushes only: a pull request from a fork would run its code on the production server.
 - **A green `main` deploys learn.halkyon.eu** through `deploy/redeploy.sh`, the only command the runner's key may run as `deploy` (pinned in `authorized_keys`, from that server itself only). It deploys the commit CI checked while it is still the tip, as a fast-forward, only if every new commit is signed by a key in `deploy/allowed-signers` (the deployed copy decides), after dumping the database (the last 3 dumps stay in `~deploy/backups`: the way back from a deploy, not backups), applies the migrations in a run of their own before switching (`--role=maintenance --apply-migrations`: a failing migration stops the deploy while the old server keeps serving, so migrations must keep the old server working until the switch), and fails unless the new server answers. A failed deploy leaves the checkout at the deployed commit; run the workflow again to retry. By hand: `ssh learn-deploy halkyon-learn/deploy/redeploy.sh <commit>`. Never `docker compose down -v` there.
+- **Sidecars** (`deploy/sidecars/README.md`, decided 2026-10-14): services an instance may run beside itself, each a compose file of its own beside `deploy/compose.yaml` (`compose.jellyfin.yaml`; later LanguageTool, Fedify), switched on in `deploy/.env` (`COMPOSE_FILE=compose.yaml:compose.jellyfin.yaml`), so `redeploy.sh` deploys them with the instance. The instance's compose has two networks: `edge` (`halkyon-learn-edge`, 172.30.80.0/24), Caddy's at a fixed address (172.30.80.2), shared with the server, and the internal default one, where the server reaches Postgres; a sidecar joins `edge` alone and publishes no port (the CI runner may reach `127.0.0.1`, never Docker's 172.16.0.0/12). A sidecar reached from outside mounts its site from `deploy/sidecars/` into Caddy's `/etc/caddy/sidecars/`, which the Caddyfile imports (none without), and `redeploy.sh` recreates Caddy when a site changes. Each runs as a user ID of its own (Jellyfin 61096), never one a person, another container or the runner uses; its files on the server are git-ignored (`deploy/jellyfin/`). The Jellyfin sidecar sets the server's `JELLYFIN_URL`, so the instance suggests it.
 - **Backups** (`deploy/backups.md`): every hour, `deploy/backup.sh` (a systemd timer, as `deploy`) dumps the database, which holds people's files too, then bundles each published studyset's repository (`git bundle`, from the server's `repositories` volume), and restic stores both in a Hetzner Storage Box (24 hourly, 7 daily, 4 weekly, 3 monthly); the box's own snapshots protect it from a taken-over server. A restore takes the instance back up to an hour, and people who started since lose what their device held: their app no longer finds its session, starts a new anonymous account and deletes the old folder. So a change that makes devices lose their session, or keeps deleted people's data longer, changes what a restore costs; say so in `deploy/backups.md`.
 - **Releases** are the project's, not an instance's: the release, the Snap Store listing and the F-Droid listing point to the project's page on Halkyon's website, halkyon.app/learn (its own repository, `project-halkyon/website`; learn.halkyon.app redirects there); the builds name learn.halkyon.eu, the project's instance (`SERVER_URL`), which a new install offers first (see *Instances*). They come from a developer's laptop, never from CI: `tool/release.sh` on `main` as pushed and green in CI. Versions follow semantic versioning, worked out from the commits since the last release (`feat` raises the minor number, `fix`/`perf` the patch number, breaking ones the major from 1.0 on; other types release nothing), so **commit messages keep their type prefix**. It writes the version and a raised build number into the app's pubspec in a signed release commit (`chore(release): vX.Y.Z`) with a signed tag (a key in `deploy/allowed-signers`), builds one universal APK from a checkout of the tag, signed with the app's key, and the snap (`snap/snapcraft.yaml`, the committed HEAD, built in LXD), both with `SERVER_URL=https://learn.halkyon.eu/`; signs `SHA256SUMS` with GPG; pushes only once both builds passed their checks; and publishes on code.halkyon.app through Forgejo's API and in the F-Droid repository. The Snap Store upload is off until `snap_store_channel` in the script names a channel. A run that stopped after its release commit and tag carries on with them. The key's password and the Forgejo token are asked for at each run and never stored. The app's key is a PKCS #12 keystore outside the repository (also in the maintainer's password manager); there is no Google Play, so it is the app's identity: lost, installed apps can't be updated. Its certificate's SHA-256 is in `tool/release/android-certificate.sha256`, and the script releases no APK signed otherwise; other release builds fall back to the debug key (`android/app/build.gradle.kts`), so `flutter run --release` works anywhere. The APK also goes into the project's own F-Droid repository, https://halkyon.app/fdroid/repo, through `tool/fdroid_repo.sh` (fdroidserver v2.4.5): its settings and the app's listing are in `tool/release/fdroid/` (the anti-features TetheredNet and NonFreeNet declared with their reasons and F-Droid's own icons); its index is signed with a key of its own (a PKCS #12 keystore beside the app's, password asked for, certificate pinned in `tool/release/fdroid-index-certificate.sha256`, since a new key makes the F-Droid app refuse the repository); and it is uploaded to the halkyon.app server (`ssh fdroid-upload`) as the user `fdroid`, whose key can only write into the folder Caddy serves (rrsync), the APKs first and the indexes last.
 

deploy/.env.example

@@ -6,6 +6,15 @@
 # `http://localhost` for the local rehearsal; `learn.halkyon.eu` on its server.
 DOMAIN=http://localhost
 
+# Sidecars switched on beside the instance (sidecars/README.md), each by its compose file
+# after the instance's own: compose.jellyfin.yaml, a Jellyfin people watch and listen to
+# learn from. Unset: the instance alone.
+#COMPOSE_FILE=compose.yaml:compose.jellyfin.yaml
+
+# With the Jellyfin sidecar: the domain it answers at, which the instance then suggests
+# to its people (the app's Jellyfin sign-in fills it in).
+#JELLYFIN_DOMAIN=jellyfin.example.org
+
 # Postgres password, shared by the postgres and server services.
 # Generate: openssl rand -base64 32
 POSTGRES_PASSWORD=

deploy/Caddyfile

@@ -19,3 +19,7 @@
 		reverse_proxy server:8080
 	}
 }
+
+# The sites of the sidecars switched on (sidecars/README.md), which their compose files
+# mount into sidecars/; none without.
+import sidecars/*.caddy

deploy/compose.jellyfin.yaml

@@ -0,0 +1,44 @@
+# The Jellyfin sidecar (decided 2026-10-14): a Jellyfin server beside the instance, which
+# people watch and listen to learn from (Halkyon Learn's Watch screen), suggested to them
+# by the instance. Switched on in .env, with the domain it answers at:
+#
+#   COMPOSE_FILE=compose.yaml:compose.jellyfin.yaml
+#   JELLYFIN_DOMAIN=jellyfin.example.org
+#
+# Its files are the server's, in ./jellyfin (config, cache, media), owned by a user ID
+# of its own, which no person, no other container and not the CI runner uses: processes
+# of one user ID can read each other's environment and files. Setting it up, and the
+# demo account: sidecars/README.md.
+
+services:
+  jellyfin:
+    # v12.2, the version Halkyon Learn's Jellyfin feature is checked with.
+    image: jellyfin/jellyfin@sha256:357724bf0ae27a672c7cbaa899db2d9abeb13dbd8657ccce750258a4c059d037
+    restart: unless-stopped
+    user: "61096:61096"
+    environment:
+      JELLYFIN_PublishedServerUrl: https://${JELLYFIN_DOMAIN:?set JELLYFIN_DOMAIN in deploy/.env}
+    volumes:
+      - ./jellyfin/config:/config
+      - ./jellyfin/cache:/cache
+      - ./jellyfin/media:/media:ro
+    # On `edge` alone, beside Caddy and the server's public API, never beside Postgres;
+    # no published port. A fixed address, for the admin's SSH tunnel (sidecars/README.md).
+    networks:
+      edge:
+        ipv4_address: 172.30.80.10
+    # Videos stream as stored (the app asks for static=true): nothing is transcoded.
+    mem_limit: 768m
+    logging:
+      driver: journald
+
+  caddy:
+    environment:
+      JELLYFIN_DOMAIN: ${JELLYFIN_DOMAIN:?set JELLYFIN_DOMAIN in deploy/.env}
+    volumes:
+      - ./sidecars/jellyfin.caddy:/etc/caddy/sidecars/jellyfin.caddy:ro
+
+  server:
+    environment:
+      # The instance suggests this Jellyfin, rather than looking for one.
+      JELLYFIN_URL: https://${JELLYFIN_DOMAIN:?set JELLYFIN_DOMAIN in deploy/.env}/

deploy/compose.yaml

@@ -8,7 +8,15 @@
 # Only Caddy publishes ports. Docker opens published ports directly in the kernel's
 # packet filter, past ufw, and the Serverpod server listens on every interface, so the
 # server and Postgres publish nothing: they're reachable only inside this project's
-# network, where Caddy finds the server by its service name.
+# networks, where Caddy finds the server by its service name.
+#
+# Two networks (decided 2026-10-14): `edge`, Caddy's, which the server shares with it,
+# and the internal default one, where the server reaches Postgres. Caddy never reaches
+# Postgres, nor does a sidecar behind Caddy.
+#
+# Sidecars (decided 2026-10-14) are each a compose file of their own beside this one,
+# switched on in .env (COMPOSE_FILE=compose.yaml:compose.jellyfin.yaml): each adds its
+# service and mounts its site into Caddy's sidecars/ folder (sidecars/README.md).
 
 name: halkyon-learn
 
@@ -32,6 +40,11 @@ services:
       - ./Caddyfile:/etc/caddy/Caddyfile:ro
       - caddy_data:/data # certificates: must survive restarts (rate limits)
       - caddy_config:/config
+    networks:
+      edge:
+        # Fixed, so containers behind Caddy can trust it as their proxy (the
+        # visitor's own address comes in X-Forwarded-For).
+        ipv4_address: 172.30.80.2
     depends_on:
       - server
     logging: *logging
@@ -72,6 +85,9 @@ services:
       # Every release of every published studyset: backed up beside the database
       # (backup.sh), never deleted with `down -v`.
       - repositories:/app/repositories
+    networks:
+      - default
+      - edge
     depends_on:
       postgres:
         condition: service_healthy
@@ -115,6 +131,13 @@ services:
       retries: 12
     logging: *logging
 
+networks:
+  edge:
+    name: halkyon-learn-edge
+    ipam:
+      config:
+        - subnet: 172.30.80.0/24
+
 volumes:
   caddy_data:
   caddy_config:

deploy/redeploy.sh

@@ -70,10 +70,10 @@ main() {
   # Caddy reads its Caddyfile when it starts, and `up` doesn't restart it for a
   # changed file it mounts. Nor would `caddy reload` see the change: Docker
   # mounts a single file by its inode, and git writes a changed file anew, so
-  # the container keeps the old one (found 2026-10-10). A changed Caddyfile
+  # the container keeps the old one (found 2026-10-10). A changed Caddyfile, or a
-  # therefore recreates the container, which mounts the new file; the
+  # sidecar's site, therefore recreates the container, which mounts the new file; the
   # certificates are in a volume, and the instance is away for seconds.
-  if ! git diff --quiet "$deployed" "$commit" -- Caddyfile; then
+  if ! git diff --quiet "$deployed" "$commit" -- Caddyfile 'sidecars/*.caddy'; then
     docker compose up -d --force-recreate --no-deps caddy
   fi
   wait_until_answering

deploy/sidecars/README.md

@@ -0,0 +1,88 @@
+# Sidecars
+
+Services an instance may run beside itself (decided 2026-10-14), each switched on by
+naming its compose file after the instance's own in `deploy/.env`:
+
+```sh
+COMPOSE_FILE=compose.yaml:compose.jellyfin.yaml
+```
+
+A sidecar's compose file adds its service and, if it is reached from outside, mounts
+its site from this folder into Caddy's `/etc/caddy/sidecars/`, which the Caddyfile
+imports. It joins the `edge` network beside Caddy and the server's public API, never the
+internal one where Postgres is, and publishes no port of its own. Its process runs as a
+user ID of its own, which no person, no other container and not the CI runner uses.
+`redeploy.sh` deploys whatever `.env` switches on, and recreates Caddy when a sidecar's
+site changes.
+
+| Sidecar | Compose file | Site |
+|---|---|---|
+| Jellyfin, which people watch and listen to learn from | `compose.jellyfin.yaml` | `jellyfin.caddy` |
+
+## Jellyfin
+
+A Jellyfin server beside the instance, which the instance then suggests to its people
+(`JELLYFIN_URL`, set by the compose file): the app's Jellyfin sign-in fills it in and
+lists the people it shows on its login screen. Videos stream as stored, nothing is
+transcoded. Caddy refuses every change to a user from outside, so an account without a
+password (a demo's) stays everyone's; the admin signs in through an SSH tunnel only.
+
+Steps, in order; ✱ marks a step as root.
+
+1. ✱ **Its user and folders** (the operator, on the instance's server, as a user with
+   sudo): the user ID 61096 is Jellyfin's alone; the media folder is `deploy`'s, so it
+   can copy files in, readable by all.
+
+   ```sh
+   sudo useradd --system --uid 61096 --user-group --no-create-home --shell /usr/sbin/nologin jellyfin
+   cd ~deploy/halkyon-learn/deploy
+   sudo mkdir -p jellyfin/config jellyfin/cache jellyfin/media
+   sudo chown -R jellyfin:jellyfin jellyfin/config jellyfin/cache
+   sudo chown -R deploy:deploy jellyfin/media
+   ```
+
+2. **DNS** (the operator, at the domain's DNS): the Jellyfin domain, such as
+   `jellyfin.example.org`, pointing at the instance's server (`A`, and `AAAA` if it has
+   an IPv6 address).
+
+3. **Switch it on** (the operator, on the instance's server, as `deploy`): in
+   `deploy/.env`,
+
+   ```sh
+   COMPOSE_FILE=compose.yaml:compose.jellyfin.yaml
+   JELLYFIN_DOMAIN=jellyfin.example.org
+   ```
+
+4. **The media** (the operator, on their own machine): a folder per film, show or
+   album, as Jellyfin expects them, into `deploy/jellyfin/media/`, readable by all:
+
+   ```sh
+   rsync -av --chmod=D755,F644 "<folder>" <deploy's ssh alias>:halkyon-learn/deploy/jellyfin/media/movies/
+   ```
+
+5. **Start it** (the operator, on the instance's server, as `deploy`):
+   `cd ~/halkyon-learn/deploy && docker compose up -d`. Caddy and the server are
+   recreated with it (the site, and the instance's suggestion); the instance is away
+   for seconds.
+
+6. **Set Jellyfin up through a tunnel** (the operator, on their own machine):
+   `ssh -N -L 8096:172.30.80.10:8096 <deploy's ssh alias>`, then http://localhost:8096.
+   - The wizard: an admin of a name nobody guesses, its password in a password
+     manager; a library of the type Movies (or Shows, Music) on `/media/…`; remote
+     connections allowed.
+   - Dashboard › Networking › **Known proxies**: `172.30.80.2` (Caddy's address on
+     `edge`). Jellyfin then sees each visitor's own address, and the tunnel as local.
+   - Dashboard › General: **Quick Connect** off, unless people sign in with it.
+   - Dashboard › Users › the admin: **Hide this user from login screens**; **Allow
+     remote connections to this server** off.
+   - For a demo, Dashboard › Users › **+**: an account without a password, shown on
+     login screens, seeing the demo's library alone; off: playback and conversion that
+     need transcoding, content downloading, remote control of other users, deleting
+     media.
+
+7. **Check from outside** (anyone, anywhere): `https://<domain>/System/Info/Public`
+   answers, `/Users/Public` lists the demo account, a `POST` to `/Users/Password` gets
+   403, and the instance's `instance.info()` names the domain as `jellyfinUrl`.
+
+Switching it off: the compose file out of `COMPOSE_FILE`, then `docker compose up -d
+--remove-orphans`; `deploy/jellyfin/` stays until removed by hand.

deploy/sidecars/jellyfin.caddy

@@ -0,0 +1,17 @@
+# The Jellyfin sidecar's site (compose.jellyfin.yaml).
+{$JELLYFIN_DOMAIN} {
+	# Its demo account (sidecars/README.md) has no password and is everyone's: nobody
+	# may change a user from outside (give it a password, change its settings or its
+	# picture), only sign in. Halkyon Learn writes nothing else under /Users. The setup
+	# wizard is closed too. The admin works through an SSH tunnel, which doesn't pass
+	# here.
+	@userChanges {
+		method POST PUT DELETE
+		path /Users/* /UserImage /UserImage/*
+		not path /Users/AuthenticateByName /Users/AuthenticateWithQuickConnect
+	}
+	respond @userChanges 403
+	respond /Startup/* 403
+
+	reverse_proxy jellyfin:8096
+}