halkyon/docs/adr/0039-halkyon-li-at-home.md

# ADR 0039: halkyon.li, the family's instance at home, published through ramiel

- **Status:** decided 8 Nov 2026 by the developer: D1 to D6 and D17 to D20 are their
  own choices, D7 to D16 the agent's recommendations, which they confirmed. Nothing is
  built yet. halkyon.li's DNS zone at netcup is in place (D7, checked 8 Nov).
- **Builds on:** Halkyon Learn's ADR 0021 (a private instance whose operator approves
  every account: "The developer may stop the public instance and run halkyon.li for
  themselves and their close family"), ADR 0016 (the instance's register), ADR 0020
  (SimpleLogin), ADR 0030, 0033 and 0038 (Train's, Todo's and Meditate's servers beside
  Learn's).
- **Supersedes:** ADR 0032 (Jellyfin at home through a tunnel to learn.halkyon.eu's
  server), whose server end is retired with sahaquiel; its open question goes with it.
- **Followed by:** the Instance console's ADR (to come), through which halkyon.li's
  regions are added and removed (D5).

## Context

- **sahaquiel**, learn.halkyon.eu's server, is retired later in November 2026. It holds
  learn.halkyon.eu (its server, PostgreSQL, the register), the Forgejo runner that
  checks every push and deploys a green `main`, and the backups' timer. The developer
  moves its data out themselves.
- **balthasar** is the developer's iMac at home (iMac17,1: 4 cores, 7.2 GB RAM). Its
  root filesystem is 19 GB, 91 % full; `/home` is a 916 GB HDD with 150 GB free. It sits
  behind a FRITZ!Box on Vodafone cable, with a dynamic IPv4 address (178.24.33.155 on
  8 Nov) and IPv6. It already runs ADR 0032's home Jellyfin (media in
  `/home/raisondetredev/Media`).
- **ramiel**, the halkyon.app server (45.129.181.128, `2a03:4000:47:9e6:…`): 1 vCPU,
  957 MiB RAM, 8.8 GB disk free, Ubuntu 24.04. The system's Caddy (v2.11.4, Ubuntu's
  package) holds 80 and 443 (HTTP/3 on UDP 443 too) for `code.halkyon.app` (Forgejo on
  `127.0.0.1:3000`), `halkyon.app` (the F-Droid repository) and `learn.halkyon.app`.
  ufw and fail2ban are active. It sent about 67 GB in its last 20 days.
- **halkyon.li**: its nameservers are Cloudflare's on 8 Nov, every record proxied, with
  a wildcard `*.halkyon.li` and an MX `mail.halkyon.li`. Nothing answers. The MX can't
  have been receiving mail: its name points at Cloudflare's proxy, which carries no
  SMTP, and there is no SPF record. There is no DNSSEC either: no DS record at the
  registry. The developer moves the zone to netcup's DNS (D7), where ramiel's
  addresses are too.
- **melchior**, the developer's laptop, is where development and the pre-push check
  run from now on.
- **The forge**: `project-halkyon/halkyon` is private since 8 Nov (the developer's
  choice: the code is hidden from the public for now; nobody but the developer has
  contributed). `appflowy-editor`, `pure-zstd` and `serverpod-offline-sync` stay
  public, as every build fetches them anonymously as git dependencies.

## Decisions

### The developer's (8 Nov 2026)

- **D1 halkyon.li is the family's instance, run on balthasar**, with Learn, Todo, Train
  and Meditate from the start, and Jellyfin. It is the instance ADR 0021 foresaw: sign-up
  by approval (`SIGN_UP=approval`). learn.halkyon.eu ends with sahaquiel.
- **D2 Sign-up and sign-in through SimpleLogin alone** (`SIGN_IN_WAYS=simplelogin`).
  No password, no TOTP and no email codes, so no SMTP service. halkyon.li gets a
  SimpleLogin app of its own (D9).
- **D3 Published through ramiel by WireGuard, with TLS passed through by name** (over TLS
  ended on ramiel, and over a port forward on the FRITZ!Box, which would show the home
  address). On ramiel's 443,
  HAProxy (Ubuntu's package, kept current by `unattended-upgrades`) reads the name a
  connection asks for (SNI) and sends halkyon.li's, unopened, through the tunnel to
  balthasar; every other name goes to ramiel's own Caddy, moved to the loopback (over
  caddy-l4, which needs a Caddy that apt no longer updates). ramiel sees no request of
  halkyon.li's, holds none of its certificates, and spends its one CPU copying bytes
  rather than encrypting streams. halkyon.li thus has no HTTP/3 (QUIC can't be routed by
  name this simply); clients use HTTP/2.
- **D4 Jellyfin is `jellyfin.halkyon.li` alone**: `jellyfin.halkyon.eu` ends with
  sahaquiel.
- **D5 Regions are added and removed in the Instance console**, for Train's map,
  routes and search and Todo's search: CoMaps' border polygons (`data/borders/*.poly`,
  below countries: *Germany_Free State of Bavaria_Upper Bavaria_South*). The console's
  ADR decides how, under two rules already agreed: the settings it keeps are not in the
  backups (what belongs to accounts stays in the database and so in them), and no server
  gets Docker's socket. Until it ships, halkyon.li runs without those sidecars.
- **D6 The CI runner moves to balthasar and runs at night**, 02:00 to 05:00, started and
  stopped by a systemd timer; Forgejo queues the jobs pushed during the day. In the
  window the heavy sidecars stop (Photon, BRouter, the tiles, Jellyfin, LanguageTool,
  the dictionaries), while Caddy, PostgreSQL and the four servers keep serving. CI only
  checks: the deploy job goes with sahaquiel, as halkyon.li deploys by hand (D13).

### Recommended by the agent, confirmed by the developer

- **D7 The names**: `learn.halkyon.li` (Learn's server, which holds the register),
  `todo.halkyon.li`, `train.halkyon.li`, `meditate.halkyon.li`, `jellyfin.halkyon.li`,
  and `halkyon.li` itself for people's handles and profile pages (`ACCOUNT_DOMAIN`), as
  halkyon.eu is for learn.halkyon.eu. Each is an `A` and `AAAA` record at ramiel, in
  netcup's DNS zone for halkyon.li (the developer's choice, 8 Nov). The zone is made at
  netcup first, then the domain's nameservers are switched to netcup's. With no DS
  record at the registry, the switch can't break DNSSEC. Cloudflare's records end with
  it, its proxy too. The new zone has no wildcard, so no name answers that isn't meant
  to, and no MX, as the old one can't have received mail. Mail for halkyon.li is
  outside this record.
- **D8 The tunnel**: `wg-quick@wg-halkyon` on both hosts, `10.61.97.1` ramiel and
  `10.61.97.2` balthasar, a /30, IPv4 alone (apart from ADR 0032's `10.61.96.0/30`,
  Docker's pools and Multipass' bridges). ramiel listens on UDP 51820, opened in ufw.
  balthasar dials out with `PersistentKeepalive = 25`, so the FRITZ!Box opens nothing
  and a new home address needs no change. Each side's `AllowedIPs` is the other's
  address alone. The keys stay in `/etc/wireguard/` (root, 0600) and in Proton Pass.
- **D9 What the tunnel may carry**, in chains of their own as ADR 0032's, living as long
  as the tunnel: on ramiel set by wg-quick's `PostUp` and removed by its `PreDown`; at
  home loaded by systemd before the interface exists and deleted after it is gone
  (`ExecStartPre`, `ExecStopPost`), as Ubuntu 26.04's AppArmor profile for wg-quick lets
  its `nft` read nothing in `/etc/wireguard` (found 8 Nov 2026):
  - on ramiel: HAProxy's connections to `10.61.97.2:443`, nothing else; nothing new
    from the tunnel into ramiel (so a taken-over balthasar can't reach ramiel's sshd or
    Forgejo);
  - on balthasar: new connections from `10.61.97.1` to Caddy's 443, and to sshd's 22
    for the jump from away (D17), and nothing else; nothing new from the tunnel into the
    rest of the host, the home network or the other containers.
- **D10 Visitors' addresses through the PROXY protocol**: HAProxy sends each
  connection's real source (PROXY v2) to balthasar's Caddy, which accepts that header
  from `10.61.97.1` alone (`proxy_protocol` listener wrapper), so the servers' per-network
  limits keep counting visitors, not ramiel. ramiel's own Caddy gets the header too,
  from the loopback, so Forgejo's and the F-Droid repository's logs keep their visitors.
- **D11 Certificates at home**: balthasar's Caddy gets halkyon.li's names' certificates
  from Let's Encrypt by the TLS-ALPN challenge, which passes through HAProxy unopened.
  Port 80 stays with ramiel's Caddy, which sends `http://*.halkyon.li` to `https://`;
  the home Caddy disables the HTTP challenge. It also switches HTTP/3 off
  (`protocols h1 h2`), so it advertises nothing that ramiel's UDP 443 would answer for
  the wrong names.
- **D12 balthasar becomes a server**:
  - the desktop no longer starts (`systemctl set-default multi-user.target`, undone by
    `graphical.target`), which frees about 1 GB; Multipass is disabled if unused;
  - Docker's data moves to the HDD, as the root filesystem has room for no image:
    `/var/lib/docker` and `/var/lib/containerd` (where Docker 29 keeps images) as bind
    mounts from `/data`, as `/home` and snapd's folder already are on this machine,
    rather than Docker's `data-root`, which would leave containerd's store behind;
  - a `deploy` user holds the checkout (`~deploy/halkyon`) and runs the stack, as on
    sahaquiel; the sidecars' user IDs are those of the sidecar README;
  - Caddy publishes 443 on the tunnel's address alone (`10.61.97.2:443`), the tunnel
    started before Docker (`Before=docker.service`), so nothing of the instance answers
    on the home network.
- **D13 Deploys by hand with `redeploy.sh`**, run as `deploy` on balthasar with the
  commit, as on sahaquiel: still only the tip of `main`, a fast-forward, every commit
  signed by a key in `deploy/allowed-signers`, the databases dumped first, migrations
  in a run of their own. A deploy waits for that night's green run of the commit. The
  script loops over the apps .env switches on (`train todo`, now `meditate` too) and
  its comments stop naming learn.halkyon.eu.
- **D14 Meditate's deploy step (ADR 0038, step 7) is built for it**:
  `deploy/compose.meditate.yaml` and `sidecars/meditate.caddy` as Todo's are, the
  `halkyon_meditate` database under a role of its own, a member of the register,
  `redeploy.sh` and `backup.sh` taking it in.
- **D15 Jellyfin as the instance's sidecar** (`compose.jellyfin.yaml`), on `edge` beside
  Caddy, with its existing configuration (`deploy/home-jellyfin/jellyfin/config`), so its
  accounts and libraries stay. Its library is mounted read-only from the host
  (`JELLYFIN_MEDIA`). Found while building: it can't stay in
  `/home/raisondetredev/Media`, as the developer's home is closed to other users and
  Jellyfin runs as its own (61096). So it moves to `/data/media` (a rename on one
  filesystem, `/home` being a bind mount of `/data/home`), readable by all, over opening
  the home to every user. ADR 0032's network settings (LAN `172.29.96.1/32`, proxy
  `10.61.96.1`) are set to the sidecar's before it starts, as they would count the
  admin's requests as remote and lock them out. `deploy/home-jellyfin/` and
  `compose.jellyfin-home.yaml` go, with ADR 0032's open question.
- **D16 The runner on balthasar**: `deploy/runner/`'s unit as it is (its own user, no
  `/home`, no Docker socket, Docker's networks denied), with these changes:
  - its folder on the HDD, `/data/forgejo-runner`, seen as `/var/lib/forgejo-runner`;
    the rest of `/data`, where `/home` really lies, hidden from it;
  - no IPv6 at all, and no private IPv4 network (`10.0.0.0/8`, which holds the tunnel,
    among them). Corrected while building: the proposal had the home network denied by
    interface, but the home network and the internet come through the same one, and the
    home network's IPv6 addresses are public ones under a prefix Vodafone changes. Every
    host CI fetches from answers over IPv4;
  - the timer's window (D6), and memory to fit beside the instance: 5 GB, swap beyond;
  - CI's `runs-on` (`balthasar`) and the Sandbox step's checks following it.

### Decided after the proposal (8 Nov 2026)

- **D17 melchior reaches balthasar at home and from away** (the developer's choice). At
  home it connects over the home network. From away it jumps through ramiel
  (`ssh -J ramiel` to `10.61.97.2`), so D9's rules also let `10.61.97.1` reach
  balthasar's sshd on TCP 22, and nothing else beyond Caddy's 443. balthasar's sshd
  takes keys alone (`PasswordAuthentication no`, `KbdInteractiveAuthentication no`) and
  lets in only the developer's user and `deploy`. A taken-over ramiel thus meets one
  more port, which answers to no password.
- **D18 Backups go to the same Hetzner Storage Box** (the developer's choice), in a
  restic repository of their own with a password of its own. The repository is reached
  through a sub-account of the box, confined to its own folder. A taken-over balthasar
  then reaches halkyon.li's backups alone, and the box's automatic snapshots, which
  only Hetzner's Console removes, still keep those safe. `backup.sh` runs hourly as it
  does on sahaquiel, inside the nightly window too, as the servers and PostgreSQL keep
  running (D6).
- **D19 Photon builds an index of our own from the chosen polygons**, as polygons below
  countries need (Photon's published dumps stop at countries): a Geofabrik extract cut
  by the polygons, imported by a short-lived Nominatim. The developer raised no
  objection. The console's ADR decides how it runs.

- **D20 balthasar's root disk is unlocked over SSH at boot** (the developer's choice,
  over Tang on ramiel, which would unlock a stolen machine that is still online, and over
  typing at the iMac's keyboard alone). Found while writing the host's steps: the root
  disk is LUKS-encrypted with a passphrase (`/etc/crypttab`: `none`), and the HDD
  unlocks from a key file on it, so every boot waits for the passphrase.
  `dropbear-initramfs` waits on the home network at boot, on port 2222 (so its host key
  never clashes with sshd's in a `known_hosts`). It takes melchior's key alone, which may
  run nothing but `cryptroot-unlock`. The USB network adapter's driver (`cdc_ncm`) is
  added to the initramfs. From away, a reboot waits for someone at home, as the tunnel
  isn't up that early. The keyboard still works as before. Ubuntu 26.04 builds its boot
  images with dracut, for which Ubuntu packages no SSH at boot; so initramfs-tools
  replaces it, with `cryptsetup-initramfs` and Ubuntu's `dropbear-initramfs` (the
  developer's choice of 8 Nov 2026, over the unpackaged dracut-sshd and over no
  unlocking by SSH). dracut's last image stays in GRUB's menu, shown for 5 seconds, as
  the way back until initramfs-tools' images have booted, so a failed image costs a
  choice at the iMac rather than a USB stick.

## Consequences

- People see ramiel's address alone; nothing names the home address, which only ramiel
  knows (`wg show`). With balthasar off or offline, ramiel's HAProxy finds nothing behind
  the tunnel and halkyon.li is away; the apps sync offline-first and catch up.
- Every byte of halkyon.li crosses the home line's upload and ramiel twice. Jellyfin's
  streams are bounded by the upload; ramiel's traffic plan is to be checked.
- ramiel gains HAProxy, a listening UDP port, a kernel interface and firewall rules; its
  Caddy moves off 443 to the loopback, its sites unchanged.
- The source links (About in each app, the snap's `source-code`, the F-Droid listings,
  `halkyon.app`'s and `learn.halkyon.app`'s redirects) lead to Forgejo's sign-in page
  while `halkyon` is private. halkyon.li leaves `SOURCE_URL` as it is.
- Learn's `AGENTS.md`, *CI and deploys*, the root's, and `deploy/backups.md` are
  rewritten for balthasar once built.