feat(deploy): the CI runner's window by hand, halkyon-ci-now (ADR 0039, D6)

Rests the heavy sidecars, runs every queued job, then stops the runner and
starts the sidecars again however it ends; from melchior through
systemd-run, so a closed laptop doesn't end it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

deploy/host/README.md

@@ -200,6 +200,7 @@ databases, the first start, the first operator, the backups). Then:
      deploy/runner/halkyon-night.service deploy/runner/halkyon-night.timer \
      deploy/runner/halkyon-morning.service deploy/runner/halkyon-morning.timer \
      /etc/systemd/system/
+   sudo install -m 755 deploy/runner/ci-now.sh /usr/local/sbin/halkyon-ci-now
    ```
 
    In Forgejo (`code.halkyon.app`, the repository's *Settings › Actions › Runners*): a new
@@ -213,6 +214,7 @@ databases, the first start, the first operator, the backups). Then:
    systemctl list-timers 'forgejo-runner*' 'halkyon-*'   # 01:55, 02:00, 05:00
    ```
 
-   To try it by day: `sudo systemctl start forgejo-runner` (it stops itself three hours
+   To run the queue by day: `sudo halkyon-ci-now` (`deploy/runner/ci-now.sh`, which says
-   later) and push a branch. The job's first step, *Sandbox*, fails if the runner
+   how from melchior too) rests the heavy sidecars, runs every queued job and starts them
+   again; `--keep-sidecars` keeps them, the job then slower, in swap. The job's first step, *Sandbox*, fails if the runner
    reaches the instance, the home network, the tunnel or any IPv6 address, or sees `/data`.

deploy/runner/ci-now.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+# The CI runner's window by hand, outside the night's (ADR 0039, D6): the heavy sidecars
+# rested (halkyon-night), the runner started, every queued job run, then the runner
+# stopped and the sidecars started again (halkyon-morning), however it ends. Jellyfin,
+# the map, routes, place search, LanguageTool and the dictionaries are down meanwhile;
+# --keep-sidecars keeps them, and the job runs slower beside them, in swap.
+#
+# Installed root's own by host step 7, never run from deploy's checkout (deploy could
+# change it, and root runs it):
+#   sudo install -m 755 deploy/runner/ci-now.sh /usr/local/sbin/halkyon-ci-now
+# At the iMac:  sudo halkyon-ci-now [--keep-sidecars]
+# From melchior, on its own on balthasar, so a closed laptop doesn't end it:
+#   ssh -t balthasar sudo systemd-run --unit halkyon-ci-now /usr/local/sbin/halkyon-ci-now
+#   ssh balthasar journalctl -fu halkyon-ci-now
+#   ssh -t balthasar sudo systemctl stop halkyon-ci-now     # to end it early
+set -euo pipefail
+
+keep=false
+case ${1:-} in
+  '') ;;
+  --keep-sidecars) keep=true ;;
+  *)
+    echo "usage: $0 [--keep-sidecars]" >&2
+    exit 2
+    ;;
+esac
+[ "$(id -u)" = 0 ] || { echo "Error: run it as root (sudo)" >&2; exit 1; }
+if systemctl is-active --quiet forgejo-runner; then
+  echo "The runner runs already (the night's window, or another start); nothing done." >&2
+  exit 1
+fi
+
+restore() {
+  trap - EXIT INT TERM HUP
+  echo "== The runner stopped$($keep || echo ', the sidecars started again')"
+  systemctl stop forgejo-runner || true
+  $keep || systemctl start halkyon-morning
+}
+trap restore EXIT INT TERM HUP
+
+if ! $keep; then
+  echo "== The heavy sidecars rested"
+  systemctl start halkyon-night
+fi
+echo "== The runner started; it stops itself after three hours at the latest"
+systemctl start forgejo-runner
+
+# The runner alone in its control group while idle; a job's processes beside it while
+# one runs. Done once it has been idle for 90 s, several of its 10 s polls of the queue.
+procs=/sys/fs/cgroup/system.slice/forgejo-runner.service/cgroup.procs
+idle=0
+jobs=0
+running=false
+while systemctl is-active --quiet forgejo-runner; do
+  if [ "$(wc -l <"$procs")" -gt 1 ]; then
+    $running || { jobs=$((jobs + 1)); echo "$(date +%H:%M) job $jobs running"; }
+    running=true
+    idle=0
+  else
+    $running && echo "$(date +%H:%M) job $jobs ended"
+    running=false
+    idle=$((idle + 10))
+    [ "$idle" -ge 90 ] && break
+  fi
+  sleep 10
+done
+echo "== $jobs job(s) run; their results are in Forgejo, the repository's Actions"

test/deploy/runner_test.dart

@@ -154,6 +154,48 @@ void main() {
     });
   });
 
+  group('a window by hand (halkyon-ci-now)', () {
+    final script = read('deploy/runner/ci-now.sh');
+
+    test('rests the heavy sidecars around the runner, unless asked to keep '
+        'them', () {
+      expect(script, contains('systemctl start halkyon-night'));
+      expect(script, contains('systemctl start halkyon-morning'));
+      expect(script, contains('--keep-sidecars'));
+    });
+
+    test('stops the runner and starts the sidecars again however it ends: '
+        'done, interrupted, stopped or its SSH session gone', () {
+      final trap = RegExp(
+        r'^trap (\S+) (.*)$',
+        multiLine: true,
+      ).firstMatch(script);
+      expect(trap, isNotNull);
+      expect(trap![2]!.split(' '), containsAll(['EXIT', 'INT', 'TERM', 'HUP']));
+      final restore = RegExp(
+        '^${trap[1]}\\(\\) \\{([^}]*)\\}',
+        multiLine: true,
+      ).firstMatch(script)![1]!;
+      expect(restore, contains('systemctl stop forgejo-runner'));
+      expect(restore, contains('systemctl start halkyon-morning'));
+    });
+
+    test('leaves the nightly window alone: refuses while the runner runs', () {
+      expect(script, contains('systemctl is-active --quiet forgejo-runner'));
+    });
+
+    test(
+      "is installed by host step 7, root's own, never deploy's checkout",
+      () {
+        final readme = read('deploy/host/README.md');
+        expect(
+          readme,
+          contains('deploy/runner/ci-now.sh /usr/local/sbin/halkyon-ci-now'),
+        );
+      },
+    );
+  });
+
   group('the workflow', () {
     final workflow = loadYaml(read('.forgejo/workflows/ci.yaml')) as YamlMap;
     final jobs = workflow['jobs'] as YamlMap;