fix(release): F-Droid's upload resumable alone, by content, and Forgejo asked again

Train's first release (31 Oct 2026) stopped twice after its tag was pushed:

- Forgejo answered 500 (a unique constraint) to the release: it enters a
  pushed tag in its releases itself, a moment after the push. Asked again a
  few seconds later, it makes the release of that entry.
- The F-Droid upload filled the halkyon.app server's disk: rsync compared by
  time, and this laptop's working folder, restored from the repository, had
  every APK anew, so all 1.9 GB went again. It compares by content now.

tool/with_signing_keys.sh fdroid [APK] runs tool/fdroid_repo.sh alone with
the index's key, for an upload that failed after its release was made, and
the script skips an APK the working folder holds already rather than adding
a second file of the same build, which fdroid refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
Tizian 「ティツィアーン」 2026-10-31 23:28:49 +01:00
commit d981747728
Signed by: raisondetredev
GPG key ID: 07F6FCF405488B64
5 changed files with 64 additions and 7 deletions

View file

@ -82,7 +82,7 @@ Checklist after doing changes, in this order:
- Check `serverpod` MCP `tail_server_logs` and `tail_flutter_logs` for any issues. - Check `serverpod` MCP `tail_server_logs` and `tail_flutter_logs` for any issues.
- **Melos** (decided 2026-10-19, ADR 0015) runs scripts across the workspace's packages: a dev dependency of the root (`dart run melos …`, its version in the lock), its scripts under `melos:` in the root `pubspec.yaml`: `analyze`; `test:dart` (the packages without Flutter, one at a time, as each server's tests start a PostgreSQL of their own on the same port); `test:flutter` (`flutter test -x requires-server`, one package at a time); `generate` (Serverpod's code generation in every package with a `config/generator.yaml`, the CLI on the PATH); `check` (`tool/check.dart`). Every script takes Melos' filters: `dart run melos run test:flutter --scope=halkyon_app`, or `--diff=origin/main --include-dependents` for what changed and what depends on it; `dart run melos list` lists the packages. The root overrides `cli_util` to 0.5.2 for it: Learn's `flutter_launcher_icons` (0.14.4, its latest) asks for ^0.4.1 and uses only the logger, which 0.5 keeps; drop the override once that package allows 0.5. - **Melos** (decided 2026-10-19, ADR 0015) runs scripts across the workspace's packages: a dev dependency of the root (`dart run melos …`, its version in the lock), its scripts under `melos:` in the root `pubspec.yaml`: `analyze`; `test:dart` (the packages without Flutter, one at a time, as each server's tests start a PostgreSQL of their own on the same port); `test:flutter` (`flutter test -x requires-server`, one package at a time); `generate` (Serverpod's code generation in every package with a `config/generator.yaml`, the CLI on the PATH); `check` (`tool/check.dart`). Every script takes Melos' filters: `dart run melos run test:flutter --scope=halkyon_app`, or `--diff=origin/main --include-dependents` for what changed and what depends on it; `dart run melos list` lists the packages. The root overrides `cli_util` to 0.5.2 for it: Learn's `flutter_launcher_icons` (0.14.4, its latest) asks for ^0.4.1 and uses only the logger, which 0.5 keeps; drop the override once that package allows 0.5.
- **Tools are Dart where they hold logic** (decided 2026-10-19): conditions, parsing, parallel work, anything worth a test, as `tool/check.dart` (its file selection in `tool/src/check_files.dart`, tested in the root's `test/`). **Bash for glue**, short and linear (`tool/pre-push`, `tool/check_signatures.sh`, `tool/workspace_only.sh`, which narrows the root pubspec's workspace to the packages an image builds), **and for everything run on the production host**, which has no Dart SDK and gets none (`deploy/redeploy.sh`, `deploy/backup.sh`). **Python only where a library exists only there** (the apps' icons with librsvg and Pillow, `tool/app_icons.py`, its Anki fixtures with Anki itself); Learn's data scripts and `packages/halkyon_app/tool/subdivision_names.py` need nothing but Python's standard library and may move to Dart when they next change. **Releases come from `tool/release.dart`** (decided 2026-10-19, ADR 0015), shared by the apps, each configured in Dart beside it (`apps/<id>/release.dart`, typed and commented, registered in `tool/src/release/apps.dart`): `dart run tool/release.dart learn [major | minor | patch] [--dry-run]`, on a developer's laptop, never in CI, through `tool/with_signing_keys.sh learn …` (Bash glue): the keystores are kept in Proton Pass alone (vault *Infra · Signing*, each a note's attachment, the password its hidden field), and it downloads the app's and the F-Droid index's into a folder in memory (`$XDG_RUNTIME_DIR`), hands their paths and, but for a dry run, their passwords (each item's hidden field *Password*, in the environment as `<keystore variable>_PASSWORD`, so the tool asks for neither) and the Forgejo token (a login in the vault *Infra*, `HALKYON_FORGEJO_TOKEN`) to the release tool and removes the files however the run ends (`test/release/signing_keys_test.dart` holds it to every app released). Tags are per app (`learn-v0.19.0`; an app's earlier tags, Learn's `v…`, count as its history), and only commits touching the app decide its version: its folder, the shared packages its packages use by path (transitively), the root pubspec and lock, its release files (key pins, F-Droid listing, snap recipe) and where its packages were before a move. Each release's *What's new* (Train's ADR 0030, D61), one file a language in the app's `assets/whats_new/` (`<version>.<language>.txt`, English and German), is drafted by it from the commits after *Go on?* and edited in `$VISUAL`/`$EDITOR` until it fits (no `TRANSLATE:` left, F-Droid's 500 characters), committed with the version, and is the app's sheet after the update, the Forgejo notes and F-Droid's changelog. `--dry-run` reads everything a release would (the version, the commits that count and what each touches, the steps, the notes it would publish), changes nothing, asks for no secret, and lists what a release would refuse. Its pure logic is in `tool/src/release/`, tested in the root's `test/release/`. `tool/fdroid_repo.sh` stays Bash (glue for fdroidserver). Build 21 (`v0.18.0`) was the last release of `tool/release.sh`, the Bash script it replaced, removed since. - **Tools are Dart where they hold logic** (decided 2026-10-19): conditions, parsing, parallel work, anything worth a test, as `tool/check.dart` (its file selection in `tool/src/check_files.dart`, tested in the root's `test/`). **Bash for glue**, short and linear (`tool/pre-push`, `tool/check_signatures.sh`, `tool/workspace_only.sh`, which narrows the root pubspec's workspace to the packages an image builds), **and for everything run on the production host**, which has no Dart SDK and gets none (`deploy/redeploy.sh`, `deploy/backup.sh`). **Python only where a library exists only there** (the apps' icons with librsvg and Pillow, `tool/app_icons.py`, its Anki fixtures with Anki itself); Learn's data scripts and `packages/halkyon_app/tool/subdivision_names.py` need nothing but Python's standard library and may move to Dart when they next change. **Releases come from `tool/release.dart`** (decided 2026-10-19, ADR 0015), shared by the apps, each configured in Dart beside it (`apps/<id>/release.dart`, typed and commented, registered in `tool/src/release/apps.dart`): `dart run tool/release.dart learn [major | minor | patch] [--dry-run]`, on a developer's laptop, never in CI, through `tool/with_signing_keys.sh learn …` (Bash glue): the keystores are kept in Proton Pass alone (vault *Infra · Signing*, each a note's attachment, the password its hidden field), and it downloads the app's and the F-Droid index's into a folder in memory (`$XDG_RUNTIME_DIR`), hands their paths and, but for a dry run, their passwords (each item's hidden field *Password*, in the environment as `<keystore variable>_PASSWORD`, so the tool asks for neither) and the Forgejo token (a login in the vault *Infra*, `HALKYON_FORGEJO_TOKEN`) to the release tool and removes the files however the run ends (`test/release/signing_keys_test.dart` holds it to every app released). Tags are per app (`learn-v0.19.0`; an app's earlier tags, Learn's `v…`, count as its history), and only commits touching the app decide its version: its folder, the shared packages its packages use by path (transitively), the root pubspec and lock, its release files (key pins, F-Droid listing, snap recipe) and where its packages were before a move. Each release's *What's new* (Train's ADR 0030, D61), one file a language in the app's `assets/whats_new/` (`<version>.<language>.txt`, English and German), is drafted by it from the commits after *Go on?* and edited in `$VISUAL`/`$EDITOR` until it fits (no `TRANSLATE:` left, F-Droid's 500 characters), committed with the version, and is the app's sheet after the update, the Forgejo notes and F-Droid's changelog. `--dry-run` reads everything a release would (the version, the commits that count and what each touches, the steps, the notes it would publish), changes nothing, asks for no secret, and lists what a release would refuse. Its pure logic is in `tool/src/release/`, tested in the root's `test/release/`. `tool/fdroid_repo.sh` stays Bash (glue for fdroidserver); `tool/with_signing_keys.sh fdroid [APK]` runs it alone with the index's key, e.g. when a release's upload failed after its release was made; a release colliding with the entry Forgejo makes of the pushed tag itself asks again, a few seconds later. Build 21 (`v0.18.0`) was the last release of `tool/release.sh`, the Bash script it replaced, removed since.
## CI and deploys ## CI and deploys

View file

@ -61,4 +61,16 @@ void main() {
expect(script, contains(r'export HALKYON_FORGEJO_TOKEN="$token"')); expect(script, contains(r'export HALKYON_FORGEJO_TOKEN="$token"'));
expect(releaser, contains("variable: 'HALKYON_FORGEJO_TOKEN'")); expect(releaser, contains("variable: 'HALKYON_FORGEJO_TOKEN'"));
}); });
test("fdroid runs the F-Droid repository's script alone, with the "
"index's key and its password only", () {
expect(script, contains(' fdroid) ;;'));
expect(script, contains(r'tool/fdroid_repo.sh "$@" || status=$?'));
final alone = script.substring(
script.indexOf(r'if [ "$app" = fdroid ]; then'),
script.indexOf(r'tool/fdroid_repo.sh "$@"'),
);
expect(alone, isNot(contains('app_item')));
expect(alone, isNot(contains('forgejo_token')));
});
} }

View file

@ -134,7 +134,15 @@ for icon in app.halkyon.learn:snap/gui/halkyon-learn.png \
done done
if [ -n "$apk" ]; then if [ -n "$apk" ]; then
[ -f "$apk" ] || fail "No APK at $apk." [ -f "$apk" ] || fail "No APK at $apk."
cp "$apk" "$working/repo/" # Once in the working folder (an earlier run whose upload failed), the same APK
# again would be a second file of the same build, which fdroid refuses.
apk_hash=$(sha256sum "$apk" | cut -d' ' -f1)
if sha256sum "$working"/repo/*.apk "$working"/archive/*.apk 2>/dev/null |
grep -q "^$apk_hash "; then
echo " $(basename "$apk") is in the repository already"
else
cp "$apk" "$working/repo/"
fi
fi fi
# Names every APK as F-Droid does (<app id>_<version code>.apk) and signs the # Names every APK as F-Droid does (<app id>_<version code>.apk) and signs the
# indexes with the repository's key. # indexes with the repository's key.
@ -142,15 +150,17 @@ fi
step "Uploading it to $upload_host" step "Uploading it to $upload_host"
# The APKs and icons first, the indexes last, so the F-Droid app never reads an # The APKs and icons first, the indexes last, so the F-Droid app never reads an
# index naming a file that isn't there yet. # index naming a file that isn't there yet. By their contents (--checksum), not their
# times: a working folder restored on another laptop has every APK with a new time,
# and would send them all again (1.9 GB in October 2026, on a disk with less free).
for dir in repo archive; do for dir in repo archive; do
[ -d "$working/$dir" ] || continue [ -d "$working/$dir" ] || continue
rsync -a --chmod=D755,F644 --exclude='index*' --exclude='entry.*' \ rsync -a --checksum --chmod=D755,F644 --exclude='index*' --exclude='entry.*' \
"$working/$dir/" "$upload_host:$dir/" "$working/$dir/" "$upload_host:$dir/"
done done
for dir in repo archive; do for dir in repo archive; do
[ -d "$working/$dir" ] || continue [ -d "$working/$dir" ] || continue
rsync -a --chmod=D755,F644 --delete-after "$working/$dir/" "$upload_host:$dir/" rsync -a --checksum --chmod=D755,F644 --delete-after "$working/$dir/" "$upload_host:$dir/"
done done
echo " $repo_url?fingerprint=$fingerprint" echo " $repo_url?fingerprint=$fingerprint"

View file

@ -1209,16 +1209,33 @@ final class _Forgejo(
} }
/// Makes the release of [tag] and returns its id. /// Makes the release of [tag] and returns its id.
///
/// Forgejo enters a pushed tag in its releases itself, a moment after the
/// push; asked for the release meanwhile, it collides with that entry
/// (500, a unique constraint, as train-v0.1.0 did on 31 Oct 2026). Once
/// the entry is there, it makes the release of it: so it is asked again,
/// a few seconds later.
Future<int> createRelease({ Future<int> createRelease({
required String tag, required String tag,
required String name, required String name,
required String body, required String body,
}) async { }) async {
final response = await _client.post( Future<http.Response> post() => _client.post(
_api('/releases'), _api('/releases'),
headers: {..._authorization, 'Content-Type': 'application/json'}, headers: {..._authorization, 'Content-Type': 'application/json'},
body: jsonEncode({'tag_name': tag, 'name': name, 'body': body}), body: jsonEncode({'tag_name': tag, 'name': name, 'body': body}),
); );
var response = await post();
for (
var attempt = 1;
attempt < 5 &&
response.statusCode == 500 &&
response.body.contains('unique constraint');
attempt++
) {
await Future<void>.delayed(const Duration(seconds: 3));
response = await post();
}
final json = jsonDecode(response.body); final json = jsonDecode(response.body);
if (response.statusCode != 201 || if (response.statusCode != 201 ||
json is! Map<String, Object?> || json is! Map<String, Object?> ||

View file

@ -10,6 +10,11 @@
# #
# tool/with_signing_keys.sh <app> [release.dart's arguments] # tool/with_signing_keys.sh <app> [release.dart's arguments]
# tool/with_signing_keys.sh train minor --dry-run # tool/with_signing_keys.sh train minor --dry-run
# tool/with_signing_keys.sh fdroid [tool/fdroid_repo.sh's arguments]
#
# The last runs tool/fdroid_repo.sh alone, with the F-Droid index's key and its password
# only: to publish the repository again, or a release's APK whose upload failed after its
# release was made (build/release/<tag>/<apk>).
# #
# Needs pass-cli, logged in (pass-cli login), and jq. Reads of the items take their IDs, # Needs pass-cli, logged in (pass-cli login), and jq. Reads of the items take their IDs,
# their attachments' names and, but for a dry run, the field "Password", which goes # their attachments' names and, but for a dry run, the field "Password", which goes
@ -24,13 +29,14 @@ fail() {
exit 1 exit 1
} }
[ $# -ge 1 ] || fail 'Usage: tool/with_signing_keys.sh <app> [release arguments]' [ $# -ge 1 ] || fail 'Usage: tool/with_signing_keys.sh <app | fdroid> [arguments]'
app="$1" app="$1"
shift shift
# Each app released (tool/src/release/apps.dart): the variable its keystore's path goes # Each app released (tool/src/release/apps.dart): the variable its keystore's path goes
# in (its AndroidSigning's keystoreVariable) and the item keeping it. # in (its AndroidSigning's keystoreVariable) and the item keeping it.
case "$app" in case "$app" in
fdroid) ;;
learn) app_variable=HALKYON_ANDROID_KEYSTORE app_item='Halkyon Learn – Android release keystore' ;; learn) app_variable=HALKYON_ANDROID_KEYSTORE app_item='Halkyon Learn – Android release keystore' ;;
train) app_variable=HALKYON_TRAIN_ANDROID_KEYSTORE app_item='Halkyon Train – Android release keystore' ;; train) app_variable=HALKYON_TRAIN_ANDROID_KEYSTORE app_item='Halkyon Train – Android release keystore' ;;
*) fail "No signing key known for $app." ;; *) fail "No signing key known for $app." ;;
@ -88,6 +94,18 @@ forgejo_token() {
printf '%s' "$value" printf '%s' "$value"
} }
if [ "$app" = fdroid ]; then
echo "▸ Fetching the F-Droid index's key from $vault"
fetch "$fdroid_item" fdroid-index.p12 "$keys/fdroid-index.p12"
export HALKYON_FDROID_KEYSTORE="$keys/fdroid-index.p12"
fdroid_password=$(password "$fdroid_item")
export HALKYON_FDROID_KEYSTORE_PASSWORD="$fdroid_password"
unset fdroid_password
status=0
tool/fdroid_repo.sh "$@" || status=$?
exit "$status"
fi
echo "▸ Fetching the signing keys from $vault" echo "▸ Fetching the signing keys from $vault"
fetch "$app_item" android-release.p12 "$keys/android-release.p12" fetch "$app_item" android-release.p12 "$keys/android-release.p12"
fetch "$fdroid_item" fdroid-index.p12 "$keys/fdroid-index.p12" fetch "$fdroid_item" fdroid-index.p12 "$keys/fdroid-index.p12"