fix(release): F-Droid's upload resumable alone, by content, and Forgejo asked again
Train's first release (31 Oct 2026) stopped twice after its tag was pushed: - Forgejo answered 500 (a unique constraint) to the release: it enters a pushed tag in its releases itself, a moment after the push. Asked again a few seconds later, it makes the release of that entry. - The F-Droid upload filled the halkyon.app server's disk: rsync compared by time, and this laptop's working folder, restored from the repository, had every APK anew, so all 1.9 GB went again. It compares by content now. tool/with_signing_keys.sh fdroid [APK] runs tool/fdroid_repo.sh alone with the index's key, for an upload that failed after its release was made, and the script skips an APK the working folder holds already rather than adding a second file of the same build, which fdroid refuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
90edf2ec84
commit
d981747728
5 changed files with 64 additions and 7 deletions
|
|
@ -82,7 +82,7 @@ Checklist after doing changes, in this order:
|
||||||
- Check `serverpod` MCP `tail_server_logs` and `tail_flutter_logs` for any issues.
|
- Check `serverpod` MCP `tail_server_logs` and `tail_flutter_logs` for any issues.
|
||||||
|
|
||||||
- **Melos** (decided 2026-10-19, ADR 0015) runs scripts across the workspace's packages: a dev dependency of the root (`dart run melos …`, its version in the lock), its scripts under `melos:` in the root `pubspec.yaml`: `analyze`; `test:dart` (the packages without Flutter, one at a time, as each server's tests start a PostgreSQL of their own on the same port); `test:flutter` (`flutter test -x requires-server`, one package at a time); `generate` (Serverpod's code generation in every package with a `config/generator.yaml`, the CLI on the PATH); `check` (`tool/check.dart`). Every script takes Melos' filters: `dart run melos run test:flutter --scope=halkyon_app`, or `--diff=origin/main --include-dependents` for what changed and what depends on it; `dart run melos list` lists the packages. The root overrides `cli_util` to 0.5.2 for it: Learn's `flutter_launcher_icons` (0.14.4, its latest) asks for ^0.4.1 and uses only the logger, which 0.5 keeps; drop the override once that package allows 0.5.
|
- **Melos** (decided 2026-10-19, ADR 0015) runs scripts across the workspace's packages: a dev dependency of the root (`dart run melos …`, its version in the lock), its scripts under `melos:` in the root `pubspec.yaml`: `analyze`; `test:dart` (the packages without Flutter, one at a time, as each server's tests start a PostgreSQL of their own on the same port); `test:flutter` (`flutter test -x requires-server`, one package at a time); `generate` (Serverpod's code generation in every package with a `config/generator.yaml`, the CLI on the PATH); `check` (`tool/check.dart`). Every script takes Melos' filters: `dart run melos run test:flutter --scope=halkyon_app`, or `--diff=origin/main --include-dependents` for what changed and what depends on it; `dart run melos list` lists the packages. The root overrides `cli_util` to 0.5.2 for it: Learn's `flutter_launcher_icons` (0.14.4, its latest) asks for ^0.4.1 and uses only the logger, which 0.5 keeps; drop the override once that package allows 0.5.
|
||||||
- **Tools are Dart where they hold logic** (decided 2026-10-19): conditions, parsing, parallel work, anything worth a test, as `tool/check.dart` (its file selection in `tool/src/check_files.dart`, tested in the root's `test/`). **Bash for glue**, short and linear (`tool/pre-push`, `tool/check_signatures.sh`, `tool/workspace_only.sh`, which narrows the root pubspec's workspace to the packages an image builds), **and for everything run on the production host**, which has no Dart SDK and gets none (`deploy/redeploy.sh`, `deploy/backup.sh`). **Python only where a library exists only there** (the apps' icons with librsvg and Pillow, `tool/app_icons.py`, its Anki fixtures with Anki itself); Learn's data scripts and `packages/halkyon_app/tool/subdivision_names.py` need nothing but Python's standard library and may move to Dart when they next change. **Releases come from `tool/release.dart`** (decided 2026-10-19, ADR 0015), shared by the apps, each configured in Dart beside it (`apps/<id>/release.dart`, typed and commented, registered in `tool/src/release/apps.dart`): `dart run tool/release.dart learn [major | minor | patch] [--dry-run]`, on a developer's laptop, never in CI, through `tool/with_signing_keys.sh learn …` (Bash glue): the keystores are kept in Proton Pass alone (vault *Infra · Signing*, each a note's attachment, the password its hidden field), and it downloads the app's and the F-Droid index's into a folder in memory (`$XDG_RUNTIME_DIR`), hands their paths and, but for a dry run, their passwords (each item's hidden field *Password*, in the environment as `<keystore variable>_PASSWORD`, so the tool asks for neither) and the Forgejo token (a login in the vault *Infra*, `HALKYON_FORGEJO_TOKEN`) to the release tool and removes the files however the run ends (`test/release/signing_keys_test.dart` holds it to every app released). Tags are per app (`learn-v0.19.0`; an app's earlier tags, Learn's `v…`, count as its history), and only commits touching the app decide its version: its folder, the shared packages its packages use by path (transitively), the root pubspec and lock, its release files (key pins, F-Droid listing, snap recipe) and where its packages were before a move. Each release's *What's new* (Train's ADR 0030, D61), one file a language in the app's `assets/whats_new/` (`<version>.<language>.txt`, English and German), is drafted by it from the commits after *Go on?* and edited in `$VISUAL`/`$EDITOR` until it fits (no `TRANSLATE:` left, F-Droid's 500 characters), committed with the version, and is the app's sheet after the update, the Forgejo notes and F-Droid's changelog. `--dry-run` reads everything a release would (the version, the commits that count and what each touches, the steps, the notes it would publish), changes nothing, asks for no secret, and lists what a release would refuse. Its pure logic is in `tool/src/release/`, tested in the root's `test/release/`. `tool/fdroid_repo.sh` stays Bash (glue for fdroidserver). Build 21 (`v0.18.0`) was the last release of `tool/release.sh`, the Bash script it replaced, removed since.
|
- **Tools are Dart where they hold logic** (decided 2026-10-19): conditions, parsing, parallel work, anything worth a test, as `tool/check.dart` (its file selection in `tool/src/check_files.dart`, tested in the root's `test/`). **Bash for glue**, short and linear (`tool/pre-push`, `tool/check_signatures.sh`, `tool/workspace_only.sh`, which narrows the root pubspec's workspace to the packages an image builds), **and for everything run on the production host**, which has no Dart SDK and gets none (`deploy/redeploy.sh`, `deploy/backup.sh`). **Python only where a library exists only there** (the apps' icons with librsvg and Pillow, `tool/app_icons.py`, its Anki fixtures with Anki itself); Learn's data scripts and `packages/halkyon_app/tool/subdivision_names.py` need nothing but Python's standard library and may move to Dart when they next change. **Releases come from `tool/release.dart`** (decided 2026-10-19, ADR 0015), shared by the apps, each configured in Dart beside it (`apps/<id>/release.dart`, typed and commented, registered in `tool/src/release/apps.dart`): `dart run tool/release.dart learn [major | minor | patch] [--dry-run]`, on a developer's laptop, never in CI, through `tool/with_signing_keys.sh learn …` (Bash glue): the keystores are kept in Proton Pass alone (vault *Infra · Signing*, each a note's attachment, the password its hidden field), and it downloads the app's and the F-Droid index's into a folder in memory (`$XDG_RUNTIME_DIR`), hands their paths and, but for a dry run, their passwords (each item's hidden field *Password*, in the environment as `<keystore variable>_PASSWORD`, so the tool asks for neither) and the Forgejo token (a login in the vault *Infra*, `HALKYON_FORGEJO_TOKEN`) to the release tool and removes the files however the run ends (`test/release/signing_keys_test.dart` holds it to every app released). Tags are per app (`learn-v0.19.0`; an app's earlier tags, Learn's `v…`, count as its history), and only commits touching the app decide its version: its folder, the shared packages its packages use by path (transitively), the root pubspec and lock, its release files (key pins, F-Droid listing, snap recipe) and where its packages were before a move. Each release's *What's new* (Train's ADR 0030, D61), one file a language in the app's `assets/whats_new/` (`<version>.<language>.txt`, English and German), is drafted by it from the commits after *Go on?* and edited in `$VISUAL`/`$EDITOR` until it fits (no `TRANSLATE:` left, F-Droid's 500 characters), committed with the version, and is the app's sheet after the update, the Forgejo notes and F-Droid's changelog. `--dry-run` reads everything a release would (the version, the commits that count and what each touches, the steps, the notes it would publish), changes nothing, asks for no secret, and lists what a release would refuse. Its pure logic is in `tool/src/release/`, tested in the root's `test/release/`. `tool/fdroid_repo.sh` stays Bash (glue for fdroidserver); `tool/with_signing_keys.sh fdroid [APK]` runs it alone with the index's key, e.g. when a release's upload failed after its release was made; a release colliding with the entry Forgejo makes of the pushed tag itself asks again, a few seconds later. Build 21 (`v0.18.0`) was the last release of `tool/release.sh`, the Bash script it replaced, removed since.
|
||||||
|
|
||||||
## CI and deploys
|
## CI and deploys
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -61,4 +61,16 @@ void main() {
|
||||||
expect(script, contains(r'export HALKYON_FORGEJO_TOKEN="$token"'));
|
expect(script, contains(r'export HALKYON_FORGEJO_TOKEN="$token"'));
|
||||||
expect(releaser, contains("variable: 'HALKYON_FORGEJO_TOKEN'"));
|
expect(releaser, contains("variable: 'HALKYON_FORGEJO_TOKEN'"));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("fdroid runs the F-Droid repository's script alone, with the "
|
||||||
|
"index's key and its password only", () {
|
||||||
|
expect(script, contains(' fdroid) ;;'));
|
||||||
|
expect(script, contains(r'tool/fdroid_repo.sh "$@" || status=$?'));
|
||||||
|
final alone = script.substring(
|
||||||
|
script.indexOf(r'if [ "$app" = fdroid ]; then'),
|
||||||
|
script.indexOf(r'tool/fdroid_repo.sh "$@"'),
|
||||||
|
);
|
||||||
|
expect(alone, isNot(contains('app_item')));
|
||||||
|
expect(alone, isNot(contains('forgejo_token')));
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -134,7 +134,15 @@ for icon in app.halkyon.learn:snap/gui/halkyon-learn.png \
|
||||||
done
|
done
|
||||||
if [ -n "$apk" ]; then
|
if [ -n "$apk" ]; then
|
||||||
[ -f "$apk" ] || fail "No APK at $apk."
|
[ -f "$apk" ] || fail "No APK at $apk."
|
||||||
cp "$apk" "$working/repo/"
|
# Once in the working folder (an earlier run whose upload failed), the same APK
|
||||||
|
# again would be a second file of the same build, which fdroid refuses.
|
||||||
|
apk_hash=$(sha256sum "$apk" | cut -d' ' -f1)
|
||||||
|
if sha256sum "$working"/repo/*.apk "$working"/archive/*.apk 2>/dev/null |
|
||||||
|
grep -q "^$apk_hash "; then
|
||||||
|
echo " $(basename "$apk") is in the repository already"
|
||||||
|
else
|
||||||
|
cp "$apk" "$working/repo/"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
# Names every APK as F-Droid does (<app id>_<version code>.apk) and signs the
|
# Names every APK as F-Droid does (<app id>_<version code>.apk) and signs the
|
||||||
# indexes with the repository's key.
|
# indexes with the repository's key.
|
||||||
|
|
@ -142,15 +150,17 @@ fi
|
||||||
|
|
||||||
step "Uploading it to $upload_host"
|
step "Uploading it to $upload_host"
|
||||||
# The APKs and icons first, the indexes last, so the F-Droid app never reads an
|
# The APKs and icons first, the indexes last, so the F-Droid app never reads an
|
||||||
# index naming a file that isn't there yet.
|
# index naming a file that isn't there yet. By their contents (--checksum), not their
|
||||||
|
# times: a working folder restored on another laptop has every APK with a new time,
|
||||||
|
# and would send them all again (1.9 GB in October 2026, on a disk with less free).
|
||||||
for dir in repo archive; do
|
for dir in repo archive; do
|
||||||
[ -d "$working/$dir" ] || continue
|
[ -d "$working/$dir" ] || continue
|
||||||
rsync -a --chmod=D755,F644 --exclude='index*' --exclude='entry.*' \
|
rsync -a --checksum --chmod=D755,F644 --exclude='index*' --exclude='entry.*' \
|
||||||
"$working/$dir/" "$upload_host:$dir/"
|
"$working/$dir/" "$upload_host:$dir/"
|
||||||
done
|
done
|
||||||
for dir in repo archive; do
|
for dir in repo archive; do
|
||||||
[ -d "$working/$dir" ] || continue
|
[ -d "$working/$dir" ] || continue
|
||||||
rsync -a --chmod=D755,F644 --delete-after "$working/$dir/" "$upload_host:$dir/"
|
rsync -a --checksum --chmod=D755,F644 --delete-after "$working/$dir/" "$upload_host:$dir/"
|
||||||
done
|
done
|
||||||
|
|
||||||
echo " $repo_url?fingerprint=$fingerprint"
|
echo " $repo_url?fingerprint=$fingerprint"
|
||||||
|
|
|
||||||
|
|
@ -1209,16 +1209,33 @@ final class _Forgejo(
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Makes the release of [tag] and returns its id.
|
/// Makes the release of [tag] and returns its id.
|
||||||
|
///
|
||||||
|
/// Forgejo enters a pushed tag in its releases itself, a moment after the
|
||||||
|
/// push; asked for the release meanwhile, it collides with that entry
|
||||||
|
/// (500, a unique constraint, as train-v0.1.0 did on 31 Oct 2026). Once
|
||||||
|
/// the entry is there, it makes the release of it: so it is asked again,
|
||||||
|
/// a few seconds later.
|
||||||
Future<int> createRelease({
|
Future<int> createRelease({
|
||||||
required String tag,
|
required String tag,
|
||||||
required String name,
|
required String name,
|
||||||
required String body,
|
required String body,
|
||||||
}) async {
|
}) async {
|
||||||
final response = await _client.post(
|
Future<http.Response> post() => _client.post(
|
||||||
_api('/releases'),
|
_api('/releases'),
|
||||||
headers: {..._authorization, 'Content-Type': 'application/json'},
|
headers: {..._authorization, 'Content-Type': 'application/json'},
|
||||||
body: jsonEncode({'tag_name': tag, 'name': name, 'body': body}),
|
body: jsonEncode({'tag_name': tag, 'name': name, 'body': body}),
|
||||||
);
|
);
|
||||||
|
var response = await post();
|
||||||
|
for (
|
||||||
|
var attempt = 1;
|
||||||
|
attempt < 5 &&
|
||||||
|
response.statusCode == 500 &&
|
||||||
|
response.body.contains('unique constraint');
|
||||||
|
attempt++
|
||||||
|
) {
|
||||||
|
await Future<void>.delayed(const Duration(seconds: 3));
|
||||||
|
response = await post();
|
||||||
|
}
|
||||||
final json = jsonDecode(response.body);
|
final json = jsonDecode(response.body);
|
||||||
if (response.statusCode != 201 ||
|
if (response.statusCode != 201 ||
|
||||||
json is! Map<String, Object?> ||
|
json is! Map<String, Object?> ||
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,11 @@
|
||||||
#
|
#
|
||||||
# tool/with_signing_keys.sh <app> [release.dart's arguments]
|
# tool/with_signing_keys.sh <app> [release.dart's arguments]
|
||||||
# tool/with_signing_keys.sh train minor --dry-run
|
# tool/with_signing_keys.sh train minor --dry-run
|
||||||
|
# tool/with_signing_keys.sh fdroid [tool/fdroid_repo.sh's arguments]
|
||||||
|
#
|
||||||
|
# The last runs tool/fdroid_repo.sh alone, with the F-Droid index's key and its password
|
||||||
|
# only: to publish the repository again, or a release's APK whose upload failed after its
|
||||||
|
# release was made (build/release/<tag>/<apk>).
|
||||||
#
|
#
|
||||||
# Needs pass-cli, logged in (pass-cli login), and jq. Reads of the items take their IDs,
|
# Needs pass-cli, logged in (pass-cli login), and jq. Reads of the items take their IDs,
|
||||||
# their attachments' names and, but for a dry run, the field "Password", which goes
|
# their attachments' names and, but for a dry run, the field "Password", which goes
|
||||||
|
|
@ -24,13 +29,14 @@ fail() {
|
||||||
exit 1
|
exit 1
|
||||||
}
|
}
|
||||||
|
|
||||||
[ $# -ge 1 ] || fail 'Usage: tool/with_signing_keys.sh <app> [release arguments]'
|
[ $# -ge 1 ] || fail 'Usage: tool/with_signing_keys.sh <app | fdroid> [arguments]'
|
||||||
app="$1"
|
app="$1"
|
||||||
shift
|
shift
|
||||||
|
|
||||||
# Each app released (tool/src/release/apps.dart): the variable its keystore's path goes
|
# Each app released (tool/src/release/apps.dart): the variable its keystore's path goes
|
||||||
# in (its AndroidSigning's keystoreVariable) and the item keeping it.
|
# in (its AndroidSigning's keystoreVariable) and the item keeping it.
|
||||||
case "$app" in
|
case "$app" in
|
||||||
|
fdroid) ;;
|
||||||
learn) app_variable=HALKYON_ANDROID_KEYSTORE app_item='Halkyon Learn – Android release keystore' ;;
|
learn) app_variable=HALKYON_ANDROID_KEYSTORE app_item='Halkyon Learn – Android release keystore' ;;
|
||||||
train) app_variable=HALKYON_TRAIN_ANDROID_KEYSTORE app_item='Halkyon Train – Android release keystore' ;;
|
train) app_variable=HALKYON_TRAIN_ANDROID_KEYSTORE app_item='Halkyon Train – Android release keystore' ;;
|
||||||
*) fail "No signing key known for $app." ;;
|
*) fail "No signing key known for $app." ;;
|
||||||
|
|
@ -88,6 +94,18 @@ forgejo_token() {
|
||||||
printf '%s' "$value"
|
printf '%s' "$value"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if [ "$app" = fdroid ]; then
|
||||||
|
echo "▸ Fetching the F-Droid index's key from $vault"
|
||||||
|
fetch "$fdroid_item" fdroid-index.p12 "$keys/fdroid-index.p12"
|
||||||
|
export HALKYON_FDROID_KEYSTORE="$keys/fdroid-index.p12"
|
||||||
|
fdroid_password=$(password "$fdroid_item")
|
||||||
|
export HALKYON_FDROID_KEYSTORE_PASSWORD="$fdroid_password"
|
||||||
|
unset fdroid_password
|
||||||
|
status=0
|
||||||
|
tool/fdroid_repo.sh "$@" || status=$?
|
||||||
|
exit "$status"
|
||||||
|
fi
|
||||||
|
|
||||||
echo "▸ Fetching the signing keys from $vault"
|
echo "▸ Fetching the signing keys from $vault"
|
||||||
fetch "$app_item" android-release.p12 "$keys/android-release.p12"
|
fetch "$app_item" android-release.p12 "$keys/android-release.p12"
|
||||||
fetch "$fdroid_item" fdroid-index.p12 "$keys/fdroid-index.p12"
|
fetch "$fdroid_item" fdroid-index.p12 "$keys/fdroid-index.p12"
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue