fix(deploy): the sidecars' downloads try again within the hour after a failed round

A failed BRouter segment or listing, a failed tile cut or an unwritable folder waited the whole refresh period, 30 days without routes or a map; they try again after an hour, doubling to a day. Photon's failed download backs off from a day to a week, and a dead Photon is noticed within a minute. The downloads' network is an ordinary bridge the three share, as the comments now say.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>

deploy/brouter/segments.sh

@@ -3,7 +3,9 @@
 # every 5°×5° segment of BRouter's project whose south-west corner lies in
 # BROUTER_REGION (`west,south,east,north` in whole degrees, multiples of 5; Europe's
 # `-25,35,40,70` by default), downloaded where the project has a newer one than the
-# volume holds, then again every BROUTER_REFRESH_DAYS days (30). Each is written beside
+# volume holds, then again every BROUTER_REFRESH_DAYS days (30); after a round where the
+# listing or a segment didn't come, again in an hour, the wait doubling up to a day
+# while rounds keep failing. Each is written beside
 # its old self and moved over it, so BRouter never reads half a file. Segments of
 # the sea, which the project doesn't have, are skipped.
 set -eu
@@ -25,8 +27,9 @@ name() {
 }
 
 refresh() {
+  complete=0
   listing=$(curl -fsS "$base/") || { echo "The segments' listing didn't come." >&2; return 0; }
-  fetched=0 kept=0
+  fetched=0 kept=0 failed=0
   lon=$west
   while [ "$lon" -le "$east" ]; do
     lat=$south
@@ -36,8 +39,11 @@ refresh() {
         # Only where the project's is newer than ours (If-Modified-Since): a
         # current one comes back empty.
         if [ -e "$file" ]; then set -- -z "$file"; else set --; fi
-        if curl -fsS -R --retry 3 "$@" -o "$file.part" "$base/$file" &&
-          [ -s "$file.part" ]; then
+        if ! curl -fsS -R --retry 3 "$@" -o "$file.part" "$base/$file"; then
+          rm -f "$file.part"
+          echo "Segment $file didn't come." >&2
+          failed=$((failed + 1))
+        elif [ -s "$file.part" ]; then
           mv "$file.part" "$file"
           fetched=$((fetched + 1))
         else
@@ -49,10 +55,30 @@ refresh() {
     done
     lon=$((lon + 5))
   done
-  echo "Segments: $fetched fetched, $kept current, $(du -sh . | cut -f1) in all."
+  echo "Segments: $fetched fetched, $kept current, $failed failed, $(du -sh . | cut -f1) in all."
+  if [ "$failed" -eq 0 ]; then complete=1; fi
 }
 
+# Seconds to wait after the $1th incomplete round in a row: $2, doubling up to $3.
+backoff() {
+  pause=$2 n=1
+  while [ "$n" -lt "$1" ] && [ "$pause" -lt "$3" ]; do
+    pause=$((pause * 2)) n=$((n + 1))
+  done
+  if [ "$pause" -gt "$3" ]; then pause=$3; fi
+  echo "$pause"
+}
+
+failures=0
 while :; do
   refresh
-  sleep $((days * 86400))
+  if [ "$complete" -eq 1 ]; then
+    failures=0
+    sleep $((days * 86400))
+  else
+    failures=$((failures + 1))
+    pause=$(backoff "$failures" 3600 86400)
+    echo "The round wasn't complete; trying again in $((pause / 60)) minutes." >&2
+    sleep "$pause"
+  fi
 done

deploy/compose.maps.yaml

@@ -42,7 +42,8 @@ services:
       MAP_REGION: ${MAP_REGION:--25,35,45,75}
     volumes:
       - ./maps/tiles:/tiles
-    # Out to Protomaps' builds alone; on a network of its own, beside nothing.
+    # Out to Protomaps' builds on `downloads`, an ordinary bridge with a way out to
+    # anywhere, shared with BRouter's segments and the search's downloads.
     networks:
       - downloads
     mem_limit: 512m

deploy/compose.search.yaml

@@ -48,7 +48,8 @@ services:
       PHOTON_REGION: ${PHOTON_REGION:-europe}
     volumes:
       - ./photon/data:/data
-    # Out to GraphHopper's downloads alone; on a network of its own, beside nothing.
+    # Out to GraphHopper's downloads on `downloads`, an ordinary bridge with a way out
+    # to anywhere, shared with BRouter's segments and the map's downloads.
     networks:
       - downloads
     mem_limit: 256m

deploy/compose.train.yaml

@@ -88,7 +88,8 @@ services:
       BROUTER_REGION: ${BROUTER_REGION:--25,35,40,70}
     volumes:
       - ./brouter/segments:/segments4
-    # Out to brouter.de alone; on a network of its own, beside nothing.
+    # Out to brouter.de on `downloads`, an ordinary bridge with a way out to anywhere,
+    # shared with the map's and the search's downloads where their compose files are on.
     networks:
       - downloads
     mem_limit: 64m

deploy/maps/tiles.sh

@@ -3,7 +3,9 @@
 # newest of Protomaps' daily builds of the planet, cut to MAP_REGION (`west,south,
 # east,north` in degrees; by default Europe's, the box BRouter's segments cover) at
 # zooms 0 to 15, into /tiles/europe.pmtiles; again every MAP_REFRESH_DAYS days (30)
-# where a newer build has come. The cut is written into /tiles/next/ and moved over
+# where a newer build has come; after a round where the builds' list or the cut
+# failed, again in an hour, the wait doubling up to a day while rounds keep failing.
+# The cut is written into /tiles/next/ and moved over
 # the old one, so the server never reads half a file; until it is done, the old one
 # serves. Both at once take twice a cut's room.
 set -eu
@@ -12,9 +14,13 @@ builds=https://build-metadata.protomaps.dev/builds.json
 region=${MAP_REGION:--25,35,45,75}
 days=${MAP_REFRESH_DAYS:-30}
 cd /tiles
-mkdir -p next
 
 refresh() {
+  complete=0
+  if ! mkdir -p next; then
+    echo "/tiles/next can't be made; the tiles stay as they are." >&2
+    return 0
+  fi
   build=$(curl -fsS "$builds" | grep -o '"key": *"[0-9]\{8\}\.pmtiles"' |
     grep -o '[0-9]\{8\}\.pmtiles' | sort | tail -n 1) || build=
   if [ -z "$build" ]; then
@@ -23,6 +29,7 @@ refresh() {
   fi
   if [ -e europe.build ] && [ "$(cat europe.build)" = "$build" ]; then
     echo "The tiles are $build's, the newest."
+    complete=1
     return 0
   fi
   rm -f next/europe.pmtiles
@@ -31,13 +38,33 @@ refresh() {
     mv next/europe.pmtiles europe.pmtiles
     echo "$build" >europe.build
     echo "The tiles are $build's now, $(du -h europe.pmtiles | cut -f1)."
+    complete=1
   else
     rm -f next/europe.pmtiles
     echo "The cut of $build failed; the tiles stay as they are." >&2
   fi
 }
 
+# Seconds to wait after the $1th incomplete round in a row: $2, doubling up to $3.
+backoff() {
+  pause=$2 n=1
+  while [ "$n" -lt "$1" ] && [ "$pause" -lt "$3" ]; do
+    pause=$((pause * 2)) n=$((n + 1))
+  done
+  if [ "$pause" -gt "$3" ]; then pause=$3; fi
+  echo "$pause"
+}
+
+failures=0
 while :; do
   refresh
-  sleep $((days * 86400))
+  if [ "$complete" -eq 1 ]; then
+    failures=0
+    sleep $((days * 86400))
+  else
+    failures=$((failures + 1))
+    pause=$(backoff "$failures" 3600 86400)
+    echo "The round wasn't complete; trying again in $((pause / 60)) minutes." >&2
+    sleep "$pause"
+  fi
 done

deploy/photon/data.sh

@@ -5,7 +5,9 @@
 # where none is there or the one there is older than PHOTON_REFRESH_DAYS days (30),
 # unpacked as it comes into /data/next and checked against its MD5 on the way, then
 # marked ready for serve.sh, which puts it in place. Its archive is never kept: only
-# the old database and the new one take room at once.
+# the old database and the new one take room at once. Looked at daily; after a failed
+# download, again a day later, the wait doubling up to a week while downloads keep
+# failing.
 set -eu
 
 base=https://download1.graphhopper.com/public
@@ -21,7 +23,9 @@ fresh() {
 }
 
 fetch() {
+  failed=1
   if [ -e next/ready ] || fresh; then
+    failed=0
     return 0
   fi
   rm -rf next
@@ -41,6 +45,7 @@ fetch() {
     : >next/fetched
     : >next/ready
     echo "Photon's database of $region is ready, $(du -sh next/photon_data | cut -f1)."
+    failed=0
   else
     wait "$sum" 2>/dev/null || true
     rm -rf next
@@ -48,7 +53,26 @@ fetch() {
   fi
 }
 
+# Seconds to wait after the $1th failed download in a row: $2, doubling up to $3.
+backoff() {
+  pause=$2 n=1
+  while [ "$n" -lt "$1" ] && [ "$pause" -lt "$3" ]; do
+    pause=$((pause * 2)) n=$((n + 1))
+  done
+  if [ "$pause" -gt "$3" ]; then pause=$3; fi
+  echo "$pause"
+}
+
+failures=0
 while :; do
   fetch
-  sleep 86400
+  if [ "$failed" -eq 0 ]; then
+    failures=0
+    sleep 86400
+  else
+    failures=$((failures + 1))
+    pause=$(backoff "$failures" 86400 604800)
+    echo "Trying again in $((pause / 3600)) hours." >&2
+    sleep "$pause"
+  fi
 done

deploy/photon/serve.sh

@@ -2,7 +2,9 @@
 # Serves Photon on the database in /data/current (Halkyon Train's ADR 0030, D50), once
 # data.sh has brought one; and where data.sh has made a newer one (/data/next/ready),
 # stops Photon, puts the new one in the old one's place and starts it again: a
-# minute or two without search, once a month. PHOTON_HEAP is Java's heap (6g).
+# minute or two without search, once a month. Looked at every minute, which also
+# starts a Photon that stopped (killed for memory) again. PHOTON_HEAP is Java's
+# heap (6g).
 set -eu
 
 cd /data
@@ -42,7 +44,7 @@ while [ ! -d current/photon_data ] && ! swap; do
 done
 start
 while :; do
-  sleep 3600 &
+  sleep 60 &
   wait $! || true
   if [ -e next/ready ]; then
     stop

deploy/sidecars/README.md

@@ -223,7 +223,8 @@ Steps, in order; ✱ marks a step as root.
    Learn's server are recreated with it (Train's site, and the register's member); the
    instance is away for seconds. Train's server applies its migrations as it starts.
    The segments come in the first hour or so (`docker compose logs brouter-segments`
-   says when); until then BRouter finds no route.
+   says when); until then BRouter finds no route. Where brouter.de doesn't answer or
+   the folder isn't BRouter's, the log says so and it tries again an hour later.
 7. **Check it**: `https://train.halkyon.eu/` answers; in Train, signed in with a Learn
    account, a route plans between two points of the region.
 
@@ -266,7 +267,7 @@ Steps, in order; ✱ marks a step as root.
 4. **Start it** (the operator, as `deploy`): `docker compose up -d --build`. Caddy and
    Train's server are recreated with it. The first cut takes hours
    (`docker compose logs -f map-tiles-refresh` says when it is done); until then Train's
-   maps stay empty.
+   maps stay empty. A failed cut is tried again an hour later.
 5. **Check it**: in Train, signed in, a workout's map is drawn and names the instance as
    its source; `curl -I https://train.halkyon.eu/tiles/europe/0/0/0.mvt` is refused
    with 401. Not yet tried on the server (29 Oct 2026): whether `pmtiles serve` reads a
@@ -309,7 +310,7 @@ Steps, in order; ✱ marks a step as root.
 4. **Start it** (the operator, as `deploy`): `docker compose up -d --build`. Train's
    server is recreated with it. The first database takes an hour or more
    (`docker compose logs -f photon-data` says when it is ready); Photon starts on it
-   within the next hour, and until then Train's planner offers no search.
+   within a minute, and until then Train's planner offers no search.
 5. **Check it**: in Train, signed in, the planner's search finds a town of the region.
    Not yet tried on the server (29 Oct 2026): the database's size unpacked, Photon's
    memory on Europe's, and that 1.3.0 reads GraphHopper's `1.0` build.