ci: a job first stops what earlier jobs left running
On 22 Oct a cancelled job left its analysis server running in the runner, 4 GB, until someone noticed. forgejo-runner v13.1.0 starts every host step as a process group and sends the group SIGKILL on a cancel or at timeout-minutes (act/common/cmdgroup.go), and after that no later step runs, `if: always()` ones neither (they get the cancelled context; only actions' post steps get a fresh one), so a cleanup at the job's end can't be relied on. The check job's second step, after the sandbox's proof, stops every process of a job's working folder (`$HOME/work/`, also one removed since) that is neither the runner nor the step itself, TERM then KILL after 5 s, and names each in the log. The runner takes one job at a time (capacity: 1), so nothing of a running job is touched. No change on the server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tizian 「ティツィアーン」 <raisondetredev@pm.me>
This commit is contained in:
parent
0fafa59f2c
commit
c4ad437fc1
1 changed files with 39 additions and 1 deletions
|
|
@ -55,6 +55,44 @@ jobs:
|
|||
fi
|
||||
echo "This job can neither see nor reach production."
|
||||
|
||||
# A job cancelled, or stopped at timeout-minutes, is killed with its step's process
|
||||
# group (forgejo-runner v13.1.0 starts every host step as a group and sends it
|
||||
# SIGKILL), and no later step runs, `if: always()` ones neither. Yet on 22 Oct a
|
||||
# cancelled job left its analysis server running, 4 GB, until someone noticed. The
|
||||
# runner takes one job at a time (capacity: 1), so a process here in a job's working
|
||||
# folder that is neither the runner nor this step is such a leftover: it is stopped
|
||||
# before anything else starts, and the log names it.
|
||||
- name: Leftovers
|
||||
run: |
|
||||
own_group=$(ps -o pgid= -p $$ | tr -d ' ')
|
||||
# This step and the runner above it.
|
||||
keep=' '
|
||||
pid=$$
|
||||
while [ -n "$pid" ] && [ "$pid" -gt 1 ]; do
|
||||
keep="$keep$pid "
|
||||
pid=$(ps -o ppid= -p "$pid" | tr -d ' ')
|
||||
done
|
||||
left=$(ps -e -o pid=,pgid= | while read -r pid pgid; do
|
||||
case "$keep" in *" $pid "*) continue ;; esac
|
||||
[ "$pgid" != "$own_group" ] || continue
|
||||
# Only what a job started: its working folder, or one since removed.
|
||||
case "$(readlink "/proc/$pid/cwd" 2>/dev/null)" in
|
||||
"$HOME/work/"*) echo "$pid" ;;
|
||||
esac
|
||||
done)
|
||||
if [ -z "$left" ]; then
|
||||
echo 'Nothing left behind by earlier jobs.'
|
||||
exit 0
|
||||
fi
|
||||
echo 'Left behind by earlier jobs, stopped now:'
|
||||
# shellcheck disable=SC2086 # one process ID a word
|
||||
ps -o pid=,etime=,rss=,args= -p "$(echo $left | tr ' ' ,)" | cut -c1-200
|
||||
# shellcheck disable=SC2086
|
||||
kill -TERM $left 2>/dev/null || true
|
||||
sleep 5
|
||||
# shellcheck disable=SC2086
|
||||
kill -KILL $left 2>/dev/null || true
|
||||
|
||||
# A commit that passed this job before isn't checked again: a fast-forward merge
|
||||
# brings the commit a branch's push checked to main unchanged, and only its deploy
|
||||
# is new. The marks live in the runner's home; only pushes run this workflow, so
|
||||
|
|
|
|||
Loading…
Reference in a new issue