SGE template: Signal SIGINT by default to allow R to catch it gracefully

[HenrikBengtsson]

When a job is approaching it's maximum run-time limit, and --notify has been specified, SGE sends a SIGUSR2 60 seconds before sending a SIGKILL (abrupt; not capturable).

R code cannot handle SIGUSR2 signals, only SIGINT. If SGE would signal SIGINT instead, we could capture it internally as an interrupt condition, and using tryCatch(..., interrupt = ...), on.exit(), and likes to gracefully exit, e.g. close connections, checkpoint intermediate results, etc.

Idea

Update the default SGE template to signal SIGINT. Unlike Slurm, there is no user-facing option to change the type of signal in SGE. But we could use Bash traps to capture the SIGUSR2 and re-signal it as SIGINT. Something like:

#! /bin/bash
...
## Request SGE to send warning signals (SIGUSR2) before run-time limit termination
#$ -notify
...

# Function to catch SGE run-time signals and re-signal as SIGINT
forward_sigint() {
  echo "Caught SGE run-time signal. Re-signal as SIGINT to R..."
  kill -2 "${R_PID}" 2>/dev/null || true
  wait "${R_PID}"
  res=$?
  echo " - exit code (from signal handler): ${res}"
  exit "${res}"
}

# Trap SGE run-time limit signals
trap forward_sigint SIGUSR2

<%= rscript_call %> -e 'batchtools::doJobCollection("<%= uri %>")' &
R_PID=$!
wait "${R_PID}"
res=$?

# Clear the trap on normal exit
trap - SIGUSR2

[HenrikBengtsson]

SGE supports both hard (h_rt) and soft (s_rt) run-time limits. The above SIGUSR2 is signaled for the hard limit (if -notify, is set). The soft limit triggers a SIGUSR1 signal, so we should handle those too.