Error Handling Tips

Detecting uninitialized variables

Bash gives you a simple mechanism to check if all your environment variables are initialized. 

E.g. you have the following sniplet in a script: 

blastall -p blastp -a $NSLOTS -i n179.fa -d nr -e 0.01 -v 500 -b 500

If you would call it like:

bash -u ./test-hostcores.sh

you get

./test-hostcores.sh: line 13: NSLOTS: unbound variable

With -u the scripts is checked before the execution of the statements in the script. For your job this means it only starts running if all variables are initialized.

To use it on the cluster you can add the following statement in your script:

#$ -S /bin/sh
set -u
module load ncbi-blast
blastall -p blastp -a $NSLOTS -i n179.fa -d nr -e 0.01 -v 500 -b 500

and you call it like:

qsub test-hostcores.sh

the job will run in the cluster if you did not forget to use (in this case) something like

#$ -pe hostcores 4-8

otherwise the job terminates immediately and you can see the error
message as above in your error file.

Catching and handling errors after program failures

Bash provides a 'trap' command to handle signals from the operating system, in our case ERRor and TERMinated. Normally if your job fails, the script is immediately terminated (this means the rest of the script after the failed statement is not executed).

Jobtemplate:

#$ -S /bin/sh
#$ -q nonofficehours.q
#$ -P test
#$ -cwd
set -u
MYCMD="not initialized"

# print host
hostname
# print date and time
date

# --- start own code ---

# initialize error handling
set -e
MYTMPDIR=/clustertmp/testtrap-$JOB_ID
trap 'echo -e "failed command:\n$MYCMD\ncleaning up my failed process" >&2; mycleanup; exit $?' ERR TERM

# local cleanup stuff
function mycleanup {
  rm -rf $MYTMPDIR
}

# command execution
MYCMD="mkdir -p $MYTMPDIR"
$MYCMD

MYCMD="sleep 120"
$MYCMD

# normal cleanup
mycleanup

# --- end own code ---

# print date and time again
date