The current installation of Grid Engine supports special programs based on OpenMPI

The OpenMPi package must be compiled with Grid Engine support:

$ ompi_info | grep grid
                 MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6) 

OpenMPI programs can run in the special runtime environment by using more cores than one node has to offer. The Grid Engine master is responsible for assigning free nodes and cores to OpenMPI and starting the OpenMPI runtime environment (orte) on these assigned nodes. After a successful start of the runtime environment, the job starts running. When the job is completed, the Grid Engine shuts down the assigned runtime environment.

Different OpenMPI jobs use different runtime environment processes, so that the jobs do not interfere with each other!!!

OpenMPI is currently used by the Imagic and Xmipp software and therefore the support is only enabled in special queues.


To use OpenMPI your environment must be updated by calling (or putting the following line into the .profile settings file):

 source /sw/lenny/etc/openmpi.bash

or if you use environment modules:

 module load openmpi

Dynamic cores request

Since OpenMPI programs tend to use lots of cores, the dynamic assignment of resources should be used:

 #$ -pe orte 20-800 
#$ -l hostname=compute-3-*

This means the program in the script needs minimum 20 cores and maximum 800 cores to run. The job is queued and will be sent to a host if the minimum requirement (20 cores) is available. Based on the cluster load the resources can be more than 20 cores. In this case the gridengine sets a special environment variable $NSLOTS for the job's execution environment. The value in $NSLOTS is the actual reserved amount of cores (e.g. 350 cores) for this job.

Since the cores are available on various different nodes, you should choose nodes with the same hardware equipment (memory, processor speed, etc.) for efficient processing. The hostname=node(s) flag ensures this.

Below is a practical example on how to use this:

 #$ -S /bin/sh
#$ -q nonofficehours.q
#$ -P test
#$ -cwd
#$ -pe orte 20-800
#$ -l hostname=compute-3-*
echo "got $NSLOTS slots"
mpiexec -n $NSLOTS $HOME/bin/mpihello