3. SLURM Job Examples

3.1. MPI job

#!/bin/bash
#SBATCH -J hellompi
#SBATCH -o output_%j.txt
#SBATCH -e errors_%j.txt
#SBATCH -t 01:30:00
# request 32 MPI tasks
#SBATCH -n 32
#SBATCH -p allgroups
#SBATCH --mem 640G

cd $SLURM_SUBMIT_DIR
spack load intel-parallel-studio@professional.2019.4

srun ./mphello

Note

spack load ... initializes the Intel MPI environment and is equivalent to module load intel-parallel-studio-professional.2019.4-gcc-8.2.1-fnvratt

3.2. OpenMP job

#!/bin/bash
#SBATCH -J helloopenmp
#SBATCH -o output_%j.txt
#SBATCH -e errors_%j.txt
#SBATCH -t 01:30:00
#SBATCH -n 1
# notice we're using the '-c' option
# to request for OMP threads
#SBATCH -c 32
#SBATCH -p allgroups
#SBATCH –mem 640G

cd $SLURM_SUBMIT_DIR
# set this to what you asked with '-c'
export OMP_NUM_THREADS=32

srun ./omphello

Note

• OMP_NUM_THREADS must be set to the same number as the -c parameter

• set -n to 1 if the program uses OpenMP only. If using both MPI and OpenMP set -n to the number of MPI tasks. The total number of slots requested, in any case, will be n*c

3.3. GPU Job

#!/bin/bash
#SBATCH -J helloGPU
#SBATCH -o output_%j.txt
#SBATCH -e errors_%j.txt
#SBATCH -t 01:30:00
#SBATCH -n 1
#SBATCH -p allgroups
#SBATCH –mem 640G
# requesting 1 GPU; set --gres=gpu:2 to use both GPUs
#SBATCH --gres=gpu:1

cd $SLURM_SUBMIT_DIR

srun ./GPUhello

Note

CAPRI is equipped with 2 GPUs: request both only if your program is capable of using more than one GPU at a time.

Important

DO NOT request GPU if you don’t use them!

3.4. Singularity Job

#!/bin/bash
#SBATCH --job-name=mysingularity
#SBATCH --error=opencv.%j.err
#SBATCH --output=opencv.%j.out
#SBATCH --partition=allgroups
#SBATCH --ntasks=1
#SBATCH --mem=1G
#SBATCH --time=00:05:00

cd $SLURM_SUBMIT_DIR

srun singularity exec ./mysingularity.sif python script.py

3.5. Singularity job using GPU

#!/bin/bash
#SBATCH -J SingGPU
#SBATCH -o output_%j.txt
#SBATCH -e errors_%j.txt
#SBATCH -t 01:30:00
#SBATCH -n 1
#SBATCH -p allgroups
#SBATCH –mem 640G
# requesting 1 GPU; set --gres=gpu:2 to use both GPUs
#SBATCH --gres=gpu:1

cd $SLURM_SUBMIT_DIR

srun singularity exec --nv ./tensorflow.sif python script.py

Important

You must request (at least) one GPU and you must pass the -\-nv flag to singularity