Shifter

Shifter with CUDA¶

Shifter works in a slightly different way on the Cori GPU nodes than it does on the Haswell or KNL nodes. First, one should add the following to ENTRYPOINT in the Shifter container:

export PATH=/opt/shifter/bin:${PATH}
export LD_LIBRARY_PATH=/opt/shifter/lib:${LD_LIBRARY_PATH}

Next, one should load the cuda/shifter module; this will define the SHIFTER_CUDA_ROOT environment variable and point it to the version of the CUDA SDK installation which works in Shifter images.

cuda modules and Shifter images

Note that the normal cuda modules will not work inside Shifter containers.

Then one should invoke shifter from the job script as follows:

srun -n <num_task> -c <num_cpu> \
    shifter \
    --image=<your_image> \
    --entrypoint \
    --volume=${SHIFTER_CUDA_ROOT}:/opt/shifter:ro \
    ./your_gpu_code.ex args

A complete is example is show below; this example is also provided in $SHIFTER_CUDA_ROOT/example.

#!/bin/bash -e

#SBATCH -A nstaff
#SBATCH -N 1
#SBATCH -C gpu
#SBATCH --gres=gpu:1
#SBATCH -t 00:10:00
#SBATCH --job-name=nvidia-shifter

# provides SHIFTER_CUDA_ROOT
module load cuda/shifter

#
# jrmadsen/tomopy:shifter container has the following in entrypoint:
#
#    export PATH=/opt/shifter/bin:${PATH}
#    export LD_LIBRARY_PATH=/opt/shifter/lib:${LD_LIBRARY_PATH}
#

export TOMOPY_NUM_THREADS=32
export NUMEXPR_MAX_THREADS=80
export OMP_NUM_THREADS=1

srun -n 1 -c 1 \
    shifter \
    --image=jrmadsen/tomopy:shifter \
    --entrypoint \
    --volume=${SHIFTER_CUDA_ROOT}:/opt/shifter:ro \
    python ${SHIFTER_CUDA_ROOT}/example/phantom.py -a mlem -i 100 -n 1 -p shepp2d -f png -s 256