Shifter

Shifter with CUDA¶

On Cori-GPU, Shifter + CUDA should just work. The default Shifter module copies in the CUDA drivers into the image and places these libraries in your LD_LIBRARY_PATH. Thus, no additional configuration should be required to execute a container which utilizes the GPU. However, the following documentation may be useful in the event that a newer version of CUDA is released and newer drivers are required than the system installed version.

Manual Insertion of CUDA Drivers¶

First, one should add the following to ENTRYPOINT in the Shifter container:

export PATH=/opt/shifter/bin:${PATH}
export LD_LIBRARY_PATH=/opt/shifter/lib:${LD_LIBRARY_PATH}

Next, one should load the cuda/shifter module; this will define the SHIFTER_CUDA_ROOT environment variable and point it to the version of the CUDA SDK installation which works in Shifter images.

cuda modules and Shifter images

Note that the normal cuda modules will not work inside Shifter containers.

Then one should invoke shifter from the job script as follows:

srun -n <num_task> -c <num_cpu> \
    shifter \
    --image=<your_image> \
    --entrypoint \
    --volume=${SHIFTER_CUDA_ROOT}:/opt/shifter:ro \
    ./your_gpu_code.ex args

A complete is example is show below; this example is also provided in $SHIFTER_CUDA_ROOT/example.

#!/bin/bash -e

#SBATCH -A nstaff
#SBATCH -C gpu
#SBATCH --gpus=1
#SBATCH -t 00:10:00
#SBATCH --job-name=nvidia-shifter

# provides SHIFTER_CUDA_ROOT
module load cuda/shifter

#
# jrmadsen/tomopy:shifter container has the following in entrypoint:
#
#    export PATH=/opt/shifter/bin:${PATH}
#    export LD_LIBRARY_PATH=/opt/shifter/lib:${LD_LIBRARY_PATH}
#

export TOMOPY_NUM_THREADS=32
export NUMEXPR_MAX_THREADS=80
export OMP_NUM_THREADS=1

srun -n 1 -c 1 \
    shifter \
    --image=jrmadsen/tomopy:shifter \
    --entrypoint \
    --volume=${SHIFTER_CUDA_ROOT}:/opt/shifter:ro \
    python ${SHIFTER_CUDA_ROOT}/example/phantom.py -a mlem -i 100 -n 1 -p shepp2d -f png -s 256