Skip to content

Slurm Access to the Cori GPU nodes

The GPU nodes are accessible via Slurm on the Cori login nodes. They are exposed as a hardware 'constraint', in the same way that the Haswell and KNL compute nodes are. They require that you use the esslurm module before you run your Slurm scripts, or else your jobs will fail.

To access a GPU node and reserve 5 physical CPU cores, 1 GPU, and approximately 30 GB of RAM for yourself:

user@cori02> module load esslurm
user@cori02> salloc -C gpu -N 1 -t 30 -c 10 --gres=gpu:1 -A m1759
salloc: Granted job allocation 12345
salloc: Waiting for resource configuration
salloc: Nodes cgpu02 are ready for job
user@cgpu02:~>

--mem is broken following July 2019 maintenance

Following the July 2019 maintenance, the Slurm --mem flag no longer works properly. If you use this flag, you will be unable to interact with the GPUs allocated to your job. Specifically, any srun command which attempts to interact with GPUs will hang forever until canceled.

You will also see this warning message following your job allocation:

srun: error: --mem and --mem-per-cpu are mutually exclusive.

The workaround is to remove the --mem flag Note that Slurm already allocates memory to your job proportial to the number of CPUs you request for your job. E.g., if you request -c 40 (half of the available CPUs), you will be allocated roughly half of the memory on the node - around 192 GB.

Like the Cori Haswell nodes, the CPUs on these nodes have two hyper-threads per core. So -c 10 reserves 5 physical cores per node for your job.

The new flag which is not used elsewhere on Cori is --gres, which is used to reserve a particular number of GPUs on the node.

Time limits for jobs are:

  • 4 hours for interactive (salloc) jobs
  • 4 hours for batch (sbatch) jobs

GPU nodes are 'shared' by default

Slurm's default behavior on the 'normal' compute nodes on Cori and Edison is to reserve each compute node entirely for yourself; every node in your job allocation is exclusively yours. However, on the GPU nodes, the default behavior is the opposite - the default behavior is to share the nodes in your job allocation with other users. If you need to reserve all CPU resources on a node for yourself, you can specify the --exclusive option in your Slurm script invocation.

Use only what you need

There are only 18 GPU nodes to satisfy the development needs of many NERSC users. If you need all CPUs and GPUs on a given number of GPU nodes for your work, you should use them. But if you only need a single GPU and a single physical core, please be mindful of others and do not reserve the entire node for yourself.

Each user is enabled on the GPU nodes via the m1759 repository. If that repository is not your default for MPP charging, you must specify that repo with -A m1759 when you access the GPU nodes, or else your job submission will fail.

Slurm commands

While the esslurm module is loaded, commands such as sinfo, squeue, sbatch, etc. will not show information or submit jobs to 'normal' Cori compute nodes. To query the 'normal' compute nodes, unload the esslurm module with module unload esslurm and then enter your desired Slurm commands.