Slurm Access to the Cori GPU nodes¶
The GPU nodes are accessible via Slurm on the Cori login nodes. They are
exposed as a hardware 'constraint', in the same way that the Haswell and KNL
compute nodes are. They require that you use the
esslurm module before you
run your Slurm scripts, or else your jobs will fail.
To access a GPU node and reserve 5 physical CPU cores, 1 GPU, and approximately 30 GB of RAM for yourself:
user@cori02> module load esslurm user@cori02> salloc -C gpu -N 1 -t 30 -c 10 --gres=gpu:1 -A m1759 salloc: Granted job allocation 12345 salloc: Waiting for resource configuration salloc: Nodes cgpu02 are ready for job user@cgpu02:~>
--mem is broken following July 2019 maintenance
Following the July 2019 maintenance, the Slurm
--mem flag no longer works
properly. If you use this flag, you will be unable to interact with the
GPUs allocated to your job. Specifically, any
srun command which attempts
to interact with GPUs will hang forever until canceled.
You will also see this warning message following your job allocation:
srun: error: --mem and --mem-per-cpu are mutually exclusive.
The workaround is to remove the
--mem flag Note that Slurm already
allocates memory to your job proportial to the number of CPUs you request
for your job. E.g., if you request
-c 40 (half of the available CPUs),
you will be allocated roughly half of the memory on the node - around 192
Like the Cori Haswell nodes, the CPUs on these nodes have two hyper-threads per
-c 10 reserves 5 physical cores per node for your job.
The new flag which is not used elsewhere on Cori is
--gres, which is used to
reserve a particular number of GPUs on the node.
Time limits for jobs are:
- 4 hours for interactive (
- 4 hours for batch (
GPU nodes are 'shared' by default
Slurm's default behavior on the 'normal' compute nodes on Cori and Edison
is to reserve each compute node entirely for yourself; every node in your
job allocation is exclusively yours. However, on the GPU nodes, the default
behavior is the opposite - the default behavior is to share the nodes in
your job allocation with other users. If you need to reserve all CPU
resources on a node for yourself, you can specify the
in your Slurm script invocation.
Use only what you need
There are only 18 GPU nodes to satisfy the development needs of many NERSC users. If you need all CPUs and GPUs on a given number of GPU nodes for your work, you should use them. But if you only need a single GPU and a single physical core, please be mindful of others and do not reserve the entire node for yourself.
Each user is enabled on the GPU nodes via the m1759 repository. If that
repository is not your default for MPP charging, you must specify that repo
-A m1759 when you access the GPU nodes, or else your job
submission will fail.
esslurm module is loaded, commands such as
sbatch, etc. will not show information or submit jobs to 'normal' Cori
compute nodes. To query the 'normal' compute nodes, unload the
module unload esslurm and then enter your desired Slurm