Skip to content

Slurm Access to the Cori GPU nodes

The GPU nodes are accessible via Slurm on the Cori login nodes. One can allocate a job on the GPU nodes via the Slurm job allocation flag --constraint, in the same way that one allocates jobs on Haswell or KNL nodes. However, one must load the esslurm module before allocating a job on the GPU nodes, or else the job allocation will fail.

Cori GPU does not charge NERSC-hours

Because Cori GPU is a development resource targeting R&D efforts for Perlmutter readiness, and is not a production resource, NERSC does not charge against project allocations for usage of these nodes.

Submitting jobs

Each node has 8 GPUs, 40 physical cores spread across 2 sockets with 2 hyper-threads per core, and 384 GB DRAM. To access approximately 1/8 of a single node's resources (generally sufficient for single-GPU code development), one can execute

user@cori02> module load esslurm
user@cori02> salloc -C gpu -N 1 -t 60 -c 10 -G 1 -A <account>
salloc: Granted job allocation 12345
salloc: Waiting for resource configuration
salloc: Nodes cgpu02 are ready for job

which will provide the user with 1 GPU, 5 physical cores (10 hyper-threads or "CPUs"), and approximately 48 GB of DRAM. Note that Slurm already allocates memory to your job proportial to the number of CPUs you request for your job. E.g., if you request -c 40 (half of the available CPUs), you will be allocated roughly half of the memory on the node - approximately 192 GB.

The new flag in the above example which is not used elsewhere on Cori is -G (equivalent to --gpus=), which is used to allocate a particular number of GPUs on the node.

Allocate GPUs with -G <N> or --gpus=<N> instead of --gres=gpu:<N>

In older versions of Slurm, allocating GPUs required using the flag --gres=gpu:<N>. However, a recent update to Slurm enabled the use of the GPU allocation flag --gpus=<N> (or -G <N> for short). These flags have similar basic functionality to --gres=gpu:<N>, but are easier to type, and also offer more flexible resource allocation. Is is recommended that scripts replace --gres with -G or --gpus.

You must make sure to specify the Slurm account with is associated with the GPU QOS for your user account. To see which Slurm accounts your user account is associated with and which QOSes are available for each account, use the command:

module load esslurm
sacctmgr show assoc user=$USER -p

which will print to screen any accounts you can submit jobs with and the allowed QOSes for the jobs. If your user account is associated with several job accounts, you'll probably want to use something like sacctmgr show assoc user=$USER -p | grep gpu to search the output.

GPU nodes are 'shared' by default

Slurm's default behavior on the 'normal' compute nodes on Cori and Edison is to reserve each compute node entirely for yourself; every node in your job allocation is exclusively yours. However, on the GPU nodes, the default behavior is the opposite - the default behavior is to share the nodes in your job allocation with other users. If you need to reserve all CPU resources on a node for yourself, you can specify the --exclusive option in your Slurm script invocation.

Although sharing nodes reduces the likelihood that you will need to wait for a node to become available, users of shared nodes may encounter signficant performance variability due to other concurrent activity on the node, particularly if PCI traffic (CPU <-> GPU memory bandwidth and network bandwidth) comprises a significant portion of an application's performance. This is because the GPU nodes do not have enough PCI bandwidth to service all PCI connections at full speed.

Use only what you need

There are only 18 GPU nodes to satisfy the development needs of many NERSC users. If you need all CPUs and GPUs on a given number of GPU nodes for your work, you should use them. But if you only need a single GPU and a single physical core, please be mindful of others and do not reserve the entire node for yourself.

Jupyter access

Cori GPU nodes may also be accessed using NERSC's JupyterHub service, which is documented here. After logging into JuypterHub using Iris credentials, one may start a Jupyter notebook which will have access to a GPU by clicking the "start" button under the column labeled "Shared GPU Node."

JupyterHub sessions may time out at startup

When all of Cori GPU's GPUs are allocated to other jobs, Jupyter sessions will be unable to start, as they always request a GPU when the notebook is allocted to the cluster. The "Spawning server" window will not progress, and after a few minutes will time out. This behavior is expected - one must simply wait until GPUs are available on the cluster before starting a Jupyter notebook.

Job constraints

Cori GPU prioritizes interactive code development during business hours in Pacific Time (UTC-7), and allows large and/or long running jobs to run on nights and weekends. Cori GPU also prioritizes jobs submitted by NESAP application teams over non-NESAP teams.

Cori GPU imposes a few job constraints, described below, to prioritize the most important workloads for this cluster. Users should be aware that jobs of nearly any size and length can be submitted to Cori GPU at any time, and that the constraints described below affect only the time jobs are elibile to start, not the time that jobs can be submitted to the queue.

Job constraints are as follows:

  1. Jobs running between 8:00 AM Pacific Time (3:00 PM UTC) and 8:00 PM Pacific Time (3:00 AM UTC) from Monday through Friday are limited to 4-hours of run time.
  2. Jobs running before 8:00 AM Pacific Time (3:00 PM UTC) or after 8:00 PM Pacific Time (3:00 AM UTC), or on weekends, can run until 8:00 AM Pacific Time on the next weekday.
  3. Members of the NESAP ERCAP project (m1759) may add an additional flag -q special to their batch and interactive jobs to be placed in a higher-priority queue.
  4. Interactive jobs are now limited to 2 GPUs. Jobs requiring more than 2 GPUs can be submitted via sbatch instead of salloc.

Slurm commands with esslurm

While the esslurm module is loaded, commands such as sinfo, squeue, sbatch, etc. will not show information or submit jobs to 'normal' Cori compute nodes. To query the 'normal' compute nodes, unload the esslurm module with module unload esslurm and then enter your desired Slurm commands.

The (ReqNodeNotAvail, May be reserved for other job) reason code

Cori GPU uses a handful of specially configured Slurm reservations to enforce the job constraints described above. In some cases, when a job is being prevented from starting due to one of those constraints (e.g., a user submits a job requesting 12 hours of runtime at 9:00 AM Pacific Time), the Slurm reason code provided in the output of squeue will be (ReqNodeNotAvail, May be reserved for other job). This reason code does not mean that anything is wrong with the job, only that it must wait to be eligible to start until at least one of the above constraints no longer applies (e.g., the current time has reached 8:00 PM Pacific Time and the 4-hour job runtime restriction is lifted).