Introduction to Slurm

Introduction to Slurm#

Slurm is an open-source workload manager used by many of the world’s most powerful supercomputers and computer clusters, including Vulcan. At its core, Slurm ensures that multiple users can share a finite amount of computing power efficiently and fairly in the HPC environment.

As the name suggests, Slurm is only available on Linux; therefore, all clusters managed by Slurm are Linux-based. The base OS of Vulcan is Ubuntu.

Core Functions of Slurm#

The key roles of Slurm covered in this document are:

  1. Resource Allocation: It provides users with access to resources on compute nodes for a specific duration to perform work.

  2. Job Execution: It starts, executes, and monitors work on the allocated nodes.

  3. Queue Management: It arbitrates contention for resources by managing a queue of pending work.

Typical Slurm Workflow#

Cluster workflows differ from local development. On Slurm clusters, you submit jobs to a queue and wait for execution based on resource availability.

The typical workflow is as follows:

  1. You submit jobs with sbatch. A job is a standard shell script containing #SBATCH directives. You use these directives to request job-level resources, execution time, and output filenames.

  2. Slurm places your jobs in a queue, where they enter the Pending (PD) state. Slurm calculates priority based on multiple factors, such as “fair share” (recent system usage) and job age.

  3. Once your job reaches the top of the queue and the requested resources (CPUs, RAM, GPUs) become available, Slurm reserves cluster resources and starts a task on one of the reserved nodes to execute the submitted shell script. Slurm relies on Linux kernel functions (cgroup) to enforce resource constraints on tasks.

  4. Inside the job script, you should use srun to launch parallel tasks. You can specify task-level resource allocation with srun.

    Note

    The sum of the task-level resources on a node must NOT exceed the total reserved resources for that node. Otherwise, your job may be terminated prematurely.

  5. When your requested execution time expires, Slurm does not immediately kill your job. Instead, it first sends a SIGTERM signal to the job script to provide advance notice during a grace period. This allows your application to perform a graceful cleanup, such as saving a checkpoint. If any process remains after the grace period, Slurm sends a SIGKILL signal to forcefully terminate it. The SIGKILL signal cannot be intercepted or ignored.

Now that you understand the workflow, let’s look at the basic commands you’ll use to submit and monitor your work.

Contents