Understanding the Basics
What is Slurm and Why Do You Need It?
Slurm is a free and open-source system that acts like a smart traffic cop for computer clusters. When many users want to run heavy artificial intelligence (AI) programs at the same time, they cannot all use the computers at once without causing crashes. Slurm solves this problem by creating an organized waiting line, or queue, for all these tasks.
Instead of people fighting over who gets to use the expensive GPUs, users simply tell Slurm what they need. They submit their programs and ask for specific resources, such as two GPUs and a certain amount of memory. Slurm looks at all the available computers, finds the right spot, and automatically runs the job as soon as the requested resources become free.
For AI clusters, Slurm is especially powerful because it keeps everything fair and safely isolated. If one person's AI model crashes or tries to use too much memory, it will not break the system for anyone else. Slurm strictly manages the GPUs so that researchers can focus on building AI instead of worrying about server management.
Major Prerequisites
- Operating System: All machines should run a Debian-based Linux distribution (like Ubuntu 20.04 or 22.04), as this guide relies on
.debpackages andaptcommands. - Internet Access: Your controller node needs an active internet connection to download the Slurm source code and install required software packages.
- NVIDIA Drivers: Your compute nodes must already have NVIDIA drivers installed so Slurm can detect the GPUs.
- Network Time: Ensure
systemd-timesyncdorchronyis actively syncing the clocks across all nodes, as time drift causes authentication failures. - Hostnames: Ensure your controller node (the main manager) and compute nodes (the workers) can resolve each other via DNS or
/etc/hosts. - Root Access: You need
sudoprivileges on all machines to install packages and configure system files.
Step-by-Step Installation
1. Synchronize Users and Groups
Run on ALL nodes (Controller and Compute). Slurm and MUNGE (the authentication daemon) require identical user IDs and group IDs across every machine in the cluster.
# Create a dedicated group and user for MUNGE (no home directory)
sudo groupadd -g 1050 munge
sudo useradd -M -c "MUNGE Uid 'N' Gid Emporium" -d /nonexistent -u 1050 -g munge -s /usr/sbin/nologin munge
# Create a dedicated group and user for Slurm
sudo groupadd -g 1100 slurm
sudo useradd -m -c "Slurm Workload Manager" -d /var/lib/slurm -u 1100 -g slurm -s /bin/bash slurm2. Install and Configure MUNGE Authentication
Run on ALL nodes. MUNGE securely handles communication between the controller and the compute nodes.
# 1. Install MUNGE on ALL nodes
sudo apt-get update
sudo apt-get install -y munge libmunge-dev
# 2. ON THE CONTROLLER ONLY: Generate a secure MUNGE key
sudo /usr/sbin/mungekey -c
# 3. Securely copy the key to ALL COMPUTE NODES
# (e.g., via scp /etc/munge/munge.key user@compute-node:/tmp/)
# 4. ON THE COMPUTE NODES: Move the key and set strict permissions
sudo mv /tmp/munge.key /etc/munge/munge.key
sudo chown munge:munge /etc/munge/munge.key
sudo chmod 400 /etc/munge/munge.key
# 5. Enable and start the MUNGE daemon on ALL nodes
sudo systemctl enable --now munged3. Install Build Dependencies
Run on the Controller Node. Instead of installing from unofficial repositories, we will compile the official Debian packages directly from SchedMD's source code.
# Install core build tools
sudo apt-get install -y build-essential fakeroot devscripts equivs
# Install libraries needed for cgroups, task affinity, and generic resources
sudo apt-get install -y libhwloc-dev libjson-c-dev libpam0g-dev libdbus-1-dev4. Download and Build Slurm
Run on the Controller Node. Download the latest stable release (23.11.x or newer) and build the installation files.
# Download and extract the source tarball
wget https://download.schedmd.com/slurm/slurm-23.11.10.tar.bz2
tar -xaf slurm-23.11.10.tar.bz2
cd slurm-23.11.10
# Automatically resolve and install any missing build dependencies
sudo mk-build-deps -i debian/control
# Compile the Debian packages (this takes a few minutes)
debuild -b -uc -us
cd ..5. Install the Compiled Packages
Use apt-get to install these local packages so it automatically manages any external dependencies.
On the Controller Node:
sudo apt-get install -y ./slurm-smd_*.deb ./slurm-smd-slurmctld_*.deb ./slurm-smd-client_*.debOn the Compute Node(s):
Copy the generated .deb files from the controller to your compute nodes, then install them:
sudo apt-get install -y ./slurm-smd_*.deb ./slurm-smd-slurmd_*.deb ./slurm-smd-client_*.deb6. Create the AI-Optimized Configuration
Run on ALL nodes. First, create the necessary system directories with the correct permissions on all nodes:
sudo mkdir -p /etc/slurm /var/spool/slurmctld /var/spool/slurmd /var/log/slurm
sudo chown slurm:slurm /var/spool/slurmctld /var/spool/slurmd /var/log/slurmCreate /etc/slurm/slurm.conf (This exact file must be copied to all nodes):
ClusterName=ai_cluster
SlurmctldHost=YOUR_CONTROLLER_HOSTNAME # Replace with actual hostname
SlurmUser=slurm
SlurmctldPort=6817
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/var/spool/slurmctld
SlurmdSpoolDir=/var/spool/slurmd
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdLogFile=/var/log/slurm/slurmd.log
# Cgroups and Process Tracking (Crucial for isolating GPU jobs)
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
# Enable GPU Support
GresTypes=gpu
# Node Definitions (Replace with your compute node's hostname and specs)
# Check CPUs/Memory via: lscpu and free -m
NodeName=YOUR_COMPUTE_HOSTNAME CPUs=32 RealMemory=64000 Gres=gpu:4 State=UNKNOWN
# Partition Definitions
PartitionName=ai_nodes Nodes=ALL Default=YES MaxTime=INFINITE State=UP7. Configure GPU Auto-Detection and Cgroups
Run on Compute Nodes. To prevent users from accidentally using GPUs assigned to other jobs, configure cgroups and enable automatic GPU detection.
Create /etc/slurm/gres.conf:
# Automatically detects NVIDIA GPUs, memory, and topology using NVML
AutoDetect=nvmlCreate /etc/slurm/cgroup.conf:
# Constrains jobs strictly to their requested CPUs, RAM, and GPUs
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes8. Start Daemons and Verify the Cluster
On the Controller Node:
sudo systemctl enable --now slurmctldOn the Compute Node(s):
sudo systemctl enable --now slurmdTesting the AI Cluster:
Run the following from your controller node to request a GPU and verify it is successfully isolated:
# Check cluster state (should say 'idle')
sinfo
# Request 1 GPU and run nvidia-smi
srun --gres=gpu:1 nvidia-smiFinal Thoughts
You have now successfully transformed a group of individual computers into a powerful, organized AI research cluster. Here is a quick recap of what you have installed and learned:
- MUNGE Authentication: Set up secure, synchronized communication between all machines in your cluster.
- Slurm Workload Manager: Compiled and installed the official core system packages directly from the source code.
- Cluster Configuration: Configured the central controller to manage the queue and the worker nodes to execute the tasks.
- GPU Isolation: Enabled cgroups and automatic GPU detection to ensure users can safely share hardware without their programs crashing into each other.
With this robust environment running smoothly, your team can now easily schedule and run intensive AI workloads fairly and efficiently.
