Frequently Asked Questions
For Management
- Is Slurm really free?
- Why should I use Slurm or other free software?
- Why should I pay for free software?
- What does "Slurm" stand for?
For Researchers
For Users
Designing Jobs
- How can I run multiple jobs from within a single script?
- How can I run a job within an existing job allocation?
- Slurm documentation refers to CPUs, cores and threads. What exactly is considered a CPU?
- How do I run specific tasks on certain nodes in my allocation?
- How can I get the task ID in the output or error file name for a batch job?
- How does Slurm establish the environment for my job?
- Can the make command utilize the resources allocated to a Slurm job?
- How can I run an Ansys program with Slurm?
Submitting Jobs
- Why are my srun options ignored?
- Why does the srun --overcommit option not permit multiple jobs to run on nodes?
- Why is the srun --u/--unbuffered option adding a carriage return to my output?
- What is the difference between the sbatch and srun commands?
- Can tasks be launched with a remote (pseudo) terminal?
- How can I get shell prompts in interactive mode?
- Can Slurm export an X11 display on an allocated compute node?
Scheduling
Killed Jobs
- Why is my job killed prematurely?
- Why is my batch job that launches no job steps being killed?
- What does "srun: Force Terminated job" indicate?
- What does this mean: "srun: First task exited 30s ago" followed by "srun Job Failed"?
Managing Jobs
- How can I temporarily prevent a job from running (e.g. place it into a hold state)?
- Can I change my job's size after it has started running?
- Why does squeue (and "scontrol show jobid") sometimes not display a job's estimated start time?
- Can squeue output be color coded?
- Why is my job/node in a COMPLETING state?
- How can a job in a complete or failed state be requeued?
- Why is sview not coloring/highlighting nodes properly?
- Why is my MPICH2 or MVAPICH2 job not running with Slurm? Why does the DAKOTA program not run with Slurm?
Resource Limits
- Why are my resource limits not propagated?
- Why are jobs not getting the appropriate memory limit?
- Why is my MPI job failing due to the locked memory (memlock) limit being too low?
For Administrators
Test Environments
- Can multiple Slurm systems be run in parallel for testing purposes?
- Can Slurm emulate a larger cluster?
- Can Slurm emulate nodes with more resources than physically exist on the node?
Build and Install
- Why aren't pam_slurm.so, auth_none.so, or other components in a Slurm RPM?
- How can I build Slurm with debugging symbols?
- How can a patch file be generated from a Slurm commit in GitHub?
- How can I apply a patch to my Slurm source?
- Why am I being offered an automatic update for Slurm?
Cluster Management
- How should I relocate the primary or backup controller?
- Do I need to maintain synchronized clocks on the cluster?
- How can I stop Slurm from scheduling jobs?
- How can I dry up the workload for a maintenance period?
- What should I be aware of when upgrading Slurm?
- Is there anything exceptional to be aware of when upgrading my database server?
- When adding a new cluster, how can the Slurm cluster configuration be copied from an existing cluster to the new cluster?
- How could some jobs submitted immediately before the slurmctld daemon crashed be lost?
- Is resource limit propagation useful on a homogeneous cluster?
- Why are the resource limits set in the database not being enforced?
- Can Slurm be configured to manage licenses?
- How easy is it to switch from PBS or Torque to Slurm?
- What might account for MPI performance being below the expected level?
- How do I safely remove partitions?
- How can a routing queue be configured?
- What happened to the "none" plugins?
Accounting Database
- Why should I use the slurmdbd instead of the regular database plugins?
- How critical is configuring high availability for my database?
- How can I use double quotes in MySQL queries?
Compute Nodes (slurmd)
- Why is a node shown in state DOWN when the node has registered for service?
- What happens when a node crashes?
- How can I control the execution of multiple jobs per node?
- Why are jobs allocated nodes and then unable to initiate programs on some nodes?
- Why does slurmctld log that some nodes are not responding even if they are not in any partition?
- How can I easily preserve drained node information between major Slurm updates?
- Does anyone have an example node health check script for Slurm?
- Why doesn't the HealthCheckProgram execute on DOWN nodes?
- How can I prevent the slurmd and slurmstepd daemons from being killed when a node's memory is exhausted?
- I see the host of my calling node as 127.0.1.1 instead of the correct IP address. Why is that?
- How should I add nodes to Slurm?
- How should I remove nodes from Slurm?
- Why is a compute node down with the reason set to "Node unexpectedly rebooted"?
- How do I convert my nodes to Control Group (cgroup) v2?
- Can Slurm be used to run jobs on Amazon's EC2?
User Management
- How can PAM be used to control a user's limits on or access to compute nodes?
- How can I exclude some users from pam_slurm?
- Can a user's account be changed in the database?
- I had to change a user's UID and now they cannot submit jobs. How do I get the new UID to take effect?
- How can I get SSSD to work with Slurm?
Jobs
- How is job suspend/resume useful?
- How can I suspend, resume, hold or release all of the jobs belonging to a specific user, partition, etc?
- After manually setting a job priority value, how can its priority value be returned to being managed by the priority/multifactor plugin?
- Can I update multiple jobs with a single scontrol command?
- How could I automatically print a job's Slurm job ID to its standard output?
- Is it possible to write to user stdout?
- Why are user processes and srun running even though the job is supposed to be completed?