SLURM gives you the option to run jobs in a specific order using the job dependency feature.
-d, --dependency=<dependency_list>
Below is a list of some common dependency types you can use:
after:job_id[:jobid...]: This job can begin execution after the specified jobs have begun execution.afterany:job_id[:jobid...]: This job can begin execution after the specified jobs have terminated.afternotok:job_id[:jobid...]: This job can begin execution after the specified jobs have terminated in some failed state (non-zero exit code, node failure, timed out, etc).afterok:job_id[:jobid...]: This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero).singleton: This job can begin execution after any previously launched jobs sharing the same job name and user have terminated. In other words, only one job by that name and owned by that user can be running or suspended at any point in time.Say you submit a job called job_1.sub
sbatch job_1.sub
Submitted batch job 101301
If you have another job, job_2.sub that needs to start running after successful completion of job_1.sub you should type:
sbatch -d afterok:101301 job_2.sub
Submitted batch job 101302