Batch jobs

The reality of computing on shared resources is that nodes are rarely available when you’re in front of the keyboard and you need to put your work into a script that can be run by the scheduler when resources become available. Your batch script may mix basic shell commands and functions that will be run serially on the first compute node in your allocation with parallel programs that are run as jobs as described in Section 2.

Submitting a basic job script with flux batch

The flux batch command allows you to submit batch scripts to a queue for later execution once resources are available. In the simplescript.sh example below, we mix shell commands to log when the job starts and that it has completed with a flux run command to launch an MPI program.

sh-4.2$ cat simplescript.sh
#!/bin/sh

date
flux run -n4 ./mpi_hellosleep
echo 'job complete'
sh-4.2$ flux batch -N2 -n4 ./simplescript.sh
fDkrou8xo
sh-4.2$ flux jobs
       JOBID USER     NAME       ST NTASKS NNODES  RUNTIME NODELIST
   fDkrou8xo day36    simplescri  R      4      2   9.366s rzalastor[5-6]
sh-4.2$ ls
flux-fDkrou8xo.out  mpi_hellosleep  mpi_hellosleep.c  simplescript.sh
sh-4.2$ tail -f flux-fDkrou8xo.out
Thu Mar 11 10:36:16 PST 2021
MASTER: Number of MPI tasks is: 4
task 0 on rzalastor5 going to sleep
task 2 on rzalastor6 going to sleep
task 1 on rzalastor5 going to sleep
task 3 on rzalastor6 going to sleep
task 0 on rzalastor5 woke up
task 2 on rzalastor6 woke up
task 1 on rzalastor5 woke up
task 3 on rzalastor6 woke up
job complete
^C
sh-4.2$

Adding job submission directives to your batch script

Many resource managers allow you to put batch submission flags in your script as comments. In Flux, you can do this by prepending the flags with #flux: in your script. For example, the job script below will run 4 tasks on two nodes.

sh-4.2$ cat simplescript.sh
#!/bin/sh

#flux: -N 2
#flux: -n 4

date
flux run -n4 ./mpi_hellosleep
echo 'job complete'

sh-4.2$ flux batch ./simplescript.sh
f3YDA4qqR
sh-4.2$ tail -f flux-f3YDA4qqR.out
Mon Mar 15 13:45:01 PDT 2021
task 2 on rzalastor5 going to sleep
MASTER: Number of MPI tasks is: 4
task 0 on rzalastor4 going to sleep
task 3 on rzalastor5 going to sleep
task 1 on rzalastor4 going to sleep
task 2 on rzalastor5 woke up
task 0 on rzalastor4 woke up
task 3 on rzalastor5 woke up
task 1 on rzalastor4 woke up
job complete
^C
sh-4.2$

Starting an interactive Flux instance with flux alloc

When you submit a job script with flux batch, you are actually starting a new Flux instance with its own resources and running your script in that instance. The flux alloc command will create a new Flux instance and start an interactive shell in it.

[day36@corona211:~]$ flux alloc -N2 -n2 -t 1h
[day36@corona177:~]$ flux resource list
     STATE NNODES   NCORES    NGPUS NODELIST
      free      2       96       16 corona[177-178]
 allocated      0        0        0
      down      0        0        0
[day36@corona177:~]$ flux run -N2 -n2 hostname
corona177
corona178
[day36@corona177:~]$

Alternatively, you can supply flux alloc with a command or script and it will run that in a new Flux instance. Unlike flux batch, flux alloc will block until the command or script returns and send the standard output and error to the terminal.

[day36@corona211:flux_test]$ cat test_batch.sh
#!/bin/bash

echo "resources"
flux resource list

echo "hosts"
flux run -N 2 -n 2 hostname
[day36@corona211:flux_test]$ flux alloc -N2 -n2 ./test_batch.sh
resources
     STATE NNODES   NCORES    NGPUS NODELIST
      free      2       96       16 corona[177-178]
 allocated      0        0        0
      down      0        0        0
hosts
corona177
corona178
[detached: session exiting]
[day36@corona211:flux_test]

Submitting jobs to an existing instance with flux proxy

Some user workflows involve getting an allocation (Flux instance) and submitting work to it from outside of that allocation. Flux can accomodate these types of workflows using the flux proxy command. You could, for example, create a two node Flux instance with flux alloc -N2 -n96 -t 1d --bg, then use a flux proxy command to submit work in that instance:

[day36@corona212:~]$ flux jobs
       JOBID USER     NAME       ST NTASKS NNODES  RUNTIME NODELIST
 ƒ8RmSm8mYW3 day36    flux        R      2      2   53.01s corona[177-178]
[day36@corona212:~]$ flux proxy ƒ8RmSm8mYW3 flux mini run -N2 -n2 hostname
corona177
corona178
[day36@corona212:~]$

Running commands in the system instance with flux --parent

Since each Flux job is a fully featured Flux instance, running a Flux command inside of a job will run that command inside of that instance. For example, if you run flux resource list on a corona login node, it will list all of the resources in that cluster:

[day36@corona82:~]$ flux resource list
     STATE NNODES   NCORES    NGPUS NODELIST
      free     45     2160      360 corona[173,194,238,253-259,261-283,285-296]
 allocated     68     3264      544 corona[171,174-193,196-201,203-207,213-216,218-237,239-247,250-252]
      down      8      384       64 corona[172,195,202,217,248-249,260,284]
[day36@corona82:~]$

If, however, you get an interactive allocation with flux alloc and run flux resource list in that, you will only see the resources allocated to your job:

[day36@corona82:~]$ flux alloc -N1
flux-job: fpyF8fye25m started                                                                                                                                                                     00:00:06
[day36@corona189:~]$ flux resource list
     STATE NNODES   NCORES    NGPUS NODELIST
      free      1       48        8 corona189
 allocated      0        0        0
      down      0        0        0
[day36@corona189:~]$

If you want to know information about the resources available in the main system instance (the parent instance), you can add a --parent flag to your flux command:

[day36@corona189:~]$ flux --parent resource list
     STATE NNODES   NCORES    NGPUS NODELIST
      free     48     2304      384 corona[194,198,204,206,238,250,253-259,261-283,285-296]
 allocated     65     3120      520 corona[171,173-193,196-197,199-201,203,205,207,213-216,218-237,239-247,251-252]
      down      8      384       64 corona[172,195,202,217,248-249,260,284]
[day36@corona189:~]$

This becomes particularly important if, for example, you want to include a command in your batch script that will submit the next job in a series of jobs. As we will discuss further in Section 5, if you put a flux batch myscript command in your batch script, that command will get run as a subjob of the current job. If you want that batch job to submitted to the main system queue, you will need to use flux --parent batch myscript.

More user facing batch options

The Flux job submission commands have many more options for doing things like running on specific nodes or queues, modifying your job environment, specifying task mappings, and more. See, for example, man flux-run for details on all of the options available. We have also put together a table of equivalent resource manager flags.


Section 2 | Section 3 | Exercise 3 | Section 4
Back to index