Workflow management software

The Flux Python bindings provide tools for managing complex workflows, but using them requires software development skills that domain scientists and other end users may not possess. There are several projects that leverage these bindings to build general, user-friendly workflow managers.

Maestro is a workflow manager developed at LLNL that uses Flux to orchestrate complex or simple workloads based on straightforward workflow descriptions. Most notably, it has enabled the Gordon Bell finalist and SC19 Best Paper MuMMI project.

Merlin is an LLNL developed workflow manager specifically for running machine learning workflows. It uses Maestro for describing tasks and Flux for task launch.

UQPipeline is an LLNL internal tool used for uncertainty quantification studies. Its Themis component can use Flux to launch large ensembles of jobs quickly.

Appendix 2 | Appendix 3
Back to index