Introducing the Cylc Workflow Manager

Hilary Oliver
National Institute of Water and Atmospheric Research, New Zealand
28 Nov, 4 pm, in 2155

Cylc ("silk") is an Open Source workflow engine for cycling systems. Cylc implements a novel meta-scheduling algorithm with no global cycle loop: it only sees the dependence between individual tasks (even across cycles) in a single ongoing workflow. Consequently Cylc does not need to wait for one cycle to finish before starting the next, and it can catch up very efficiently from failures and delays. Other features include an intuitive dependency graph notation with ISO 8601 date-time (or integer) recurrence expressions for cycling configuration; multiple inheritance for efficient sharing of all common task run time settings; and the Jinja2 template processor for efficient programmatic generation of workflow definitions. Cylc has comprehensive event handling capabilities and robust inter-workflow triggering to support large operations, but as a distributed system with no central server it is also easy to use for individuals and small groups.

The entire production and research workload of the UK Met Office (NWP and climate) is managed with Cylc, and it is now part of the official software infrastructure of the Unified Model system. In the U.S. Cylc is used by the NRL Marine Meteorology research division; by the 557th Weather Wing of the Air Force (with the Unified Model); GFDL Princeton is rewriting its FRE infrastructure around Cylc; and NCAR has adopted it for CESM CMIP6 production runs.

Cross-planet video conferencing bandwidth permitting, project lead Hilary Oliver (NIWA, New Zealand) will give a general introduction to Cylc, with a live demo.