National Institute of Water and
Atmospheric Research, New Zealand
28 Nov, 4 pm, in 2155
Abstract:
Cylc ("silk") is an Open Source workflow engine for cycling systems.
Cylc implements a novel meta-scheduling algorithm with no global cycle
loop: it only sees the dependence between individual tasks (even across
cycles) in a single ongoing workflow. Consequently Cylc does not need
to wait for one cycle to finish before starting the next, and it can
catch up very efficiently from failures and delays. Other features
include an intuitive dependency graph notation with ISO 8601 date-time
(or integer) recurrence expressions for cycling configuration; multiple
inheritance for efficient sharing of all common task run time settings;
and the Jinja2 template processor for efficient programmatic generation
of workflow definitions. Cylc has comprehensive event handling
capabilities and robust inter-workflow triggering to support large
operations, but as a distributed system with no central server it is
also easy to use for individuals and small groups.
The entire production and research workload of the UK Met Office (NWP
and climate) is managed with Cylc, and it is now part of the official
software infrastructure of the Unified Model system. In the U.S. Cylc
is used by the NRL Marine Meteorology research division; by the 557th
Weather Wing of the Air Force (with the Unified Model); GFDL Princeton
is rewriting its FRE infrastructure around Cylc; and NCAR has adopted
it for CESM CMIP6 production runs.
Cross-planet video conferencing bandwidth permitting, project lead
Hilary Oliver (NIWA, New Zealand) will give a general introduction to
Cylc, with a live demo.