Rose and Managing Workflows

How would you like a tool that would allow you to:

• Write and maintain workflows more efficient and sophisticated than an existing job submission script.
• Create dependency trees of tasks that will run at the click of a button, with automatic optimisation so that multiple tasks may be run in parallel.
• Revision control your tasks so that multiple people can easily collaborate on a piece of work with full logging, backup, and documentation.
• Make available your tasks for students and colleagues via a workflow discovery browser so that they can create their own copy and run it with a click of a button.
• Alert you to problems and allow you to re-run steps of your work without having to restart from scratch.
• Show you the progress of your work in a web browser or graphical tool.

If any of that sounds appealing then you may be interested in a new workflow manager that we’re currently testing called Rose.

Strictly speaking, Rose consists of two components: The aforementioned Rose, developed by the Met Office as a series of wrappers and tools around Cylc, an intelligent work scheduler and task manager. Together, these two packages allow you to design workflows as simple as Hello World, or as complex as the National Institute of Water and Atmospheric Research in New Zealand’s cycling weather forecast suite, for which this should give you a taster.

Rose: http://metomi.github.io/rose/doc/rose.html

Cylc: http://cylc.github.io/cylc/

Other features, such as a built-in scripting language for complex and dynamically generated tasks, an internal automated build system for fetching source code and compling it, and a metadata-based suite editor for constructing easy to use graphical user interfaces, ease the pain of updating work, allowing a suitably written workflow to work for anybody with minimal, or ideally zero, editing.

We at ARCCA have been working to implement Rose and Cylc on Raven for a number of months and are nearly ready to roll out a test system for interested parties. If you have a complex series of tasks that are currently bundled up in one or more job submission scripts, perhaps with manual operations like moving data around that ordinarily have to be done by hand, then we’d love to take a look at Rosifying them, making them easier to maintain and run. Contact ARCCA for more information and a grand tour, and keep an eye out for some examples to try out once we have our Rose server open for trial.