Lightweight Container Integration into Workflow Systems: A First Look at Singularity and Makeflow
Workshop: WORKS 2017 (12th Workshop on Workflows in Support of Large-Scale Science)
Authors: Kyle Sweeney (University of Notre Dame)
Abstract: Traditionally, HPC centers have relied on professional system administrators to install, upgrade, and manage operating systems and application software suitable for a particular site. This relieves end users of the burden of managing these details, but also limits users’ freedom to deploy software environments specialized for their task. Container technology addresses this limitation by allowing the end user to create and deploy custom software environments for a task. However, when used at large scale with a workflow management system in the most obvious way, container images can result in considerable waste in both filesystem space and network transfers. To address this, we consider two methods of composing software and data elements into a container – stacked and flat composition – and use these to create an efficient architecture for integrating containers into workflow systems. We demonstrate these techniques using the Singularity lightweight container environment and the Makeflow workflow system, demonstrating that flat composition results in reduced storage consumption and network transfers, on both synthetic and bioinformatics workflow.
Workshop Index