A07: Scalable Parallel Scripting in the Cloud
Student: Benjamin H. Glick (Lewis & Clark College)
Supervisor: Kyle Chard (University of Chicago, Argonne National Laboratory)
Abstract: It’s often complicated, time consuming, but frequently necessary to successfully port complex workflows to multiple high-performance environments. Parsl is a Python-based parallel scripting library that provides a simple model for describing and executing dataflow-based scripts over arbitrary execution resources such as clouds, campus clusters, and high-performance systems. Parsl’s execution layer abstracts the differences between providers enabling provisioning and management of compute nodes for use with a pilot system. In this poster, we describe the development of a new execution provider designed to support Amazon Web Services (AWS) and Microsoft’s Azure. This provider supports the transparent execution of implicitly parallel Python-based scripts using elastic cloud resources. We demonstrate that Parsl is capable of executing thousands of applications per second over this elastic execution fabric.
ACM-SRC Semi-Finalist: no
Poster: pdf
Two-page extended abstract: pdf
Poster Index