P62: How To Do Machine Learning on Big Clusters
SessionPoster Reception
Event Type
ACM Student Research Competition
Poster
Reception
TimeTuesday, November 14th5:15pm - 7pm
LocationFour Seasons Ballroom
DescriptionScientific pipelines, such those in chemogenomics machine learning applications, often compose of multiple interdependent data processing tasks. We are developing HyperLoom - a platform for defining and executing workflow pipelines in large-scale distributed environments. HyperLoom users can easily define dependencies between computational tasks and create a pipeline which can then be executed on HPC systems. The high-performance core of HyperLoom dynamically orchestrates the tasks over available resources respecting task requirements. The entire system was designed to have a minimal overhead and to efficiently deal with varying computational times of the tasks. HyperLoom allows to execute pipelines that contain basic built-in tasks, user-defined Python tasks, tasks wrapping third-party applications or a combination of those.
