Extreme Scale Data Management for In-Situ Scientific Workflows
Workshop: WORKS 2017 (12th Workshop on Workflows in Support of Large-Scale Science)
Abstract: Data staging and in-situ/in-transit data processing are emerging as attractive approaches for supporting extreme scale scientific workflows. These approaches can improve end-to-end performance by enabling efficient data sharing between coupled simulations and data analytics components of an in-situ workflow. However, complex and dynamic data access/exchange patterns coupled with architectural trends toward smaller memory per core and deeper memory hierarchies threaten to impact the effectiveness of this approach. In this talk, I will explore a policy-based autonomic data management approach that can adaptively respond at runtime to dynamic data management requirements. Specifically, I will formulate the autonomic data management approach and present the design and implementation of autonomic policies as well as cross layer mechanisms, and will experimentally demonstrate how these autonomic adaptations can tune the application behaviors and resource allocations at runtime while meeting the data management requirements and constraints. This research is part of the DataSpaces project at the Rutgers Discovery Informatics Institute.