SC17 Denver, CO

P99: The Intersection of Big Data and HPC: Using Asynchronous Many Task Runtime Systems for HPC and Big Data


Authors: Joshua Daniel Suetterlein (Pacific Northwest National Laboratory), Joshua Landwehr (Trovares Inc), Andres Marquez (Pacific Northwest National Laboratory), Joseph Manzano (Pacific Northwest National Laboratory), Kevin Barker (Pacific Northwest National Laboratory), Guang Gao (University of Delaware)

Abstract: Although the primary objectives of the HPC and Big data fields seem disparate, HPC is beginning to suffer from a growing size of its workloads and the limitation of its techniques to handle large amount of data. This places interesting research challenges for both HPC and Big Data on how to marriage both fields together. This poster presents a case study which uses Asynchronous Many Task Runtimes (AMTs) as an exploratory vehicle to highlight possible solutions to these challenges. AMTs presents the unique opportunity for better load balancing, reconfigurable schedulers and data layouts that can take advantage of introspection frameworks, and the ability to exploit a massive amount of concurrency. We use the Performance Open Community Runtime (P-OCR) as a vehicle to port MapReduce operators to the HPC realm. We conduct experiments with both strong and weak scaling experimental format using WordCount and TeraSort as our kernels.
Award: Best Poster Finalist (BP): no

Poster: pdf
Two-page extended abstract: pdf


Poster Index