SC17 Denver, CO

Ceph Applications in HPC Environments

Authors: Dr. Benjamin Lynch (University of Minnesota)

Abstract: With a series of lightning talks, we will present and discuss the use of Ceph in HPC environments. Ceph is an open-source distributed object store with an associated file system widely used in cloud and distributed computing. In addition, both the object store and file system components are seeing increasing deployments as primary data storage for traditional HPC. Ceph is backed by a robust, worldwide open source community effort with broad participation from major HPC and storage vendors.

Long Description: Most modern research is limited in some way by challenges related to collecting, storing, analyzing, and sharing large and small data sets. To address these challenges, data center managers are increasingly exploring storage platforms that can be deployed to supplement their traditional high performance storage systems. To be effective, a supplemental storage system must be inexpensive, easily scale to support multi- Petabytes to Exabytes of data, support a wide range of application interfaces (i.e., object, file, and block), and be highly fault-tolerant. The Ceph system satisfies many of these requirements and as a result many multi-petabyte installations of Ceph can now be found in academic, government, and commercial environments.Ceph has a wide range of potential use cases (object, block, file, and other semantic storage frontends), deployment configurations, tunable parameters, security options, and other characteristics that can affect its scalability, performance, and fault-tolerance.

This BoF will bring together Ceph implementors to share their deployment experiences, as well as provide feedback to the developer community on needed features and enhancements specific to the HPC community. HPC centers initiating or considering Ceph deployments will gain information from their peers about use cases and deployment strategy. Ceph developers will gain critical insights and feedback from the HPC user community and present the current roadmap and vision for Ceph in HPC.

Conference Presentation: pdf

Birds of a Feather Index