SC17 Denver, CO

P58: Wharf: Sharing Docker Images across Hosts from a Distributed Filesystem

Authors: Chao Zheng (University of Notre Dame), Lukas Rupprecht (IBM), Vasily Tarasov (IBM), Mohamed Mohamed (IBM), Dimitrios Skourtis (IBM), Amit S. Warke (IBM), Dean Hildebrand (IBM), Douglas Thain (University of Notre Dame)

Abstract: Due to their portability and less overhead compared to traditional virtual machines, containers are becoming an attractive solution for running HPC workloads. Docker is a popular toolset which enables convenient provisioning and management of containers and their corresponding images. However, Docker does not natively support running on shared storage, a crucial requirement in large-scale HPC clusters which are often diskless or access data via a shared burst buffer layer. This lack of distributed storage support can lead to overhead when running containerized HPC applications. In this work, we explore how Docker images can be served efficiently from a shared distributed storage layer. We implement a distributed layer on top of Docker that allows multiple Docker daemons to access container images from a shared filesystem such as IBM Spectrum Scale or NFS. Our design is independent of the underlying storage layer and minimizes the synchronization overhead between different daemons.
Award: Best Poster Finalist (BP): no

Poster: pdf
Two-page extended abstract: pdf

Poster Index