Adaptive Compression to Improve I/O Performance for Climate Simulations
Workshop: The 2nd International Workshop on Data Reduction for Big Scientific Data (DRBSD-2)
Abstract: We present an adaptive compression tool for scientific applications that automatically determines and adapts the best among a set of well-known effective compression schemes to each data variable and enables optimizing both compression ratio and compression overhead. Our adaptive compression library ACOMPS integrates several lossless compression algorithms and also reorganizes data variables in a preprocessing step to enable the compression schemes to work well for various types of data, including floating point values. Our library can be tuned to pick the best compression schemes based on compression ratio, compression speed or a combination of both. For ease of use, we also provide our library as a plugin for the widely used ADIOS middleware I/O system. In our experiments with a climate simulation application, we show that ACOMPS performs well compared to other lossless compression methods and has low overhead.