By Kris Garrett, SC17 SCC Reproducibility Chair from Los Alamos National Laboratory
For years, computational science research literature has relied upon computational results. While this mimics relying on experimental results in other sciences, the ability to reproduce computational results has been lacking. At SC16, the Student Cluster Competition (SCC) began taking the lead in pushing for reproducibility of computational results.
The SCC consists of four computing challenges, each year to be solved in real time over a 48-hour period during the SC conference. Last year, one of the challenges was to replicate the results from the SC15 paper “A parallel connectivity algorithm for de Bruijn graphs in metagenomic applications” by Flick, Jain, Pan and Aluru. Using the authors’ software, the results of the paper were replicated by the 14 student teams.
Both the paper authors and the competitors were awarded for this effort. The paper was awarded a Results Replicated badge by the ACM and the authors were awarded a certificate of appreciation from SIGHPC at the SC16 awards ceremony. The top scoring teams from the SCC reproducibility challenge were given the opportunity to publish their results from the competition in the Parallel Computing journal.
The SCC is continuing its push into reproducibility with a new challenge at SC17. Authors of SC16 papers were given the opportunity to submit an Artifact Description appendix describing the software used to create the results in their papers. The papers that participated in this effort received an Artifact Available badge for their efforts.
In all, nine papers were submitted. Of those papers, the SCC chose “The vectorization of the Tersoff multi-body potential: An exercise in performance portability” by Höhnerbach, Ismail and Bientinesi for the 2017 reproducibility challenge. One unique aspect of this paper is that it reports on optimizations for several hardware architectures.
Given the diversity of architectures brought to the competition, the SCC may be the only venue able to replicate all the results from the paper. As in the previous year, this paper will receive the Results Replicated badge from the ACM and a certificate of appreciation from SIGHPC at the SC17 awards ceremony.
The SCC reproducibility program is part of a wider effort to encourage authors submitting papers to the conference to voluntarily complete an appendix to their paper that described the details of their software environment and computational experiments to the extent that an independent person could replicate their results. Read the SC17 blog post on this initiative.
About the author:
Kris Garrett is the SC17 SCC Reproducibility Chair and is currently a postdoc at Los Alamos National Laboratory in the Computational and Physics Methods Group. His research interests include both mathematics and HPC research for kinetic transport problems. He is also a co-lead of the Parallel Computing Summer Research Internship at Los Alamos National Laboratory.