Tracking of Online Parameter Fine-Tuning in Scientific Workflows
Workshop: WORKS 2017 (12th Workshop on Workflows in Support of Large-Scale Science)
Authors: Marta Mattoso (Federal University of Rio de Janeiro)
Abstract: In long-lasting large-scale workflow executions, computational scientists need to adapt the workflow by fine-tuning several parameters of complex computational models. These specific tunings may significantly reduce overall execution time. In executions that last for weeks, for instance, they can easily lose track of what has been tuned at previous simulation stages if the adaptations are not registered properly. In this work, we propose a solution for tracking parameter fine-tunings at runtime. With support of sophisticated online data analysis, scientists get a detailed view of the execution, providing insights to determine when and how to tune parameters. We developed DfAdapter*, a tool that collects human adaptations in the dataflow, while the workflow runs with or without a Scientific Workflow Management System. It controls and stores specific parameter-tunings in a provenance database, relating the human adaptation actions with data for: domain, dataflow provenance, execution, and performance. An extended PROV-compliant data diagram records the adaptation data. We evaluate DfAdapter by plugging it into a high performance workflow built with the libMesh library. The experiments with real data, from the Oil and Gas domain, showed that tunings significantly reduced the simulation time.
* DfAdapter Repository. Available at: https://github.com/hpcdb/DfAdapter