FAIR Data Pipeline: provenance-driven data management for traceable scientific workflows
Authors:
Sonia Natalie Mitchell,
Andrew Lahiff,
Nathan Cummings,
Jonathan Hollocombe,
Bram Boskamp,
Ryan Field,
Dennis Reddyhoff,
Kristian Zarebski,
Antony Wilson,
Bruno Viola,
Martin Burke,
Blair Archibald,
Paul Bessell,
Richard Blackwell,
Lisa A Boden,
Alys Brett,
Sam Brett,
Ruth Dundas,
Jessica Enright,
Alejandra N. Gonzalez-Beltran,
Claire Harris,
Ian Hinder,
Christopher David Hughes,
Martin Knight,
Vino Mano
, et al. (13 additional authors not shown)
Abstract:
Modern epidemiological analyses to understand and combat the spread of disease depend critically on access to, and use of, data. Rapidly evolving data, such as data streams changing during a disease outbreak, are particularly challenging. Data management is further complicated by data being imprecisely identified when used. Public trust in policy decisions resulting from such analyses is easily da…
▽ More
Modern epidemiological analyses to understand and combat the spread of disease depend critically on access to, and use of, data. Rapidly evolving data, such as data streams changing during a disease outbreak, are particularly challenging. Data management is further complicated by data being imprecisely identified when used. Public trust in policy decisions resulting from such analyses is easily damaged and is often low, with cynicism arising where claims of "following the science" are made without accompanying evidence. Tracing the provenance of such decisions back through open software to primary data would clarify this evidence, enhancing the transparency of the decision-making process. Here, we demonstrate a Findable, Accessible, Interoperable and Reusable (FAIR) data pipeline developed during the COVID-19 pandemic that allows easy annotation of data as they are consumed by analyses, while tracing the provenance of scientific outputs back through the analytical source code to data sources. Such a tool provides a mechanism for the public, and fellow scientists, to better assess the trust that should be placed in scientific evidence, while allowing scientists to support policy-makers in openly justifying their decisions. We believe that tools such as this should be promoted for use across all areas of policy-facing research.
△ Less
Submitted 4 May, 2022; v1 submitted 13 October, 2021;
originally announced October 2021.
From Physics Model to Results: An Optimizing Framework for Cross-Architecture Code Generation
Authors:
Marek Blazewicz,
Ian Hinder,
David M. Koppelman,
Steven R. Brandt,
Milosz Ciznicki,
Michal Kierzynka,
Frank Löffler,
Erik Schnetter,
Jian Tao
Abstract:
Starting from a high-level problem description in terms of partial differential equations using abstract tensor notation, the Chemora framework discretizes, optimizes, and generates complete high performance codes for a wide range of compute architectures. Chemora extends the capabilities of Cactus, facilitating the usage of large-scale CPU/GPU systems in an efficient manner for complex applicatio…
▽ More
Starting from a high-level problem description in terms of partial differential equations using abstract tensor notation, the Chemora framework discretizes, optimizes, and generates complete high performance codes for a wide range of compute architectures. Chemora extends the capabilities of Cactus, facilitating the usage of large-scale CPU/GPU systems in an efficient manner for complex applications, without low-level code tuning. Chemora achieves parallelism through MPI and multi-threading, combining OpenMP and CUDA. Optimizations include high-level code transformations, efficient loop traversal strategies, dynamically selected data and instruction cache usage strategies, and JIT compilation of GPU code tailored to the problem characteristics. The discretization is based on higher-order finite differences on multi-block domains. Chemora's capabilities are demonstrated by simulations of black hole collisions. This problem provides an acid test of the framework, as the Einstein equations contain hundreds of variables and thousands of terms.
△ Less
Submitted 24 July, 2013;
originally announced July 2013.
