REFORMS: Reporting Standards for Machine Learning Based Science
Authors:
Sayash Kapoor,
Emily Cantrell,
Kenny Peng,
Thanh Hien Pham,
Christopher A. Bail,
Odd Erik Gundersen,
Jake M. Hofman,
Jessica Hullman,
Michael A. Lones,
Momin M. Malik,
Priyanka Nanayakkara,
Russell A. Poldrack,
Inioluwa Deborah Raji,
Michael Roberts,
Matthew J. Salganik,
Marta Serra-Garcia,
Brandon M. Stewart,
Gilles Vandewiele,
Arvind Narayanan
Abstract:
Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways acros…
▽ More
Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear reporting standards for ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist ($\textbf{Re}$porting Standards $\textbf{For}$ $\textbf{M}$achine Learning Based $\textbf{S}$cience). It consists of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.
△ Less
Submitted 19 September, 2023; v1 submitted 15 August, 2023;
originally announced August 2023.
