Ilmu Komputer & AI editorial
Open AccessOA2024
REFORMS: Consensus-based Recommendations for Machine-learning-based Science
REFORMS is a consensus-based checklist of 32 questions and guidelines developed by 19 researchers across disciplines to address failures in machine-learning-based science. It aims to help researchers design and report studies, referees review papers, and journals enforce transparency and reproducibility standards.
Sayash Kapoor; Emily M. Cantrell; Kenny Peng; Thanh Hien Pham; Christopher A. Bail; Odd Erik Gundersen; J. Hofman; J. Hullman; M. Lones; Momin M. Malik; Priyanka Nanayakkara; R. Poldrack; Inioluwa Deborah Raji; Michael Roberts; Matthew J. Salganik; Marta Serra-Garcia; Brandon M. Stewart; Gilles Vandewiele; Arvind Narayananยท Science Advancesยท 2024ยท DOI 10.1126/sciadv.adk3452
The core problem
Machine learning (ML) methods are increasingly used across scientific disciplines, but their adoption has been accompanied by failures of validity, reproducibility, and generalizability. Such failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. The authors observe that ML methods often fail in similar ways across disciplines, motivating the need for clear recommendations for conducting and reporting ML-based science. Drawing from an extensive literature review, they present the REFORMS checklist (recommendations for machine-learning-based science), consisting of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers from computer science, data science, mathematics, social sciences, and biomedical sciences. It can serve as a resource for researchers designing and implementing studies, for referees reviewing papers, and for journals enforcing standards for transparency and reproducibility.
Innovation
The development of REFORMS followed a consensus-based process. The authors conducted an extensive review of past literature to identify common pitfalls and best practices in ML-based science. A panel of 19 researchers with expertise across computer science, data science, mathematics, social sciences, and biomedical sciences participated in the consensus process. Through iterative discussions and synthesis, they distilled the findings into a checklist of 32 questions, each paired with guidelines. The checklist is designed to be applicable across disciplines, addressing issues that arise in the design, implementation, and reporting of ML studies. The consensus approach ensures that the recommendations are broadly representative and grounded in collective expertise.
Introduction
Machine learning (ML) methods are increasingly used across scientific disciplines, but their adoption has been accompanied by failures of validity, reproducibility, and generalizability. Such failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. The authors observe that ML methods often fail in similar ways across disciplines, motivating the need for clear recommendations for conducting and reporting ML-based science. Drawing from an extensive literature review, they present the REFORMS checklist (recommendations for machine-learning-based science), consisting of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers from computer science, data science, mathematics, social sciences, and biomedical sciences. It can serve as a resource for researchers designing and implementing studies, for referees reviewing papers, and for journals enforcing standards for transparency and reproducibility.
Why it matters
The authors argue that the widespread adoption of ML in science necessitates standardized reporting and evaluation practices to ensure the credibility of findings. REFORMS addresses this need by providing a consensus-based framework that highlights common failure modes and offers actionable recommendations. By using REFORMS, researchers can improve the validity and reproducibility of their work, referees can more effectively assess ML-based claims, and journals can enforce higher standards for transparency. The checklist also facilitates cross-disciplinary communication by establishing a shared vocabulary and set of expectations. The authors acknowledge that implementing REFORMS may require effort and resources, but they believe the benefits for scientific progress outweigh the costs. They call for broader adoption and further refinement of the checklist as ML methods continue to evolve.
Who should read this
Pembaca Research Digest dan praktisi rumpun ini.
Opening member contentโฆ