Ilmu Sosial editorial
Open AccessOA2024
Measuring Transparency in the Social Sciences: Political Science and International Relations
Using machine learning classifiers on 93,931 articles from 160 top political science and international relations journals (2010โ2021), researchers find that only ~21% of statistical inference papers have open data and ~5% of experiments are preregistered. Leading journals demonstrate that rapid improvement is feasible.
B. Scoggins; Matthew P. Robertsonยท Royal Society Open Scienceยท 2024ยท DOI 10.1098/rsos.240313
The core problem
The scientific method relies on transparency, yet the adoption of transparent research practices remains slow. The replication crisis in psychology highlighted that published findings using statistical inference are vulnerable to undetected errors, data manipulation, and data falsification. To address these issues and enhance research credibility, open data and preregistration have gained traction across the natural and social sciences. However, the extent of their adoption in different disciplines is unknown. This study introduces computational procedures to identify the transparency of a research field using large-scale text analysis and machine learning classifiers. Using political science and international relations as an illustrative case, the authors examine 93,931 articles across the top 160 political science and international relations journals between 2010 and 2021. The research questions are: (1) What is the prevalence of open data and preregistration in political science and international relations? (2) How has this prevalence changed over time? (3) Which journals lead in transparency practices?
Innovation
The authors developed computational procedures to detect transparency practices from article text. They employed machine learning classifiers trained to identify mentions of open data and preregistration. The dataset comprises 93,931 articles published between 2010 and 2021 in the top 160 political science and international relations journals. The classifiers were used to classify articles as employing statistical inference and to detect whether they provided open data or preregistration. The approach enables large-scale, automated assessment of transparency practices across a field. The study focuses on two key indicators: open data (for papers using statistical inference) and preregistration (for experiments). The methodology leverages text analysis to overcome the limitations of manual review, allowing for a comprehensive scan of the literature. The classifiers' performance was validated, though specific metrics are not detailed in the abstract. The computational approach is generalizable to other disciplines.
Introduction
The scientific method relies on transparency, yet the adoption of transparent research practices remains slow. The replication crisis in psychology highlighted that published findings using statistical inference are vulnerable to undetected errors, data manipulation, and data falsification. To address these issues and enhance research credibility, open data and preregistration have gained traction across the natural and social sciences. However, the extent of their adoption in different disciplines is unknown. This study introduces computational procedures to identify the transparency of a research field using large-scale text analysis and machine learning classifiers. Using political science and international relations as an illustrative case, the authors examine 93,931 articles across the top 160 political science and international relations journals between 2010 and 2021. The research questions are: (1) What is the prevalence of open data and preregistration in political science and international relations? (2) How has this prevalence changed over time? (3) Which journals lead in transparency practices?
Why it matters
The low adoption rates of open data (21%) and preregistration (5%) in political science and international relations mirror challenges seen in other social sciences. The replication crisis in psychology serves as a cautionary tale: without transparency, undetected errors and questionable research practices can undermine credibility. The study's computational approach offers a scalable way to monitor transparency across disciplines. The example of leading journals shows that change is possible and can happen quickly, likely due to editorial policies and incentives. The authors argue that broader adoption requires systemic changes, such as journal requirements and funding agency mandates. The findings have implications for research credibility and the reproducibility of social science. The study's limitations include reliance on text classifiers and the focus on two indicators. Future research could expand to other transparency practices and disciplines. Overall, the study provides a benchmark and a tool for tracking progress in transparency.
Who should read this
Pembaca Research Digest dan praktisi rumpun ini.
Opening member contentโฆ