Back to AI Research

AI Research

The embrace of open science: An analysis of a decad... | AI Research

Key Takeaways

  • The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers This research investigates how documentation practices in ar...
  • The reproducibility crisis has directed the AI research community toward improving documentation practices.
  • Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists.
  • We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade.
  • Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications.
Paper AbstractExpand

The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.

The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers
This research investigates how documentation practices in artificial intelligence have evolved over the last decade. As the field of AI has grown rapidly, concerns regarding the "reproducibility crisis"—the difficulty of verifying scientific results—have intensified. By analyzing 56,800 papers published across five leading AI conferences between 2014 and 2024, the authors sought to determine if researchers are becoming more transparent and if formal requirements, such as reproducibility checklists, have effectively driven this change.

A Large-Scale Automated Approach

To manage the massive volume of research, the authors moved beyond traditional, time-consuming manual reviews. They developed an automated framework using large language models (LLMs) to scan thousands of papers for seven key indicators of reproducibility. These indicators include the availability of open-source code, the use of open datasets, clear descriptions of hardware and software dependencies, and detailed experimental setups. By validating their automated method against manually annotated samples, the researchers were able to conduct a comprehensive, longitudinal study that would have been impossible to perform by hand.

Significant Gains in Transparency

The study reveals a clear cultural shift toward open science. Over the decade, the practice of sharing both code and data—a combination strongly linked to successful reproduction—increased nearly sixfold, rising from 11% in 2014 to 64% in 2024. Furthermore, the number of papers that failed to document any of the seven reproducibility variables dropped significantly, becoming nearly non-existent by 2019. Based on these improvements in documentation, the authors estimate that the overall reproducibility rate of empirical AI research has more than doubled, growing from 28% to 64% over the same period.

The Role of Reproducibility Checklists

A key question addressed by the study is whether the introduction of formal reproducibility checklists by major conferences actually caused these improvements. The data suggests that the trend toward better documentation was already well underway before these checklists were implemented. Statistical analysis shows that the introduction of these requirements did not lead to a systematic, field-wide acceleration in documentation quality. Instead, the findings indicate that the AI community’s move toward transparency is part of a broader, pre-existing movement toward open science rather than a direct response to new procedural mandates.

Important Considerations

While the results show a positive trajectory, the authors note that their findings are based on documentation practices rather than direct experimental testing. The estimated reproducibility rates are approximations derived from applying previously established empirical success rates to the current data. Additionally, because the study focuses on documentation, it cannot guarantee that any individual paper is reproducible; rather, it measures the removal of barriers that typically prevent independent researchers from verifying scientific claims.

Comments (0)

No comments yet

Be the first to share your thoughts!