Reproduction knowledge often disappears
ReproBridge studies a practical problem in machine-learning research: a successful reproduction can leave its most important decisions buried in a session history. The next person or agent then has to reconstruct the same environment and procedure.
The proposed artifact design includes a refined codebase, environment setup, experiment launchers and a reproduction guide. An agent workflow assembles those materials so a later session can start from more than the raw repository.
What the benchmark reports
ReproBridge-Bench contains 1,029 tasks from 387 experiments across 36 papers, with five human-verified reference scores per task. Agents work in fresh sessions, while a separate Auditor examines execution evidence.
Across four models, the researchers report mean reproduction success increasing from 76.4% with raw code to 92.6% with refined artifacts. Downstream API spending falls by 19.7% on average in the evaluated setup. The spread in success rates between models also narrows.
The reported success and cost results concern those tasks, models and artifact conditions. They do not establish a guaranteed reduction for every research repository or include every possible cost of constructing and maintaining the artifacts.
What researchers can take from the design
The ablation study reports that removing any artifact component reduces success, with the largest losses associated with code and environment removal. Franklin’s analysis is that a guide alone cannot compensate for an unusable executable setup.
A useful release checklist would preserve the exact launch path, dependency assumptions and scoring procedure together with the code. When assessing ReproBridge, inspect how the Auditor establishes that an experiment actually ran and matched its reference. The preprint offers evidence for retaining execution knowledge, rather than treating a plausible written explanation as proof of reproduction.
Comments