Back to AI Research

AI Research

ReuScope separates data reuse from its effect on benchmark scores

Key Takeaways

  • An audit of Capsule Vision 2024 finds shared files and recordings, while showing why source-removal score changes cannot directly measure leakage inflation.
  • ReuScope examines a common problem in AI evaluation: a dataset can contain reused material, yet measuring the effect of that reuse requires more than deleting some rows and comparing scores.
  • The [paper](https://arxiv.org/abs/2610.41835) develops an audit protocol using Capsule Vision 2024 as its case study.
  • The authors identify byte-identical files across the public training and validation boundary.
  • All 41 contributing Kvasir-Capsule recordings occur in both partitions.

ReuScope examines a common problem in AI evaluation: a dataset can contain reused material, yet measuring the effect of that reuse requires more than deleting some rows and comparing scores. The paper develops an audit protocol using Capsule Vision 2024 as its case study.

What the audit finds

The authors identify byte-identical files across the public training and validation boundary. All 41 contributing Kvasir-Capsule recordings occur in both partitions. They distinguish that shared recording evidence from a count of shared patients, which the available cross-recording identifiers cannot establish.
That distinction prevents a stronger privacy or independence claim from being inferred from evidence that supports only a narrower conclusion. Matching a file, a frame, a recording and a person are related but different checks.

Why the score comparison changes direction

Across ten paired seeds, removing the Kvasir-Capsule training rows and five flagged near-matches lowers public-validation balanced accuracy from 82.5% to 61.2%. When the same fitted models are evaluated on validation rows from the other sources, the comparison reverses: removal improves balanced accuracy by 1.38 percentage points.
The public-validation drop is sensitivity to source removal, rather than a direct estimate of leakage-induced inflation. Removing a source also changes the useful training material and class support. The paper uses matched controls to investigate these competing explanations and reports an exploratory local-removal experiment whose accuracy interval spans zero.

What benchmark authors can reuse

Keep the data-relationship audit separate from the intervention used to study performance. Report the evaluation population, class coverage and uncertainty unit for each conclusion. A useful review should ask which exact files or recordings overlap and then ask a second question: what comparison would isolate the effect under study? ReuScope demonstrates why those questions need separate answers before a leaderboard result is interpreted as evidence of generalization.

Comments