More Criticism Does Not Make a Better Review: EquiR... | AI Research

More Criticism Does Not Make a Better Review: EquiReview-R
AI-assisted peer review is becoming common, but simply generating more feedback does not necessarily lead to a higher-quality review. Current systems often struggle to distinguish between missing a genuine problem and making an unfair or unsupported claim. This paper introduces EquiReview-R, a system that treats the review process as an evidence-guided refinement task rather than a simple generation task. By focusing on revising existing concerns before searching for new ones, the system aims to reduce "overcritique"—the tendency to include unsupported or overly broad allegations—while maintaining high standards for identifying actual scientific weaknesses. The same ai systems question is explored in ProgRouter, which adds a research perspective.

The Problem with "More"

The authors identify a critical bottleneck: as AI systems generate more candidate objections, they often accumulate claims that have not been properly verified against the paper’s evidence. A review might be long and appear thorough, yet still contain errors that impose unfair burdens on authors. The researchers found that existing systems often fail to revise initial, unverified concerns, leading to a buildup of "noise." They argue that a review should be viewed as a structured set of concerns that can expand when a real issue is found, but must also contract when a claim is refuted, merged, or proven to be unsupported by the evidence.

How EquiReview-R Works

EquiReview-R follows a specific, logical sequence to ensure the review remains accurate: 1. Evidence-Guided Revision: Before looking for new issues, the system re-evaluates every existing concern against the paper’s text. It categorizes each claim as supported, narrowed, refuted, merged, or resolved. 2. Complementary Search: Once the current review is cleaned up, the system searches for missing issues from two perspectives: one independent of the current review and one specifically targeting areas the current review has not yet covered. 3. Selective Stopping: The system uses a decision rule to determine whether to stop, continue, or defer. It only stops when it is confident that no major, high-consequence issues remain unresolved. The ai search story also surfaces in Stanford AI discovery identifies natural weight..., adding another angle.

Key Findings

In tests on a cohort of previously unseen scientific papers, EquiReview-R demonstrated that it could significantly improve the quality of reviews without sacrificing the ability to find important issues. Specifically, it reduced major overcritique from 15.5% to 8.1% while maintaining a similar level of "recall" (the ability to catch major omissions) as high-recall models. The researchers also found that simply adding more computational power or generating more text did not achieve these results; the improvement came specifically from the revision process, which allowed the system to prioritize accuracy over volume.

A New Resource for Research

To support further study into how AI reviews can be improved, the authors released "ReviewTrace." This is a new, evidence-linked corpus that records the entire trajectory of how concerns are proposed, challenged, and revised. By providing this data, the researchers hope to help the community better understand the nuances of review revision, disagreement, and the provenance of scientific criticism. The ai search story also surfaces in Google AI Releases TimesFM 3 for..., adding another angle. as detailed in the full paper on Arxiv