Navigating Sparse Evidence: Agentic Visual RAG via... | AI Research

Navigating Sparse Evidence: Agentic Visual RAG via Explicit Context Selection and Consolidation introduces a new framework called SCoRE to improve how AI models process and reason over visually rich documents like slides, reports, and PDFs. While current Visual Retrieval-Augmented Generation (VRAG) systems can retrieve images, they often struggle when relevant information is scattered across many pages or hidden in small document regions. SCoRE addresses this by creating a unified agent loop that explicitly selects, preserves, and organizes visual evidence before generating a final answer, ensuring the model's reasoning is grounded in clear, relevant source material.

The Challenge of Visual Evidence

In many documents, the information needed to answer a query is not neatly organized. It may be concentrated in a tiny chart on one page or spread across several different documents. Existing AI agents often fail here because they rely on raw, noisy exploration trajectories or compressed text memories. This leads to two main problems: the agent may lose track of where the evidence came from, and the final answer may be based on a disorganized "trial-and-error" process rather than a coherent set of supporting facts. The same reasoning question is explored in A Unified Physics-Aware Quantum Machine Learning..., which adds a research perspective.

How SCoRE Works

SCoRE functions as a unified agent loop that manages evidence in two distinct stages:

• Exploration and Selection: As the agent searches through documents, it maintains a "textual ledger." It records only the information it deems relevant to the user's query, along with precise pointers to the original images. By keeping this ledger compact, the agent avoids overwhelming its memory with irrelevant visual data.

• Consolidation: Once the agent decides it has enough information, it triggers a "consolidate" action. It reloads the original, high-quality images referenced in its ledger, filters out any noise, and arranges the evidence into a logical sequence. This step separates the messy process of searching from the final, structured act of answering, ensuring the model provides a response backed by specific, indexed visual sources. The ai agents story also surfaces in Stanford Researchers Develop TRACE to Fix..., adding another angle.

Training for Better Reasoning

To ensure the agent learns to organize evidence effectively, the researchers implemented a two-stage training process. First, they used "cold-start" trajectory distillation, where the model learns from high-quality examples that contain both correct answers and complete evidence coverage. Second, they applied reinforcement learning with an "evidence-aware" reward. This reward system specifically encourages the model to find all necessary gold-standard pages (coverage), keep its evidence chain concise (compactness), and ensure the final answer is accurate.

Performance and Impact

When tested on three major VRAG benchmarks—ViDoSeek, SlideVQA, and MMLongBench—SCoRE achieved state-of-the-art accuracy across various model scales. The results demonstrate that by explicitly separating the discovery of evidence from its final organization, the model becomes more reliable and efficient. The study confirms that both the selection and consolidation phases are essential for improving the model's ability to ground its answers in the actual visual content of the documents it retrieves. The ai agents story also surfaces in Andrew Ng Launches OpenWorker to Deliver..., adding another angle. as detailed in the full paper on Arxiv