Back to AI Research

AI Research

Beyond Top-K: Replacing Black-Box Retrieval with In... | AI Research

Key Takeaways

  • Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations introduces a method for document retrieval that replaces traditional "chunk...
  • Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query.
  • We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable.
  • A table-aware chunker built as a steelman fixes the unit problem but leaves 27-30% of numeric chunks with no fiscal-year header at every chunk size we tried.
  • On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5) -- or 35.3% tuned, which READ still leads by 23.5 points (p_Holm = 0.017).
Paper AbstractExpand

Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query. We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable. On a 780-page government financial report, 86.8% of content lines are table rows, thousands of near-identical figures compete in one embedding space, and a figure inherits its unit from a header a median of 13 lines above it -- so a chunk boundary routinely separates a number from whether it is in lakh or crore, an error of two orders of magnitude. A table-aware chunker built as a steelman fixes the unit problem but leaves 27-30% of numeric chunks with no fiscal-year header at every chunk size we tried. We propose READ (Reliable Embedding-free Agentic Document-search), in which an agent reads the raw document through three deterministic operations -- normalized lexical search, structural navigation, and bounded span reads -- exposed over the Model Context Protocol, so a trajectory is a replayable audit trail, not an opaque similarity score. On 51 verified questions READ answers 58.8% against dense retrieval's 15.7% (p_Holm = 2 x 10^-5) -- or 35.3% tuned, which READ still leads by 23.5 points (p_Holm = 0.017). An agent given the same loop but a top-k tool reaches only 27.5%, locating the gain in the interface rather than in iteration. We also report what the evidence does not support: BM25 is statistically indistinguishable from READ, so our result separates embedding-based from embedding-free retrieval, not agentic from lexical search.

Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations introduces a method for document retrieval that replaces traditional "chunk-and-embed" systems with an agentic, deterministic approach. The authors argue that for structured, exactness-critical documents like financial statements, standard retrieval methods are structurally unsound because they partition text into chunks that often separate figures from their governing units and fiscal years.

The Failure of Chunk-Based Retrieval

The authors, Sagar Tamang, Ayush Vyas, and Tabarakul Hazarika, identify that standard retrieval-augmented generation (RAG) relies on splitting documents into chunks and ranking them by similarity. In financial reports, this process frequently fails because 86.8% of content lines are table rows. Because units (like "lakh" or "crore") are often declared in headers many lines above the relevant figures, a fixed-size chunk boundary routinely separates a number from its unit, leading to errors of two orders of magnitude. Even with "steelman" table-aware chunking, approximately 28% of numeric chunks remain disconnected from their fiscal-year headers, a problem that persists regardless of the chunk size used.

The READ Approach

To address these structural issues, the authors propose READ (Reliable Embedding-free Agentic Document-search). Instead of relying on opaque similarity scores, READ uses an agent that interacts with the raw document through three deterministic operations:

  • Normalized lexical search: Locates specific patterns while accounting for conversion artifacts like digit grouping or split words.

  • Structural navigation: Uses an outline of the document to identify relevant sections.

  • Bounded span reads: Allows the agent to retrieve specific line ranges after identifying where the evidence lies, ensuring that governing headers are included in the context.
    Because these operations are deterministic and exposed via the Model Context Protocol, every retrieval trajectory serves as a replayable audit trail.

Performance and Results

The authors tested READ against dense retrieval baselines using 51 verified questions based on a 780-page government financial report. READ achieved an accuracy of 58.8%, significantly outperforming the dense retrieval baseline, which reached 15.7%. Even when the dense baseline was tuned for optimal chunk size and retrieval depth, it reached only 35.3%, leaving READ with a 23.5-point lead. The authors note that an agent using a "top-k" tool instead of READ’s operations reached only 27.5% accuracy, suggesting that the performance gain stems from the interface design rather than the iterative nature of the agent.

Limitations and Findings

The research highlights that the effectiveness of retrieval is bounded by the quality of document conversion. The authors found that PDF-to-Markdown conversion can introduce errors, such as splitting a decimal point from its digits, which no retrieval method can resolve. They categorize these as "conversion-limited" failures rather than retrieval failures. Additionally, the authors report that their evidence does not support the claim that agentic search is inherently superior to lexical search; they found that BM25 (a traditional lexical search method) was statistically indistinguishable from READ, indicating that the primary advantage of their approach is the move away from embedding-based retrieval toward a more interpretable, structure-aware interface.

Comments (0)

No comments yet

Be the first to share your thoughts!