Back to AI Research

AI Research

Token-Efficient Data Reasoning Agents via Adaptive... | AI Research

Key Takeaways

  • Agentic data cracking is a method designed to reduce the high computational costs of using LLM agents to reason over large, unstructured document collections...
  • Valuable data remains embedded in unstructured sources: web pages, reports, contracts, filings, earnings calls, and PDFs.
  • The big bet in enterprise AI is deploying LLM agents that reason over this data to answer complex questions for every knowledge worker.
  • Agents can do this today, but at prohibitive cost.
  • Each question repeatedly opens large documents to recover scattered evidence, consuming up to a million tokens.
Paper AbstractExpand

Valuable data remains embedded in unstructured sources: web pages, reports, contracts, filings, earnings calls, and PDFs. The big bet in enterprise AI is deploying LLM agents that reason over this data to answer complex questions for every knowledge worker. Agents can do this today, but at prohibitive cost. Each question repeatedly opens large documents to recover scattered evidence, consuming up to a million tokens. However, if the data were already structured, the same question would reduce to a cheap database lookup. For example, on FanOutQA benchmark, reasoning over an ideal pre-structured store is 28X cheaper, and the gap grows to orders of magnitude as questions fan out over more documents. Yet structuring everything in advance is not viable: documents hold vastly more possible structure than any workload will use, and the useful structure and documents are unknown until queries arrive. We propose agentic data cracking, a method that structures unstructured data adaptively and speculatively as a byproduct of reasoning itself. Structuring is adaptive because observed queries decide when it happens and what matters, and speculative because it goes beyond the current question. Whenever the agent opens a document to answer, a cracking sub-agent forks from the already-loaded context at marginal cost and extracts grounded structure likely to serve related future queries. Over time, an increasing share of queries is fully covered by structured data and answered without opening a document, keeping agentic accuracy at close to RAG cost. On FanOutQA, extended with merely one related question per test question, cracking cuts cost by 53% while preserving accuracy. Agentic data cracking is a first step toward next-generation data infrastructure for agentic reasoning over unstructured data: a shared substrate beneath the model where knowledge that reasoning already paid to uncover accumulates.

Agentic data cracking is a method designed to reduce the high computational costs of using LLM agents to reason over large, unstructured document collections. By adaptively and speculatively structuring data as a byproduct of the reasoning process, the system allows future queries to bypass expensive document loading, instead retrieving answers from a structured store.

The Cost of Data Reasoning

LLM agents are highly effective at answering complex questions by searching through documents, extracting facts, and performing multi-step reasoning. However, this process is "prefill-intensive," meaning the agent must repeatedly load large documents into its context window to extract scattered evidence. According to researchers Milad Rezaei Hajidehi, Qitong Wang, and Stratos Idreos of Harvard University, a single question can consume up to one million tokens and cost nearly one dollar. While pre-structuring all data into a database would make these tasks significantly cheaper, it is infeasible because documents contain far more information than any specific workload requires, and the necessary structure is unknown until queries arrive.

How Agentic Data Cracking Works

The proposed system functions by integrating a "cracking sub-agent" into the reasoning workflow. When a primary agent opens a document to answer a question, the sub-agent forks from the existing context at a marginal cost. It uses semantic reasoning to identify and extract "cracked objects"—grounded entities, attributes, and relations—that are likely to be useful for future, related queries.
These objects are stored in a catalog. When a subsequent query is made, the agent first checks this catalog. If the required information is already structured, the agent performs a low-cost database lookup instead of re-opening and re-processing the raw document. If the information is missing, the system falls back to the original document, ensuring that accuracy is maintained.

Performance and Efficiency

In experiments using the FanOutQA benchmark, the researchers found that reasoning over an ideal pre-structured store is 28 times cheaper than reasoning over raw documents. When applying agentic data cracking to the benchmark—extended to include at least one related question per test question—the system reduced costs by 53% while preserving answer accuracy. At the 10th percentile of savings, the system performed 9 times cheaper than the baseline. The researchers note that these savings accumulate over time as the system builds a "data moat," where the structured knowledge becomes a durable asset that improves with use.

Considerations for Implementation

The effectiveness of this approach relies on "semantic locality," where users or automated systems tend to ask related questions that target the same documents or overlapping sets of entities. The system is designed to be adaptive, meaning it only extracts structure that is actually demanded by the workload, avoiding the waste of exhaustive, upfront extraction.
A key limitation is that the system requires a "miss" on the initial query to trigger the extraction of new structure. Furthermore, the researchers note that the current schema is limited to strings, integers, and dates. Because the system relies on the agent's ability to resolve entity names and relation labels, mismatches in these labels can prevent the retrieval of stored objects, necessitating a fallback to raw document access.

Comments (0)

No comments yet

Be the first to share your thoughts!