RAFT: A Stateful Retrieval-Augmented Framework for... | AI Research

RAFT: A Stateful Retrieval-Augmented Framework for Troubleshooting Agents
Troubleshooting enterprise customer support issues is a complex, multi-stage process that evolves as engineers gather logs, test hypotheses, and identify root causes. Traditional Retrieval-Augmented Generation (RAG) systems often fail here because they treat support cases as static documents, ignoring the chronological progression of an investigation. This paper introduces RAFT, a framework designed to handle the stateful nature of troubleshooting by organizing historical cases into directed chains of timeline entries. By retrieving information based on the specific stage of an active investigation, RAFT helps agents find relevant historical guidance more effectively than standard RAG methods. The same ai evaluation question is explored in Multi-Step Tool-Calling over Korean Open Public..., which adds a research perspective.

How RAFT Works

RAFT transforms raw, closed support cases into structured, multi-stage timelines. Each case is broken down into "timeline entries," which represent meaningful shifts in the investigation—such as the initial symptom report, the testing of a hypothesis, or the final resolution. When an agent is working on a new ticket, RAFT does not just search for similar cases; it searches for similar states within those cases. This allows the system to return the most relevant part of a historical case, providing the agent with both immediate tactical guidance for the current step and the broader context of how that case was eventually resolved. Additionally, RAFT can use an optional case-level graph to link cases that share similar root causes or resolutions, even if their initial symptoms differ.

Evaluating Troubleshooting Performance

Because public data for multi-stage troubleshooting is rare, the authors developed a new synthetic benchmark based on Microsoft Learn Windows Server documentation. They also tested the framework against real-world Apache Jira issues that included human-labeled duplicates. To ensure the evaluation was rigorous and reproducible, the team tested the retrieval layer independently of a full agent system. They measured performance at three different stages of an investigation—0%, 30%, and 60% completion—to see how well the system adapts as more information about a problem becomes available. The same ai evaluation question is explored in Beyond Aggregate Scores, which adds a research perspective.

Key Results

The experiments show that RAFT consistently outperforms both vanilla RAG and existing GraphRAG approaches. In the synthetic benchmark, RAFT achieved a significantly higher "Case Hit" rate—the ability to retrieve a truly relevant historical case—at every stage of the investigation. For example, at the initial symptom stage (0% progress), RAFT achieved an 84.2% Case Hit rate, compared to 67.3% for vanilla RAG. The results indicate that for troubleshooting tasks, focusing on the stateful progression of a case is more effective than the entity-centric graph approaches used in general-purpose RAG systems.

Considerations for Implementation

While RAFT shows strong performance, the authors note that effective retrieval in this domain requires careful handling of data. Real-world support cases are often noisy, containing irrelevant information that can distract an agent. RAFT addresses this by using a reviewer-based filtering process to ensure that only actionable, high-quality cases are indexed. Furthermore, the framework is designed to be modular, allowing organizations to apply metadata filters—such as product version or error code—to narrow the search space and ensure that retrieved guidance is relevant to the specific technical environment of the active case. The same ai evaluation question is explored in MAPLE, which adds a research perspective. as detailed in the full paper on Arxiv