VAKRA: Evaluating Multi-Hop Reasoning Across APIs a... | AI Research

VAKRA (eValuating API and Knowledge Retrieval Agents) is a benchmark designed to measure how well AI agents perform multi-step reasoning when they must interact with both structured APIs and unstructured document collections. While previous benchmarks have evaluated tool use, document retrieval, or policy adherence in isolation, VAKRA tests these capabilities together in complex, enterprise-style workflows that require agents to navigate multiple sources of information to solve a single request.

The Challenge of Multi-Source Reasoning

The researchers at IBM developed VAKRA to address the gap between existing benchmarks and the requirements of real-world enterprise deployment. In these settings, an agent might need to resolve a customer complaint by first identifying a record in a CRM, extracting tracking data from a document, and then using that information to query a logistics API—all while adhering to specific natural-language policies. VAKRA provides over 8,000 executable APIs across 62 domains, paired with domain-aligned document collections, to simulate these heterogeneous environments.

How VAKRA Evaluates Agents

To ensure a fair comparison, the researchers used a fixed ReAct harness, which forces models to follow a consistent "reason-act-observe" loop. This isolates the model's inherent reasoning capabilities from the specific agent architecture. The benchmark evaluates performance across three increasingly difficult settings:

• API Interaction Styles: Testing how models handle different interface abstractions, ranging from general-purpose tools to highly specific endpoints.

• Multi-Hop Reasoning: Requiring agents to chain 2–5 steps where the output of one tool call determines the input for the next.

• Multi-Source Reasoning with Policies: Requiring agents to integrate information from both APIs and documents while following natural-language constraints that dictate which sources are permissible.
  Correctness is verified by re-executing the agent's predicted tool calls against live, locally-hosted databases and indices. This allows the benchmark to accept multiple valid paths to a solution rather than forcing a single, rigid sequence.

Performance and Bottlenecks

The evaluation of various frontier and open-weight models revealed significant performance gaps. Even the strongest model tested, GPT-5.5, achieved 70.4% accuracy on single-hop tasks, but this dropped to approximately 50% for more complex, compositional API tasks. The researchers observed that performance degrades by more than 50% as the depth of reasoning increases.
Franklin analysis: The data indicates that the primary bottleneck for these agents is not the mechanics of invoking a tool, but rather the language-mediated reasoning required to use them. Trace analysis shows that failures frequently occur during entity disambiguation, cross-source grounding, and schema alignment—the steps where the agent must interpret and reconcile information from different systems.

Limitations in Policy Adherence

A notable finding is the difficulty models have with policy-constrained queries. When agents were asked to follow natural-language rules regarding which tools or sources they were permitted to use, performance was poor, with accuracy on unanswerable queries falling as low as 2.4%. This suggests that while current models are becoming more proficient at basic tool interaction, they struggle to reliably interpret and apply abstract constraints within a multi-step reasoning process.