Comparative Approaches to Agent Retrieval over Larg... | AI Research

Comparative Approaches to Agent Retrieval over Large Skill Libraries explores how AI agents can efficiently access specific capabilities from a large library of 690 skills without the high cost of loading every file into context. Researchers Indivara Kolluru and Nathan Sportsman of Praetorian compare two retrieval methods: a hybrid ranker that uses lexical and dense-embedding search, and a typed knowledge graph that maps workflow relationships like prerequisites and data flow.

The Retrieval Problem

Loading an entire library of skills is computationally expensive, costing approximately 46,900 tokens per task. To solve this, the authors developed a "pull" route where an agent queries a library to retrieve only the necessary skills. The study evaluates whether adding a knowledge graph—which encodes structural dependencies—improves retrieval accuracy compared to a standard hybrid ranker that relies on keyword and embedding similarity.

How the Systems Work

The hybrid ranker uses BM25 lexical scoring combined with MiniLM vector similarity to identify relevant skills. The knowledge graph, by contrast, uses an LLM-based pipeline to generate 1,421 typed edges (such as "requires" or "feeds_into") between skills. This graph generation process is designed to be cost-effective, using a "coarse-to-fine" approach where the system first identifies embedding neighbors and then uses an LLM to assign specific relationship types.

Key Findings and Limitations

The study concludes that the knowledge graph does not improve retrieval performance over the hybrid ranker. Several factors contribute to this:

• The Pre-filter Topology Bound: The authors identified that because the graph’s candidate edges are drawn from the same embedding neighbors the ranker already searches, the graph is confined to the embedding’s existing topology. The LLM can add semantic labels to these connections, but it cannot extend the reach of the retrieval system.

• Performance Deficit: At a matched token budget, substituting graph neighbors for additional ranked results resulted in a performance decrease of 11.2 points.

• Redundancy: The LLM-generated edge layer provided no measurable improvement over the "free" neighbors already obtained from a local embedding pass.

• Unreachable Targets: 73% of the queries missed by the ranker were found to be unreachable through the graph entirely, suggesting that structural improvements cannot compensate for fundamental gaps in the retrieval signal.

The Impact of Evaluation Methods

The researchers found that the choice of query set significantly impacts reported accuracy. When they evaluated their system using a set of 37 queries written by the authors while they were reading the corpus, the hit@5 accuracy was 44 points higher than when using a set of 117 "non-echoing" queries. This inflation occurs because author-written queries often reuse the vocabulary found in the skill descriptions, effectively turning the retrieval task into a simple keyword match. The authors argue that this methodological finding is critical, as a standard author-written evaluation would have masked the system's actual performance gaps.