Relevance does not establish availability
RATED addresses a problem in agent tool selection: a tool can match a request semantically while being unavailable when called. The researchers propose combining semantic retrieval with an adaptation step informed by execution feedback.
The framework uses a dual encoder for matching and a lightweight contextual bandit to re-rank candidates. The paper describes this as a way to respond to runtime failures without expensive model retraining.
The benchmark includes changing reliability
The authors introduce EcoAgent, a benchmark containing 5,172 tools and more than 210,000 query-tool pairs. They evaluate retrieval and behavior under changing tool availability, rather than assuming every candidate remains callable.
The paper reports improvements over its comparison retriever and a lower fault-selection rate in the specified outage experiment. Those are results from the authors' setup. They should not be read as a guaranteed failure rate for a different tool catalog, outage pattern or agent application.
Test recovery with your own tool set
Franklin's interpretation is that runtime availability deserves an explicit place in a retrieval evaluation. Begin with a request that several tools can answer, then make one candidate unavailable in a controlled test. Observe whether the selector learns from the failure and chooses a suitable alternative.
Also measure whether the alternative actually completes the task. Avoiding an unavailable tool is helpful only if the replacement has the required capability and permissions. Keep relevance and availability results separate so one cannot conceal a weakness in the other.
Before adopting the approach, inspect the paper's feedback signals and outage conditions. Compare it with a simple availability-aware baseline on your own workload. The proposed mechanism is a research contribution to tool selection; it does not remove the need for error handling, limits and verification around the resulting action.
Comments