Back to AI Research

AI Research

Real-Time Detection and Repair of LLM Agent Failures | AI Research

Key Takeaways

  • Real-Time Detection and Repair of LLM Agent Failures, authored by Sunny Dubey, introduces a lightweight, high-speed monitoring system designed to detect and...
  • We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs.
  • Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at =9), predicting its own failure region out of sample on AFTraj-2K.
  • Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779).
  • Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate.
Paper AbstractExpand

LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run&#39;s stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor&#39;s 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.

Real-Time Detection and Repair of LLM Agent Failures, authored by Sunny Dubey, introduces a lightweight, high-speed monitoring system designed to detect and repair LLM agent failures mid-episode. By analyzing observable step telemetry—such as semantic embeddings, token uncertainty, and action metadata—the system identifies issues like looping, goal drift, and tool errors without requiring a second LLM to judge every step, which the paper notes is often more expensive than the agent itself.

How the Monitoring System Works

The system uses a one-class echo-state-network (ESN) ensemble that is trained only on healthy agent runs. It functions as a "watchdog" that monitors the agent's telemetry stream in real time. Because the reservoir computing component is fixed at initialization and only requires a ridge readout to be fitted, the monitor can score a step in approximately 200 microseconds. This is three orders of magnitude faster than using an LLM-based auditor.
To address specific failure modes, the system employs three layers:

  • Statistical Monitors: An ESN ensemble that uses CUSUM alarms to detect persistent shifts in behavior.

  • Deterministic Verification: A layer that recomputes a run’s total from received tool results and confirms required calls were made. This layer requires no calibration and is used to catch fabrication and coverage errors.

  • Repair Mechanism: When the monitor flags a run, the system rolls the agent back to its last fact-gathering step and re-runs it.

Performance and Results

The research validates this approach across 2,823 agent episodes using three frameworks, three local model families (Qwen2.5, Llama3.1), and the Gemini-2.5-flash API. Key findings include:

  • Detection Capability: The system recovers 45% of failures compared to a 16% resampling control, increasing overall task success from 52% to 73% at the cost of roughly one extra model call per run.

  • Efficiency: The entire monitoring path operates in under one millisecond per step.

  • Deterministic Accuracy: The deterministic verification layer caught 60% of failures (96% when including coverage checks) with zero false positives across 63 test episodes.

Limitations and Considerations

The paper emphasizes that these monitors are not "plug-and-play" across different environments. A primary limitation is that monitors calibrated on one deployment often perform at chance levels when moved to another without recalibration.
Furthermore, the monitors have specific blind spots:

  • Content Corruption: Statistical monitors struggle with content-based failures, such as malformed JSON or incorrect documents, which necessitated the addition of a content-grounding telemetry channel.

  • Organic Failures: The monitors transfer only weakly to organic (non-injected) failures, such as fabrications, which the author notes are better handled by the deterministic verifier.

  • Horizon Dependency: The ESN’s advantage over simpler, memoryless baselines is a monotone function of the post-onset horizon. If a failure ends too quickly after it begins, the ESN has less time to integrate evidence, making simpler methods more competitive.

Why This Matters

This research provides a framework for making LLM agents more reliable without the prohibitive costs associated with constant LLM-based auditing. By separating behavioral monitoring (statistical) from factual verification (deterministic), the system offers a tiered approach to safety that balances speed, cost, and accuracy. The author notes that while internal-state probes might offer earlier detection, this telemetry-based approach is intended for deployments where the user does not have access to the model's internal weights.

Comments