Back to AI Research

AI Research

TRAJDEBUG: Tracing Error Lifecycle to Identify Crit... | AI Research

Key Takeaways

  • TRAJDEBUG is a framework designed to identify the specific, earliest error in a long-horizon agent trajectory that causes a final task failure.
  • LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging.
  • Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure.
  • However, progress faces two main challenges.
  • First, long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context.
Paper AbstractExpand

LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure. However, progress faces two main challenges. First, long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context. Second, failed trajectories often contain multiple local errors with different downstream effects, only some of which remain responsible for the final failure. In this work, we propose TrajDebug, an error-lifecycle tracing framework that addresses long-trajectory error discovery with multi-granularity history compression and evidence-based error identification, and supports critical attribution by tracing each error's resolution status and terminal impact. We further construct TrajErrBench, a benchmark of 486 manually annotated failed trajectories from Tau2Bench and SWE-Bench Pro, covering realistic tool-use and coding scenarios. Experiments across diverse agent benchmarks show that TrajDebug achieves the best overall performance over existing baselines, and application studies further demonstrate that its diagnoses provide actionable feedback for improving downstream agent success. We will release the codes and data to facilitate further research.

TRAJDEBUG is a framework designed to identify the specific, earliest error in a long-horizon agent trajectory that causes a final task failure. By tracing the lifecycle of errors—from their initial trigger to their downstream impact—the system helps developers pinpoint why an agent failed, rather than getting lost in the multiple, often inconsequential, mistakes that occur during complex tasks.

The Challenge of Long-Horizon Debugging

LLM-based agents often perform hundreds of steps involving planning, tool use, and environment feedback. When these agents fail, identifying the root cause is difficult because evidence is scattered across distant instructions and observations. Furthermore, trajectories often contain many "local" errors—some are fixed by the agent, some are harmless, and others are merely symptoms of a deeper issue. Existing methods often struggle to distinguish these minor mistakes from the single, critical error responsible for the final failure.

How TrajDebug Works

The framework processes trajectories in three distinct stages:

  • Multi-Granularity Compression: To manage long contexts, the system creates three views of each step: high-detail (for local verification), medium-detail (for intent and action), and low-detail (for coarse progress). This allows the model to maintain necessary evidence while keeping the overall context manageable.

  • Error Trigger Detection: The system identifies "triggers," which are mismatches between the agent's actions and task instructions, history, or environment feedback. Each trigger must be supported by citable evidence to prevent the model from hallucinating errors.

  • Lifecycle Classification: Related triggers are grouped into "error instances." The system then classifies these instances based on whether they were resolved or if they left a "terminal footprint," such as an irreversible state change or a persistent violation. Only instances that remain relevant to the final failure are passed to the final stage for causal attribution.

TrajErrBench: A New Benchmark

To evaluate the system, the researchers introduced TrajErrBench, a dataset of 486 manually annotated failed trajectories. This benchmark includes 400 cases from τ2-Bench and 86 cases from SWE-Bench Pro, covering diverse scenarios like tool use and complex software engineering tasks. The average length of the SWE-Bench Pro trajectories is approximately 119.7 steps, providing a rigorous test for long-horizon error detection.

Results and Practical Application

Experiments show that TrajDebug outperforms existing prompting baselines and diagnostic systems in identifying critical errors. Beyond simple diagnosis, the researchers tested the framework in two practical scenarios:

  • Targeted Guidance: Using diagnoses to provide feedback before re-executing a task improved success rates by 10.80% on average.

  • Failure Memory: Aggregating diagnoses from historical failures into a reusable memory bank for new tasks yielded a 5.70% average improvement.
    These findings suggest that TrajDebug functions as both a diagnostic tool and a method for converting past failures into actionable experience for future agent performance.

Comments (0)

No comments yet

Be the first to share your thoughts!