CTBench: Evaluating Troubleshooting Capabilities of...

CTBench: Evaluating Troubleshooting Capabilities of... | AI Research

Key Takeaways

CTBench is a public benchmark designed to evaluate how well AI agents perform as telecom troubleshooting engineers.
However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces.
In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer.
CTBench focuses on root cause analysis and path restoration.
Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps.

Paper AbstractExpand

Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.

CTBench is a public benchmark designed to evaluate how well AI agents perform as telecom troubleshooting engineers. Developed by researchers from Huawei Technologies and Queen Mary University of London, the benchmark assesses an agent's ability to diagnose network faults and restore service paths within complex, partially observable telecom environments that feature diverse vendors, devices, and protocols.

Evaluating Operational Competence

Existing benchmarks for AI agents often assume full network visibility or focus on static question-answering, which does not reflect the reality of network operations and maintenance (NetO&M). In practice, engineers must perform sequential decision-making, such as forming hypotheses, selecting diagnostic actions, and inspecting command outputs. CTBench addresses this by requiring agents to operate under partial observability, where they must navigate heterogeneous interfaces and justify their conclusions with evidence rather than just providing a final answer.

Benchmark Tasks and Data

The benchmark consists of 234 expert-curated tasks: 126 for Root Cause Analysis (RCA) and 108 for path restoration. These tasks were constructed by 15 senior telecom experts and validated by independent reviewers to ensure they reflect real-world operational challenges. Each task includes:

Golden Evidence Steps: A set of expert-validated actions required to solve the problem, allowing for the evaluation of the agent's diagnostic process.
Rich Metadata: Annotations covering fault-propagation chains, protocol complexity, device heterogeneity, and evidence observability. This metadata allows for fine-grained analysis of why an agent might succeed or fail in specific scenarios.

Evaluation Metrics

CTBench moves beyond simple accuracy by using expert-grounded metrics that evaluate the entire diagnostic trajectory. For RCA, the benchmark measures:

RCA-Loc: The ability to identify the affected node and object.
RCA-ID: The ability to assign the correct root-cause label.
RCA-Evidence: The F1 score of the agent’s actions compared to the golden evidence steps.
Similar metrics are applied to path-restoration tasks, focusing on endpoint identification, path reconstruction, and the use of required diagnostic evidence.

Performance and Limitations

Experiments with representative agent-model combinations revealed that while current state-of-the-art agents are effective at identifying endpoints in path-restoration tasks, they generally underperform in root cause analysis. Agents struggled specifically with interface state, link-layer, and service-management faults.
A key finding is that agents often fail to provide the evidence-grounded diagnoses necessary for operational trust, even when they arrive at a correct final answer. Furthermore, the results indicate that while path-restoration tasks are more resource-intensive, increased resource usage does not necessarily lead to better diagnostic performance.