A root-cause-analysis preprint compares simple language-model harnesses with specialised agents for investigating microservice failures. The authors ask whether hand-crafted investigation steps improve results over letting a capable model choose how to examine the evidence.
A shared set of failure cases
The study uses RCAtlas, a collection of 90 validated cases from eight public datasets. It compares three simple harnesses with five published LLM root-cause agents, alongside statistical and causal methods, across six models and three reasoning-effort settings.
The simple harnesses include one call over an evidence package, a SQL agent and the Codex coding agent. They differ in the tools and interaction available, but they do not prescribe every investigative call in advance.
Accuracy and investigation freedom
The authors report PR@1 of 65–77% for simple harnesses on three mid-tier models. Published agents scored 16–53% and non-LLM methods 6–39% in the reported comparisons. The simple harnesses led on every model where both agent families were run.
A case study describes fixed procedures skipping decisive evidence. That offers an explanation for a particular failure mode, while the broader results remain specific to this case collection and tested configurations. They do not prove that all structure is harmful in every operational environment.
More reasoning also costs time
The paper reports that additional reasoning effort changed accuracy by only a few points at most while increasing runtime by as much as sevenfold. A stronger model mainly improved reliability, while multi-turn interaction mainly improved aptitude in the authors’ analysis.
For an engineering team, the useful next comparison is a flexible baseline against its existing runbook agent on the same incidents. Record evidence found, accepted diagnoses and investigation time together. The paper’s results make that baseline worth including before assuming a more elaborate agent design is better.
Comments