Back to AI Research

AI Research

TIDE turns benchmark results into a tree of behavioral differences

Key Takeaways

  • A diagnostic framework uses model behavior to select capability splits, exposing contrasts that disappear in aggregate benchmark scores.
  • Tree-Induced Diagnostic Evaluation, or TIDE, asks what a benchmark’s overall score hides.
  • Two models can achieve similar aggregate accuracy while succeeding on different kinds of questions.
  • The framework builds a hierarchy of behavioral distinctions to make those differences easier to inspect.
  • ## Build categories from observed behavior

Tree-Induced Diagnostic Evaluation, or TIDE, asks what a benchmark’s overall score hides. Two models can achieve similar aggregate accuracy while succeeding on different kinds of questions. The framework builds a hierarchy of behavioral distinctions to make those differences easier to inspect.

Build categories from observed behavior

The TIDE paper starts with benchmark samples and model scores. At each node, a language-model judge proposes candidate divisions and assigns samples to groups. Behavior-based information gain then selects among the proposed splits.
The result is a capability tree rather than one manually chosen list of categories. That can reveal useful slices of performance, but it also means the diagnostic structure depends on the judge’s grouping and the evidence available in the benchmark. A meaningful label still needs examples that justify what it claims to measure.

Aggregate rankings can conceal reversals

The authors apply TIDE to five models across four benchmarks. One reported contrast has Claude Opus 4.6 ahead of GPT-5.4 on a microeconomic-market-reasoning slice, while GPT-5.4 leads on a contract-law-reasoning slice.
Those percentages describe the paper’s particular benchmark groups. They should not become a general recommendation that one model is always better for economics and the other for law. A slice’s size, composition and construction matter when interpreting a seemingly large difference.

Use the tree to ask better questions

For a team evaluating models, the practical use is to locate a failure pattern and inspect the underlying cases. A diagnostic tree can suggest where to expand testing, collect more examples or reconsider a broad aggregate ranking.
The paper presents a way to extract more information from existing evaluation results, rather than a replacement for verifying individual answers. Its strongest contribution is making behavioral contrasts visible. The next step is to check that the identified categories remain useful on new tasks and do not merely describe the accidental composition of one benchmark.

Comments