Back to AI Research

AI Research

STEPBENCH diagnoses missing, incorrect and misordered steps in AI-generated procedures

Key Takeaways

  • The benchmark uses 6,885 wikiHow procedures to separate different planning skills, with long step-ordering tasks proving difficult.
  • STEPBENCH evaluates whether AI systems understand the steps in a procedure, rather than judging only a complete plan’s final appearance.
  • Its diagnostic tasks are designed to expose different ways a sequence can fail.
  • The benchmark contains 6,885 verified wikiHow procedures across 81 categories.
  • From each procedure, the authors derive five tasks: four with deterministic answers involving withheld, incorrect or shuffled steps, and one that asks for the next step.

STEPBENCH evaluates whether AI systems understand the steps in a procedure, rather than judging only a complete plan’s final appearance. Its diagnostic tasks are designed to expose different ways a sequence can fail.

Five tasks from shared procedures

The benchmark contains 6,885 verified wikiHow procedures across 81 categories. From each procedure, the authors derive five tasks: four with deterministic answers involving withheld, incorrect or shuffled steps, and one that asks for the next step.
Using the same underlying procedures makes it possible to compare these capabilities without changing the subject matter each time. The study also examines full-plan generation, using STEPEVAL to identify failures that would prevent completion of the goal.

Ordering is a particular weakness

Across fourteen configurations of twelve models, the strongest step-ordering result was 63.0%. For procedures with nine or more steps, it fell to 33.1%.
The authors also find that incorrect-step detection predicts judged plan validity across models. That suggests a potentially cheaper diagnostic, though the reported relationship does not mean detecting an error is identical to executing a correct plan.
A separate set of 214 procedures published in 2026 produced stable comprehension scores and rankings. The authors interpret this as suggesting limited influence from memorisation, rather than proving that all contamination concerns have been eliminated.

What an applied evaluation should preserve

Franklin’s interpretation is that a planning assessment should record the kind of mistake, not merely whether a reviewer liked the answer. Missing a prerequisite, inserting an incorrect action and swapping two steps can require different fixes.
For a practical workflow, inspect dependencies and failure-critical steps before treating fluent instructions as usable. The benchmark’s judged procedures are also different from validating actions through execution, especially where safety, specialised equipment or local conditions matter.
Read the STEPBENCH paper for the task construction and the relationship between diagnostic scores and plan validity.

Comments