Back to AI Research

AI Research

RTL repair scores can change meaning when reset semantics change

Key Takeaways

  • A benchmark audit shows why hardware-repair evaluations need explicit scoring semantics and a specification-only baseline.
  • A study of register-transfer-level repair asks whether benchmark scores actually measure a model’s ability to fix faulty hardware descriptions.
  • The authors build a dataset and evaluation harness, then use them to investigate how scoring choices and access to the specification affect the apparent result.
  • ## The benchmark separates types of faults
  • The [RTL repair paper](https://arxiv.org/abs/2610.28595) introduces 5,784 broken-and-fixed training pairs and 459 held-out cases.

A study of register-transfer-level repair asks whether benchmark scores actually measure a model’s ability to fix faulty hardware descriptions. The authors build a dataset and evaluation harness, then use them to investigate how scoring choices and access to the specification affect the apparent result.

The benchmark separates types of faults

The RTL repair paper introduces 5,784 broken-and-fixed training pairs and 459 held-out cases. The held-out set contains 374 syntactic mutations and 85 behavioral mutations, with support for simulation and formal-equivalence evaluation.
That distinction matters because repairing syntax and recovering the intended behavior are different accomplishments. A model that produces code accepted by a parser has not necessarily restored the design described in the specification.

Scoring can reverse the apparent effect

In one evaluated cohort, simulation and formal scoring give opposite signs for the apparent benefit of supplying a specification. Reset-aware rescoring attributes almost all of that disagreement to reset and first-cycle comparison semantics.
The finding is a measurement warning. If two scoring systems interpret initialization differently, a change in score may reflect the evaluation boundary rather than a clean improvement in repair ability. The paper therefore calls for explicit reporting of initialization, reset, first-cycle and unknown-value handling.

Test whether the faulty code mattered

The authors also examine 274 mined model failures and compare specification-only recovery with recovery conditioned on the faulty RTL. The direction and size of the difference vary across six models.
A conventional repair score can consequently mix several abilities: interpreting the specification, generating a fresh solution and using the defective implementation. Including a specification-only baseline helps separate them. The proposed controls also disclose oracle coverage and undecided cases.
For readers comparing coding systems, the useful question is what evidence a passing case actually supplies. These results concern the constructed hardware-repair tasks and scoring setups; they do not establish that every RTL benchmark is invalid or that one evaluation method always settles correctness.

Comments