Back to AI Research

AI Research

When Shared Rollouts Fail in Defensive Driving Eval... | AI Research

Key Takeaways

  • When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit investigates a structural validity risk in the NAVSIM v2.2 benchmarking...
  • Defensive driving scores are useful only when they preserve distinctions between policies that observe surrounding actors and those that do not.
  • Re-simulation benchmarks may use reference-conditioned forgiveness, under which an agent receives credit when the logged human reference fails a compliance channel.
  • When agent and reference share an unstable rollout transformation, this rule can propagate shared reference failures into broad compliance credit.
  • We audit this risk in NAVSIM v2.2 original scene single-stage scoring.
Paper AbstractExpand

Defensive driving scores are useful only when they preserve distinctions between policies that observe surrounding actors and those that do not. Re-simulation benchmarks may use reference-conditioned forgiveness, under which an agent receives credit when the logged human reference fails a compliance channel. When agent and reference share an unstable rollout transformation, this rule can propagate shared reference failures into broad compliance credit. We audit this risk in NAVSIM v2.2 original scene single-stage scoring. Under the affected documented-stack condition on the audited numerical backend, the route-blind Ignore-All probe and a route-aware actor-blind probe outrank human replay and PDM-Closed over the complete 12,146-token navtest split. A fresh installation following the public specification reproduces rollout divergence on a fixed 32-token diagnostic set. A same-source dependency stack control and an exact-input diagnostic isolate dependency-sensitive numerical behavior in the shared velocity refit. On a 450-token control pool, replacing only the solver eliminates rollout divergence and restores blind-last ordering while keeping forgiveness enabled. Thus, the numerical instability is the direct trigger. Reference-conditioned forgiveness propagates the resulting shared reference failures into compliance credit. We contribute an audit protocol requiring score basis and stack disclosure, blind probes, overwrite reporting, and rollout stability tests before using such scores for defensive driving claims.

When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit investigates a structural validity risk in the NAVSIM v2.2 benchmarking framework. The researchers demonstrate that a specific scoring configuration can cause actor-blind driving policies to outrank sophisticated, actor-aware models, undermining the benchmark's ability to evaluate defensive driving competence.

The Problem of Shared-Failure Forgiveness

Defensive driving benchmarks often use "reference-conditioned forgiveness," a rule designed to prevent penalizing an agent for safety violations that also occur in the logged human reference data. The authors identify a phenomenon they call "shared-failure forgiveness collapse." This occurs when both the agent’s proposed trajectory and the human reference trajectory share an unstable numerical transformation—specifically a velocity refit process. When this shared process fails, it triggers the forgiveness rule, effectively converting structural failures into "pass" credits. This process erases the distinctions between policies that observe surrounding traffic and those that do not.

Audit Findings

The authors audited the NAVSIM v2.2 single-stage scoring configuration and found that it fails basic actor-blind sanity checks. On the 12,146-token navtest split, two actor-blind probes—one route-blind and one route-aware—outranked human replay and the PDM-Closed model.
The researchers isolated the cause by using a two-sided finite-difference diagnostic that bypassed the unstable shared refit. When this diagnostic was applied, the ranking reversed, placing human replay and PDM-Closed above the actor-blind probes. This confirms that the high scores for blind policies were not due to superior driving performance, but rather a numerical instability in the scorer’s shared rollout path that triggered widespread forgiveness.

Mechanism and Controls

The study identifies the default pseudoinverse used in the velocity refit as the direct trigger for the instability. Because the agent and the reference share this same path, the scorer frequently encounters "shared reference failures."
To validate this, the authors employed several controls:

  • Solver replacement: Replacing the solver eliminated rollout divergence and restored the expected "blind-last" ranking while keeping the forgiveness rule enabled.

  • Dependency-stack control: By changing binary dependencies to run the same scorer source, the researchers confirmed that the behavior is sensitive to the underlying numerical environment.

  • Overwrite reporting: The audit showed that for the route-blind "Ignore-All" probe, raw failures were overwritten as passes on over 99% of Drivable Area Compliance, Driving Direction Compliance, and lane-keeping evaluations.

Implications for Benchmarking

The authors argue that aggregate scores in re-simulation benchmarks are only credible if they preserve necessary distinctions between actor-aware and actor-blind policies. They contribute an audit protocol for future defensive driving evaluations, which includes:

  • Disclosure of the full score basis, including rollout transformations and forgiveness rules.

  • Implementation of blind probes to test for ranking anomalies.

  • Reporting of overwrite rates to identify when forgiveness is masking structural failures.

  • Rollout stability tests to ensure the scoring backend is not introducing dependency-sensitive numerical errors.

Comments (0)

No comments yet

Be the first to share your thoughts!