Wrong Prediction, Right Answer: Recovering Evidence... | AI Research

Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores investigates why large language models (LLMs) often fail reasoning tasks. The authors, Qiyao Yan, Chenpeng Wang, and Liangming Pan, argue that these failures are frequently not due to a lack of reasoning capability, but rather a "late-stage output bottleneck" where structural biases in the model's final generation process overwrite correct internal logic.

The Gap Between Internal Logic and Output

The researchers observe a consistent "readout gap" across benchmarks like ProofWriter, ANLI, and FOLIO. While hidden-state probes can successfully decode correct answers from the model's internal layers, the model’s native sequence scoring—the process of calculating the probability of the final answer—often collapses. This collapse occurs because the model's unembedding matrix and softmax layer are susceptible to structural biases, such as a preference for frequent labels or specific prompt phrasing. Consequently, the model may compute the correct reasoning steps internally but fail to express them in its final output.

Diagnostic Protocol and Correction

To verify that the model possesses intact internal logic, the authors introduce a diagnostic protocol using a target-label-free additive correction. By fitting just two parameters on as few as 25 unlabeled examples, the researchers can shift the decision boundaries of the model's output. Because this intervention is mathematically restricted to a global shift, it cannot teach the model new reasoning skills. If accuracy improves after this adjustment, it serves as evidence that the correct logical alignment was already present in the model's native outputs and was merely obscured by scoring distortions.

Key Findings and Performance

Testing on Qwen3.5, OLMo-2-1B, and Llama-3.1-8B models, the authors found that this simple correction recovered between 9 and 34 accuracy points. These improvements were not the result of shallow word-matching; the recovered decisions persisted on "hard" instances that could not be solved by simple lexical overlap. Furthermore, the recovery was consistent across different reasoning environments, ranging from controlled symbolic deduction to natural-language benchmarks. The authors note that the correction is highly sample-efficient, with performance gains saturating quickly after using only a small number of unlabeled examples.

Implications for Benchmark Evaluation

The study suggests that current benchmark evaluations may be misleading if they equate a wrong final answer with an absence of reasoning capacity. The authors urge a narrower interpretation of these results, noting that many apparent zero-shot reasoning deficits are actually "expression failures." By distinguishing between a model's internal representation and its outward scoring, researchers can better identify whether a model truly lacks a capability or is simply failing to translate its internal knowledge into the required output format.