CUSUM-Shaped Inference-Time Monitoring and Targeted...

CUSUM-Shaped Inference-Time Monitoring and Targeted... | AI Research

Key Takeaways

CUSUM-Shaped Inference-Time Monitoring and Targeted Re-Decoding for Quantized Small Language Model Reasoning This paper introduces MGT-B (Monitoring-Guided T...
Quantized small autoregressive reasoning models can enter long, repetitive, or unproductive trajectories, yet inference-time compute is usually allocated without observing how a trajectory develops.
On this set, accuracy changes from 82/240 to 88/240 (+2.50 percentage points; 13 corrections, 7 regressions; exact McNemar p = 0.2632; paired bootstrap 95% interval [-1.25, +6.25]).
All 316 no-alarm outputs in the 467-pair set are identical to vanilla, while the 151 alarmed trajectories contain 29 corrections and 8 regressions.
Neither analysis is confirmatory, and the empirical factors are not established as a valid e-process or e-detector.

Paper AbstractExpand

Quantized small autoregressive reasoning models can enter long, repetitive, or unproductive trajectories, yet inference-time compute is usually allocated without observing how a trajectory develops. Building on an earlier token-level e-CUSUM controller, we develop MGT-B (Monitoring-Guided Test-time Backtracking), a revised external controller that maps overlapping windows of pre-sampling uncertainty and degeneration features to position-conditional empirical tail probabilities, accumulates mixture betting factors with a CUSUM-shaped reset, and responds to an alarm by estimating a rollback point, restoring token and key-value-cache state, and performing constrained re-decoding. To audit whether the effect persists on problem identities first observed after the manual choice of log threshold h = 10, we retrospectively exclude 260 IDs present in pre-threshold artifacts and retain the chronologically first post-threshold pair for each remaining ID, yielding a 240-pair chronology-audit set. On this set, accuracy changes from 82/240 to 88/240 (+2.50 percentage points; 13 corrections, 7 regressions; exact McNemar p = 0.2632; paired bootstrap 95% interval [-1.25, +6.25]). A broader 467-pair historical-coverage set of seed-matched pairs changes accuracy from 146/467 to 167/467 (+4.50 points; McNemar p = 0.000753), but includes 200 seed-1 IDs available before or during threshold selection and is reported only as an exploratory estimate. All 316 no-alarm outputs in the 467-pair set are identical to vanilla, while the 151 alarmed trajectories contain 29 corrections and 8 regressions. Neither analysis is confirmatory, and the empirical factors are not established as a valid e-process or e-detector. The results support a selective monitoring-and-repair mechanism for the studied MATH-500 setting, rather than a general or theoretically certified reasoning improvement.

CUSUM-Shaped Inference-Time Monitoring and Targeted Re-Decoding for Quantized Small Language Model Reasoning
This paper introduces MGT-B (Monitoring-Guided Test-time Backtracking), a system designed to improve the reasoning accuracy of small, quantized language models. These models often struggle with long, repetitive, or unproductive reasoning chains. Instead of simply generating an answer and hoping for the best, MGT-B monitors the model's output in real-time. If it detects signs that the model is entering a "loop" or a low-quality reasoning path, it triggers an intervention to roll back the generation and attempt a more successful path using constrained re-decoding.

How the Monitoring Works

The system does not rely on a single metric to detect errors. Instead, it tracks several features simultaneously, including the model's uncertainty (entropy), the probability of the chosen tokens, and the frequency of repeated n-grams. These features are aggregated over overlapping windows of text. The system uses a "CUSUM-shaped" (Cumulative Sum) statistic, which is a method for detecting changes in a sequence. By accumulating evidence over time, the monitor can identify when a trajectory has drifted into an unproductive state. When this statistic crosses a specific threshold, the system sounds an alarm.

The Intervention Process

When an alarm is triggered, MGT-B performs a targeted repair. It estimates a "rollback point"—a position in the text where the reasoning likely went off track—and clears the model's current state, including its key-value cache and internal monitoring data. It then restarts the generation from that earlier point, but with stricter settings: it lowers the temperature (making the model more deterministic), applies a penalty to discourage repetition, and explicitly blocks the n-grams that were identified as problematic. This allows the model a second chance to generate a coherent reasoning path.

Results and Performance

The researchers evaluated MGT-B on the MATH-500 dataset. In a "chronology-audit" set of 240 problems, the model’s accuracy improved from 82/240 to 88/240. A broader exploratory set of 467 problems showed a larger increase in accuracy, though the authors note this set was influenced by the manual selection of the alarm threshold and should be viewed with caution. Across the 467-pair set, 151 trajectories triggered an alarm, resulting in 29 corrections and 8 regressions, while all 316 trajectories that did not trigger an alarm remained identical to the original model output.

Important Considerations

The authors emphasize that this research is not a definitive proof of a general reasoning improvement. Because the threshold for triggering an alarm was chosen manually and the system was tested on specific data, the results are not considered confirmatory. Furthermore, the statistical methods used—such as the empirical tail probabilities—are not established as theoretically certified "e-processes." The study serves as an exploration of a selective monitoring-and-repair mechanism for quantized models, rather than a universal solution for all language model reasoning errors.