Penelope: Localized Latent Recurrence for Efficient...

Penelope: Localized Latent Recurrence for Efficient... | AI Research

Key Takeaways

Penelope is a framework designed to improve the efficiency of structured reasoning in decoder-only Transformers.
Complex structured reasoning tasks often require additional computation, yet current language models obtain it mainly by increasing parameter scale or by serializing intermediate steps as chain-of-thought (CoT) tokens.
The former raises training and deployment costs, while the latter ties reasoning computation to autoregressive output length.
We introduce Penelope, an efficient latent-reasoning framework for pretrained decoder-only Transformers that localizes recurrent computation to a selected decoder interval.
The lower decoder prefix is evaluated once to construct a problem-conditioned boundary memory, which is then iteratively refined through time-modulated GRU dynamics and recurrent readout states before answer generation.

Paper AbstractExpand

Complex structured reasoning tasks often require additional computation, yet current language models obtain it mainly by increasing parameter scale or by serializing intermediate steps as chain-of-thought (CoT) tokens. The former raises training and deployment costs, while the latter ties reasoning computation to autoregressive output length. We introduce Penelope, an efficient latent-reasoning framework for pretrained decoder-only Transformers that localizes recurrent computation to a selected decoder interval. The lower decoder prefix is evaluated once to construct a problem-conditioned boundary memory, which is then iteratively refined through time-modulated GRU dynamics and recurrent readout states before answer generation. A progressive CoT-to-latent curriculum transfers visible reasoning into this internal recurrent path, allowing additional computation to be allocated in latent space without repeatedly executing the complete decoder or generating a long intermediate trace. Experiments on open-source structured-reasoning benchmarks show that, at validation-selected latent budgets, Penelope attains competitive accuracy relative to established latent-reasoning models while reducing measured inference latency. These results show that latent refinement can be localized to a narrow decoder interval, reducing repeated full-decoder execution without generating a long visible reasoning trace and providing a practical accuracy-efficiency tradeoff for decoder-only Transformer models.

Penelope is a framework designed to improve the efficiency of structured reasoning in decoder-only Transformers. Current models typically achieve reasoning by either increasing parameter scale or by generating long, explicit chains of thought (CoT) that increase output length. Penelope instead performs reasoning within the model's internal latent space, localizing recurrent computation to a specific, narrow interval of the decoder to reduce the need for repeated full-model execution.

How Penelope Works

The framework decomposes a Transformer into three parts: a lower prefix that runs once to establish a problem-conditioned foundation, a shared output-side interval that performs iterative refinements, and a final interface that prepares the state for answer generation.
Instead of re-running the entire decoder to refine reasoning, Penelope caches the prompt-side context and repeatedly applies only the selected decoder interval. This process uses a fixed-size latent memory and recurrent readout states. A time-modulated GRU (Gated Recurrent Unit) updates these states over $K$ steps, allowing the model to refine its internal representation of the problem without generating a long, visible textual trace.

Training and Reasoning

Penelope utilizes a progressive curriculum to transfer reasoning capabilities from explicit CoT tokens into its internal recurrent path. During training, the model is taught to replace visible reasoning steps with latent refinements. As training progresses, the model learns to consolidate its reasoning into the latent interface, eventually reaching a stage where it can generate answers directly from the refined latent state. This approach allows the model to allocate additional computation to the latent space while maintaining compatibility with standard autoregressive answer generation.

Efficiency and Performance

According to the authors, Penelope provides a practical trade-off between accuracy and inference latency. By limiting recurrent computation to a narrow decoder interval, the marginal cost of adding reasoning steps is reduced compared to full-decoder recurrence. Experiments on structured-reasoning benchmarks—including Deep ListOps, ProsQA, and PrOntoQA—show that Penelope achieves competitive accuracy relative to established latent-reasoning models while reducing the number of sequential decoder-layer applications.

Considerations

The authors note that the effectiveness of this approach depends on the specific decoder interval chosen for recurrence. While the evaluated output-side configuration demonstrates efficiency gains, the optimal placement of this interval remains dependent on the specific backbone model being used. Furthermore, while Penelope reduces the serial decoder depth required for reasoning, the total parameter count and memory footprint differ from other methods due to the inclusion of the recurrent modules, memory, and readout states.