NVIDIA Releases Gated DeltaNet-2 for Efficient Recurrent Memory

NVIDIA AI has introduced Gated DeltaNet-2, a new linear attention layer designed to improve how recurrent models manage memory. By decoupling the processes of erasing old information and writing new data, the model addresses a significant bottleneck in delta-rule architectures. Trained at 1.3B parameters on 100B FineWeb-Edu tokens, Gated DeltaNet-2 demonstrates superior performance across language modeling, commonsense reasoning, and long-context retrieval tasks compared to existing models like Mamba-2, Gated DeltaNet, KDA, and Mamba-3.

Refining the Delta Rule

In traditional recurrent linear attention, models often rely on a single scalar gate to manage memory updates. This scalar typically controls both the erasure of existing content on the key side and the commitment of new content on the value side. Because these two operations act on different axes of the state, tying them to a single scalar restricts the model's flexibility.
Gated DeltaNet-2 resolves this by implementing the Gated Delta Rule-2, which introduces two distinct channel-wise gates. A channel-wise erase gate operates on the key axis to determine what is removed from the state, while a channel-wise write gate manages the value axis for new information. By using sigmoid projections of the token representation to generate these gates, the model allows for more granular control over memory edits, effectively preserving the delta-rule write direction while enabling channel-selective updates.

Efficient Training and Architecture

To maintain computational efficiency, Gated DeltaNet-2 utilizes a chunkwise WY form that allows for parallel training. The implementation leverages fused Triton kernels to handle the recurrence, with the backward pass explicitly deriving a gate-aware vector-Jacobian product to account for the separate erase and write gates. This design ensures that the model remains efficient even as it adds complexity to the update rule.
The architecture is integrated into a standard Transformer-style block, utilizing linear projections, short causal convolutions, and normalization layers. For applications requiring high-precision local interactions, a hybrid variant is available that incorporates Sliding-Window Attention (SWA). In this configuration, the recurrent mixer handles the compression of long histories, while SWA manages exact local interactions, maintaining linear sequence scaling with a bounded attention cache.

Performance Gains

Experimental results highlight the effectiveness of the decoupled gate design. When matched against baselines with identical recurrent state sizes, Gated DeltaNet-2 achieves the highest average scores in both recurrent and hybrid settings for language modeling and commonsense reasoning.
The most significant improvements are observed in long-context retrieval tasks. On the RULER benchmark, Gated DeltaNet-2 shows substantial gains over KDA, with S-NIAH-3 performance at 2K rising from 63.2 to 89.8, and MK-NIAH-1 performance at 4K increasing from 28.0 to 37.8. These results suggest that the enhanced update rule provides a more effective way to manage and retrieve information from compressed memory states.