Back to AI Research

AI Research

DAMI initializes linear-attention memory for a wider range of retrieval distances

Key Takeaways

  • Joint decay-parameter initialization improves controlled retrieval benchmarks without adding trainable parameters.
  • ### Memory behavior starts before training
  • [Dual-Axis Memory Initialization](https://arxiv.org/abs/2610.41994), or DAMI, investigates how the starting values of memory-gating parameters affect long-context retrieval in linear-attention models.
  • The paper begins with controlled Gated DeltaNet experiments in which trained gate parameters remain strongly correlated with their initial values.
  • That motivates changing the initialization itself, rather than adding a new memory module after training.

Memory behavior starts before training

Dual-Axis Memory Initialization, or DAMI, investigates how the starting values of memory-gating parameters affect long-context retrieval in linear-attention models.
The paper begins with controlled Gated DeltaNet experiments in which trained gate parameters remain strongly correlated with their initial values. That motivates changing the initialization itself, rather than adding a new memory module after training.

Set different timescales and sensitivities

DAMI jointly configures two aspects of memory behavior. Across heads, it assigns geometric timescales to cover different retrieval distances. Across layers, it uses a low–high–low input-sensitivity profile.
A small sample of activations calibrates the Gated DeltaNet-family implementation to those targets without adding trainable parameters. The paper also describes architecture-specific extensions for Mamba-2 and RWKV-7, so readers should not assume every model receives an identical parameter substitution.
The controlled comparisons span GDN, KDA, GDN2, Mamba-2 and RWKV-7. The authors report average RULER-style retrieval gains of 5.29 to 16.71 percentage points over the corresponding random-initialization baselines. These are benchmark differences for the tested architectures, not percentage reductions in application error.

Keep retrieval gains separate from broader capability

The experiments use approximately 100-million-parameter models trained on five billion tokens. Language modeling, commonsense reasoning and shorter real-world retrieval remain broadly comparable to the baselines.
That qualification matters: improving a controlled long-distance retrieval task does not establish a uniform improvement in every use of context. The paper's contribution is a structured initialization method with targeted experimental evidence.
For a reproduction, compare the joint initialization with both a random baseline and the relevant single-axis alternatives under the same training budget. Measure retrieval by distance, then check whether the gain persists on the intended workload without harming the other capabilities that workload needs.

Comments