State space model
An alternative to transformers that reads long sequences in linear time by carrying a compact running memory.
What it is
State space models (SSMs), such as Mamba, process a sequence step by step while keeping a fixed-size internal state, instead of comparing every token with every other token like attention does. That makes them efficient on very long inputs. Many recent models mix SSM layers with attention layers.
How it works
Each layer updates a hidden state from the previous state and the new input using learned, input-dependent rules, then produces an output. The computation can be run as a fast parallel scan during training and as a cheap recurrence during generation, so memory does not grow with length.
What it is good for
- Very long documents, audio or genomic sequences
- Fast generation with constant memory
- Hybrid models that combine SSM and attention
Limits to know
- Weaker than attention at exact recall from far back
- Smaller ecosystem and tooling
- Usually paired with some attention layers
