AI Glossary

Memory bandwidth

What does “Memory bandwidth” mean?

Memory bandwidth: The rate at which data can move to or from memory, usually measured in bytes per second.

Memory bandwidth visual guide: The rate at which data can move to or from memory, usually measured in bytes per second.
Visual guide Open full size (new tab)

What is an example of “Memory bandwidth”?

A language model's generation speed can be limited by reading its weights from memory.

Keep learning

See also

GPUTermTPUTermNPUTermCompute-bound workloadTermMemory-bound workloadTerm

Same topic

Matrix multiplicationTermMixed-precision trainingTermKV cacheTermModel compressionTermKernel fusionTerm