Attention

Mamba / state-space block (SSM)

A recurrent state-space scan replaces attention: constant memory per token, linear time in sequence length.

First page of Mamba: Linear-Time Sequence Modeling with Selective State SpacesIntroduced inDec 2023Mamba: Linear-Time Sequence Modeling with Selective State SpacesGu & Dao · arXiv 2312.00752 ↗
Attentionthe KV cache grows with every tokenSelective state (Mamba)conv, then a gated scan rewrites SScost per token growscost stays constant

State-space blocks (Mamba-2’s SSD scan here) carry information through a fixed-size hidden state that is updated token by token, instead of comparing every token against every other one. That makes both compute and memory linear in sequence length. Hybrid models such as Nemotron-H use mostly Mamba blocks with a few attention layers mixed in for tasks that need exact token recall.

Adoption over time

Share of new models that have included Mamba/SSM blocks over time.

2%202220232024202520262%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4text-generation · ↓ 2.3M · ♡ 420Open in visualizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 architecture graphnvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4any-to-any · ↓ 1.5M · ♡ 177Open in visualizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 architecture graphnvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8any-to-any · ↓ 1.3M · ♡ 61Open in visualizer ai21labs/Jamba-tiny-dev architecture graphai21labs/Jamba-tiny-dev↓ 1.2M · ♡ 14Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16text-generation · ↓ 956k · ♡ 811Open in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16text-generation · ↓ 865k · ♡ 417Open in visualizer state-spaces/mamba-130m-hf architecture graphstate-spaces/mamba-130m-hftext-generation · ↓ 699k · ♡ 72Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-4B-BF16text-generation · ↓ 672k · ♡ 106Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4text-generation · ↓ 628k · ♡ 174Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8text-generation · ↓ 624k · ♡ 357Open in visualizer microsoft/Mage-VL architecture graphmicrosoft/Mage-VLimage-text-to-text · ↓ 482k · ♡ 343Open in visualizer nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16text-generation · ↓ 448k · ♡ 324Open in visualizer

Browse all 54 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Short convolution block Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →