Attention

Short convolution block

A gated, depthwise causal convolution mixes only a few neighboring tokens. A very cheap substitute for attention.

First page of Hungry Hungry Hippos: Towards Language Modeling with State Space ModelsGoes back toDec 2022Hungry Hungry Hippos: Towards Language Modeling with State Space ModelsFu et al. · arXiv 2212.14052 ↗
Short conv block: mix only nearby tokensa gated depthwise causal conv · each output sees just its few neighboursinputsoutputs3-wide causal window

Some hybrid models (LFM2, Nemotron-H) replace most attention layers with short causal convolutions: each position mixes information from just the last few tokens through a depthwise conv, with multiplicative gates deciding what passes through. It is dramatically cheaper than attention and surprisingly capable for local patterns; the few remaining attention layers supply global context.

Adoption over time

Share of new models that have included short convolution blocks over time.

4%202220232024202520264%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4text-generation · ↓ 2.3M · ♡ 420Open in visualizer moonshotai/Kimi-K3 architecture graphmoonshotai/Kimi-K3image-text-to-text · ↓ 2.0M · ♡ 11kOpen in visualizer Qwen/Qwen3-Omni-30B-A3B-Instruct architecture graphQwen/Qwen3-Omni-30B-A3B-Instructany-to-any · ↓ 1.1M · ♡ 982Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16text-generation · ↓ 956k · ♡ 811Open in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16text-generation · ↓ 865k · ♡ 417Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-4B-BF16text-generation · ↓ 672k · ♡ 106Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4text-generation · ↓ 628k · ♡ 174Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8text-generation · ↓ 624k · ♡ 357Open in visualizer nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16text-generation · ↓ 448k · ♡ 324Open in visualizer nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4text-generation · ↓ 431k · ♡ 128Open in visualizer nvidia/NVIDIA-Nemotron-Nano-9B-v2 architecture graphnvidia/NVIDIA-Nemotron-Nano-9B-v2text-generation · ↓ 332k · ♡ 513Open in visualizer nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4text-generation · ↓ 327k · ♡ 301Open in visualizer

Browse all 53 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →