Attention

Sparse attention (top-k token selection)

A cheap ‘indexer’ scores past tokens and full attention runs only over the top-scoring ones.

First page of Generating Long Sequences with Sparse TransformersGoes back toApr 2019Generating Long Sequences with Sparse TransformersChild et al. · arXiv 1904.10509 ↗
Sparse attention: attend to the few that mattera cheap indexer scores the past · full attention runs only over the top-kqueryindexer scoresattention only visits the top-k

Instead of attending to every previous token, a lightweight indexer ranks the context and selects a fixed budget of the most relevant tokens; the expensive attention pass then runs only over that subset. DeepSeek V4 and GLM 5.x use this to keep per-token cost nearly flat while the context grows to a million tokens. Quality holds up because most attention mass concentrates on few tokens anyway.

Adoption over time

Share of new models that have included sparse top-k attention over time.

5%202220232024202520265%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 8.0M · ♡ 14kOpen in visualizer zai-org/GLM-5.2 architecture graphzai-org/GLM-5.2text-generation · ↓ 2.7M · ♡ 5kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 2.1M · ♡ 2kOpen in visualizer zai-org/GLM-5.2-FP8 architecture graphzai-org/GLM-5.2-FP8text-generation · ↓ 1.9M · ♡ 252Open in visualizer deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 1.6M · ♡ 3kOpen in visualizer nvidia/GLM-5.2-NVFP4 architecture graphnvidia/GLM-5.2-NVFP4text-generation · ↓ 1.4M · ♡ 308Open in visualizer deepseek-ai/DeepSeek-V4-Pro architecture graphdeepseek-ai/DeepSeek-V4-Protext-generation · ↓ 1.3M · ♡ 5kOpen in visualizer deepseek-ai/DeepSeek-V3-0324 architecture graphdeepseek-ai/DeepSeek-V3-0324text-generation · ↓ 1.2M · ♡ 3kOpen in visualizer deepseek-ai/DeepSeek-V3.2 architecture graphdeepseek-ai/DeepSeek-V3.2text-generation · ↓ 1.1M · ♡ 1kOpen in visualizer deepseek-ai/DeepSeek-V3 architecture graphdeepseek-ai/DeepSeek-V3text-generation · ↓ 1.0M · ♡ 4kOpen in visualizer zai-org/GLM-5-FP8 architecture graphzai-org/GLM-5-FP8text-generation · ↓ 1.0M · ♡ 182Open in visualizer nvidia/DeepSeek-V4-Flash-NVFP4 architecture graphnvidia/DeepSeek-V4-Flash-NVFP4text-generation · ↓ 747k · ♡ 98Open in visualizer

Browse all 78 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →