Attention

Attention sink

A learned ‘nowhere’ slot lets a head cleanly attend to nothing instead of smearing weight over random tokens.

First page of Efficient Streaming Language Models with Attention SinksIntroduced inSep 2023Efficient Streaming Language Models with Attention SinksXiao et al. · arXiv 2309.17453 ↗
Attention sink: a place to point at nothingwith no good match, weight would smear · the sink absorbs it cleanly insteadsinknothing relevant -weight smearsit all landson the sink

Softmax forces attention weights to sum to one, so a head that has nothing relevant to look at must still put its weight somewhere. Often degrading quality. An attention sink adds a learned logit (or a pinned first token) that absorbs that leftover probability. gpt-oss bakes per-head sink parameters into every attention layer; the same idea is why streaming-inference tricks keep the first tokens around.

Adoption over time

Share of new models that have included attention sinks over time.

3%202220232024202520263%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 2.1M · ♡ 2kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 1.6M · ♡ 3kOpen in visualizer deepseek-ai/DeepSeek-V4-Pro architecture graphdeepseek-ai/DeepSeek-V4-Protext-generation · ↓ 1.3M · ♡ 5kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash-DSpark architecture graphdeepseek-ai/DeepSeek-V4-Flash-DSparktext-generation · ↓ 554k · ♡ 254Open in visualizer sgl-project/DeepSeek-V4-Flash-FP8 architecture graphsgl-project/DeepSeek-V4-Flash-FP8↓ 342k · ♡ 15Open in visualizer AtlasCloud/DeepSeek-V4-Flash-0731-FP8-DSpark architecture graphAtlasCloud/DeepSeek-V4-Flash-0731-FP8-DSpark↓ 293k · ♡ 5Open in visualizer MJPansa/DeepSeek-V4-Flash-0731-NVFP4 architecture graphMJPansa/DeepSeek-V4-Flash-0731-NVFP4text-generation · ↓ 201k · ♡ 11Open in visualizer 0xSero/deepseek-v4-flash-0731-spark architecture graph0xSero/deepseek-v4-flash-0731-sparktext-generation · ↓ 39k · ♡ 29Open in visualizer deepseek-ai/DeepSeek-V4-Pro-DSpark architecture graphdeepseek-ai/DeepSeek-V4-Pro-DSparktext-generation · ↓ 23k · ♡ 539Open in visualizer deepseek-ai/DeepSeek-V4-Pro-0813 architecture graphdeepseek-ai/DeepSeek-V4-Pro-0813text-generation · ↓ 20k · ♡ 489Open in visualizer apetersson/DeepSeek-V4-Flash-0731-Abliterated-FP8 architecture graphapetersson/DeepSeek-V4-Flash-0731-Abliterated-FP8text-generation · ↓ 12k · ♡ 39Open in visualizer unsloth/DeepSeek-V4-Flash-0731 architecture graphunsloth/DeepSeek-V4-Flash-0731text-generation · ↓ 5k · ♡ 33Open in visualizer

Browse all 25 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA)

hfviewer renders the full architecture of 3,600+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →