Attention

Self-attention

Every token builds queries, keys and values, then gathers information from the other tokens that match its query.

First page of Attention Is All You NeedIntroduced inJun 2017Attention Is All You NeedVaswani et al. · arXiv 1706.03762 ↗
Self-attention: every token asks, all others answerthe query compares with each earlier token · weights (bars) set the blendquery

Self-attention is the core transformer operation: each position emits a query that is compared against the keys of the other positions, and the resulting weights average their values. It is what lets the model relate any token to any other, at a cost that grows with the square of sequence length. Which is why so many of the surrounding tricks (GQA, sliding windows, sparse and linear attention) exist to tame it.

Adoption over time

Share of new models that have included self-attention over time.

23%2022202320242025202623%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

google-bert/bert-base-uncased architecture graphgoogle-bert/bert-base-uncasedfill-mask · ↓ 115.9M · ♡ 3kOpen in visualizer cross-encoder/ms-marco-MiniLM-L6-v2 architecture graphcross-encoder/ms-marco-MiniLM-L6-v2text-ranking · ↓ 88.0M · ♡ 300Open in visualizer BAAI/bge-small-en-v1.5 architecture graphBAAI/bge-small-en-v1.5feature-extraction · ↓ 72.7M · ♡ 532Open in visualizer sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 architecture graphsentence-transformers/paraphrase-multilingual-MiniLM-L12-v2sentence-similarity · ↓ 56.5M · ♡ 1kOpen in visualizer google/electra-base-discriminator architecture graphgoogle/electra-base-discriminator↓ 54.8M · ♡ 153Open in visualizer BAAI/bge-m3 architecture graphBAAI/bge-m3sentence-similarity · ↓ 34.9M · ♡ 3kOpen in visualizer google-t5/t5-small architecture graphgoogle-t5/t5-smalltranslation · ↓ 22.8M · ♡ 592Open in visualizer openai/clip-vit-base-patch32 architecture graphopenai/clip-vit-base-patch32zero-shot-image-classification · ↓ 20.8M · ♡ 1kOpen in visualizer BAAI/bge-reranker-v2-m3 architecture graphBAAI/bge-reranker-v2-m3text-classification · ↓ 18.6M · ♡ 1kOpen in visualizer FacebookAI/xlm-roberta-base architecture graphFacebookAI/xlm-roberta-basefill-mask · ↓ 17.8M · ♡ 882Open in visualizer facebook/opt-125m architecture graphfacebook/opt-125mtext-generation · ↓ 17.5M · ♡ 292Open in visualizer openai-community/gpt2 architecture graphopenai-community/gpt2text-generation · ↓ 13.6M · ♡ 3kOpen in visualizer

Browse all 1181 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA)

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →