Norms & blocks

Transformer block

The model’s repeating unit: attention mixes information between tokens, an MLP transforms each token, with norms and residuals holding it together.

First page of Attention Is All You NeedIntroduced inJun 2017Attention Is All You NeedVaswani et al. · arXiv 1706.03762 ↗
One block: mix, transform, and carry the residualattention:tokens exchangeMLP:each token aloneresidualresidual× N layers

Nearly every modern model is a stack of identical blocks like this one. Each block first lets tokens exchange information through some variant of attention, then transforms every token independently through a (usually gated) MLP, with a normalization layer before each part and residual connections around them. Expand the block in the graph to see the exact wiring this model uses inside.

Adoption over time

Share of new models that have included transformer blocks over time.

47%2022202320242025202647%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

google-bert/bert-base-uncased architecture graphgoogle-bert/bert-base-uncasedfill-mask · ↓ 115.9M · ♡ 3kOpen in visualizer cross-encoder/ms-marco-MiniLM-L6-v2 architecture graphcross-encoder/ms-marco-MiniLM-L6-v2text-ranking · ↓ 88.0M · ♡ 300Open in visualizer BAAI/bge-small-en-v1.5 architecture graphBAAI/bge-small-en-v1.5feature-extraction · ↓ 72.7M · ♡ 532Open in visualizer sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 architecture graphsentence-transformers/paraphrase-multilingual-MiniLM-L12-v2sentence-similarity · ↓ 56.5M · ♡ 1kOpen in visualizer google/electra-base-discriminator architecture graphgoogle/electra-base-discriminator↓ 54.8M · ♡ 153Open in visualizer BAAI/bge-m3 architecture graphBAAI/bge-m3sentence-similarity · ↓ 34.9M · ♡ 3kOpen in visualizer Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 29.7M · ♡ 2kOpen in visualizer BAAI/bge-reranker-v2-m3 architecture graphBAAI/bge-reranker-v2-m3text-classification · ↓ 18.6M · ♡ 1kOpen in visualizer FacebookAI/xlm-roberta-base architecture graphFacebookAI/xlm-roberta-basefill-mask · ↓ 17.8M · ♡ 882Open in visualizer facebook/opt-125m architecture graphfacebook/opt-125mtext-generation · ↓ 17.5M · ♡ 292Open in visualizer FacebookAI/roberta-base architecture graphFacebookAI/roberta-basefill-mask · ↓ 13.3M · ♡ 638Open in visualizer BAAI/bge-large-en-v1.5 architecture graphBAAI/bge-large-en-v1.5feature-extraction · ↓ 12.4M · ♡ 713Open in visualizer

Browse all 1641 models with this in the catalog →

Related concepts

RMSNorm Attention residuals (AttnRes) Residual (skip) connection Logit softcapping Gated MLP (SwiGLU) Hyper-connections (mHC)

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →