Model architecture glossary.

Plain-English explanations of the blocks you see in modern model architectures. The same explanations appear when you hover nodes in hfviewer’s interactive graphs, grounded in each model’s real numbers.

Attention

Oct 2020QK-NormNormalizes queries and keys right before attention so their dot products can’t blow up. share of new models using this
May 2024Multi-head latent attention (MLA)Compresses keys and values into one small latent vector per token. The cache stores the latent, not full K/V. share of new models using this
Apr 2019Sparse attention (top-k token selection)A cheap ‘indexer’ scores past tokens and full attention runs only over the top-scoring ones. share of new models using this
Apr 2020Sliding-window attentionEach token only attends to a fixed window of recent tokens, keeping cost and cache flat as context grows. share of new models using this
Jun 2020Linear attention (gated DeltaNet)Replaces the attention matrix with a fixed-size recurrent state. Cost stays constant per token, no KV cache growth. share of new models using this
Dec 2023Mamba / state-space block (SSM)A recurrent state-space scan replaces attention: constant memory per token, linear time in sequence length. share of new models using this
Dec 2022Short convolution blockA gated, depthwise causal convolution mixes only a few neighboring tokens. A very cheap substitute for attention. share of new models using this
May 2023Grouped-query attention (GQA)Several query heads share one key/value head, shrinking the KV cache several-fold with little quality loss. share of new models using this
Jun 2017Self-attentionEvery token builds queries, keys and values, then gathers information from the other tokens that match its query. share of new models using this
Sep 2023Attention sinkA learned ‘nowhere’ slot lets a head cleanly attend to nothing instead of smearing weight over random tokens. share of new models using this
KV cacheGeneration reuses cached keys/values of all previous tokens instead of recomputing them. Memory becomes the bottleneck.

Positions & context

Norms & blocks

Mixture of experts

Heads & prediction

Multimodal

Diffusion & generation

Audio

Every concept here links to real models where you can see the block in context - zoom, expand, and inspect the actual architecture.

Browse all 3,500+ model graphs →