Attention

Grouped-query attention (GQA)

Several query heads share one key/value head, shrinking the KV cache several-fold with little quality loss.

First page of GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsIntroduced inMay 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsAinslie et al. · arXiv 2305.13245 ↗
Multi-head: 8 Q → 8 KVGQA: 8 Q share 2 KVMHA cacheGQA cachesame tokens,4× less memory

In classic multi-head attention every head has its own keys and values, and all of them must be cached for every generated token. GQA keeps many query heads but lets groups of them share a single key/value head. With one shared KV head this becomes multi-query attention. Since KV-cache size is what limits batch size and context length at inference time, nearly every recent LLM uses it.

Adoption over time

Share of new models that have included GQA over time.

65%2022202320242025202665%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 29.7M · ♡ 2kOpen in visualizer Qwen/Qwen3-8B architecture graphQwen/Qwen3-8Btext-generation · ↓ 16.3M · ♡ 1kOpen in visualizer Qwen/Qwen2.5-7B-Instruct architecture graphQwen/Qwen2.5-7B-Instructtext-generation · ↓ 12.1M · ♡ 2kOpen in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 11.7M · ♡ 553Open in visualizer Qwen/Qwen2.5-1.5B-Instruct architecture graphQwen/Qwen2.5-1.5B-Instructtext-generation · ↓ 11.6M · ♡ 796Open in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 11.1M · ♡ 351Open in visualizer google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 10.1M · ♡ 1kOpen in visualizer google/gemma-4-31B-it architecture graphgoogle/gemma-4-31B-itimage-text-to-text · ↓ 10.0M · ♡ 4kOpen in visualizer meta-llama/Llama-3.2-1B-Instruct architecture graphmeta-llama/Llama-3.2-1B-Instructtext-generation · ↓ 9.2M · ♡ 2kOpen in visualizer Qwen/Qwen3.6-27B-FP8 architecture graphQwen/Qwen3.6-27B-FP8image-text-to-text · ↓ 9.0M · ♡ 346Open in visualizer openai/gpt-oss-20b architecture graphopenai/gpt-oss-20btext-generation · ↓ 8.1M · ♡ 5kOpen in visualizer Qwen/Qwen3-32B architecture graphQwen/Qwen3-32Btext-generation · ↓ 8.0M · ♡ 732Open in visualizer

Browse all 1860 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Self-attention

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →