Norms & blocks

RMSNorm

Rescales activations by their RMS magnitude. Like LayerNorm without mean-centering, and cheaper.

First page of Root Mean Square Layer NormalizationIntroduced inOct 2019Root Mean Square Layer NormalizationZhang & Sennrich · arXiv 1910.07467 ↗
RMSNorm: one divide, no mean subtractionone scale for the whole vector, then a learned per-channel gain (green)× learned gainRMSRMS = 1

RMSNorm divides each activation vector by its root-mean-square and multiplies by a learned scale, skipping the mean subtraction and bias that LayerNorm carries. It stabilizes training just as well in practice while being simpler and faster, which is why almost every modern LLM (Llama onward) uses it, typically placed before attention and before the MLP in each block (‘pre-norm’).

Adoption over time

Share of new models that have included RMSNorm over time.

83%2022202320242025202683%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 29.7M · ♡ 2kOpen in visualizer Qwen/Qwen3-8B architecture graphQwen/Qwen3-8Btext-generation · ↓ 16.3M · ♡ 1kOpen in visualizer Qwen/Qwen3.5-9B architecture graphQwen/Qwen3.5-9Bimage-text-to-text · ↓ 13.9M · ♡ 2kOpen in visualizer Qwen/Qwen2.5-7B-Instruct architecture graphQwen/Qwen2.5-7B-Instructtext-generation · ↓ 12.1M · ♡ 2kOpen in visualizer Comfy-Org/MiniMax-H3 architecture graphComfy-Org/MiniMax-H3↓ 11.8M · ♡ 1kOpen in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 11.7M · ♡ 553Open in visualizer Qwen/Qwen2.5-1.5B-Instruct architecture graphQwen/Qwen2.5-1.5B-Instructtext-generation · ↓ 11.6M · ♡ 796Open in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 11.1M · ♡ 351Open in visualizer google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 10.1M · ♡ 1kOpen in visualizer google/gemma-4-31B-it architecture graphgoogle/gemma-4-31B-itimage-text-to-text · ↓ 10.0M · ♡ 4kOpen in visualizer meta-llama/Llama-3.2-1B-Instruct architecture graphmeta-llama/Llama-3.2-1B-Instructtext-generation · ↓ 9.2M · ♡ 2kOpen in visualizer Qwen/Qwen2.5-VL-7B-Instruct architecture graphQwen/Qwen2.5-VL-7B-Instructimage-text-to-text · ↓ 9.1M · ♡ 2kOpen in visualizer

Browse all 2312 models with this in the catalog →

Related concepts

Attention residuals (AttnRes) Residual (skip) connection Logit softcapping Gated MLP (SwiGLU) Hyper-connections (mHC) Transformer block

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →