Logit softcapping
Squashes attention or output logits through a scaled tanh so no single value can run away.
Popularized inJul 2024Gemma 2: Improving Open Language Models at a Practical Size
Softcapping bounds logits to a fixed range by passing them through tanh scaled by a cap value. Gemma-2 popularized it for both attention scores and final output logits as a training stabilizer. Extreme logits otherwise cause saturation and loss spikes. Newer models often replace it with QK-Norm, but you will still find it in several current architectures.
See it in real models
Open any of these on hfviewer to find this block in the interactive architecture graph.
meta-models/Muse-Glimmer-30Bimage-text-to-text · ↓ 165k · ♡ 2kOpen in visualizer
unsloth/Muse-Glimmer-30Bimage-text-to-text · ↓ 58k · ♡ 12Open in visualizer
RedHatAI/Muse-Glimmer-30B-FP8-blockimage-text-to-text · ↓ 26k · ♡ 9Open in visualizer
RadixArk/Muse-Glimmer-NVFP4image-text-to-text · ↓ 7k · ♡ 5Open in visualizer
unsloth/Muse-Glimmer-30B-unsloth-bnb-4bitimage-text-to-text · ↓ 5k · ♡ 10Open in visualizer
RedHatAI/Muse-Glimmer-30B-NVFP4↓ 4k · ♡ 16Open in visualizer
Preyazz/Muse-Glimmer-30B-NVFP4image-text-to-text · ↓ 1k · ♡ 6Open in visualizer
darkc0de/Muse-Glimmer-30B-hereticimage-text-to-text · ↓ 1k · ♡ 10Open in visualizer
Related concepts
hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.
Browse all model graphs →