Norms & blocks

Hyper-connections (mHC)

Widens the single residual stream into several parallel streams with learned mixing between them.

First page of Hyper-ConnectionsIntroduced inSep 2024Hyper-ConnectionsZhu et al. · arXiv 2409.19606 ↗
Hyper-connections: a wider residual streamone stream becomes several in parallel, with learned mixing between themone streamn streams, learned mixing

Hyper-connections replace the transformer’s single residual stream with several parallel streams; at each block, learned matrices mix the streams before and after the block runs. DeepSeek V4’s mHC variant constrains that mixing to be doubly stochastic (computed with a few Sinkhorn iterations) so the streams stay balanced. The effect is richer gradient routing than one shared residual. One of the most distinctive pieces of the V4 architecture.

Adoption over time

Share of new models that have included hyper-connections over time.

4%202220232024202520264%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 2.1M · ♡ 2kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 1.6M · ♡ 3kOpen in visualizer deepseek-ai/DeepSeek-V4-Pro architecture graphdeepseek-ai/DeepSeek-V4-Protext-generation · ↓ 1.3M · ♡ 5kOpen in visualizer nvidia/DeepSeek-V4-Flash-NVFP4 architecture graphnvidia/DeepSeek-V4-Flash-NVFP4text-generation · ↓ 747k · ♡ 98Open in visualizer deepseek-ai/DeepSeek-V4-Flash-DSpark architecture graphdeepseek-ai/DeepSeek-V4-Flash-DSparktext-generation · ↓ 554k · ♡ 254Open in visualizer deepseek-ai/DeepSeek-V4-Flash-Base architecture graphdeepseek-ai/DeepSeek-V4-Flash-Base↓ 386k · ♡ 293Open in visualizer sgl-project/DeepSeek-V4-Flash-FP8 architecture graphsgl-project/DeepSeek-V4-Flash-FP8↓ 342k · ♡ 15Open in visualizer AtlasCloud/DeepSeek-V4-Flash-0731-FP8-DSpark architecture graphAtlasCloud/DeepSeek-V4-Flash-0731-FP8-DSpark↓ 293k · ♡ 5Open in visualizer MJPansa/DeepSeek-V4-Flash-0731-NVFP4 architecture graphMJPansa/DeepSeek-V4-Flash-0731-NVFP4text-generation · ↓ 201k · ♡ 11Open in visualizer nvidia/DeepSeek-V4-Pro-NVFP4 architecture graphnvidia/DeepSeek-V4-Pro-NVFP4text-generation · ↓ 102k · ♡ 76Open in visualizer 0xSero/deepseek-v4-flash-0731-spark architecture graph0xSero/deepseek-v4-flash-0731-sparktext-generation · ↓ 39k · ♡ 29Open in visualizer deepseek-ai/DeepSeek-V4-Pro-DSpark architecture graphdeepseek-ai/DeepSeek-V4-Pro-DSparktext-generation · ↓ 23k · ♡ 539Open in visualizer

Browse all 40 models with this in the catalog →

Related concepts

RMSNorm Attention residuals (AttnRes) Residual (skip) connection Logit softcapping Gated MLP (SwiGLU) Transformer block

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →