Mixture of experts

Shared expert

An always-on expert that every token passes through, alongside whatever the router picks.

First page of DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsIntroduced inJan 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDai et al. · arXiv 2401.06066 ↗
The shared expert runs for every tokenrouterexpert 1expert 2expert 3expert 4expert 5expert 6expert 7expert 8always on →shared expertrouted experts specialize; the shared one carries common knowledge

DeepSeek-style MoE adds one or more shared experts that process every token unconditionally, in parallel with the routed top-k experts. The shared expert captures common knowledge every token needs, freeing the routed experts to specialize harder. Which measurably improves quality at the same compute. Most 2025-generation MoE models (DeepSeek, GLM, Qwen-MoE, Nemotron) include one.

Adoption over time

Share of new models that have included a shared expert over time.

21%2022202320242025202621%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 11.7M · ♡ 553Open in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 11.1M · ♡ 351Open in visualizer deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 8.0M · ♡ 14kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 5.8M · ♡ 3kOpen in visualizer ornith-ai/Ornith-1.0-35B architecture graphornith-ai/Ornith-1.0-35Btext-generation · ↓ 3.0M · ♡ 493Open in visualizer deepreinforce-ai/Ornith-1.0-35B architecture graphdeepreinforce-ai/Ornith-1.0-35Btext-generation · ↓ 3.0M · ♡ 493Open in visualizer baidu/Unlimited-OCR architecture graphbaidu/Unlimited-OCRimage-text-to-text · ↓ 2.9M · ♡ 4kOpen in visualizer zai-org/GLM-5.2 architecture graphzai-org/GLM-5.2text-generation · ↓ 2.7M · ♡ 5kOpen in visualizer unsloth/Qwen3.6-35B-A3B-NVFP4 architecture graphunsloth/Qwen3.6-35B-A3B-NVFP4image-text-to-text · ↓ 2.4M · ♡ 109Open in visualizer Qwen/Qwen3.5-35B-A3B architecture graphQwen/Qwen3.5-35B-A3Bimage-text-to-text · ↓ 2.4M · ♡ 1kOpen in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4text-generation · ↓ 2.3M · ♡ 420Open in visualizer deepseek-ai/DeepSeek-OCR architecture graphdeepseek-ai/DeepSeek-OCRimage-text-to-text · ↓ 2.2M · ♡ 3kOpen in visualizer

Browse all 345 models with this in the catalog →

Related concepts

Grouped expert routing MoE router MoE experts

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →