Mixture of experts

MoE experts

A bank of parallel feed-forward networks; each token runs through only the few the router selected.

First page of Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerIntroduced inJan 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerShazeer et al. · arXiv 1701.06538 ↗
Mixture of experts: top-2 of 8routerexpert 1expert 2expert 3expert 4expert 5expert 6expert 7expert 8stored: 8 experts · used per token: 2

Each expert is an independent gated MLP. Because only the top-k experts execute per token, total parameter count and per-token compute decouple: a model can store hundreds of experts' worth of knowledge while running just a handful. Different experts end up specializing. By language, domain, or syntax. Although the specialization is learned, not assigned.

Adoption over time

Share of new models that have included MoE experts over time.

27%2022202320242025202627%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 11.7M · ♡ 553Open in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 11.1M · ♡ 351Open in visualizer google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 10.1M · ♡ 1kOpen in visualizer openai/gpt-oss-20b architecture graphopenai/gpt-oss-20btext-generation · ↓ 8.1M · ♡ 5kOpen in visualizer deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 8.0M · ♡ 14kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 5.8M · ♡ 3kOpen in visualizer openai/gpt-oss-120b architecture graphopenai/gpt-oss-120btext-generation · ↓ 4.3M · ♡ 5kOpen in visualizer ornith-ai/Ornith-1.0-35B architecture graphornith-ai/Ornith-1.0-35Btext-generation · ↓ 3.0M · ♡ 493Open in visualizer deepreinforce-ai/Ornith-1.0-35B architecture graphdeepreinforce-ai/Ornith-1.0-35Btext-generation · ↓ 3.0M · ♡ 493Open in visualizer baidu/Unlimited-OCR architecture graphbaidu/Unlimited-OCRimage-text-to-text · ↓ 2.9M · ♡ 4kOpen in visualizer cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit architecture graphcyankiwi/gemma-4-26B-A4B-it-AWQ-4bitimage-text-to-text · ↓ 2.7M · ♡ 91Open in visualizer zai-org/GLM-5.2 architecture graphzai-org/GLM-5.2text-generation · ↓ 2.7M · ♡ 5kOpen in visualizer

Browse all 482 models with this in the catalog →

Related concepts

Grouped expert routing MoE router Shared expert

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →