Heads & prediction

Multi-token prediction (MTP)

An extra mini-block predicts the token after next. Used to train richer representations and to speed up decoding.

First page of Better & Faster Large Language Models via Multi-token PredictionIntroduced inApr 2024Better & Faster Large Language Models via Multi-token PredictionGloeckle et al. · arXiv 2404.19737 ↗
Multi-token prediction: guess further aheadthe main head predicts the next token · a small block drafts the one aftermain LM headMTP blockt+1t+2 draftdrafts are verified by the main model · speculative decoding for free

MTP attaches one or more small extra transformer blocks that predict token t+2 (and beyond) alongside the main next-token head. At training time this densifies the learning signal; at inference the MTP head can draft tokens that the main model then verifies. Self-speculative decoding that speeds generation without a separate draft model. DeepSeek V3/V4, GLM and Qwen3.5 all ship MTP blocks.

Adoption over time

Share of new models that have included MTP over time.

11%2022202320242025202611%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 8.0M · ♡ 14kOpen in visualizer Qwen/Qwen3-TTS-12Hz-1.7B-Base architecture graphQwen/Qwen3-TTS-12Hz-1.7B-Base↓ 3.0M · ♡ 484Open in visualizer DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF architecture graphDavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUFimage-text-to-text · ↓ 2.9M · ♡ 2kOpen in visualizer zai-org/GLM-5.2 architecture graphzai-org/GLM-5.2text-generation · ↓ 2.7M · ♡ 5kOpen in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4text-generation · ↓ 2.3M · ♡ 420Open in visualizer Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice architecture graphQwen/Qwen3-TTS-12Hz-1.7B-CustomVoicetext-to-speech · ↓ 2.2M · ♡ 2kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 2.1M · ♡ 2kOpen in visualizer zai-org/GLM-4.7-Flash architecture graphzai-org/GLM-4.7-Flashtext-generation · ↓ 2.0M · ♡ 2kOpen in visualizer zai-org/GLM-5.2-FP8 architecture graphzai-org/GLM-5.2-FP8text-generation · ↓ 1.9M · ♡ 252Open in visualizer deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 1.6M · ♡ 3kOpen in visualizer Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice architecture graphQwen/Qwen3-TTS-12Hz-0.6B-CustomVoicetext-to-speech · ↓ 1.5M · ♡ 175Open in visualizer nvidia/GLM-5.2-NVFP4 architecture graphnvidia/GLM-5.2-NVFP4text-generation · ↓ 1.4M · ♡ 308Open in visualizer

Browse all 159 models with this in the catalog →

Related concepts

LM head (output projection)

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →