Heads & prediction

LM head (output projection)

Projects the final hidden state onto vocabulary logits. Often reusing the input embedding matrix (‘tied weights’).

First page of Using the Output Embedding to Improve Language ModelsIntroduced inAug 2016Using the Output Embedding to Improve Language ModelsPress & Wolf · arXiv 1608.05859 ↗
LM head: one matrix, read twicetied weights: tokens→vectors at the input, hidden→vocab logits at the outputembedding matrix (shared)'cat'vectorhidden statelogits over the vocab

The LM head is a single linear map from the model’s hidden width to one logit per vocabulary token; softmax over those logits gives next-token probabilities. Many models tie it to the input embedding matrix, saving hidden×vocab parameters (often several hundred million) at negligible quality cost. Common in small and mid-size models, less so at the largest scales.

Adoption over time

Share of new models that have included an LM head over time.

75%2022202320242025202675%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

amazon/chronos-2 architecture graphamazon/chronos-2time-series-forecasting · ↓ 36.8M · ♡ 398Open in visualizer Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 29.7M · ♡ 2kOpen in visualizer sentence-transformers/all-mpnet-base-v2 architecture graphsentence-transformers/all-mpnet-base-v2sentence-similarity · ↓ 24.6M · ♡ 1kOpen in visualizer google-t5/t5-small architecture graphgoogle-t5/t5-smalltranslation · ↓ 22.8M · ♡ 592Open in visualizer FacebookAI/xlm-roberta-base architecture graphFacebookAI/xlm-roberta-basefill-mask · ↓ 17.8M · ♡ 882Open in visualizer facebook/opt-125m architecture graphfacebook/opt-125mtext-generation · ↓ 17.5M · ♡ 292Open in visualizer Qwen/Qwen3-8B architecture graphQwen/Qwen3-8Btext-generation · ↓ 16.3M · ♡ 1kOpen in visualizer Qwen/Qwen3.5-9B architecture graphQwen/Qwen3.5-9Bimage-text-to-text · ↓ 13.9M · ♡ 2kOpen in visualizer openai-community/gpt2 architecture graphopenai-community/gpt2text-generation · ↓ 13.6M · ♡ 3kOpen in visualizer FacebookAI/roberta-base architecture graphFacebookAI/roberta-basefill-mask · ↓ 13.3M · ♡ 638Open in visualizer Qwen/Qwen2.5-7B-Instruct architecture graphQwen/Qwen2.5-7B-Instructtext-generation · ↓ 12.1M · ♡ 2kOpen in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 11.7M · ♡ 553Open in visualizer

Browse all 2344 models with this in the catalog →

Related concepts

Multi-token prediction (MTP)

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →