Multimodal

Vision encoder (ViT)

A transformer over image patches that turns pictures into a sequence of visual embeddings.

First page of An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleIntroduced inOct 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleDosovitskiy et al. · arXiv 2010.11929 ↗
Vision encoder: pictures in, tokens outpatches flow through a ViT · out comes a sequence of visual embeddingsViT ×Nvisual tokens for the LLM

The vision tower of a multimodal model is usually a ViT-family encoder (SigLIP and CLIP variants dominate) pretrained on image-text pairs. It converts patch embeddings into contextualized visual features that a projector then aligns with the language model’s embedding space. Multimodal capability lives or dies by this component and how it was pretrained.

Adoption over time

Share of new models that have included a vision encoder over time.

2%202220232024202520262%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

llava-hf/llava-1.5-7b-hf architecture graphllava-hf/llava-1.5-7b-hfimage-text-to-text · ↓ 2.9M · ♡ 370Open in visualizer microsoft/TRELLIS-image-large architecture graphmicrosoft/TRELLIS-image-largeimage-to-3d · ↓ 1.7M · ♡ 670Open in visualizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 architecture graphnvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4any-to-any · ↓ 1.5M · ♡ 177Open in visualizer Qwen/Qwen2.5-Omni-3B architecture graphQwen/Qwen2.5-Omni-3Bany-to-any · ↓ 1.4M · ♡ 345Open in visualizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 architecture graphnvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8any-to-any · ↓ 1.3M · ♡ 61Open in visualizer google/gemma-3-4b-it architecture graphgoogle/gemma-3-4b-itimage-text-to-text · ↓ 1.3M · ♡ 1kOpen in visualizer google/gemma-3-12b-it architecture graphgoogle/gemma-3-12b-itimage-text-to-text · ↓ 1.2M · ♡ 806Open in visualizer microsoft/Phi-3.5-vision-instruct architecture graphmicrosoft/Phi-3.5-vision-instructimage-text-to-text · ↓ 1.2M · ♡ 738Open in visualizer google/gemma-3-27b-it architecture graphgoogle/gemma-3-27b-itimage-text-to-text · ↓ 771k · ♡ 2kOpen in visualizer black-forest-labs/FLUX.1-dev architecture graphblack-forest-labs/FLUX.1-devtext-to-image · ↓ 517k · ♡ 14kOpen in visualizer google/siglip2-so400m-patch16-naflex architecture graphgoogle/siglip2-so400m-patch16-naflexzero-shot-image-classification · ↓ 434k · ♡ 81Open in visualizer google/gemma-3n-E2B-it architecture graphgoogle/gemma-3n-E2B-itimage-text-to-text · ↓ 391k · ♡ 319Open in visualizer

Browse all 143 models with this in the catalog →

Related concepts

Patch embedding Patch merger / pixel shuffle Multimodal projector Cross-attention

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →