Multimodal

Multimodal projector

A small MLP that translates vision-encoder features into the language model’s embedding space.

First page of Visual Instruction TuningPopularized inApr 2023Visual Instruction TuningLiu et al. · arXiv 2304.08485 ↗
Multimodal projector: the small bridgeone or two linear layers map vision features into the LLM’s embedding spaceMLPvision featuresLLM-space tokens

The projector (often just one or two linear layers) maps visual features into vectors that look like text-token embeddings to the LLM, which then attends over them like any other tokens. It is tiny compared to the towers it connects but is where ‘alignment’ between modalities is learned. Many multimodal training recipes tune the projector first, alone.

Adoption over time

Share of new models that have included a multimodal projector over time.

5%202220232024202520265%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

llava-hf/llava-1.5-7b-hf architecture graphllava-hf/llava-1.5-7b-hfimage-text-to-text · ↓ 2.9M · ♡ 370Open in visualizer microsoft/Florence-2-base architecture graphmicrosoft/Florence-2-baseimage-text-to-text · ↓ 2.8M · ♡ 392Open in visualizer moonshotai/Kimi-K3 architecture graphmoonshotai/Kimi-K3image-text-to-text · ↓ 2.0M · ♡ 11kOpen in visualizer IDEA-Research/grounding-dino-base architecture graphIDEA-Research/grounding-dino-basezero-shot-object-detection · ↓ 1.3M · ♡ 202Open in visualizer google/gemma-3-4b-it architecture graphgoogle/gemma-3-4b-itimage-text-to-text · ↓ 1.3M · ♡ 1kOpen in visualizer google/gemma-3-12b-it architecture graphgoogle/gemma-3-12b-itimage-text-to-text · ↓ 1.2M · ♡ 806Open in visualizer nvidia/Kimi-K2.7-Code-NVFP4 architecture graphnvidia/Kimi-K2.7-Code-NVFP4text-generation · ↓ 1.1M · ♡ 9Open in visualizer IDEA-Research/grounding-dino-tiny architecture graphIDEA-Research/grounding-dino-tinyzero-shot-object-detection · ↓ 1.0M · ♡ 111Open in visualizer moonshotai/Kimi-K2.5 architecture graphmoonshotai/Kimi-K2.5image-text-to-text · ↓ 858k · ♡ 3kOpen in visualizer moonshotai/Kimi-K2.6 architecture graphmoonshotai/Kimi-K2.6image-text-to-text · ↓ 820k · ♡ 2kOpen in visualizer google/gemma-3-27b-it architecture graphgoogle/gemma-3-27b-itimage-text-to-text · ↓ 771k · ♡ 2kOpen in visualizer microsoft/Florence-2-large architecture graphmicrosoft/Florence-2-largeimage-text-to-text · ↓ 709k · ♡ 2kOpen in visualizer

Browse all 164 models with this in the catalog →

Related concepts

Patch embedding Patch merger / pixel shuffle Vision encoder (ViT) Cross-attention

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →