Multimodal

Patch merger / pixel shuffle

Merges neighboring visual tokens into fewer, wider ones so the language model reads a shorter sequence.

First page of Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural NetworkGoes back toSep 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural NetworkShi et al. · arXiv 1609.05158 ↗
Patch merger: fewer, wider tokens2×2 neighborhoods of visual tokens fuse into one · the LLM reads 4× less16 tokens4 tokens, each wider

A full-resolution vision encoder emits far more tokens than a language model wants to attend over. Patch mergers (or pixel-shuffle/unshuffle stages) fuse each small neighborhood of visual tokens into one wider token. Commonly a 4× reduction. Before handing them to the LLM. It is a large inference-cost saving with modest information loss.

Adoption over time

Share of new models that have included a patch merger over time.

24%2022202320242025202624%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3.5-9B architecture graphQwen/Qwen3.5-9Bimage-text-to-text · ↓ 13.9M · ♡ 2kOpen in visualizer Comfy-Org/MiniMax-H3 architecture graphComfy-Org/MiniMax-H3↓ 11.8M · ♡ 1kOpen in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 11.7M · ♡ 553Open in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 11.1M · ♡ 351Open in visualizer Qwen/Qwen2.5-VL-7B-Instruct architecture graphQwen/Qwen2.5-VL-7B-Instructimage-text-to-text · ↓ 9.1M · ♡ 2kOpen in visualizer Qwen/Qwen3.6-27B-FP8 architecture graphQwen/Qwen3.6-27B-FP8image-text-to-text · ↓ 9.0M · ♡ 346Open in visualizer Qwen/Qwen3.5-4B architecture graphQwen/Qwen3.5-4Bimage-text-to-text · ↓ 7.5M · ♡ 812Open in visualizer Qwen/Qwen2.5-VL-3B-Instruct architecture graphQwen/Qwen2.5-VL-3B-Instructimage-text-to-text · ↓ 7.1M · ♡ 684Open in visualizer Qwen/Qwen3.6-27B architecture graphQwen/Qwen3.6-27Bimage-text-to-text · ↓ 6.9M · ♡ 2kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 5.8M · ♡ 3kOpen in visualizer Qwen/Qwen3-VL-8B-Instruct architecture graphQwen/Qwen3-VL-8B-Instructimage-text-to-text · ↓ 4.9M · ♡ 1kOpen in visualizer unsloth/Qwen3.6-27B-NVFP4 architecture graphunsloth/Qwen3.6-27B-NVFP4image-text-to-text · ↓ 4.1M · ♡ 273Open in visualizer

Browse all 539 models with this in the catalog →

Related concepts

Patch embedding Vision encoder (ViT) Multimodal projector Cross-attention

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →