Multimodal

Cross-attention

Queries come from one sequence, keys/values from another. How a decoder reads an encoder’s output.

First page of Neural Machine Translation by Jointly Learning to Align and TranslateIntroduced inSep 2014Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau et al. · arXiv 1409.0473 ↗
Cross-attention: reading a different sequencequeries come from the current stream · keys and values from the other onedecoderencoder output (audio, image, prompt)

In cross-attention the queries belong to the current stream (say, generated text or noisy image latents) while keys and values come from a different one (encoder output, a text prompt, audio features). It is the bridge in encoder-decoder models like Whisper and the conditioning mechanism in diffusion U-Nets, where image latents cross-attend into the text prompt’s embeddings.

Adoption over time

Share of new models that have included cross-attention over time.

3%202220232024202520263%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

openai/whisper-large-v3-turbo architecture graphopenai/whisper-large-v3-turboautomatic-speech-recognition · ↓ 7.9M · ♡ 3kOpen in visualizer openai/whisper-large-v3 architecture graphopenai/whisper-large-v3automatic-speech-recognition · ↓ 5.0M · ♡ 6kOpen in visualizer openai/whisper-base architecture graphopenai/whisper-baseautomatic-speech-recognition · ↓ 3.3M · ♡ 285Open in visualizer facebook/bart-large-mnli architecture graphfacebook/bart-large-mnlizero-shot-classification · ↓ 3.2M · ♡ 2kOpen in visualizer microsoft/Florence-2-base architecture graphmicrosoft/Florence-2-baseimage-text-to-text · ↓ 2.8M · ♡ 392Open in visualizer openai/whisper-small architecture graphopenai/whisper-smallautomatic-speech-recognition · ↓ 2.5M · ♡ 582Open in visualizer facebook/sam3 architecture graphfacebook/sam3mask-generation · ↓ 2.3M · ♡ 3kOpen in visualizer facebook/musicgen-medium architecture graphfacebook/musicgen-mediumtext-to-audio · ↓ 1.9M · ♡ 164Open in visualizer openai/whisper-tiny architecture graphopenai/whisper-tinyautomatic-speech-recognition · ↓ 1.5M · ♡ 437Open in visualizer facebook/bart-large-cnn architecture graphfacebook/bart-large-cnnsummarization · ↓ 1.5M · ♡ 2kOpen in visualizer stabilityai/stable-diffusion-xl-base-1.0 architecture graphstabilityai/stable-diffusion-xl-base-1.0text-to-image · ↓ 1.5M · ♡ 8kOpen in visualizer microsoft/table-transformer-structure-recognition architecture graphmicrosoft/table-transformer-structure-recognitionobject-detection · ↓ 1.3M · ♡ 226Open in visualizer

Browse all 177 models with this in the catalog →

Related concepts

Patch embedding Patch merger / pixel shuffle Vision encoder (ViT) Multimodal projector

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →