Attention

Sliding-window attention

Each token only attends to a fixed window of recent tokens, keeping cost and cache flat as context grows.

First page of Longformer: The Long-Document TransformerIntroduced inApr 2020Longformer: The Long-Document TransformerBeltagy et al. · arXiv 2004.05150 ↗
Full attentionSliding windowevery token sees all before iteach token sees the last 4hybrid stacks alternate:SSSFSSSF

Rather than looking at the entire history, a sliding-window layer attends only to the last N tokens. Models like Gemma and gpt-oss interleave many sliding layers with occasional full-attention layers: the sliding layers handle local structure cheaply while the periodic full layers carry long-range information. This hybrid keeps the KV cache small without giving up long-context ability.

Adoption over time

Share of new models that have included sliding-window attention over time.

15%2022202320242025202615%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 10.1M · ♡ 1kOpen in visualizer google/gemma-4-31B-it architecture graphgoogle/gemma-4-31B-itimage-text-to-text · ↓ 10.0M · ♡ 4kOpen in visualizer pyannote/speaker-diarization-3.1 architecture graphpyannote/speaker-diarization-3.1automatic-speech-recognition · ↓ 9.1M · ♡ 3kOpen in visualizer openai/gpt-oss-20b architecture graphopenai/gpt-oss-20btext-generation · ↓ 8.1M · ♡ 5kOpen in visualizer answerdotai/ModernBERT-base architecture graphanswerdotai/ModernBERT-basefill-mask · ↓ 5.6M · ♡ 1kOpen in visualizer google/gemma-4-E4B-it architecture graphgoogle/gemma-4-E4B-itany-to-any · ↓ 5.2M · ♡ 1kOpen in visualizer ibm-granite/granite-embedding-small-english-r2 architecture graphibm-granite/granite-embedding-small-english-r2feature-extraction · ↓ 4.3M · ♡ 75Open in visualizer openai/gpt-oss-120b architecture graphopenai/gpt-oss-120btext-generation · ↓ 4.3M · ♡ 5kOpen in visualizer google/gemma-4-E2B-it architecture graphgoogle/gemma-4-E2B-itany-to-any · ↓ 3.9M · ♡ 894Open in visualizer google/gemma-4-12B-it architecture graphgoogle/gemma-4-12B-itany-to-any · ↓ 3.3M · ♡ 1kOpen in visualizer cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit architecture graphcyankiwi/gemma-4-26B-A4B-it-AWQ-4bitimage-text-to-text · ↓ 2.7M · ♡ 91Open in visualizer google/embeddinggemma-300m architecture graphgoogle/embeddinggemma-300msentence-similarity · ↓ 2.5M · ♡ 2kOpen in visualizer

Browse all 366 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →