Positions & context

Rotary position embedding (RoPE)

Encodes token positions by rotating query/key vectors. Relative distances fall out of the math for free.

First page of RoFormer: Enhanced Transformer with Rotary Position EmbeddingIntroduced inApr 2021RoFormer: Enhanced Transformer with Rotary Position EmbeddingSu et al. · arXiv 2104.09864 ↗
RoPE: position becomes rotationtokens 0..4: each position rotates Q/K a fixed angle furthersame angle step between neighborshere: +24° per position012340→22→4same gap, same angleattention only feels the distance

RoPE rotates each query and key vector by an angle proportional to its position, so the dot product between two tokens depends only on their relative distance. It needs no learned position table, extrapolates better than absolute embeddings, and is the de-facto standard in modern LLMs. Long-context variants rescale the rotation frequencies (see YaRN) to stretch the same mechanism to million-token windows.

Adoption over time

Share of new models that have included RoPE over time.

64%2022202320242025202664%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 29.7M · ♡ 2kOpen in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 11.7M · ♡ 553Open in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 11.1M · ♡ 351Open in visualizer google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 10.1M · ♡ 1kOpen in visualizer google/gemma-4-31B-it architecture graphgoogle/gemma-4-31B-itimage-text-to-text · ↓ 10.0M · ♡ 4kOpen in visualizer Qwen/Qwen3.6-27B-FP8 architecture graphQwen/Qwen3.6-27B-FP8image-text-to-text · ↓ 9.0M · ♡ 346Open in visualizer openai/gpt-oss-20b architecture graphopenai/gpt-oss-20btext-generation · ↓ 8.1M · ♡ 5kOpen in visualizer Qwen/Qwen3.5-4B architecture graphQwen/Qwen3.5-4Bimage-text-to-text · ↓ 7.5M · ♡ 812Open in visualizer Qwen/Qwen3-1.7B architecture graphQwen/Qwen3-1.7Btext-generation · ↓ 7.3M · ♡ 517Open in visualizer Qwen/Qwen2.5-VL-3B-Instruct architecture graphQwen/Qwen2.5-VL-3B-Instructimage-text-to-text · ↓ 7.1M · ♡ 684Open in visualizer Qwen/Qwen3.6-27B architecture graphQwen/Qwen3.6-27Bimage-text-to-text · ↓ 6.9M · ♡ 2kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 5.8M · ♡ 3kOpen in visualizer

Browse all 1654 models with this in the catalog →

Related concepts

NoPE (no position embedding) RoPE scaling (YaRN)

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →