Positions & context

RoPE scaling (YaRN)

Rescales RoPE frequencies so a model trained on short contexts can operate on much longer ones.

First page of YaRN: Efficient Context Window Extension of Large Language ModelsIntroduced inAug 2023YaRN: Efficient Context Window Extension of Large Language ModelsPeng et al. · arXiv 2309.00071 ↗
RoPE scaling: the same waves, stretchedYaRN slows the rotations so trained phases cover a much longer context4k trainedsame 5 phases now span 128k

YaRN and related RoPE-scaling schemes stretch the rotation frequencies of rotary embeddings so that positions beyond the training length still land in a range the model understands, usually with a temperature correction on attention. It is the standard way models advertise 128k+ contexts after being pretrained on much shorter windows.

Related concepts

Rotary position embedding (RoPE) NoPE (no position embedding)

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →