Positions & context

NoPE (no position embedding)

This path deliberately skips position encoding. The model relies on causal masking alone to sense order.

Introduced inMay 2023The Impact of Positional Encoding on Length Generalization in TransformersKazemnejad et al. · arXiv 2305.19466 ↗

Some recent models apply RoPE only to part of the head dimensions or only in some layers, leaving the rest position-free (‘NoPE’). Because causal masking already gives a weak sense of order, dropping explicit positions in places can improve length generalization: the model overfits less to the exact positions it saw in training. You will see graphs split query/key paths into an explicitly rotated part and a NoPE part.

Adoption over time

Share of new models that have included NoPE over time.

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Related concepts