Generation reuses cached keys/values of all previous tokens instead of recomputing them. Memory becomes the bottleneck.
When a model generates token by token, the keys and values of every earlier token are stored so each new token only computes attention against the cache instead of re-running the whole sequence. The cache grows with context length × layers × KV heads × head dim, and it. Not compute. Is usually what limits batch size and context in serving. Most modern attention variants (GQA, MLA, sliding windows, linear attention) are primarily KV-cache-size optimizations.
hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.
Browse all model graphs →