← Back to Glossary
KV Cache (Key-Value Cache)
Caching previously calculated Key and Value attention states in RAM so the model doesn't recompute earlier tokens for every new word generated.
Why it exists
Generating text token-by-token requires attention over all prior tokens. Without caching, generating token N requires recomputing tokens 1 to N-1.
How it works
Saves Key and Value vectors in GPU memory during generation. Each new step only computes Key/Value for the incoming token, reducing complexity from O(N^2) to O(N).
Related terms
Where you'll use this
roadmap