Learn · Concepts

What is KV cache? (why long conversations eat your VRAM)

By Bitpute · Published 12 July 2026 · Updated 24 July 2026 · How we estimate · Sources · Editorial policy · Version history · Report an error

On this page

It sits on top of the weights
How big it actually gets
Why modern models are lighter here
What you can actually do about it

The weights aren't the only thing in VRAM. Every token in your context is cached too — and at long context that cache can grow larger than the model itself. Here's why, and what controls it.

Concept guideLast reviewed: July 2026

When a model reads or generates a token, it computes attention keys and values for it — and keeps them so it doesn't recompute the whole conversation on every new token. That store is the KV cache. It's what makes generation fast, and it's why a model that "fits" at short context can run out of memory in a long chat.

1It sits on top of the weights

Your total VRAM need is weights + overhead + KV cache. The first two are fixed once you pick a model and quant. The KV cache is the variable — it grows linearly with how long your context is:

KV cache (GiB)kv = 2 × layers × kv_heads × head_dim × context × bytes ÷ 2³⁰

The leading 2 is the two tensors (keys and values); bytes is 2 for an FP16 cache. Everything except context is fixed by the model's architecture — so in practice, KV cache scales with your context length.

2How big it actually gets

Llama 3.1 8B — 32 layers, 8 KV heads, head_dim 128

At 8K context ≈ 1.0 GiB. At 32K ≈ 4 GiB. At 128K ≈ 16 GiB — larger than the whole model at Q4. The context is now your biggest VRAM line item.

This is why "does it fit?" has two answers — one for a quick prompt, one for a 100K-token document. The calculator lets you set the context and watch the total move.

3Why modern models are lighter here

Notice the formula depends on kv_heads, not the number of attention heads. Older models (multi-head attention) cache one K/V per attention head. Modern models use grouped-query attention (GQA), sharing K/V across groups — often 4–8 KV heads instead of 32+. That cuts the cache 4–8× for the same context, which is a big reason recent 7-8B models handle long context on modest cards while older ones choke.

4What you can actually do about it

Use only the context you need. Loading a model with a 128K window when your prompts are 4K wastes gigabytes. Most runtimes let you cap it.
Prefer GQA models for long-context work — far smaller cache per token.
Quantize the KV cache. Many runtimes (llama.cpp, vLLM) can store it at 8-bit or even 4-bit, roughly halving or quartering the bytes term.
Keep batch at 1 for single-user local use — the cache multiplies with concurrent requests.

Rule of thumb

If a model just barely fits at short context, it won't fit at long context. Leave headroom for the cache, or the first long document will OOM you.

Size it with context → Why VRAM matters The formula

5Common questions

What is KV cache in an LLM?

The key and value tensors cached for every token already in the context window. It is stored in VRAM alongside the model weights and grows linearly with conversation length, which is why a long chat can run out of memory on a setup that started fine.

How much VRAM does KV cache use?

It depends on layer count, attention scheme and context length rather than parameter count alone. On a small model with a long window it can exceed the weights themselves; on a large model with a short window it is a minor term. The GPU Memory Calculator sizes it for a specific model and context.

How do I reduce KV cache memory?

Shorten the context, or use a runtime that quantizes the cache. Models using grouped-query attention are also far lighter here than older multi-head designs, because fewer key/value heads are stored per layer.