A KV cache (key-value cache) is the buffer a transformer keeps, during a single request, of the attention "keys" and "values" it has already computed for every token in the context so far. When the model generates the next token, it doesn't recompute attention over the whole prompt from scratch — it reuses those stored keys and values and only computes the new ones. That's the whole trick that makes autoregressive generation tractable instead of quadratically miserable.I find it's the piece of LLM internals that people billing for tokens most often confuse with prompt caching. They're related but not the same thing, and the distinction matters once you're paying real money.
Why the KV cache matters in 2026
It matters because the KV cache is where your context window physically lives at inference time, and it scales linearly with the number of tokens in flight. Every token you keep in context is a row in that cache occupying GPU memory for the duration of the turn. A long agentic session with a fat system prompt, a pile of tool definitions, and three files the agent re-read for no reason isn't just expensive on the invoice — it's a chunk of HBM the provider has to hold open for you, and that cost gets passed through.This is also the mechanism that makes prompt caching possible at the API level. When a provider lets you reuse a stable prefix, what they're really doing is persisting the KV cache for that prefix across requests so they don't have to "prefill" it again. That's why cache reads are billed at roughly 10% of the input rate rather than free: the keys and values still have to be loaded and attended to, they just don't have to be recomputed. On Claude Opus 4.8 that turns a $5/MTok input prefix into about $0.50/MTok on a hit — worth structuring your context to earn.The practical takeaway is the same one I keep coming back to: the cheapest token is the one you never put in the cache. Pruning what the agent reads — via semantic code search instead of whole-file dumps, and output filtering on noisy command logs — shrinks the KV cache, the bill, and the latency at the same time. That's the lever Tokenade pulls automatically, and you can watch the saved tokens add up on the dashboard.
When the KV cache is the wrong thing to optimise
When the real problem is too much context, not caching. If your agent is slow and pricey because it loads everything every turn, no caching strategy saves you — you need to put less in the window. Fix the input first; see context engineering.
For one-shot calls with no shared prefix. A single isolated request builds its KV cache, uses it once, and throws it away. There's nothing to reuse and nothing to tune; the cache is just an implementation detail you'll never feel.
When you'd be hand-managing it yourself. Unless you run your own inference stack, you don't control the KV cache directly — you influence it through what you send. Reach for prompt caching and a smaller context, not for knobs the API doesn't expose.