KV Cache

The KV in KV cache stands for Key Value. Since LLMs need to infer their next token and need to look at all the previous tokens most LLMs nowadays keep a KV cache. If we didn’t do this LLMs would be very slow. According to this blog post, KV caching gave a 5x speedup.

Effects of Caching

As you can guess, caching the context stores some extra data, usually this is stored on VRAM (the graphics card’s RAM) and you can calculate the storage: $\text{KV cache bytes}=2\times n\_layers\times n\_kv\_heads\times head\_dim\times seq\_len\times batch\_size\times bytes\_per\_value$ For example, for Llama-3-8B (for one 8192 token long sequence) these values are:

Name	Value
$n\_layers$	32
$n\_kv\_heads$	8
$head\_dim$	128
$seq\_len$	8192
$batch\_size$	1
$bytes\_per\_value$	2
And putting that into our equation:
$1,073,741,824=2\times 32\times 8\times 128\times 8192\times 1\times 2$
Aka: 1.0737 gigabytes.

---

Since the KV cache has to be read for every token, KV caching increases the amount of bandwidth needed to run the model. This is why most KV caches run on the GPU rather than the CPU’s RAM or (even worse) on disk. It is also important to realise that if you have multiple inferences going on at the same time, you will need to store both the KV caches.

There is an upside to using a KV cache though, we do not have to calculate the value of all the tokens that came before which is usually worth it.

Most LLM inference providers usually store the KV cache for a limited time for you, this usually gives you a price decrease (or gives you more bang for the buck on subscription plans) which is nice! :)