The KV Cache Explained: Why LLM Decoding Is Memory-Bound

Описание: Why does a GPU that can do hundreds of trillions of operations per second still crawl through text one token at a time? The answer is the KV cache and memory bandwidth.

📖 Read this lesson as an article, chapter by chapter: https://easy-explain-app.vercel.app/l...
📱 Free Android app: https://easy-explain-app.vercel.app/d...

In this lesson we recall attention, split inference into prefill and decode, derive the KV cache size formula, and work through a Llama-style 7B example in fp16 with stated assumptions. Then we derive the arithmetic intensity of decode, use a roofline argument to show that bandwidth, not FLOPs, sets tokens per second, and look at mitigations like grouped-query attention, sliding windows and cache quantization. We finish with a PyTorch and Hugging Face loop that times generation with and without a cache.

This is episode 5 of the Generative AI and LLM Engineering series. Next up: why LLMs hallucinate.

0:00 Intro
0:39 Don't Recompute the Past
2:21 Prefill vs Decode
3:17 Deriving the Cache Size
4:22 Worked Example
5:26 Why Decode Is Memory-Bound
6:51 The Roofline Argument
8:49 Mitigations
10:00 Python: Manual KV Cache
11:46 Common Mistakes
12:29 Recap and Next Episode

#LLM #KVCache #Transformers #GenerativeAI #MachineLearning