Quantization & the KV Cache: Fitting a Giant Model into a Tiny GPU

Описание: A 7B parameter model needs 28GB of memory at full precision, more than any consumer GPU has. Quantized to 4-bit, it fits in 3.5GB. This video works through exactly where those numbers come from, number by number.

Covers: FP32/FP16/INT8/INT4 and what each actually costs, how quantization maps real weights to integers with a shared scale factor, why the precision loss is usually tolerable, real methods (GPTQ, AWQ, GGUF), what the KV cache is and why it can cost more memory than the model's weights, why it grows with context length, and how weight precision and KV cache size together determine how large a model, how long a context, and how many users a single GPU can actually serve.

Every number on screen is computed, not just illustrated.

Chapters:
0:00 Intro
1:00 1. Precision and Memory
2:15 2. Worked Example: A 7B Model at Every Precision
3:35 3. How Quantization Works
5:21 4. Why the Precision Loss Is Tolerable
6:40 5. Real Quantization Methods
8:10 6. What the KV Cache Is
9:19 7. Worked Example: KV Cache Size
10:28 8. Why the KV Cache Grows
11:47 9. Putting It Together
13:18 10. Why This Mattered
14:37 Recap
15:46 Outro