Why LLM Inference Memory Grows With Context | KV Cache Explained Visually
Автор: Ankit Wahane
Загружено: 2026-08-18
Просмотров: 480
Описание:
Your LLM fits comfortably in GPU memory. Then the conversation gets longer. More users arrive. And suddenly 𝗖𝗨𝗗𝗔 𝗢𝘂𝘁 𝗼𝗳 𝗠𝗲𝗺𝗼𝗿𝘆.
What changed?
One of the biggest reasons is the 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲.
In this video, we understand KV Cache from first principles and see why GPU memory grows as context length and concurrency increase during LLM inference.
Chapter:
0:00 Same Model, Same GPU, Sudden Out-of-Memory
1:05 What Is the KV Cache? (Query, Key, Value Explained)
3:41 KV Cache Size Formula (Qwen2.5-7B Example)
4:56 What You Multiply the Cache By (Context Length + Concurrency)
7:57 Prefill vs Decode: Quadratic vs Linear Attention
9:40 Shrinking the Cache: GQA and Quantization Myths
11:41 4 Other Causes of the Same CUDA OOM Error
14:08 What's Next: Predicting Real GPU Crashes
We start with attention itself:
• Query → what the current token is looking for
• Key → what previous tokens can be matched against
• Value → the information those tokens carry
During autoregressive generation, the model repeatedly attends to tokens it has already processed.
Recomputing their Keys and Values at every generation step would waste a huge amount of computation.
So instead:
𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗼𝗻𝗰𝗲 → 𝗦𝘁𝗼𝗿𝗲 → 𝗥𝗲𝘂𝘀𝗲.
That stored memory is the 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲.
Using a 7B-class LLM configuration as a running example, we calculate how much KV-cache memory is required per token and then see what happens as we increase:
• Context length
• Number of generated tokens
• Batch size
• Concurrent users
𝗬𝗼𝘂’𝗹𝗹 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱:
✓ What Query, Key and Value actually represent
✓ Why Keys and Values are cached
✓ Why Queries do not need to be stored in the same way
✓ How KV Cache speeds up autoregressive generation
✓ How to estimate KV-cache memory per token
✓ Why longer conversations consume more GPU memory
✓ Why concurrent users multiply KV-cache usage
✓ Why a model can fit in VRAM but still OOM during inference
✓ The difference between prefill and decode
✓ Why KV-cache memory grows linearly with the number of cached tokens
✓ Why long-context prefill becomes computationally expensive
✓ Why KV Cache is only one part of the complete inference-memory picture
The key idea is simple:
𝗠𝗼𝗱𝗲𝗹 𝘄𝗲𝗶𝗴𝗵𝘁𝘀 𝗮𝗿𝗲 𝗺𝗼𝘀𝘁𝗹𝘆 𝗳𝗶𝘅𝗲𝗱.
𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗴𝗿𝗼𝘄𝘀 𝗮𝘀 𝘁𝗵𝗲 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗴𝗿𝗼𝘄𝘀.
And when multiple conversations are active at the same time, those caches quickly add up.
This is 𝗣𝗮𝗿𝘁 𝟮 of my 𝗟𝗟𝗠 𝗚𝗣𝗨 𝗠𝗲𝗺𝗼𝗿𝘆 series.
Part 1 → Where does GPU memory go during LLM training?
Part 2 → KV Cache and inference memory
𝗔𝗜 𝗕𝗲𝘆𝗼𝗻𝗱 𝘁𝗵𝗲 𝗧𝘂𝘁𝗼𝗿𝗶𝗮𝗹𝘀.
𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝗦𝘆𝘀𝘁𝗲𝗺. 𝗕𝘂𝗶𝗹𝗱 𝗕𝗲𝘁𝘁𝗲𝗿 𝗔𝗜.
— Ankit Wahane
#KVCache #LLM #LLMInference #GPUMemory #VRAM #CUDA #Transformers #Attention #GenerativeAI #AIEngineering #vLLM
#GenAIEducationIndia #aiinterviews #genai #cuda #finetuning
Повторяем попытку...
Доступные форматы для скачивания:
Скачать видео
-
Информация по загрузке: