ycliper

Популярное

Музыка Кино и Анимация Автомобили Животные Спорт Путешествия Игры Юмор

Интересные видео

2025 Сериалы Трейлеры Новости Как сделать Видеоуроки Diy своими руками

Топ запросов

смотреть а4 schoolboy runaway турецкий сериал смотреть мультфильмы эдисон
Скачать

Why LLM Inference Memory Grows With Context | KV Cache Explained Visually

KV cache

KV cache explained

key value cache LLM

what is KV cache

transformer KV cache memory

CUDA out of memory

LLM inference memory

why does GPU memory keep increasing

query key value attention

grouped query attention

prefill vs decode

attention quadratic vs linear

FlashAttention

KV cache quantization

LLM serving memory

VRAM sizing LLM inference

batch size context length

PyTorch memory fragmentation

reduce KV cache size

LLM inference optimization

Автор: Ankit Wahane

Загружено: 2026-08-18

Просмотров: 480

Описание: Your LLM fits comfortably in GPU memory. Then the conversation gets longer. More users arrive. And suddenly 𝗖𝗨𝗗𝗔 𝗢𝘂𝘁 𝗼𝗳 𝗠𝗲𝗺𝗼𝗿𝘆.

What changed?

One of the biggest reasons is the 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲.

In this video, we understand KV Cache from first principles and see why GPU memory grows as context length and concurrency increase during LLM inference.

Chapter:
0:00 Same Model, Same GPU, Sudden Out-of-Memory
1:05 What Is the KV Cache? (Query, Key, Value Explained)
3:41 KV Cache Size Formula (Qwen2.5-7B Example)
4:56 What You Multiply the Cache By (Context Length + Concurrency)
7:57 Prefill vs Decode: Quadratic vs Linear Attention
9:40 Shrinking the Cache: GQA and Quantization Myths
11:41 4 Other Causes of the Same CUDA OOM Error
14:08 What's Next: Predicting Real GPU Crashes

We start with attention itself:

• Query → what the current token is looking for
• Key → what previous tokens can be matched against
• Value → the information those tokens carry

During autoregressive generation, the model repeatedly attends to tokens it has already processed.

Recomputing their Keys and Values at every generation step would waste a huge amount of computation.

So instead:

𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗼𝗻𝗰𝗲 → 𝗦𝘁𝗼𝗿𝗲 → 𝗥𝗲𝘂𝘀𝗲.

That stored memory is the 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲.

Using a 7B-class LLM configuration as a running example, we calculate how much KV-cache memory is required per token and then see what happens as we increase:

• Context length
• Number of generated tokens
• Batch size
• Concurrent users

𝗬𝗼𝘂’𝗹𝗹 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱:

✓ What Query, Key and Value actually represent
✓ Why Keys and Values are cached
✓ Why Queries do not need to be stored in the same way
✓ How KV Cache speeds up autoregressive generation
✓ How to estimate KV-cache memory per token
✓ Why longer conversations consume more GPU memory
✓ Why concurrent users multiply KV-cache usage
✓ Why a model can fit in VRAM but still OOM during inference
✓ The difference between prefill and decode
✓ Why KV-cache memory grows linearly with the number of cached tokens
✓ Why long-context prefill becomes computationally expensive
✓ Why KV Cache is only one part of the complete inference-memory picture

The key idea is simple:

𝗠𝗼𝗱𝗲𝗹 𝘄𝗲𝗶𝗴𝗵𝘁𝘀 𝗮𝗿𝗲 𝗺𝗼𝘀𝘁𝗹𝘆 𝗳𝗶𝘅𝗲𝗱.

𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗴𝗿𝗼𝘄𝘀 𝗮𝘀 𝘁𝗵𝗲 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗴𝗿𝗼𝘄𝘀.

And when multiple conversations are active at the same time, those caches quickly add up.

This is 𝗣𝗮𝗿𝘁 𝟮 of my 𝗟𝗟𝗠 𝗚𝗣𝗨 𝗠𝗲𝗺𝗼𝗿𝘆 series.

Part 1 → Where does GPU memory go during LLM training?
Part 2 → KV Cache and inference memory

𝗔𝗜 𝗕𝗲𝘆𝗼𝗻𝗱 𝘁𝗵𝗲 𝗧𝘂𝘁𝗼𝗿𝗶𝗮𝗹𝘀.

𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝗦𝘆𝘀𝘁𝗲𝗺. 𝗕𝘂𝗶𝗹𝗱 𝗕𝗲𝘁𝘁𝗲𝗿 𝗔𝗜.

— Ankit Wahane

#KVCache #LLM #LLMInference #GPUMemory #VRAM #CUDA #Transformers #Attention #GenerativeAI #AIEngineering #vLLM
#GenAIEducationIndia #aiinterviews #genai #cuda #finetuning

Не удается загрузить Youtube-плеер. Проверьте блокировку Youtube в вашей сети.
Повторяем попытку...
Why LLM Inference Memory Grows With Context | KV Cache Explained Visually

Поделиться в:

Доступные форматы для скачивания:

Скачать видео

  • Информация по загрузке:

Скачать аудио

Похожие видео

© 2025 ycliper. Все права защищены.



  • Контакты
  • О нас
  • Политика конфиденциальности



Контакты для правообладателей: [email protected]