Which LLM can you run on your machine? (Understand Local AI GPU Limits)

Описание: Will It Run? How to Know if a Local AI Model Fits Your Hardware (Before You Download 40GB)

DESCRIPTION
You found the perfect local model. Free, private, runs on your own machine. You copy the command, hit run, and 40GB later your computer just dies: "out of memory."

There's a way to know the answer in ten seconds of mental math, before you download anything. This video builds the whole mental model from scratch: one simple formula, the two things that quietly break it, and why "it fits" and "it's fast" are two completely different questions.

By the end you'll be able to look at any model, any GPU or Mac, and say with confidence FITS, SLOW, or WON'T LOAD.

What you'll learn:
• The one memory formula: parameters × bytes-per-parameter
• Quantization (Q8, Q4): why almost everyone runs a "compressed" model, and where it breaks
• The KV cache: how long context can silently double your memory
• Headroom: why 16GB of VRAM never gives you 16GB
• Discrete VRAM vs. unified memory (Apple Silicon), and why a bigger tank changes everything
• Offloading to system RAM: the wall, and the performance cliff behind it
• Fits ≠ fast: why memory BANDWIDTH decides your tokens-per-second
• The tools that estimate this for you, and the blind spots they miss

CHAPTERS
0:00 The trap: "it should fit"
0:42 Why the simple rule breaks (in both directions)
1:36 Weights: parameters × bytes
2:35 Quantization: the discount that rescues everyone
4:19 The KV cache: how context eats memory
6:05 Headroom: you never get all your VRAM
7:20 Two kinds of tank: VRAM vs. unified memory
8:40 Offloading: the wall and the cliff
10:05 Fits isn't fast: bandwidth is the speed limit
11:55 The whole rule, in one line
12:28 Three real examples (14B, 32B, 70B)
14:05 The tools, and their blind spots
14:57 The one-breath recap

KEY NUMBERS (rules of thumb)
• Bytes per parameter: FP16 ≈ 2, Q8 ≈ 1, Q4 ≈ 0.5
• Rough memory ≈ params(billions) × bytes-per-param, then add the KV cache + ~20% headroom
• Usable memory ≈ ~90% of total (OS/driver reserve the rest)
• Bandwidth sets decode speed: DDR5 system RAM ~50–100 GB/s · Apple unified ~400–800 GB/s · high-end GPU ~1000+ GB/s

SOURCES / GO DEEPER
• Quantization & GGUF bytes-per-weight — llama.cpp & the GGUF format docs: https://github.com/ggerganov/llama.cpp · Hugging Face quantization guide: https://huggingface.co/docs/transform...
• KV cache grows linearly with context (and why it dominates at long context) — "Efficient Memory Management for LLM Serving with PagedAttention" (vLLM), Kwon et al. 2023: https://arxiv.org/abs/2309.06180 · HF KV-cache explainer: https://huggingface.co/docs/transform...
• LLM decoding is memory-bandwidth bound (not compute bound) — background on the roofline/latency of transformer inference: https://kipp.ly/transformer-inference...
• Unified memory on Apple Silicon (shared CPU/GPU pool) — Apple: https://developer.apple.com/documenta...
• Offloading / partial GPU layers (the RAM cliff) — llama.cpp `--n-gpu-layers` & Ollama docs: https://github.com/ollama/ollama/blob...
• Estimate it yourself — Hugging Face Model Memory Calculator: https://huggingface.co/spaces/hf-acce... · check your live VRAM with `nvidia-smi`

Bandwidth figures are from published manufacturer specs (DDR5, Apple M-series, and current desktop GPUs) and vary by exact part; treat them as ballpark, not gospel.

#LocalAI #LLM #Quantization #GPU #RunLLMsLocally