How to make Big AI models fit on small hardware

Описание: A walkthrough of EXL3 quantization and compression — why it preserves more
intelligence than the alternatives, and how to use it to run models several
times larger than your hardware should allow.

I go through the bit math from scratch (why a 320B model needs 640 GB at
16-bit), how EXL3 differs from NVFP4, mixed-bit quantization, and pruning a
model by feeding it calibration data and cutting whatever doesn't light up.

The worked example: DeepSeek V4 Flash, 284B parameters, compressed to ~3 bits
per weight and pruned 18.5% until it fits on a single $4,500 DGX Spark. On
GLM-5.2 the same recipe scored 78.8% on TerminalBench 2.1 (vs 81% reported)
and 88.7% on GPQA Diamond, with a KLD of ~0.11 — roughly a fifth of the size.

Recorded live, so there's Q&A throughout: NVFP4 KV cache and long context,
what hardware you need to quantize, and where to start if you're new to this.

CHAPTERS
00:00 What EXL3 is and why it's better
00:37 The bit math: why a 320B model needs 640 GB
03:10 NVFP4 vs EXL3
03:50 What hardware EXL3 runs on
04:35 How NVFP4 scalars actually work
05:16 Why grouping weights beats quantizing them one by one
06:53 Why EXL3 took years to catch on
07:58 What you need to quantize your own models
08:40 The breakthrough: mixed-bit EXL3 and the B12X kernels
09:40 3-bit GLM-5.2: TerminalBench, GPQA and KLD results
10:16 Mixed-bit explained: different widths for different weights
11:10 Case study: DeepSeek V4 Flash
12:17 Making it fit on a single DGX Spark
13:03 Pruning 18.5% using calibration data
14:36 How close the compressed model stays to the original
15:30 Q&A: NVFP4 KV cache and long context
16:27 Managing a GPU fleet with agents — the real time sink
17:04 Rule of thumb: quantize a model 3× your compute
18:08 Q&A: my hardware — 6000s, DGX Sparks, 3090s, B70s
18:58 Building all of this with local models
19:45 Live demo: pointing an agent at DeepSeek V4 Flash Vision
20:34 Auto-research loops that run without me
22:38 The local AI hardware + model registry
24:19 Built into Omarchy: scan your hardware, get a launch command
25:30 Q&A: what hardware you need to quantize
26:03 LoRA and "caveman" reasoning traces (NEXN2)
26:54 Open PRs and incentivising better quants
27:18 LocalMaxxing, and why this needs to exist
28:47 Q&A: where to start — turboderp, ExLlamaV3, LM Studio
29:52 Closing

LINKS
Quantized models on Hugging Face: [your HF URL]
Local AI registry (GitHub): [repo URL]
LocalMaxxing: [URL]
Julia Turk's channel (quantization deep dives): [URL]
ExLlamaV3 by turboderp: [URL]
Discord: [invite]