I Tested NVFP4 vs FP8 KV Cache on Blackwell (One Refused to Boot)

Описание: The key-value (KV) cache is now the single largest consumer of GPU memory in LLM inference. In a 128k-token agent context on a 20B model, the KV cache alone demands 3 GB of HBM for a single conversation—and reading that entire cache at 273 GB/s on every generated token strangles throughput.

Naturally, the industry's answer is quantization: compress the cache from 16-bit BF16 down to 8-bit FP8, or all the way to NVIDIA's block-scaled 4.5-bit NVFP4.

In this Skildcore deep dive (Run 26: KV Cache Precision), we put FP8 and NVFP4 to the test on a local NVIDIA Grace Blackwell GB10 workstation (128 GB unified memory, SM121). We trace the exact byte math of gpt-oss-20b, explain why NVFP4 hard-failed at boot with a capability mismatch, unpack the "Three Locks" that gate sub-8-bit quantization, and reveal the surprise benchmark: why cutting the cache footprint in half delivered zero decode speedup.

🔥 WHAT YOU'LL LEARN:
The KV Cache Squeeze: Why every token burns 24,576 bytes in BF16 and how memory bandwidth caps decode speed.
FP8 vs. NVFP4 Format Breakdown: E4M3 (1 byte/num) vs. block-scaled 4-bit float with FP8 scale factors (4.5 bits/num, 1.8x smaller than FP8).
The SM121 Wall: Why vLLM 0.27.1 refuses to boot NVFP4 KV on GB10 / SM120 workstations while allowing SM100 / SM103 datacenter parts.
The TCGen05 Hardware Missing Link: Tensor memory instructions vs. unpacking FP4 through register files.
The Three Locks: The silicon ISA, the kernel library (FlashInfer SM100 binaries), and the serving stack checks.
Measured Bytes on GB10: 3 GB (BF16) down to 1.5 GB (FP8) and 0.75 GB (NVFP4 rate) at 128k context.
The Zero-Speedup Surprise: Why 65k-token decode took 18.2 ms on BF16 and 18.1 ms on FP8 (177 GB/s vs 172 GB/s scan rate).
Capacity vs. Speed: Why quantized KV is fundamentally an agent-capacity tool, not a latency accelerator.
The Accuracy Trap: How Hopper suffered a 91% to 13% needle-in-a-haystack crash before two-level FP32 accumulation was introduced.
Architectural Verdict: When to deploy NVFP4 (Datacenter SM100) vs. FP8 / llama.cpp Q8_0 (Workstations & Edge).

⏱️ CHAPTERS
00:00 - The Scratchpad That Eats GPU Memory: FP8 vs. NVFP4
00:51 - The Squeeze: Cache Geometry & gpt-oss-20b Math
01:43 - Two Formats: 8-Bit FP8 vs. 4.5-Bit Block-Scaled NVFP4
02:43 - The Wall: SM121 Rejects NVFP4 (vLLM Startup Check Failure)
04:04 - Three Locks on One Format: Silicon, Kernel Library, Serving Stack
04:51 - Measured on GB10: Exact Cache Footprint at 128k Context
05:27 - The Surprise Benchmark: Half the Bytes, Zero Speedup (18.2 ms vs 18.1 ms)
05:43 - Capacity Product, Not a Speed Product: The Missing Fused Dequant Kernel
06:21 - Live Proof on Hardware: llama.cpp Q8_0 at 262k Window
06:38 - The Accuracy Question: Hopper's 91% to 13% Needle Crash & FP32 Accumulation
07:19 - The Precision Ladder: Q8_0 Near-Lossless vs. Q4 Formats
07:37 - Architectural Verdict: Datacenter Blackwell (SM100) vs. Edge / Workstation (GB10)
08:04 - The Real Lesson: What Silicon, Kernels, and Stack Actually Permit

📄 TOPICS & SOFTWARE EVALUATED:
Formats: FP8 (e4m3), NVFP4 (4-bit block-scaled with FP8 scale factor), BF16
Architecture: NVIDIA Blackwell (SM100, SM103, SM120, SM121 / GB10)
Runtimes & Libraries: vLLM 0.27.1+, FlashInfer, TensorRT-LLM, llama.cpp
Model Workloads: gpt-oss-20b, 128k/262k context windows

🔔 Subscribe to Skildcore for deep architectural breakdowns of GPU hardware, LLM serving engines, and inference runtime infrastructure.

#KVCache #NVFP4 #FP8 #vLLM #Blackwell #GPUArchitecture #LLMInference #NVIDIA #AIInfrastructure #DeepLearning