CUDA graphs explained: why the GPU waits on the CPU

Описание: CUDA graphs explained: why an LLM's GPU still sits idle waiting on the CPU. Kernel launches, a decode round recorded once and replayed, and the host-side gaps left in every round, measured on one RTX 5090, plus why batching dilutes that cost and a single conversation pays all of it.

Full write-up: https://gpillon.github.io/ignis/why-i...
ignis on GitHub: https://github.com/gpillon/ignis (Apache-2.0)

Chapters
0:00 The GPU waits
0:19 What a launch is
0:37 CUDA graphs
0:52 Where the millisecond goes
1:10 Per round, not per lane
1:28 Prefill has no graph
1:50 The price
2:08 In short

The numbers in this video
Measured with ignis on one RTX 5090, Qwen3.8-27B NVFP4, hq-e8-2b KV, 262,144-token context, DFlash2 with 7 drafts (every round a width-8 verify), Nsight Systems over 8 s of steady decode, one capture per width:
1,166 graph nodes per decode round at one lane; submitting the graph takes about 370 µs, 2.2% of the round. A graph launch costs 0.36-0.73 µs per node (standalone benchmark of graphs of trivial kernels).
The card idles 1,014 µs per round at one lane and 847 µs at eight: 4.91% of the timeline at one lane, 2.27% at eight (the window includes the client's prefills). The same five host-side gaps recur every round; the largest, 389-406 µs, sits right before the graph launch.
A round at eight lanes takes 33.54 ms of graph replay against 19.18 ms at one: 1.75x the device time for 8x the lanes.
Removing the 224 copy and memset nodes per round would return about 0.16% of decode; the largest available kernel fusion about 0.5% (estimates from the captures).
Prefill, 1,024-token chunks (build of 2026-09-11; today's default chunk is 2,048): a chunk is 98.6 ms of device timeline, 90.2% kernels executing, 9.3% launch gaps over ~1,160 kernels, 0.5% the forced synchronization. About 9.2 ms of launch idle per pass is 23% of a 256-token pass (39.3 ms).
Sources: docs/findings/ in the repo (decode-round-host-idle, decode-round-anatomy, prefill-chunk-wall-time) and docs/user/why-ignis/decode-round.md.

How this video was made
Motion graphics only, every figure from the measurements above. Narration: Qwen3-TTS (AI voice). Music: ACE-Step 1.5 (AI-generated). Built with HyperFrames.

#LLM #CUDA #GPU