Kernel Fusion and CUDA Graphs, From the Ground Up

Описание: A multiply, an add, and a sigmoid make eight trips across a tensor in GPU memory. Fused into one kernel, the same arithmetic makes four — and finishes sooner without getting any smarter.

This episode is about the two fixes for the costs we counted in episode 4:

• Kernel fusion: eight memory operations become four, three launches become one, and two temporary tensors never get allocated
• RMSNorm as eight kernels, then as one, and the residual add vLLM folds in with it
• The three patterns — vertical, horizontal, and the epilogue a compiler attaches to a matrix multiply
• Why nobody fuses the matrix multiplication itself, and what cuBLAS already does on the inside
• torch.compile and Inductor: what gets fused for you, and the Triton name clash that trips people up
• CUDA graphs: record the launches once, replay them, and why that recording is frozen to exact shapes and addresses
• How to watch both of them happen on your own code

Chapters
0:00 Round trips to memory
1:46 Fusing a transformer layer
4:52 Who writes fused kernels
6:39 The other half: launch overhead

SpeechRevolutions — the speech-to-text API I build: https://speechrevolutions.com
X: https://x.com/realameenahmed
Instagram:   / sandandlightning  
TikTok:   / sandandlightning  

I'm Ameen. I build and run a production speech-to-text API, and this channel is the engineering I use every day, explained from the ground up.