JUST FUSE IT: Fixing GPU Memory Bottlenecks with kernel fusion (RMSNorm & Softmax)

Описание: Fixing GPU memory bottlenecks with kernel fusion is the key to speeding up LLMs. In this video, we write custom Triton kernels for RMSNorm & Softmax to beat PyTorch by 2X and bypass the Memory Wall.

Your GPU is sitting idle 98% of the time during these operations. Instead of relying on standard, unfused PyTorch functions that waste valuable VRAM bandwidth, we are building blazing-fast, single-pass operations. We'll process data directly in ultra-fast SRAM.

The result? Our custom causal mask implementation scales flawlessly, proving that optimizing data movement—not just compute—is mandatory for modern AI models.

Code from this episode (GitHub): https://github.com/qooba/bielik-anato...
Previous episode (MatMul from scratch):    • How to Beat PyTorch? Writing a Fast MatMul...  

Chapters:
00:00 - Fixing GPU Memory Bottlenecks (Intro)
00:44 - The Memory Wall Explained
01:23 - What Kernel Fusion Actually Means
02:30 - Writing a Single-Pass RMSNorm in Triton
04:12 - Benchmark: Custom Triton Kernel vs PyTorch
05:45 - Causal Softmax: The Ultimate Single-Pass Fusion
06:21 - Fusing the Causal Mask in Fast SRAM
07:10 - Benchmark: Beating PyTorch Softmax by 2X!
08:49 - Wrapping Up & What's Next (RoPE)

Tools & Tech used: Python, PyTorch, OpenAI Triton, CUDA GPU.
Series: Bielik Anatomy