llama.cpp's New MoE Expert Cache: Up To 2.2x Faster on One GPU

Описание: llama.cpp introduced a new GPU expert cache flag (--moe-cache-mib in PR #29887) that keeps least-recently-used Mixture-of-Experts weights in VRAM to speed up offloaded inference. In benchmark testing on an RTX 5090, this new cache delivered up to 2.20x faster generation speeds for offloaded MoE models like Qwen 3.8 Flash-Next.

How the new dynamic least-recently-used expert cache works compared to static flags like --n-cpu-moe and --fit.
Benchmark results across RTX 5090, RTX 4090, RTX 5060 Ti, and Radeon AI PRO R9700 GPUs.
Key trade-offs, including prompt processing speed impact and why hit rates below 75% hurt performance.
A step-by-step checklist to test --moe-cache-mib on your own GPU in 15 minutes.

What hit rate and generation speed do you get when testing the new expert cache on your setup?

Chapters:
0:00 Intro
0:44 How the Expert Cache Works
2:35 Performance Expectations
3:18 GPU Benchmark Results
6:00 Surprises and Gotchas
7:36 Which Setup to Pick
8:46 How to Try It Yourself