llama.cpp Just Made 125B MoE Models 4× Faster!

Описание: llama.cpp's new MoE expert cache took a 125B model from 9.7 to 37.2 tokens a second on one 24 GB GPU. Here's where that 4× really comes from, and which cards get slower.

The 4× is from lich-77's experimental expert-cache branch (llama.cpp issue #29949), measured on an RX 7900 XTX running Qwen3.8-Flash-Next. Its baseline was -ncmoe 48, which keeps every one of the model's 24,576 experts in system RAM, so the card started out holding none of them. The cache that actually shipped (PR #29887, release 0.7.0, flag --moe-cache-mib) posts 2.2× on an RTX 5090 against llama.cpp's default --fit. On the same machine, the experimental branch was 1.55× faster than the merged one, and pinned memory plus 48 routing round trips per token explain most of that gap.

Whether this speeds up your local LLM depends on two things: how much of the expert pile your spare VRAM can hold, which sets your hit rate, and how fast your PCIe slot moves the misses. Below about an 85% hit rate the cache costs you speed. An 8 GB RTX 4060 laptop went from 23.4 to 15 tokens a second, a 12 GB RTX 3060 from 32 to 12, and on a 12 GB RTX 4070 SUPER plain llama.cpp beat every cache size. On a 32 GB RTX 5090 in a full x16 slot it roughly doubled decode. We also cover the Windows pinned-memory fix from October 16th (37.76 to 62.73 tokens a second), what the cache does to prompt speed and speculative decoding, how Strata compares on Qwen3.8-Flash-Next, and a 15-minute test that gives you your own number.

CHAPTERS
0:00 9.7 tokens a second on a 24 GB card
3:25 What an MoE model is
6:10 Why the GPU sits idle
7:43 The old fix: whole layers
9:12 How the expert cache works
12:13 Where the 4× actually comes from
13:44 1.55×: pinned memory and 48 waits
16:20 8 and 12 GB cards get slower
20:28 Your PCIe slot
23:03 Prompts, drafts and loops
26:40 The 15-minute test
29:20 Strata vs llama.cpp
31:15 Who wins, who loses

SOURCES
lich-77's GPU-resident LRU cache (the 4× run), issue #29949: https://github.com/ggml-org/llama.cpp...
Merged expert cache, PR #29887 (RTX 4090/5090 table, owner reports in the thread): https://github.com/ggml-org/llama.cpp...
Multi-GPU cache and the prompt-speed table, PR #30112: https://github.com/ggml-org/llama.cpp...
Pinned memory on Windows, PR #30449: https://github.com/ggml-org/llama.cpp...
Release v0.7.0: https://github.com/ggml-org/llama.cpp...
First build with the cache, b11480: https://github.com/ggml-org/llama.cpp...
Qwen3.8-Flash-Next model card: https://huggingface.co/Qwen/Qwen3.8-F...
r/LocalLLaMA PR thread (RTX 5090 and RTX 3060 runs):   / 1x03xkc  
RTX 4060 laptop runs:   / 1x0wyge  
Pinned RAM, 48 syncs per token, Strata same-PC test:   / 1wz88oq  
80B on 16 GB cache sweep:   / 1x84r67  
NoCodeAbode, 12 GB RTX 4070 SUPER test:    • llama.cpp Just Got an Expert Cache...  Doe...  
Strata README (RTX 5070 speeds): https://github.com/Niko1221/Strata

#llamacpp #LocalLLM #LocalAI