SGLang vs llama.cpp: The Slot Table That Unlocked 205 tok/s MoE Decode
Автор: SkildCore
Загружено: 2026-11-09
Просмотров: 15
Описание:
A 30-billion parameter Mixture-of-Experts (MoE) model does not fit into 16 GB of VRAM. The classic fix was naive CPU offloading: spill overflow layers to host system RAM, stream them back across the PCIe bus during every forward pass, and watch generation crawl as the GPU sits starved waiting on the wire.
In SGLang v0.5.23, a new architecture called PagedExperts fundamentally changes this equation. Because MoE models only activate a tiny fraction of their experts per token (e.g., 8 out of 128), consecutive tokens naturally exhibit strong temporal locality, repeatedly querying the same working set. By keeping a dynamic slot table directly in GPU VRAM and treating system RAM as a page-locked backing store, PagedExperts turns full weight streaming into demand-paged cache hits.
On an RTX 5070 Ti (16 GB), PagedExperts decodes INT4 MoE at 205.6 tok/s (vs 16.6 tok/s on classic offload) and FP8 MoE at 87.7 tok/s (vs 2.9 tok/s—a 30x speedup).
In this episode of Skildcore, we trace a token's journey through the GPU slot table, break down the LRU eviction mechanics, compare benchmarks against tuned llama.cpp, and analyze the four failure modes where paging collapses.
---
TIMESTAMPS:
0:00 - The 16GB VRAM Wall: Why Classic MoE Offloading Collapses
0:30 - Chapter 01: Introducing PagedExperts (SGLang v0.5.23)
0:59 - The Bottleneck: Why the Wire Stalls the GPU (Qwen3-30B-A3B Breakdown)
1:44 - The Shift: Moving From Streaming to Demand Paging
2:10 - Fused Kernels on GPU: Bit-Identical Deterministic Inference
2:30 - Chapter 02: A Token's Journey Through the Slot Table (Hit vs Miss)
3:18 - Eviction Policy: What Happens When the Table Fills (LRU)
3:44 - Chapter 03: The Benchmarks (RTX 5070 Ti 16GB & Laptop GPU)
4:05 - The SGLang vs llama.cpp Comparison (INT4 vs FP8)
5:02 - Chapter 04: The 4 Limits: When Paging Stops Helping
5:08 - Limit 1: Cache Churn & Unfocused Prompt Routing
5:22 - Limit 2: Pinned Host Memory & WSL2 Hyper-V Ceilings
5:34 - Limit 3: CUDA Graph Concurrency Caps (Slots / Top-k)
6:13 - Limit 4: The Shared VRAM Budget: Resident Experts vs KV Cache
6:25 - Architectural Verdict: VRAM Is a Cache, Not a Box
---
KEY TECHNICAL TAKEAWAYS:
• Why streaming offload stalls: Autoregressive decode finishes arithmetic in microseconds, but pulling gigabytes of weights over PCIe leaves streaming multiprocessors (SMs) completely idle.
• Demand-paged slot tables: SGLang allocates a fixed slot table in GPU VRAM (e.g., 92 of 128 experts per layer) while keeping the full model in pinned host RAM. Math executes exclusively on GPU via fused kernels—only storage moves.
• Zero-transfer hits: When a token activates an expert already resident in a slot, zero bytes cross PCIe. On a miss, only the missing expert (e.g., 2.43 MB) is copied.
• INT4 vs FP8 scaling:
INT4 (RTX 5070 Ti 16GB, 92 slots): 205.6 tok/s paged vs 16.6 tok/s classic offload (llama.cpp sits at 194 tok/s).
FP8 (RTX 5070 Ti 16GB, 40 slots): 87.7 tok/s paged vs 2.9 tok/s classic offload (30x faster; llama.cpp hits 45 tok/s).
• The concurrency cap: CUDA graph capture is limited to (Slots / Top-k) concurrent requests (11 requests at K=92, 5 requests at K=40). Beyond that, steps run without graphs and incur latency penalties.
• The KV cache tradeoff: Slots and KV cache share the exact same GPU VRAM pool. Every resident expert slot reduces available context sequence capacity.
CITATIONS & IMPLEMENTATION:
• SGLang v0.5.23 Release Notes & PR (PagedExperts Implementation, Nov 2026)
• Evaluated Hardware: NVIDIA RTX 5070 Ti 16GB, RTX 4070 Laptop GPU 8GB
• Architecture Tested: Qwen3-30B-A3B (48 MoE layers, 128 experts, Top-8 routing)
#SGLang #MixtureOfExperts #MoE #GPUArchitecture #LocalLLM #MachineLearning #HardwareEngineering #RTX5070Ti #llamacpp #Skildcore
Повторяем попытку...
Доступные форматы для скачивания:
Скачать видео
-
Информация по загрузке: