llama.cpp Cut 31 Wasted GPU Passes
Автор: CodeMotion
Загружено: 2026-08-21
Просмотров: 1192
Описание:
llama.cpp release b10517 changes a specific Vulkan flash-attention path: q8_0 key and value cache data is dequantized once for coopmat1, written into per-head-contiguous f16 scratch, and then read by the existing f16 attention path. The pull request says the old prefill path repeated that dequantization thirty-two times, once per workgroup.
The supplied A/B table for Qwen3 MoE 30B.A3B reports prompt processing rising from 200.17 to 281.99 tokens per second at 32K context, and from 99.08 to 166.17 at 65K. Token generation changes far less: 36.71 to 36.73 at 32K, and 26.27 to 27.50 at 65K. A deterministic greedy comparison at temperature zero and seed one produced byte-identical output for the reported approximately 15.5K-token prompt.
This episode also covers the limits. Scratch grows with KV-cache size; the author reports about 268 MB at 128K in one setup. The release falls back instead of aborting when scratch exceeds maxStorageBufferRange, requires a compatible KV-cache layout, skips coopmat2, adds contiguous quantized K/V tests, and excludes Intel Xe1. Community comments include positive Radeon reports and Intel A770 regressions, so the honest conclusion is to benchmark your exact model, context depth, cache format, and hardware.
Verified sources:
llama.cpp release b10517: https://github.com/ggml-org/llama.cpp...
llama.cpp pull request #25494: https://github.com/ggml-org/llama.cpp...
No universal speedup is claimed. The measured headline is faster long-context prompt processing on the cited eligible Vulkan path, not a dramatic increase in token generation.
Sources used:
https://github.com/ggml-org/llama.cpp... https://github.com/ggml-org/llama.cpp...
https://github.com/ggml-org/llama.cpp... https://github.com/ggml-org/llama.cpp...
https://api.github.com/repos/ggml-org... https://api.github.com/repos/ggml-org...
https://api.github.com/repos/ggml-org... https://api.github.com/repos/ggml-org...
Subscribe to CodeMotion for visual explainers about local AI, AI agents, model architecture, developer tools, and the systems behind modern AI.
Повторяем попытку...
Доступные форматы для скачивания:
Скачать видео
-
Информация по загрузке: