ycliper

Популярное

Музыка Кино и Анимация Автомобили Животные Спорт Путешествия Игры Юмор

Интересные видео

2025 Сериалы Трейлеры Новости Как сделать Видеоуроки Diy своими руками

Топ запросов

смотреть а4 schoolboy runaway турецкий сериал смотреть мультфильмы эдисон
Скачать

llama.cpp Cut 31 Wasted GPU Passes

AI performance

AMD Radeon

CodeMotion

GPU inference

Intel GPU

KV cache

Qwen3 MoE

Vulkan

b10517

flash attention

llama.cpp

local AI

local inference

long context

non-NVIDIA GPU

open source AI

prefill

prompt processing

q8_0

quantization

Автор: CodeMotion

Загружено: 2026-08-21

Просмотров: 1192

Описание: llama.cpp release b10517 changes a specific Vulkan flash-attention path: q8_0 key and value cache data is dequantized once for coopmat1, written into per-head-contiguous f16 scratch, and then read by the existing f16 attention path. The pull request says the old prefill path repeated that dequantization thirty-two times, once per workgroup.

The supplied A/B table for Qwen3 MoE 30B.A3B reports prompt processing rising from 200.17 to 281.99 tokens per second at 32K context, and from 99.08 to 166.17 at 65K. Token generation changes far less: 36.71 to 36.73 at 32K, and 26.27 to 27.50 at 65K. A deterministic greedy comparison at temperature zero and seed one produced byte-identical output for the reported approximately 15.5K-token prompt.

This episode also covers the limits. Scratch grows with KV-cache size; the author reports about 268 MB at 128K in one setup. The release falls back instead of aborting when scratch exceeds maxStorageBufferRange, requires a compatible KV-cache layout, skips coopmat2, adds contiguous quantized K/V tests, and excludes Intel Xe1. Community comments include positive Radeon reports and Intel A770 regressions, so the honest conclusion is to benchmark your exact model, context depth, cache format, and hardware.

Verified sources:
llama.cpp release b10517: https://github.com/ggml-org/llama.cpp...
llama.cpp pull request #25494: https://github.com/ggml-org/llama.cpp...

No universal speedup is claimed. The measured headline is faster long-context prompt processing on the cited eligible Vulkan path, not a dramatic increase in token generation.

Sources used:
https://github.com/ggml-org/llama.cpp... https://github.com/ggml-org/llama.cpp...
https://github.com/ggml-org/llama.cpp... https://github.com/ggml-org/llama.cpp...
https://api.github.com/repos/ggml-org... https://api.github.com/repos/ggml-org...
https://api.github.com/repos/ggml-org... https://api.github.com/repos/ggml-org...

Subscribe to CodeMotion for visual explainers about local AI, AI agents, model architecture, developer tools, and the systems behind modern AI.

Не удается загрузить Youtube-плеер. Проверьте блокировку Youtube в вашей сети.
Повторяем попытку...
llama.cpp Cut 31 Wasted GPU Passes

Поделиться в:

Доступные форматы для скачивания:

Скачать видео

  • Информация по загрузке:

Скачать аудио

Похожие видео

© 2025 ycliper. Все права защищены.



  • Контакты
  • О нас
  • Политика конфиденциальности



Контакты для правообладателей: [email protected]