llama.cpp 1.0 Is Finally Out... And It Just Got Twice as Fast
Автор: NetworkCoder
Загружено: 2026-11-16
Просмотров: 10266
Описание:
Llama.cpp 1.0 update testing on a 12GB GPU. I show how two ignored settings increased my performance by 68 percent.
Updating to llama.cpp 1.0 initially yielded identical speeds until I toggled two flags that ship disabled by default. This guide covers how to fit a 125B model on limited VRAM, what breaks during the upgrade, and how to use the built-in tuning script to find your optimal configuration. By adjusting these specific local LLM settings, I reduced prompt processing time from 150 seconds to 80 seconds.
I also break down the performance impact of FP8 KV cache and pinned memory loading. If you are running high parameter models on consumer hardware, these optimizations are essential for maintaining usable token speeds without additional cloud costs.
Subscribe for weekly local AI hardware breakdowns, and comment whether you saw a similar speed boost after your own update.
00:00 Updated to 1.0… same speed
00:55 What llama.cpp 1.0 actually is
02:06 What breaks when you upgrade
03:15 What's new (FP8 KV, server, speech)
04:13 How a 125B model fits on 12GB
06:24 The test machine & method
07:28 Test 1: just upgraded
08:00 Test 2: cache on
08:30 Test 3: the hidden setting (+52%)
09:00 Test 4: pinned loading
10:10 What didn't help
11:00 Let the tuning script decide
12:08 How much RAM do you need?
13:20 Only 64GB? What still works
14:25 The recipe & verdict
📚 Sources
• llama.cpp v1.0.0 release: https://github.com/ggml-org/llama.cpp...
• Hugging Face blog: https://huggingface.co/blog/ggml-org/...
• MoE cache docs: https://github.com/ggml-org/llama.cpp...
• Model: https://huggingface.co/ukisai/Swift-1...
🧱 The AI stack
• Engine: llama.cpp v1.0.0 (b1b770d), built from source with CUDA
• Model: Swift 1.5 125B MoE (Qwen 3.8 Flash Next based), GSQ-RCO IQ3_XXS, 76 GB
• GPU: RTX 4070 Super 12GB (PCIe Gen 3)
• CPU: 2× Xeon E5-2673 v4 (80 threads) · RAM: 125 GB
• Context 64K · q8_0 KV cache · flash attention
• Cloud API cost: $0
💻 All commands, logs and results:
https://github.com/network-tocoder/ll...
Повторяем попытку...
Доступные форматы для скачивания:
Скачать видео
-
Информация по загрузке: