They Shrank Qwen 27B from 54GB to 6GB. How Much Did It Keep?

Описание: PrismML's Bonsai 2 compresses Qwen3.8-27B's language-model weights to 5.9GB. PrismML reports retaining 98.2% of its average benchmark score—but the losses vary substantially by task.

You'll learn:

Why ordinary 2-bit compression usually breaks long reasoning first
How ternary weights (−1, 0, +1 with a shared scale) get to about 1.7 bits per weight
Why a Hadamard rotation helps when weights have outliers
Why Bonsai 2 needs custom kernels and PrismML's own runtime (stock llama.cpp can't run it)
What 98.2% does and doesn't mean, including about three-quarters of the original on long-running agent tasks
What "6 GB" counts, and what it leaves out

All benchmark, speed and size figures are PrismML's own measurements; we haven't found an independent reproduction. We didn't run the model for this video. Diagrams are illustrations.

Sources:

PrismML whitepaper: https://github.com/PrismML-Eng/Bonsai...
PrismML launch post: https://prismml.com/news/bonsai-2-27b
Model cards: https://huggingface.co/prism-ml/Terna... and https://huggingface.co/prism-ml/Terna...
SpinQuant (Liu et al., 2024): https://arxiv.org/abs/2405.16406

Chapters:
0:00 27B in 6 GB: the result
1:07 What's actually being shrunk
1:46 Why 2-bit usually breaks reasoning
2:55 Ternary: −1, 0, +1
4:13 The Hadamard rotation
5:51 Custom kernels
7:45 98% of what?
9:46 What it changes for you
10:30 The answer

Subscribe to Model and Machine for clear explanations of what makes AI systems work, where they fail, and what that changes for you.

#AI #LocalAI #LLM