Can We Actually Run 27B Models on Your Laptop?

Описание: How does a 27B LLM fit in 6GB? Ternary Bonsai 2 27B (1.58-bit weights), GSQ-RCO quantization and Cloudflare Clef, the decision model that answers in one pass. The math of local AI on a 16GB Mac.

Qwen 3.8 27B weighs 54 GB at 16 bits, and every generated token reads the whole model. Memory traffic = weights x bits x passes. Quantization cuts the bits: 4-bit keeps 98.7% but is still 17.6 GB, while a standard 2-bit build collapses on reasoning (AIME26 drops from 95 to 57). Prism ML's Bonsai 2 uses ternary weights (-1, 0, +1, log2(3) = 1.58 bits) with Hadamard rotation: 5.95 GB and 98.2% of the full model's score, 28.7 tok/s on an M5 Pro. ISTA's GSQ-RCO gives each tensor its own precision: 3.5 bits matches the full model on AIME25 and LiveCodeBench.

Cloudflare Clef cuts the passes: a decision model built on Qwen that reads the input once and returns a probability for every allowed answer, with no text generation. Clef Flash: 39 ms median latency vs 524 ms for Jev System One. At 4 bits it runs on a 16GB Mac in 0.31 s. Then the receipt: a million decisions on one Mac.

Subscribe for daily breakdowns of local AI with real numbers, and tell me in the comments which model you'd run and on what hardware.

Chapters:
0:00 Intro
0:35 Why 54 GB
1:25 Shrinking a weight
3:08 1.58 bits
7:08 The model that can't talk
9:02 Stack both

#LocalAI #LLM #Quantization