Ternary Weights, 3-Bit KV Caches, and the Limits of Quantization | Bangalore Paper Club

Описание: Ternary weights, post-training quantization, and a 3-bit KV cache. Three papers in one evening at the Bangalore Paper Club by Conscious Engines.

The Research Room brings together a small group of researchers, engineers, and builders every few weeks to work through influential ML papers. Not just discuss them, but unpack the ideas, assumptions, and math underneath.

This edition is all about making models smaller without making them worse: Microsoft's 1-bit LLM line, post-training ternarization from 512 calibration samples, and the KV cache half of the memory problem that weight quantization never touches.

━━━━━━━━━━━━━━━━━━━━
CHAPTERS
━━━━━━━━━━━━━━━━━━━━
00:00:00 Introduction
00:02:00 BitNet: Official Inference Framework for 1-bit LLMs (Microsoft)
00:18:10 CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs (ICML 2026 Oral)
00:32:40 TurboQuant: Redefining AI Efficiency with Extreme Compression (Google Research, ICLR 2026)

━━━━━━━━━━━━━━━━━━━━
PAPERS DISCUSSED
━━━━━━━━━━━━━━━━━━━━
1. BitNet: Official Inference Framework for 1-bit LLMs
Presented by Sabesh Bharathi, Researcher. What if the weights were only -1, 0, and +1? BitNet is Microsoft's ternary stack end to end: the b1.58 formulation, a 2.4B model trained on 4T tokens, and the kernels that make it actually run. Up to 6.17x faster on CPU with 55% to 82% less energy, and a 100B model running on a single laptop-class CPU.
https://github.com/microsoft/bitnet

2. CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
Presented by Shahir Abdul, Researcher. Ternary models have always meant training that way from scratch. CAT-Q makes it post-training instead, ternarizing an existing checkpoint from just 512 calibration samples while beating BitNet b1.58 with roughly 100,000x fewer training tokens. Scales to 235B parameters in under 60 GPU-hours.
https://arxiv.org/abs/2606.26650

3. TurboQuant: Redefining AI Efficiency with Extreme Compression
Presented by Rishikesh Panda, Research Lead. Weights are only half the memory problem. TurboQuant pairs PolarQuant, which eliminates the per-block quantization constants everyone else pays for, with a 1-bit Johnson-Lindenstrauss correction, giving a 3-bit KV cache with no training and no accuracy loss. Attention logits get up to 8x faster on H100.
https://research.google/blog/turboqua...

━━━━━━━━━━━━━━━━━━━━
ABOUT CONSCIOUS ENGINES
━━━━━━━━━━━━━━━━━━━━
Conscious Engines is an AI research lab building small, fast, application-specific models. Intelligence that specialises, shrinks, and disappears into products. Small. Instant. Invisible.

Website: https://consciousengines.com
Blog: https://consciousengines.com/blog
X: https://x.com/c_engines
Join us: https://consciousengines.com/join

Want to present at the next Bangalore Paper Club? Subscribe and turn on notifications. We post each edition's reading list ahead of the session.

#BitNet #Quantization #ModelCompression #EdgeAI #SLM #MachineLearning #PaperClub