NCM Is this the End of LLMs?

Описание: Is Stanford's Next-Concept Modeling (NCM) the end of LLMs? It writes text 31x faster, but against which model, and does any of it reach your GPU?

Every chatbot you use, ChatGPT and Gemini included, writes one token at a time, and every token makes your GPU read the model's full weights out of memory. That is why large language models feel slow on your own machine. NCM, from Stanford and Together AI, skips that. It writes in levels, a rough sketch first and the words last, using a multiscale residual VQ-VAE and an ordinary transformer that fills a whole level per pass. In this video I read the paper, the GitHub repo and the Hugging Face checkpoints to find out what it actually beats, what it loses to, and what it changes for anyone running local AI.

Watch next from this series:
The End of VRAM-Bottlenecked LLMs: Qwen3.8-Flash-Next →    • The End of VRAM-Bottlenecked LLMs: Qwen3.8...  
The End of AI Hallucinations? Meta's New Fix (CLM's) →    • The End of AI Hallucinations? Meta's New F...  
Why Your Mac Is Slow at Local AI (And It's Not the Chip) →    • Why Your Mac Is Slow at Local AI (And It's...  

Chapters
0:00 The 2 AM cursor: why AI writes so slowly
0:52 31x faster? The post half a million people read
2:00 One token, one full weight read
4:13 1% vs 86%: the Sudoku table
5:55 How NCM works: sketch first, words last
9:20 NCM vs diffusion language models
11:40 What the 31x actually measures
13:49 Translation: word by word still wins
15:49 Count the weight reads on your GPU
18:41 Why the data center doesn't need this
20:39 Five bytes: what keeps NCM off your machine
23:11 DiffusionGemma: what you can run tonight
25:12 Verdict: buy bandwidth today, read the compute line

What's inside:
• Why one token at a time means one full read of the weights. A 16 GB model writing 1,024 tokens moves about 16,000 GB, roughly 9 seconds of memory traffic on an RTX 5090 at 1,792 GB/s.
• The Sudoku result: a 25M NCM solved 86% of hard boards, where a same-size word-by-word model solved 1%.
• How NCM differs from diffusion LLMs like Inception's Mercury and Google's DiffusionGemma: diffusion rewrites the whole draft many times, NCM writes the outline once and the detail once.
• The 31x is one test: 48-token XSum summaries, 8 passes vs 128 for the AURORA-LM diffusion model, on one H100 at batch 16. The paper never times NCM against a word-by-word model.
• The 88M generator needs a second 61M network (149M total), and the number of passes is fixed per model (8, 11, 32).
• Where plain word by word wins: 25.2 vs 22.6 BLEU on German-to-English. NCM only ties at 132 passes against about 64.
• The part that matters for local AI: 32 weight reads instead of 1,024. The bottleneck moves from memory bandwidth to compute, which helps one person at batch size 1 more than a data center.
• The limits: a fixed-size output box, 725M parameters at most, evaluation-only code, CUDA only, and a web-text checkpoint that is a 5-byte TODO file.
• DiffusionGemma today: 25.2B total, 3.8B active, Apache 2.0, 15–20 tokens per pass, and 8–19 points behind Gemma 4 (AIME 2026 69.1 vs 88.3).

🔗 Sources
NCM paper (arXiv 2610.14162): https://arxiv.org/abs/2610.14162
NCM code: https://github.com/HazyResearch/ncm
NCM checkpoints: https://huggingface.co/hazyresearch/n...
Hazy Research blog: https://hazyresearch.stanford.edu/blo...
Hermann Kumbong's launch thread: https://x.com/KumbongHermann/status/2...
Meta Large Concept Models: https://arxiv.org/abs/2412.08821
Inception Mercury: https://www.inceptionlabs.ai/blog/int...
DiffusionGemma model card: https://huggingface.co/google/diffusi...
Google DiffusionGemma blog: https://blog.google/innovation-and-ai...
ORL / LLaDA 2.1 mini on r/LocalLLaMA:   / 1x7polk  
RTX 50 series memory bandwidth: https://en.wikipedia.org/wiki/GeForce...

#llm #ai #localai