Janas: Run MoE Models Larger Than RAM from Your SSD?

Описание: Can a 32 GB laptop run a 48.4 GB mixture-of-experts model from its SSD? Janas kept the weights on a 5.7 GB/s NVMe drive, cached about 19 GB of experts in RAM, and generated at 22 tokens/s, or 37 tokens/s with drafting.

This video explains how expert streaming changes the memory problem and compares Janas with llama.cpp and resident models. It also examines cache behavior, prompt versus generation speed, quantization support, PCIe limits, RAM headroom, storage capacity, and SSD endurance.

Topics covered:

• How Janas runs a 48.4 GB MoE model on a 32 GB laptop
• Why sparse MoE routing changes the memory requirement
• NVMe expert streaming, caching, prefetching, and cache misses
• Qwen3-Next-80B-A3B prompt and generation benchmarks
• Janas versus llama.cpp for models larger than RAM
• Supported model families, architectures, and quantization paths
• PCIe bandwidth, desktop contention, and disk-wait limits
• RAM headroom, conversion workspace, and SSD capacity planning
• Resident small models versus larger streamed models
• SSD read traffic, writes, and endurance ratings

For those of you using 32 GB laptops, would you stream the larger model or keep a smaller model resident for steadier latency?

Every number in this video comes from the sources linked below. Corrections and your own measurements are welcome in the comments.

Subscribe to Tensor Garage for practical local AI, GPU and open-model breakdowns without the marketing fluff.

Research and pricing were checked in October 2026. Hardware prices, availability and runtime support change quickly, so verify current details before purchasing.

Narration uses a synthetic voice blended from synthetic voice samples.

Chapters:
00:00 - How Expert Streaming Works
01:57 - Models and Quant Formats
03:18 - What the Speed Means
05:29 - Planning RAM and Capacity
07:07 - When Resident Models Win
08:28 - SSD Wear Without Guesswork

Sources:
[1] GitHub - prabanta-dev/janas: 37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU. — https://github.com/prabanta-dev/janas
[2] deepseek-ai/DeepSeek-V3 · Hugging Face — https://huggingface.co/deepseek-ai/De...
[3] Qwen/Qwen3-235B-A22B · Hugging Face — https://huggingface.co/Qwen/Qwen3-235...
[4] mistralai/Mixtral-8x7B-v0.1 · Hugging Face — https://huggingface.co/mistralai/Mixt...
[5] unsloth/Qwen3-235B-A22B-GGUF · Hugging Face — https://huggingface.co/unsloth/Qwen3-...
[6] Performance of llama.cpp on Apple Silicon M-series · ggml-org llama.cpp · Discussion #4167 — https://github.com/ggerganov/llama.cp...
[7] MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache — https://arxiv.org/abs/2401.14361
[8] LLM in a flash: Efficient Large Language Model Inference with Limited Memory — https://arxiv.org/abs/2312.11514
[9] Samsung 990 Pro SSD Review: The Return of the King — https://www.tomshardware.com/reviews/...

#Janas #MoE #LocalLLM #Qwen3 #NVMe