Interviewer: Dense vs MoE — what's the difference?

Описание: Top ML interview question: Dense vs MoE — what's the actual difference? Surface-level answers get you cut.

Four progressive layers:
1. Essence: Dense activates all params, MoE activates a subset
2. Apples-to-apples: param-matched vs FLOPs-matched give opposite conclusions
3. Architecture: 32B Dense vs 30B MoE differ in how params are distributed
4. Load balancing evolution: aux loss → z-loss → shared experts → no aux loss (DeepSeek-V3)

Llama 3.1 405B training cost, DeepSeek-V3 flagship, Qwen3 dual-track, Qwen3-Next 27B counter-example—all here.

Layer 2 = pass. Layer 4 = offer.

⏱ Chapters
00:00 Interview Trap & Core Difference
00:40 Layer 1: Compute & Cost
02:11 Layer 2: MoE Isn't Subpar
04:04 Layer 3: Architectural Trade-offs
05:40 Layer 4: Load Balancing Evolution
07:20 Industry Trends & Summary

🌐 中文版:    • 面试官问：Dense 和 MoE 区别在哪？