How Mixture-of-Experts Routes Every Token

Описание: A mixture-of-experts (MoE) model uses a tiny learned router to choose which expert networks run for each token. This explainer traces top-k routing, sparse activation, gate renormalization, load balancing, and why parameters stored are not parameters activated.

Mixtral stores about 47 billion parameters, but only about 13 billion are active for one token. The rest are not pruned or missing—they simply are not selected on that token's path. We follow the hidden state into the router, execute a four-expert top-2 example, and show exactly how probabilities 0.5 / 0.3 / 0.15 / 0.05 become gates 0.625 / 0.375 and an output of [5, 3].

You'll learn:
• Where the experts sit inside a transformer block
• Why 8×7B does not mean 56B stored parameters
• How one linear router scores every expert
• What top-2 selection skips—and what still stays in memory
• Why selected gate weights are renormalized
• How router collapse and expert imbalance reinforce themselves
• What auxiliary balancing losses and capacity limits do
• Why Mixtral's experts did not become clean topic specialists
• How Mixtral and DeepSeek-V3 separate stored from active parameters

Scope: the worked arithmetic is an executed, hand-set PyTorch teaching example, not a measurement of a shipped model. The 47B/13B pair and routing observations are Mixtral-specific. Switch, Mixtral, and DeepSeek-V3 use related but distinct routing and balancing designs; hash routing and expert-choice routing are outside this video's scope.

CHAPTERS
00:00 Thirteen of forty-seven
00:36 Where the parameters live
01:43 The router is a matrix
02:49 Keep two, drop two
03:36 The weighted mix
04:23 Every token, every layer
05:00 The routing feedback loop
05:48 Put a price on imbalance
06:46 Nobody hired a math professor
07:48 Ask how many run

NEXT: How the KV Cache Makes LLM Inference Fast
   • How the KV Cache Makes LLM Inference Fast  

PRIMARY SOURCES
Mixtral of Experts: https://arxiv.org/abs/2401.04088
Switch Transformer: https://arxiv.org/abs/2101.03961
Sparse expert gating: https://arxiv.org/abs/1701.06538
GShard: https://arxiv.org/abs/2006.16668
ST-MoE: https://arxiv.org/abs/2202.08906
DeepSeek-V3: https://arxiv.org/abs/2412.19437
OLMoE: https://arxiv.org/abs/2409.02060
Transformer feed-forward parameter analysis: https://arxiv.org/abs/2012.14913

Ground Truth — not just how, but why.

#MixtureOfExperts #LLM #MachineLearning