Mixture of experts (MoE): why big AI models only use part of themselves
A mixture-of-experts model splits part of its network into many smaller “experts” and adds a router that sends each token to only a few of them. So a model can be huge in total but cheap to run per token: DeepSeek-V3 has 671B parameters but uses 37B for each token. The catch is memory: every expert must still be loaded, so speed follows the active parameters and memory follows the total.
- 1
What is a mixture of experts?
A normal (dense) model runs every one of its parameters for every token it reads. A mixture-of-experts model replaces one part of each transformer layer, the feed-forward block, with many smaller copies called experts, plus a small router that picks which experts each token should use.
Think of a kitchen with 128 cooks where each order goes to only 2 or 8 of them. The kitchen holds a lot of skill, but each dish costs the time of a few cooks.
One warning about that picture: experts are not topic specialists. Studying Mixtral’s router, Mistral did “not observe obvious patterns in the assignment of experts based on the topic”; routing followed syntax, such as code indentation, more than subject.
- Token
- Router scores the experts
- Top few experts run
- Outputs mixed by score
Key ideaMoE gives a model many parameters to store knowledge in, but only runs a few of them for each token.
- 2
Where it came from: 1991 to 2017 to today
The idea is old. In 1991 Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton described systems “composed of many separate networks”, each learning part of a task. It became practical for language in 2017, when a Google Brain team led by Noam Shazeer put thousands of experts inside one layer and reported “greater than 1000x improvements in model capacity” with small losses in efficiency.
In 2021 Google’s Switch Transformer simplified it to one expert per token and reported up to 7x faster pre-training than a dense model with the same compute. Mistral’s Mixtral 8x7B (Dec 2023) made open MoE models popular, and many large open-weight releases since are MoE, including DeepSeek-V3, Llama 4, Qwen3, gpt-oss and Kimi.
Timeline 1991 Jacobs, Jordan, Nowlan, Hinton "adaptive mixtures of local experts" 2017 Shazeer et al. (Google Brain) sparsely-gated MoE, 137B params 2021 Switch Transformer 1 expert per token, 1.6T params 2023 Mixtral 8x7B (open weights) 2024 DeepSeek-V3: 671B total, 37B active 2026 Kimi K3: 2.8T total, 104B active (K3.1 Pro, reported Nov 2026: same) - 3
Total vs active parameters: the two numbers on every model card
Model cards for MoE models now give two sizes. Total parameters is everything stored in the file. Active parameters is how many are used to process one token, which sets the compute cost and most of the speed.
Mixtral 8x7B is a good first example. It has 8 experts per layer, but only the feed-forward part is split into experts and the rest is shared, so the total is 46.7B, not 8 × 7 = 56B. The router picks 2 experts per layer for each token, so it uses 12.9B parameters per token.
Newer models are much sparser. Mixtral used 27.6% of itself per token; DeepSeek-V3 uses 5.5%, Kimi K2 3.2% and Kimi K3 3.7% (computed from the makers’ numbers in the table below).
Active share per token (maker numbers) Mixtral 8x7B 12.9 / 46.7B = 27.6% Llama 4 Scout 17 / 109B = 15.6% DeepSeek-V3 37 / 671B = 5.5% gpt-oss-120b 5.1 / 117B = 4.4% Kimi K3 104 / 2,800B = 3.7%Key ideaSpeed and cost per token follow ACTIVE parameters. File size follows TOTAL.
- 4
The catch: memory, balance and fine-tuning
Memory does not shrink. The router can pick any expert for the next token, so every expert must be loaded. Hugging Face puts it plainly: MoE models “require high VRAM as all experts are loaded in memory”. Mixtral needs the memory of a 46.7B model while each token costs the compute of a 12.9B one. Some local tools work around this: llama.cpp can “keep all Mixture of Experts (MoE) weights in the CPU” RAM and run the rest on the GPU, which is slower but lets bigger models run.
Balance is hard. Shazeer’s team found the router tends to favour the same few experts, and “this imbalance is self-reinforcing”, so they added an extra training loss to spread the work. DeepSeek-V3 mainly uses a small bias on each expert’s routing score, nudged up or down during training, plus a light “complementary sequence-wise balance loss”.
Fine-tuning is trickier. Hugging Face’s 2023 guide notes sparse models “are more prone to overfitting”, and reports that freezing the expert layers and tuning the rest worked almost as well as tuning everything.
- Memory: plan for the total parameters
- Speed: expect roughly the speed of a dense model the size of the active parameters
- Training: needs load balancing so experts share the work
- Fine-tuning: try freezing the expert layers first (Hugging Face, 2023)
Key ideaMoE trades memory for speed. It is cheaper to run, not cheaper to store.
- 5
How to use this: reading model names and picking models
Some names carry both numbers. Qwen3-235B-A22B means 235B total, 22B active (A for active). When a launch post says “2.8T parameters”, look for the active figure before you compare it with a dense model.
For your own laptop, memory decides. At 4-bit, weights take about half a gigabyte per billion total parameters, so gpt-oss-20b (21B total) needs at least 10.5 GB and OpenAI says it runs within 16 GB, while Llama 4 Scout (109B) needs at least 54.5 GB. Real 4-bit files are a little bigger, because they store scaling numbers and keep some layers at higher precision. Use the calculator below to check a model before you download it.
Weights at 4-bit = total billions × 0.5 GB gpt-oss-20b 21B → ~10.5 GB Mixtral 8x7B 46.7B → ~23.4 GB Llama 4 Scout 109B → ~54.5 GB DeepSeek-V3 671B → ~335.5 GB (a floor: real files are a bit bigger, plus room for the conversation and apps)Key ideaTo judge an MoE model: total parameters tell you if it fits; active parameters tell you how fast it runs.
Straight from the source
The original idea, then the same idea taken apart, then how you use it today.
- 1 · Understand
- Use conditional computation, where “parts of the network are active on a per-example basis”. A trainable gating network picks a sparse combination of up to thousands of experts, which grew capacity more than 1000x with small efficiency losses.
- 2 · Break it down
- Conditional computation: run only the parts of the network a given input needs.
- Noisy top-k gating: score all experts, add a little noise, keep the top k, ignore the rest.
- Their largest MoE layer held up to 137 billion parameters, huge for 2017.
- Problem found: the router favours a few experts, so they added a loss to balance the load.
- 3 · Use it
- When you read “1T parameters” in a launch post, remember this paper: size alone no longer tells you the cost. Ask how many parameters are active per token.
- 1 · Understand
- A 671B-parameter MoE with 37B activated per token, trained on 14.8 trillion tokens using 2.788M H800 GPU hours, with load balancing done mainly by a routing bias rather than a large extra loss.
- 2 · Break it down
- Each MoE layer has 1 shared expert every token uses, plus 256 routed experts; 8 are picked per token.
- A bias added to each expert’s score is raised for underused experts and lowered for busy ones; the paper notes “the bias term is only used for routing”.
- It keeps a light “complementary sequence-wise balance loss” so no single sequence overloads a few experts.
- The GPU-hour figure covers the final training run only; the paper says it excludes earlier research and experiments.
- 3 · Use it
- Use DeepSeek-V3 as your reference example: 671B total, 37B active, about 5.5% of the model per token. Compare any new MoE model against it.
Words to know
- Mixture of experts (MoE)
- A model design where many expert sub-networks share a layer and a router sends each token to only a few of them.
- Expert
- One of the smaller feed-forward networks inside an MoE layer. Experts are learned, not assigned topics by people.
- Router (gating network)
- A small network that scores every expert for each token and keeps the top few (top-k).
- Active parameters
- The parameters used to process one token. They set compute cost and most of the speed.
- Total parameters
- Every parameter stored in the model. They set how much memory you need.
- Dense model
- A normal model that uses all of its parameters for every token.
Watch
Hand-picked from the people who explain this best. The video loads only when you press play.
IBM’s Martin Keen explains experts, the router and sparse activation with a whiteboard. Start here if the idea is new.
A walk through the Mixtral paper itself: the router, top-2 selection and what the experts actually specialise in. Watch after the lesson if you want the paper-level detail.
Total vs active: will it fit?
Pick a model or type the two numbers from any model card. Memory comes from the total; speed comes from the active parameters.
- Memory for the weights (at least)
- 10.5 GB
- 21B total × 0.5 bytes
- Compute per token, like a dense model of
- 3.6B
- if the weights sit in fast memory
- Share used per token
- 17.1%
- 3.6B of 21B
The weights fit in 16 GB with 5.5 GB left. You still need room for the conversation (the KV cache, which grows with context) and your other apps, and real quantized files run a little bigger, so leave a margin.
| Model | Total | Active | Used per token | Weights at 4-bit | Experts |
|---|---|---|---|---|---|
| Mixtral 8x7BMistral AI · Dec 2023 | 46.7B | 12.9B | 27.6% | ~23.4 GB | 2 of 8 per layer |
| DeepSeek-V3DeepSeek · Dec 2024 | 671B | 37B | 5.5% | ~335.5 GB | 8 of 256 routed + 1 shared |
| Llama 4 ScoutMeta · Apr 2025 | 109B | 17B | 15.6% | ~54.5 GB | 16 experts |
| Llama 4 MaverickMeta · Apr 2025 | 400B | 17B | 4.3% | ~200 GB | 1 of 128 routed + 1 shared |
| Qwen3-235B-A22BAlibaba Qwen · 2025 | 235B | 22B | 9.4% | ~117.5 GB | 8 of 128 |
| Kimi K2Moonshot AI · 2025 | 1,000B | 32B | 3.2% | ~500 GB | 8 of 384 + 1 shared |
| gpt-oss-20bOpenAI · Aug 2025 | 21B | 3.6B | 17.1% | ~10.5 GB | 4 of 32 |
| gpt-oss-120bOpenAI · Aug 2025 | 117B | 5.1B | 4.4% | ~58.5 GB | 4 of 128 |
| Kimi K3Moonshot AI · 2026 | 2,800B | 104B | 3.7% | ~1,400 GB | 16 of 896 + 2 shared |
Try it
Check if an MoE model fits your machine (10 minutes)
- 1Find how much memory you have: on Windows open Task Manager → Performance → Memory; on a Mac open About This Mac.
- 2In the calculator above, pick gpt-oss-20b at 4-bit. Compare the weights memory with your RAM.
- 3Now pick Llama 4 Scout. Note how the active parameters stay at 17B while memory jumps to about 54.5 GB.
- 4Open the model card of any new model you saw in the news this week and find its total and active numbers. Type them into the calculator.
- 5Write one sentence you could say in an interview: “It has X total and Y active parameters, so it needs about Z GB at 4-bit but runs like a Y-billion model.”
Check yourself
Pick an answer to see if you are right, and why.
Done? Keep going
Mark this lesson complete to keep your place. Progress is saved on this device, no login.
Questions people ask
What is a mixture of experts in simple terms?
It is a model built from many smaller expert networks plus a router. For each token the router picks a few experts to run, so the model can be very large while each token costs only a fraction of the compute.
Does a mixture-of-experts model need less memory?
No. All experts must be loaded because any of them can be picked for the next token. Memory follows the total parameters; speed and compute follow the active parameters.
Which AI models use mixture of experts?
As of Nov 2026, many large open-weight models do, including Mixtral, DeepSeek-V3, Llama 4 Scout and Maverick, Qwen3-235B-A22B, OpenAI’s gpt-oss and Moonshot’s Kimi K2 and K3. Closed models often do not publish their architecture, so you cannot always tell.
Sources and further reading
- Shazeer et al.: The Sparsely-Gated Mixture-of-Experts Layer (arXiv 1701.06538, 2017)
- Jacobs, Jordan, Nowlan, Hinton: Adaptive Mixtures of Local Experts (1991)
- Fedus, Zoph, Shazeer: Switch Transformers (arXiv 2101.03961, 2021)
- Mistral AI: Mixtral of experts (Dec 11, 2023)
- Jiang et al.: Mixtral of Experts (arXiv 2401.04088, routing analysis)
- Hugging Face: Mixture of Experts Explained (Dec 2023)
- DeepSeek-V3 Technical Report (arXiv 2412.19437)
- Meta: The Llama 4 herd (Apr 5, 2025)
- Qwen3-235B-A22B model card (Hugging Face)
- OpenAI: gpt-oss on GitHub
- OpenAI: gpt-oss-120b and gpt-oss-20b model card (arXiv 2508.10925)
- Moonshot AI: Kimi K2 on GitHub
- Moonshot AI: Kimi K3 on GitHub
- llama.cpp: llama-server options (--cpu-moe, --n-cpu-moe)
- ThursdAI: AI releases, November 2026 (Kimi K3.1 Pro, reported)