← Back to Glossary
Mixture of Experts (MoE)
An LLM architecture that routes each token to a small subset of specialized sub-networks ('experts') instead of running the whole model.
Why it exists
Allows scaling model parameter capacity to hundreds of billions without increasing compute cost per token.
How it works
A routing gate network evaluates incoming tokens and routes them to 2 out of 8 expert layers (e.g. Mixtral 8x7B), giving huge capacity at fast inference speeds.
Related terms
Where you'll use this
roadmap