Mixture of Experts:
Why the Biggest AI Models Are Going Sparse

For years, scaling a neural network meant one thing: make every layer bigger, and pay for every single one of those extra parameters on every single forward pass. Mixture of Experts (MoE) breaks that assumption. It’s the architecture behind many of the largest and most efficient language models today, and its core trick is deceptively simple: don’t use the whole network for every token.

Dense vs. sparse models

In a standard (“dense”) transformer, every token passes through every parameter in every layer. Doubling the model size doubles the compute cost for every token, every time. MoE replaces certain layers — typically the feed-forward blocks — with a bank of smaller sub-networks called experts, plus a small gating network that decides, per token, which experts should handle it.

A gating network routes each token to a small subset of experts, then combines their outputs. (Image by author)

How routing works

The gate is usually just a linear layer followed by a softmax over the experts, producing a probability distribution. Rather than mixing all experts’ outputs, most implementations use top-k routing — commonly top-1 or top-2 — so only one or two experts actually run for a given token, and their outputs are combined using the gate’s weights. If a model has 64 experts but only activates 2 per token, you get access to a huge pool of specialized parameters while paying the compute cost of a much smaller dense model.

This is why MoE papers report two different parameter counts: total parameters (everything stored on disk) and active parameters (what’s actually used per token). A model might have hundreds of billions of total parameters but only activate a tenth of that per forward pass — closer in inference cost to a much smaller dense model, while retaining the capacity of a much larger one.

The load balancing problem

Left alone, gating networks tend to collapse onto a small number of “favorite” experts, leaving the rest undertrained and effectively wasted. This is addressed with an auxiliary load-balancing loss added during training, which penalizes the model when token traffic is too unevenly distributed across experts. Getting this balance right is one of the trickier parts of training MoE models — too much balancing pressure and the router stops making meaningful specialization decisions; too little and capacity goes unused.

What experts actually learn

It’s tempting to imagine experts specializing along human-interpretable lines — one for code, one for poetry, one for math. In practice, interpretability studies of MoE models have generally found messier, less intuitive specialization: experts often split along surface-level token patterns rather than clean semantic categories. The efficiency gains are real regardless of whether the specialization is human-legible.

The trade-offs

• Memory — even though only a few experts run per token, all of them have to be loaded into memory, so MoE models still demand substantial hardware.
• Communication overhead — in distributed training, routing tokens to experts that live on different devices adds networking cost that dense models don’t have.
• Training instability — router collapse and uneven expert utilization can make MoE models finickier to train than their dense counterparts.
• Fine-tuning quirks — MoE models can behave differently under fine-tuning, sometimes requiring different learning rates or regularization for the router versus the experts.

Conclusion

Mixture of Experts is fundamentally a bet that capacity and compute don’t have to scale together. By routing each token to only a handful of specialized sub-networks, MoE architectures let model builders add parameters — and therefore capacity — without a proportional increase in inference cost. That trade-off, more than any single technique, is why sparse architectures have become the default choice for the largest models being built today.