AI Foundations

Mixture of Experts (MoE)

Mixture of Experts (MoE) is a neural network architecture that replaces a dense layer with many parallel sub-networks called experts, plus a router that activates only a small subset per token. This lets a model hold huge total parameters while keeping per-token compute low.

What is a Mixture of Experts (MoE)?

A Mixture of Experts (MoE) is a neural network design in which a single computational layer is split into many parallel sub-networks, called experts, and a small gating network decides which experts process each input. Only a few experts run for any given token, so the model performs far less computation than its total size would suggest. This idea is a form of conditional computation: different parts of the network activate for different inputs rather than every parameter firing on every example.

Modern MoE is most often applied inside transformer language models, where the experts replace the feed-forward (also called MLP) block of selected layers. The architecture was popularized for deep learning by the 2017 paper 'Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer' by Shazeer and colleagues, which showed that a sparsely gated MoE could increase model capacity by more than 1000 times with only minor added compute cost.

The core promise of MoE is decoupling model capacity from inference cost. A dense model that doubles its parameters roughly doubles the work per token. An MoE can grow its parameter count by adding experts while holding the number of active experts (and therefore the per-token compute) nearly fixed.

Experts: parallel sub-networks, each typically a feed-forward block, that specialize during training.
Router (gating network): a small learned layer that scores experts and selects which ones handle each token.
Sparse activation: only a subset of experts runs per token, keeping compute low relative to total size.

Dense vs. sparse models: total parameters vs. active parameters

In a dense model, every parameter participates in processing every token. Total parameters and active parameters are the same number, so doubling capacity doubles compute. MoE breaks this link by introducing two distinct counts: total parameters (all weights stored, across every expert) and active parameters (the weights actually used for a single token's forward pass).

This is why MoE models are described with notation like '671B total, 37B active' or names such as Qwen3-235B-A22B, where A22B signals roughly 22 billion active parameters. The model holds a large knowledge capacity in its full parameter set, but each token only pays the compute cost of the active fraction. Inference speed and FLOPs track the active count, while memory and storage track the total count.

Sparse activation is the term for this selective use. A sparse MoE routes each token to a small number of experts, in contrast to a dense layer (or a hypothetical dense MoE that uses all experts), trading a modest routing overhead for a large reduction in compute per token.

Total parameters: every weight stored on disk and loaded into memory, summed across all experts.
Active parameters: the weights used for one token, which set the inference compute cost.
Naming convention: figures like 'A22B' or '37B active' indicate the active-parameter budget.

Inside an MoE layer: experts replacing the feed-forward network

In a standard transformer block, attention is followed by a single feed-forward network. An MoE layer replaces that single feed-forward network with a set of N experts, each an independent feed-forward network with its own weights. The attention sublayers and other components usually remain dense and shared; sparsity is introduced specifically at the expert layer.

When a token arrives, the router scores the experts and selects a small set (often two). Each selected expert computes its output, and the results are combined as a weighted sum, where the weights come from the router's gating scores. Tokens that are not routed to an expert simply do not invoke its weights, which is the source of the compute savings.

Many recent designs add shared experts that always run for every token, alongside the routed experts that are selected per token. The shared expert captures common, general-purpose transformations, while routed experts handle more specialized patterns. DeepSeek's architecture is a prominent example of combining one or more shared experts with a large pool of finer-grained routed experts.

Each expert is a separate feed-forward network with independent weights.
Outputs of selected experts are combined as a gating-weighted sum.
Shared experts (when present) run for every token; routed experts are selected per token.

The router and top-k token routing

The router, also called the gating network, is usually a simple linear layer that maps a token's representation to one logit per expert. A softmax (or similar normalization) turns those logits into preference scores, and a top-k operation keeps the k highest-scoring experts. The standard mechanism, established by the 2017 sparsely gated layer, is top-k gating, with k commonly set to 2.

Mixtral 8x7B from Mistral AI is a clean illustration: each layer has 8 experts, and the router selects the top 2 per token. The selected experts can differ at every layer and every token, so a sequence flows through many different expert combinations. Top-2 routing is popular because it gives smoother gradients and some redundancy compared with selecting only a single expert.

An alternative is expert-choice routing, where each expert selects the top-k tokens it wants rather than each token selecting experts. This naturally balances load because every expert receives a fixed number of tokens, though it changes how tokens are guaranteed coverage. As a single neutral connection, the same gating idea, scoring candidates and keeping only the top few, also underlies how second-brain apps like MemX rank and retrieve the most relevant stored notes for a query.

y = Σ_{i ∈ TopK(W_r·x)}  g_i · E_i(x),   where  g = softmax( TopK(W_r·x) )

The router scores every expert, keeps the top-k, and renormalizes those logits with softmax so the active experts' gating weights sum to 1. The layer output is the weighted sum of just those experts.

Router: typically a linear layer producing one logit per expert, normalized into gating weights.
Top-k (token choice): each token selects its k highest-scoring experts; k is often 2.
Expert choice: each expert selects its top tokens, which evens out load by construction.

Load balancing, auxiliary losses, and expert collapse

Left unconstrained, a router tends to favor a small subset of experts, sending most tokens to them while others receive few or none. This failure mode is called expert collapse (or routing collapse): the favored experts train faster, get chosen more often, and the imbalance reinforces itself, wasting the capacity of the neglected experts.

The classic remedy is a load-balancing auxiliary loss, introduced in the 2017 paper, which is minimized when tokens are spread evenly across experts and when experts receive comparable importance. A separate auxiliary term, the router z-loss (introduced in the ST-MoE work), penalizes overly large router logits to keep the softmax numerically stable; these two losses address orthogonal problems (balance versus numerical stability) and are often used together.

Auxiliary losses add a tuning burden and can slightly distort the primary training objective. DeepSeek-V3 introduced an auxiliary-loss-free load balancing strategy that adjusts per-expert routing biases dynamically instead of adding a balancing loss term, aiming to keep experts balanced without sacrificing model quality.

Expert collapse: the router over-uses a few experts and starves the rest, wasting capacity.
Load-balancing auxiliary loss: encourages even token distribution across experts.
Router z-loss: keeps router logits bounded for numerical stability during training.

Benefits: capacity and inference efficiency tradeoffs

The headline benefit of MoE is large capacity at modest compute. Because only the active experts run per token, an MoE can match or beat a much larger dense model in quality while using a fraction of the inference FLOPs. Mistral AI reported that Mixtral 8x7B, with about 12.9B active parameters, matched or outperformed Llama 2 70B (a dense model) on most benchmarks while running far faster.

This efficiency compounds at scale. Frontier-class open MoE models reach hundreds of billions or even a trillion total parameters while activating only tens of billions per token, putting strong performance within reach of inference budgets that could never run an equally capable dense model.

MoE also enables specialization. Different experts can learn to handle different kinds of tokens or patterns, which can improve quality on diverse data such as multiple languages, code, and reasoning, although learned specialization is often less human-interpretable than the name 'expert' suggests.

Quality per FLOP: an MoE can rival a larger dense model at lower per-token compute.
Scalable capacity: total parameters can grow by adding experts without raising active compute much.
Specialization: experts can adapt to distinct data patterns, aiding diverse and multilingual tasks.

Challenges: memory footprint, training instability, and routing overhead

MoE efficiency applies to compute, not memory. All experts must be stored and typically loaded into memory even though only a few run per token, so a 671B-total MoE needs roughly the memory of a 671B dense model regardless of its small active count. This makes MoE models demanding to host and often requires distributing experts across multiple accelerators.

Training MoE is also more delicate than training dense models. Routing collapse, numerical instability in the router softmax, and sensitivity to the balance between the main loss and auxiliary losses can all destabilize training, which is why z-loss and balancing strategies are standard practice.

Finally, sparse routing introduces systems overhead. Distributing experts across devices requires all-to-all communication to send tokens to the right expert and gather results, and uneven load can leave some experts idle while others are saturated. Expert-parallel infrastructure and capacity limits per expert are used to manage this, but they add engineering complexity that dense models avoid.

Memory footprint tracks total parameters, so all experts must be stored even if rarely used.
Training instability: routing collapse and router numerical issues need careful regularization.
Routing overhead: cross-device token dispatch and load imbalance add systems complexity.

MoE in modern models (as of 2026) and the trend toward many fine-grained experts

As of 2026, sparse MoE is the dominant architecture for large open-weight language models. DeepSeek-V3, released in December 2024, uses 671B total parameters with about 37B active per token, drawn from 256 routed experts plus 1 shared expert with 8 routed experts selected per token, and pioneered auxiliary-loss-free load balancing. Alibaba's Qwen3-235B-A22B, released in April 2025, holds 235B total and roughly 22B active, with 128 experts and 8 selected per token.

Meta's Llama 4 family, released in 2025, applies MoE across tiers: Llama 4 Scout uses about 109B total with 17B active across 16 experts, while Llama 4 Maverick uses about 400B total with 17B active across 128 experts. Moonshot AI's Kimi K2, released in mid-2025, pushes to roughly 1 trillion total parameters with about 32B active per token. These specific figures are current as of 2026 and individual model details evolve, but the architectural pattern is stable.

The clear trend is toward more, smaller experts, often called fine-grained expert segmentation. Splitting the feed-forward capacity into many narrow experts (sometimes hundreds), frequently combined with always-on shared experts, gives the router finer control over specialization and tends to improve quality per active parameter, which is why so many 2026-era frontier models share this design.

DeepSeek-V3 (as of 2026): 671B total, about 37B active, 256 routed + 1 shared expert, 8 routed selected.
Qwen3-235B-A22B (as of 2026): 235B total, about 22B active, 128 experts, 8 selected per token.
Llama 4 (as of 2026): Scout about 109B total / 16 experts; Maverick about 400B total / 128 experts; both 17B active.
Kimi K2 (as of 2026): roughly 1T total parameters with about 32B active per token.
Trend: many fine-grained experts plus shared experts, improving quality per active parameter.

Key takeaways

Mixture of Experts replaces a dense layer with many expert sub-networks and a router that activates only a few per token, decoupling total capacity from per-token compute.
The key distinction is total parameters (all experts, which set memory cost) versus active parameters (the few used per token, which set inference compute).
A gating network scores experts and uses top-k routing (often top-2) to select which experts process each token; expert-choice routing is an alternative that balances load by design.
Load-balancing auxiliary losses and router z-loss combat expert collapse and training instability; DeepSeek-V3 introduced an auxiliary-loss-free balancing approach.
As of 2026, MoE dominates large open models (DeepSeek-V3, Qwen3, Llama 4, Kimi K2), with a clear trend toward many fine-grained experts plus shared experts.

Frequently asked questions

Total parameters count every weight stored across all experts and set the model's memory footprint. Active parameters count only the weights used for a single token, which set the inference compute cost. For example, DeepSeek-V3 has about 671B total but activates only about 37B per token (as of 2026).

A small gating network (usually a linear layer) produces one score per expert for each token, then a top-k operation keeps the k highest-scoring experts (often k equals 2). Those experts run and their outputs are combined as a gating-weighted sum; all other experts stay inactive for that token.

Because only a few experts run per token, an MoE performs far less computation than its total parameter count implies. This lets it reach the quality of a much larger dense model while using a fraction of the per-token FLOPs, though it still needs memory to store all experts.

Expert collapse is when the router over-uses a few experts and starves the rest, wasting capacity. It is countered with a load-balancing auxiliary loss that encourages even token distribution, a router z-loss for numerical stability, or auxiliary-loss-free strategies like the dynamic routing bias used in DeepSeek-V3.

No. MoE saves compute, not memory. Every expert must be stored and usually loaded, so an MoE's memory footprint tracks its total parameter count, not its small active count. A 671B-total MoE needs roughly the memory of a 671B dense model even though far fewer parameters run per token.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free