Sticky Routing MoE: Reducing Expert Switching for More Efficient Inference

Mixture-of-experts (MoE) models have become a popular architecture for scaling language models, offering strong performance with sparse activation. However, their inference efficiency is hampered by the need to route tokens to different experts at each layer, causing costly data movement and load imbalance. A new preprint on arXiv proposes Sticky Routing MoE, a simple but effective modification that encourages tokens to stay on the same expert across consecutive layers.

The core idea is to add a small penalty in the routing function that discourages expert switching between layers. Tokens that remain on the same expert from one layer to the next receive a routing bonus, while those that switch incur a cost. This bias is controlled by a hyperparameter that can be tuned to balance fidelity and efficiency.

How It Works

In standard top-1 or top-2 routing, each token independently selects the best experts per layer. Sticky routing instead biases the gating function toward the previously chosen expert. Formally, the logits for each expert at layer (l) are adjusted by adding a bonus term if the expert matches the token's assignment at layer (l-1). The authors implement this as a simple additive term during the softmax computation, requiring no changes to the underlying MoE architecture.

Results and Implications

Experiments on language modeling tasks show that sticky routing reduces expert switches by 30–50% with negligible loss in model quality (perplexity changes within 0.1). The reduction in switching translates directly to lower inter-device communication and more balanced expert loads, enabling up to 1.5× higher inference throughput on GPU clusters. The method is complementary to other MoE optimizations like expert parallelism and can be applied to any existing MoE model with minimal code changes.

Why It Matters for Developers

For engineers deploying large MoE models in production, every latency improvement counts. Sticky routing offers a drop-in optimization that can improve hardware utilization without retraining or altering the model structure. It also suggests that routing policies are an under-explored axis for efficiency gains, opening the door to further research into learned or adaptive routing strategies.