Maximum Score Routing For Mixture-of-Experts (2025)

Abstract: Routing networks in sparsely activated mixture-of-experts (MoE) dynamically allocate input tokens to top-k experts through differentiable sparse transformations, enabling scalable model capacity while preserving computational efficiency. Traditional MoE networks impose an expert capacity constraint to ensure GPU-friendly computation. However, this leads to token dropping when capacity is saturated and results in low hardware efficiency due to padding in underutilized experts. Removing the capacity constraint, in turn, compromises load balancing and computational efficiency.To address these issues, we propose Maximum Score Routing (**MaxScore**), a novel MoE routing paradigm that models routing as a minimum-cost maximum-flow problem and integrates a SoftTopk operator. MaxScore resolves the fundamental limitations of iterative rerouting and optimal transport formulations, achieving lower training losses and higher evaluation scores at equivalent FLOPs compared to both constrained and unconstrained baselines.

## introduction the mixture of experts (moe) paradigm has emerged as a compelling architectural strategy for scaling neural networks while maintaining computational efficiency. this approach dynamically combines multiple subsets of parameters (experts) by a learnable routing network, aiming to improve model capacity and computational efficiency. the routing network of sparsely activated moe @xcite dynamically allocates input tokens to top-k experts through differentiable sparse transformations, enabling conditional com- † indicates corresponding authors. softmax is conventionally employed to compute token-expert affinity coefficients in moe routing networks, which promotes inter-expert competition. to mitigate winner-takes-all and preserve load balance, both hard constraints using expert capacity @xcite , and soft constraints using auxiliary losses @xcite , are incorporated into the routing network @xcite . gshard @xcite pioneers the integration of moe with transformer architectures @xcite , where expert capacity constraints enable gpu-friendly computation patterns. expertchoice @xcite directly enables experts to select tokens based on capacity constraints. however, token dropping occurs when inputs are routed to capacity-saturated experts, while padding operations in underutilized experts create hardware inefficiencies. empirical analysis reveals that approaches such as expanding capacity @xcite or removing capacity constraints altogether @xcite effectively eliminate token dropping, but inevitably introduce a trade-off between computational efficiency and load balancing performance. efforts to prevent token dropping via refined routing strategies @xcite have not yielded performance improvements, highlighting unresolved challenges in dynamic resource allocation. this work introduces maximum score routing (maxscore), a novel moe routing paradigm that formulates token-expert routing as a minimumcost maximum-flow problem @xcite , integrated with a softtopk operator. to the best of our knowledge, this is the first successful integration of network flow modeling and softtopk in moe routing. maxscore preserves gpu-compatible expert capacity constraints and achieves better load balanc-ing. under the same flops, maxscore exhibits lower training loss and higher evaluation scores compared to both constrained and unconstrained baselines. ablation studies demonstrate the necessity of both network flow modeling and the softtopk operator, revealing fundamental limitations in the iterative rerouting mechanism of @xcite and the optimal transport-based routing of @xcite . the synergistic combination of two methodological enhancements yields superadditive performance gains, with empirical results demonstrating that their integrated efficacy surpasses the linear summation of individual improvements. scaling experiments show that maxscore delivers consistent performance improvements with larger activated parameter budgets, and achieves more gains when increasing the number of experts, compared with standard moe approaches. ## methodology we investigate the fundamental reasons why the iterative rerouting mechanism (iter) and the optimal transport formulation (sinkhorn) fail to improve model quality, and propose maximum score routing (maxscore), a novel mixture-of-experts routing strategy that integrates network flow modeling and a differentiable softtopk(•) operator. ## conclusion and future work this work introduces maxscore moe, a novel mixture-of-experts routing paradigm formulated via minimum-cost maximum flow modeling and the integration of a differentiable softtopk(•) operator. to our knowledge, this is the first successful integration of network flow modeling and softtopk within moe routing. the synergistic combination of these components yields superadditive performance gains, with empirical evidence showing that their joint effect surpasses the linear sum of individual contributions. future work will focus on evaluating the method at larger model scales and across more diverse benchmarks to validate its generality and robustness.