Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models (2025)

Abstract: The performance of MoE-based LLMs depends on the router’s ability to select suitable experts; however, the router is typically not explicitly supervised to acquire this routing ability. We propose Exploration-Driven Reinforcement Learning (ERL), which explicitly optimizes the router by exploration of alternative routing paths. For every input, ERL evaluates by (i) the original routing path and (ii) paths in which an \alpha -fraction of routing decisions is randomly perturbed, and treats their performance gap as an advantage signal in a reinforcement learning. Moreover, MoE-ERL _{wPL} mitigates the risk of performance collapse caused by routing reinforcement learning–induced expert over-specialization by intentionally enforcing overlap in experts’ knowledge. Without adding parameters or external reward models, our method improves summarization (SAMSum, XSUM), question answering (SQuAD), and language modeling (WikiText-2), and raises routing quality, delivering up to 8.9 × higher MRR than baselines over 100 perturbed routing paths. Code is available at our github.

## introduction the rapid advancement of large language models (llms) has significantly improved artificial intelligence, with conversational systems such as chatgpt @xcite have already brought significant changes to our daily lives @xcite . empirical studies show scaling training data and parameters to billions-even trillions-consistently leads to improvements in ai performance. @xcite . however, increasing model size incurs higher memory consumption and computational costs. to alleviate this, recent llms like deepseekmoe @xcite and mixtral @xcite ) have adopted the mixture-of-experts (moe) architecture @xcite . moe dynamically activates only a subset of experts per input, enabling conditional sparse computation. this approach maintains large model capacity while significantly improving computational efficiency @xcite . the switch transformer @xcite ) extends the standard transformer @xcite by applying a mixture-of-experts architecture. each feed-forward network (ffn) layer is split into multiple experts, and a learned router selects the most suitable expert for each token. during training, it combines the language modeling loss with a z-loss and an auxiliary load-balancing loss to encourage balanced expert utilization. despite providing no explicit supervision for expert selection, most previous moe studies nonetheless rest on the unvalidated assumption that jointly minimizing these three losses is sufficient for the router to implicitly learn optimal routing decisions. however, moe performance heavily depends on the router's ability to accurately select appropriate experts. our experiments ( in this work, we explicitly optimize the router using reinforcement learning to select optimal experts at each token and layer. specifically, we encourage the moe-based llm to explore multiple alternative routing paths and compute an advantage signal based on how each path impacts prediction performance. using this advantage, we directly train the existing router to enhance its routing accuracy. crucially, our method introduces no additional parameters or external models, achieving improved moe-based llm performance solely through more suitable expert selection. furthermore, because we leave the standard single-layer mlp router-adopted by most moebased llms-completely unchanged in structure and mechanism, our approach can be seamlessly integrated into most pretrained moe models without any architectural modification. many prior studies applying reinforcement learning to llms @xcite @xcite typically train separate reward models or rely on external evaluation sources such as apis or human judgments, thereby increasing model size and pipeline complexity. in contrast, our approach computes reward signals directly from the router's own prediction probabilities, allowing the model to autonomously assess performance and explore alternative routing paths without introducing additional modules or external evaluators. ## methodology many moe-based llms assume that the router will autonomously learn to select suitable experts. however, existing methods do not explicitly train the router for optimality, which makes it challenging to reliably achieve optimal expert selection. so, we propose exploration-driven reinforcement learning (erl), which explicitly trains the router toward improved expert selection. erl explores perturbed (alternative) routing paths and compares their performance against the original path. if an perturbed routing path yields better predictions, the router is reinforced perturbed routing decision; otherwise, the original routing is strengthened. the overall reinforcement learning procedure is illustrated in figure 1 . ## main results tables 2-3 indicate that moe-erl and moe-erl wp l outperform conventional baselines on the majority of datasets. summarization. on both samsum and xsum, moe-erl outperforms all baselines across rouge metrics. in rouge-2, it improves over the strongest baseline by 0.26 (samsum) and 0.08 (xsum). the variant moe-erl wp l achieves improves of 0.52 and 0.20, respectively. these results demonstrate that reinforcement learning improves the summarization capability of moe models, and incorporating the lm loss for perturbed forwards yields additional benefit. ## analysis 6.1 analysis of routing to evaluate whether the rl-trained router selects improved routing paths, we conduct an additional analysis. for each fine-tuned model, we record the rouge-2 score (summarization) or perplexity (language modeling) from the original forward. we then generate 99 perturbed forward per data by randomly altering decoder routing 33 times at each α ∈ 0.25, 0.50, 0.75, yielding 100 scores per data. we report the rank of the original path using mean reciprocal rank (mrr) and hit@1 (the proportion of cases where the original forward ranks first). evaluations are conducted on 300 randomly sampled data per dataset, with results shown in ## conclusion we proposed moe-erl, a reinforcement learning method that explicitly optimizes expert routing in moe-based llms without introducing additional parameters or external reward models. moe-erl leverages the performance gap between original and perturbed routing paths as an advantage signal to guide the router toward optimal expert selection. we further proposed moe-erl wp l , which incorporates an auxiliary lm loss on perturbed routing paths to mitigate the risk of over-specialization. most of experiments on samsum, xsum, squad, and wikitext-2 demonstrated that both variants outperform dense, moe, and moe-share baselines, improving routing quality (mrr and hit@1) by up to 8.9×. we also identified the expert misclassification risk due to reduced knowledge overlap, which moe-erl wp l effectively alleviates. future work may explore additional strategies to further reduce misclassification risk while maintaining routing precision.