Probing Semantic Routing in Large Mixture-of-Expert Models (2025)

Abstract: In the past year, large ( >100 B parameter) mixture-of-expert (MoE) models have become increasingly common in the open domain. While their advantages are often framed in terms of efficiency, prior work has also explored functional differentiation through routing behavior. We investigate whether expert routing in large MoE models is influenced by the semantics of the inputs. To test this, we design two controlled experiments. First, we compare activations on sentence pairs with a shared target word used in the same or different senses. Second, we fix context and substitute the target word with semantically similar or dissimilar alternatives. Comparing expert overlap across these conditions reveals clear, statistically significant evidence of semantic routing in large MoE models.

## introduction since their popularization in @xcite , the mixture-of-experts (moe) architecture @xcite has been integrated into many frontier large language models (llms) @xcite @xcite @xcite . the moe architecture offers the ability to train far larger models than would normally be possible with dense architectures. designers can then modulate performance by varying the number of active experts to access a greater portion of the greater model, with the trend being to increase the total and active expert counts. several prior studies have explored expert activation patterns in moe models, hypothesizing that each expert may specialize in specific domains, tasks, or topics @xcite @xcite . while it is intuitive to expect some degree of semantic specialization, previous research has not found clear evidence of routing on you must your camping gear carry sound carries well over water he your camping gear carries sound carries well over water expert 2 expert 1 expert k … router target word is held constant, but the meaning is different you must your camping gear carry sound carries well over water will you me today forgive will you absolve me today expert 2 expert 1 expert k … router context is held constant. we substitute the target word for similar, and different meaning words sound carries well over water will you efface me today the basis of semantics, concluding instead that expert activation is primarily token-dependent rather than being driven by deeper semantic relationships. given that recent large-scale moe models have achieved state-of-the-art performance while increasing total expert counts, we investigate whether these models' expert routing behavior exhibits semantic specialization. we design two controlled experiments. first, we use a word sense disambiguation (wsd) task from the wic benchmark @xcite , where the same target word appears in two different sentences, either preserving or changing its meaning. this allows us to measure whether expert activation remains stable when the word's sense is preserved. second, we study a complementary setting using the lexical substitution benchmark swords @xcite , where we fix the surrounding context but vary the target word, comparing expert overlap between semantically similar and dissimilar word substitutions. we compare the rate of overlap, between models with differing numbers of active and total experts, via a normalized metric based on cohen's κ that controls for the baseline probability of overlap. we apply these experiments to six moe models from three model families: deepseek-r1 @xcite , deepseek-v2-lite @xcite , @xcite , llama-4-scout and llama-4-maverick (ai, 2025). for all models, we find that the rate of expert overlap is significantly higher when the meaning of the target word is equal in two sentences than when the meaning of the target word is different. we also find that model scale influences the strength of this specialization: larger models generally exhibit stronger semantic routing signals-with llama-4 scout (ai, 2025) standing out as an exception, showing a pronounced effect despite its smaller total parameter count. finally, semantic differentiation in expert routing is most prominent in the middle layers, where deepseek-r1 exhibits the clearest and most consistent specialization pattern. in summary, our contributions are threefold: (1) we design two complementary semantic probing setups, based on word sense disambiguation and semantic substitution, to systematically assess expert specialization in recent moe models. (2) we introduce an expert overlap metric to quantify routing similarity and demonstrate its alignment with lexical relationships. (3) we conduct extensive experiments across three moe model families (deepseek, mixtral, and llama-4) at various scales, uncovering clear empirical evidence of semantic routing and highlighting its dependence on model size and layer depth. ## related work current research on expert specialization in moe models is sparse, yet available studies reveal little evidence of semantic-level differentiation. for example, @xcite tracked token routing pat-terns across datasets segmented by different topics, languages, and tasks, but failed to find any coherent pattern at such high-level semantics. rather, they found indications of token-level specialization, mainly concerning low-level semantic features like special characters or auxiliary verbs. similar findings have been reported in studies using independently developed moe models (e.g., @xcite @xcite . while some neuroscience research has provided evidence that the brain functions like a mixture of experts @xcite -suggesting the possibility of semantic-level specialization-other studies have shown that moe models with random routing can perform comparably to those using the more common top-k routing approach @xcite @xcite . one potential explanation for these mixed results is that prior models (using 8 to 32 experts) might not have been sufficiently expressive to capture fine-grained specialization patterns. the recently-released deepseek v3 and llama 4 maverick, featuring an extensive network of experts (256 and 128 routed specialists, respectively), provide us with a unique opportunity. hence, in this study, we test whether a more capable moe architecture exhibits semantic-level expert specialization. ## experiment results and analysis word-in-context for 1k pairs of sentences in wic, we collect router activations for each moe model ( figure 3 reports, for each layer and model, the mean number of overlapping experts across sentence pairs in the two conditions. we find strong evidence for semantic specialization in these experiments; expert overlap is lower for sentence pairs where the target word has different senses than when they are the same. this effect is statistically for all models the difference in overlap increases in intermediary layers. this supports prior findings that semantic features are more salient in the intermediary layers of llms @xcite . our results are also suggestive that this pattern emerges at scale; the difference in expert overlap increases with model size. ## conclusion our study provides the first systematic evidence that expert routing in modern mixture-of-experts (moe) language models is sensitive to semantic content. across two complementary tasks-word sense disambiguation and lexical substitution-we show that expert overlap increases when meaning is preserved and decreases when it changes. this effect is robust across six models from three moe families and persists across model scales and configurations. we find that semantic routing signals are strongest in the middle layers with these signals scaling via model size, suggesting semantic specialization in routing may be a learned, emergent behavior. our findings challenge assumptions that routing is primarily token-based and offer a new view on how sparse models organize computation. by linking routing to semantic similarity, this work enables new directions for interpretability, control, and efficiency in moe deployment.