C artesian M o E : Boosting Knowledge Sharing among Experts via C artesian Product Routing in Mixture-of-Experts (2025)

Abstract: Large language models (LLM) have been attracting much attention from the community recently, due to their remarkable performance in all kinds of downstream tasks. According to the well-known scaling law, scaling up a dense LLM enhances its capabilities, but also significantly increases the computational complexity. Mixture-of-Experts (MoE) models address that by allowing the model size to grow without substantially raising training or inference costs. Yet MoE models face challenges regarding knowledge sharing among experts, making their performance somehow sensitive to routing accuracy. To tackle that, previous works introduced shared experts and combined their outputs with those of the top K routed experts in an addition manner. In this paper, inspired by collective matrix factorization to learn shared knowledge among data, we propose CartesianMoE, which implements more effective knowledge sharing among experts in more like a multiplication manner. Extensive experimental results indicate that CartesianMoE outperforms previous MoE models for building LLMs, in terms of both perplexity and downstream task performance. And we also find that CartesianMoE achieves better expert routing robustness.

## introduction large language models (llm) have demonstrated impressive performance across various downstream natural language tasks @xcite @xcite @xcite @xcite . moreover, the well-known scaling law suggests that, as the model size increases, the model capabilities will continue to improve @xcite . however, for dense llms, the computational costs of scaling up their model sizes can become prohibitively high. to tackle that, sparse activation networks are proposed @xcite . they reduce computational costs by activating only a subset of parameters for each input. a prominent approach among them is the mixture-of-experts (moe) @xcite @xcite @xcite , which involves training multiple experts but using only a subset to process each input, with each expert generally being a feedforward network (ffn). compared to dense llms of equivalent sizes, moe llms effectively reduces computational costs while delivering comparable results, in terms of both perplexity (ppl) and downstream task performance @xcite @xcite @xcite @xcite . conventional moe models, like @xcite @xcite , activate the top k routed experts among the total n experts. due to the independent training of all experts, they rarely share learned knowledge, and thus routing fluctuations can affect the output substantially, making the performance of such moe models somehow sensitive to the routing accuracy. to tackle that, @xcite suggests using several fixed-activated shared experts to store shared knowledge, in addition to the topk routed experts. and it has been well-validated to improve moe model performance. with shared experts, @xcite further split full-sized experts into more fine-grained experts to enhance representation specialization and gain additional performance improvement, where the n experts are split into mn smaller ones, and the top mk routed ones of them are activated. the remarkable shared-expert method essentially merges the shared knowledge (i.e., outputs of shared experts) with the specific knowledge (i.e., outputs of the routed experts) in an "addition" manner. for instance, with a shared expert ffn a and several routed experts ffn b , ffn c , and ffn d , the knowledge sharing among experts can be represented as: ffn a + ffn b , ffn a + ffn c , and ffn a + ffn d . inspired by collective matrix factorization to learn shared knowledge among data (singh and gordon, 2008), in this paper we propose to represent knowledge sharing among experts in an alternative "multiplication" manner, i.e., ffn a • ffn b , ffn a • ffn c , and ffn a • ffn d . specifically, by defining two sets of sub-experts {ffn 1 a , ffn 1 b , . . .} and {ffn 2 a , ffn 2 b , . . .}, we derive each expert to be the combination of any two sub-experts from both sets respectively, like ffn in that sense, each expert share an identical sub-expert with many others. it can also be seen that, all the experts can be derived by the cartesian product between both sub-expert sets, and thus we term our proposed method as cartesianmoe. specifically, in our proposed cartesianmoe, we replace the conventional moe layer as a cartesian product layer, which consists of two sequential moe sub-layers, each denoting a set of sub-experts, as illustrated in fig. 1 . then the routing process to select routed experts is also divided into the two moe sub-layers, termed as cartesian product routing. extensive experiments on building moe llms show that cartesianmoe yields superior performance than previous counterparts, using the same number of model parameters and activated parameters. cartesianmoe also shows better routing robustness. we argue that the superiority of cartesianmoe comes from its more fine-grained knowledge sharing among experts. specifically, compared to the shared-expert method that requests all routed experts to always share the same global knowledge held by the fixed shared experts, cartesianmoe allows to divide experts into groups with each sharing some group-wise knowledge. in that sense, cartesianmoe is supposed to be also equipped with shared experts, so as to form a "global shared knowledge + group-wise shared knowledge + expert-specific knowledge" system. our contributions are summarized as follows: the concept of moe models was first introduced by @xcite . then, @xcite extended the moe model to multiple layers. later, @xcite extended that idea to long short-term memory (lstm) networks @xcite , training an lstm model with up to 137 billion parameters. with the advent of the transformer architecture @xcite , the gshard model @xcite applied moe techniques to transformers, paving the way for the development of more advanced moe models like glam @xcite and switch transformer @xcite . in early works @xcite @xcite @xcite , when extending dense models to moe models, the moe layer of the transformer consists of multiple ffns that are of the same size as those in the dense models. recent works @xcite @xcite show that splitting a fully-sized ffn into several smaller, fine-grained experts facilitates representation specialization. additionally, shared experts are commonly adopted @xcite @xcite , to enhance knowledge sharing among experts for performance improvement. the shared-expert method combines shared knowledge (i.e., the outputs of the shared experts) with specialized knowledge (i.e., the outputs of the routed experts) in an "addition" manner. inspired by collective matrix factorization to learn shared knowledge among data, here we propose an alternative "multiplication" manner to share expert knowledge, which demonstrates superiority over previous moe methods. ## analyses 6.1 impact of fixed-activated shared expert under the moe-large setting, we remove the fixedactivated shared experts from cartesianmoe to investigate its impact on the model performance. as shown in ## conclusions inspired by collective matrix factorization to capture shared knowledge within data, we introduce cartesianmoe, a "multiplication"-manner knowledge sharing method among experts in moe models. cartesianmoe categorizes fine-grained subexperts into two distinct sets, and uses their cartesian product to build experts that facilitate group-wise knowledge sharing. equipped with shared experts as previous works, cartesianmoe builds a more thorough knowledge sharing system among experts, i.e., "global shared knowledge + groupwise shared knowledge + expert-specific knowledge". extensive experiments well demonstrate that cartesianmoe outperforms previous moe models across various settings, in terms of language modeling perplexity and downstream task performance. it also presents much better routing robustness due to enhanced knowledge sharing.