C o M o E : Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning (2025)

Abstract: In parameter-efficient fine-tuning, mixture-of-experts (MoE), which involves specializing functionalities into different experts and sparsely activating them appropriately, has been widely adopted as a promising approach to trade-off between model capacity and computation overhead. However, current MoE variants fall short on heterogeneous datasets, ignoring the fact that experts may learn similar knowledge, resulting in the underutilization of MoE’s capacity. In this paper, we propose Contrastive Representation for MoE (CoMoE), a novel method to promote modularization and specialization in MoE, where the experts are trained along with a contrastive objective by sampling from activated and inactivated experts in top-k routing. We demonstrate that such a contrastive objective recovers the mutual-information gap between inputs and the two types of experts. Experiments on several benchmarks and in multi-task settings demonstrate that CoMoE can consistently enhance MoE’s capacity and promote modularization among the experts.

## introduction parameter-efficient fine-tuning (peft) has emerged to efficiently adapt large language models (llms) to downstream tasks by updating only a subset of parameters, significantly reducing computational and memory overhead @xcite @xcite . however, it struggles with substantially increased dataset sizes, especially heterogeneous training datasets, which poses a significant practical challenge @xcite . mixture-of-experts (moe) offers a versatile solution to the challenge for its modular design @xcite . thus, low-rank adaptation (lora), as a popular and effective peft method, has been widely integrated with moe @xcite , leveraging moe's modularity to enhance the model's capacity and performance. by sparsely activating a subset of experts, lora's moe variants achieve efficient training on heterogeneous datasets and allocate the experts adaptively @xcite . specifically, the sparse activation is controlled through a router mechanism (e.g., topk routing) that dispatches inputs to the activated experts. basically, given an input token, only a subset of specialized experts contribute to the output, while other irrelevant experts remain inactive. ideally, each expert should specialize in distinct representation subspaces and semantic skills, thereby collaboratively enhancing the model's representational capacity and enabling a broader spectrum of knowledge @xcite . however, despite the explicit division into multiple experts in moe architecture, its modularization degree remains questionable. two issues persist: (1) expert knowledge redundancy, where insufficient specialization constraints lead to overlapping functionalities among experts, limiting the model's capacity @xcite ; (2) expert load imbalance, where inadequate modularity and specialization during training result in frequent activation of only a subset of experts, which underutilizes other experts and contradicts its original design intent. consequently, as some studies have indicated @xcite , simply stacking more experts does not linearly improve performance; instead, it leads to a performance bottleneck. existing studies propose load balance loss @xcite and localized balancing constraint @xcite to alleviate the mentioned issues, but that is still far from enough. in this paper, we propose a novel perspective to promote the specialization of experts. as illustrated in fig. 1 , building upon top-k routing, we categorize the experts into activated experts and inactivated experts. then, we quantify the specialization of experts by mutual information (mi) between the input token and the two types of experts. to promote expert specialization, we define an mi gap, which is derived from the aforementioned mi, and aim to maximize it. in practice, based on the infonce theory @xcite , such an mi gap can be approximated via a contrastive objective by using positive samples from the activated experts and negative samples from the inactive experts @xcite . the contrastive objective is incorporated as an auxiliary objective during training, encouraging specialization and modularization among experts. we name the proposed method contrastive representation for moe (comoe), a novel moe variant. empirically, we evaluate comoe on diverse benchmarks, showcasing its remarkable performance on heterogeneous tasks. summary of our contributions: ## preliminaries lora basics lora @xcite introduces a pair of low-rank matrices a and b to reparameterize the pretrained weights w 0 in a residual manner. specifically, input x is processed through both the frozen weights and the low-rank matrices: ## methods in this section, we first define an mi gap for topk routing. then we derive a contrastive objective to estimate the gap with learning expert representations. finally, we present a training approach that incorporates the contrastive objective as an auxiliary loss. ## experiments in this section, we conduct extensive experiments coupled with ablation and visualization experiments to evaluate the effectiveness of comoe, accompanied by concise analyses. ## conclusion in this paper, we focus on the problem of redundant knowledge learning in moe, which leads to the underutilization of its capacity. to address this issue, we propose contrastive representation for moe (comoe), a novel moe variant that promotes modularization and specialization. specifically, we layer:32, task:obqa expert 1 expert 2 expert 3 expert 4 (a) w/o contrastive loss layer:32, task:obqa expert 1 expert 2 expert 3 expert 4 (b) w/ contrastive loss figure 5: comparison of expert representations in obqa before and after contrastive loss incorporation in a multitask setting. (a) without contrastive loss. (b) with contrastive loss.