G ui L o M o: Allocating Experts and Ranks for L o RA - M o E via Bilevel Optimization with G uided S election Vectors (2025)

Abstract: Parameter-efficient fine-tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), offer an efficient way to adapt large language models with reduced computational costs. However, their performance is limited by the small number of trainable parameters. Recent work combines LoRA with the Mixture-of-Experts (MoE), i.e., LoRA-MoE, to enhance capacity, but two limitations remain in hindering the full exploitation of its potential: 1) the influence of downstream tasks when assigning expert numbers, and 2) the uniform rank assignment across all LoRA experts, which restricts representational diversity.To mitigate these gaps, we propose GuiLoMo, a fine-grained layer-wise expert numbers and ranks allocation strategy with GuidedSelection Vectors (GSVs). GSVs are learned via a prior bilevel optimization process to capture both model- and task-specific needs, and are then used to allocate optimal expert numbers and ranks.Experiments on three backbone models across diverse benchmarks show that GuiLoMo consistently achieves superior or comparable performance to all baselines. Further analysis offers key insights into how expert numbers and ranks vary across layers and tasks, highlighting the benefits of adaptive expert configuration. Our code is available at https://anonymous.4open.science/r/GuiLoMo-034 .

## introduction although large language models (llms) have demonstrated remarkable performance across a wide range of general tasks @xcite @xcite @xcite , they still fall short in certain tasks or domains, such as reasoning @xcite @xcite @xcite 2025), multilingualism @xcite @xcite , and text generation in specialized contexts @xcite @xcite @xcite . to enhance the performance of llms in these challenging areas, a common practice is finetuning. however, with the growing size of current llms, full fine-tuning faces significant challenges in terms of computational efficiency and memory consumption. to mitigate these issues, parameterefficient fine-tuning (peft) methods have gained considerable attention @xcite @xcite @xcite @xcite . among these methods, low-rank adaptation (lora) @xcite ) is regarded as one of the most efficient approaches. nonetheless, its performance remains constrained due to the relatively small number of trainable parameters @xcite . recent studies suggest that combining lora with the mixture-of-experts (moe) paradigm, referred to as lora-moe, by incorporating multiple lora modules, offers a promising solution to this limitation @xcite @xcite @xcite . however, fully exploiting the potential of lora-moe remains an open research question. first, @xcite considered that uniformly allocating the number of experts across all layers is suboptimal, as different layers play distinct roles in the model. over-allocating experts to certain layers can lead to redundancy and degraded performance. to address this, they proposed a group-wise expert allocation strategy (mola), which divides all layers into four groups and assigns varying numbers of experts to each group, ensuring that layers within the same group share the same number of experts. building on this, @xcite introduced a layer-wise allocation strategy (alphalora), which theoretically determines the expert numbers for each layer based on its training quality. despite these advancements, two critical limitations remain, as shown in to address these limitations, we propose guilomo, a fine-grained strategy for jointly allocating layer-wise expert numbers and ranks in lora-moe based on bilevel optimization with guidedselection vectors. guilomo operates in two steps: 1) obtaining guidedselection vectors (gsvs): through an initial optimization, gsvs are learned to guide lora-moe in selecting the optimal expert numbers and ranks tailored to both the model backbone and the downstream task; 2) allocating expert numbers and ranks: after the prior optimization, the optimized gsvs are used to allocate expert numbers and ranks for lora-moe, followed by the final training phase. to summarize, our contributions are as follows: ## preliminary lora-moe framework lora-moe integrates multiple vanilla lora experts into each pre-trained llm submodule. vanilla lora @xcite efficiently adapts large models to downstream tasks by lowering computational and memory costs. for a pre-trained weight matrix w 0 ∈ r m×n , lora creates two low-rank trainable matrices a and b, where @xmath0, @xmath1, where r ≪ min(m, n). during training, w 0 remains fixed while a and b are updated via gradient descent. the output representation h is defined as follows: every traditional lora-moe layer incorporates n lora experts. the forward pass through the layer can be formulated as: applying lora-moe for llms lora-moe is applied to key modules of llms, namely multihead attention (mha) and feed-forward networks (ffns). in mha, inputs are projected via w q , w k , w v , and w @xmath2. each ffn uses gateand up-projection matrices w g , w @xmath3, a activation (e.g., gelu), and a down-projection w @xmath4, where d ′ > d. guilomo assigns optimal expert number and rank to these matrices. ffn norm layer mha norm layer input ×@xmath5: exploring the optimal experts number and rank step 2: allocate optimal expert number and rank based on guidedselection vector ... 🔥 🔥 router selected lora expert r i,3 =1 r i,3 =2 r i,3 =3 r i,3 =4 r i,3 =r max r 1,1 r 1,2 r 2,1 r 2,3 r 2,4 r 3,3 r 3,1 expert guidedselection vector g e rank guidedselection vector g r selected lora rank r i,3 r i,3 ... ... argmax(ge)=3 ei,2 ei,3 e max argmax(gr)=4 e e non-selected lora expert non-selected lora rank ... x r 2,2 r max r 3,2 ... ... ... ... ... ei,3 ei,1 ei,2 ei,3 ... ei,3 ei,1 ei,2 ei,3 ... ei,3 ei,1 ei,2 ei,3 ei,3 ei,1 ... ... ## method in this section, we present our guilomo strategy, which consists of two main steps: 1) a bilevel optimization algorithm is employed to obtain guided-selection vectors (gsvs) of expert and rank for each module, tailored to the specific downstream task and model ( § 3.1); 2) based on the obtained gsvs, the optimal expert number and rank for each module in lora-moe are allocated, and the final training is then conducted ( § 3.4). see § 3.2 and § 3.3 for details of gsvs. ## experiment in this section, we conduct extensive experiments to examine the performance of guilomo. we also conduct extra experimental analyses to gain deeper insights into this field, as presented in § 4.3. implementation details can be found in appendix d. ## related work lora-moe framework recent research explores the integration of moe @xcite and @xcite , referred to as lora-moe, to boost performance of llms in both singletask and multi-task scenarios in an efficient manner @xcite @xcite @xcite . for instance, @xcite leveraged lora-moe to reduce catastrophic forgetting problem during supervised fine-tuning. @xcite uitlized lora-moe with a hierarchical gating mechanism for efficient fusion across nlp and vision & language tasks. however, existing works merely allocate expert numbers and ranks uniformly for lora-moe, failing to fully exploit its potential. ## conclusion in this work, we propose guilomo, a fine-grained allocation strategy designed to fully exploit the potential of lora-moe. guilomo jointly determines expert number and rank through a bilevel optimization process. unlike prior methods that rely on uniform or task-agnostic configurations, it introduces a guidedselection mechanism that guides the layer-wise allocation of expert number and rank in lora-moe, tailored to both modelspecific and task-specific needs. extensive experiments demonstrate that guilomo consistently improves model performance across a wide range of tasks. furthermore, our analysis reveals how optimal expert configurations vary across layers and tasks, offering deeper insights into this field. we believe guilomo paves the way for more flexible and efficient expert allocation strategies in future research.