A lpha L o RA : Assigning L o RA Experts Based on Layer Training Quality (2024)

Abstract: Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), are known to enhance training efficiency in Large Language Models (LLMs). Due to the limited parameters of LoRA, recent studies seek to combine LoRA with Mixture-of-Experts (MoE) to boost performance across various tasks. However, inspired by the observed redundancy in traditional MoE structures, prior studies find that LoRA experts within the MoE architecture also exhibit redundancy, suggesting a need to vary the allocation of LoRA experts across different layers. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory to design a fine-grained allocation strategy. Our analysis reveals that the number of experts per layer correlates with layer training quality, which exhibits significant variability across layers. Based on this, we introduce AlphaLoRA, a theoretically principled and training-free method for allocating LoRA experts to reduce redundancy further. Experiments on three models across ten language processing and reasoning benchmarks demonstrate that AlphaLoRA achieves comparable or superior performance over all baselines. Our code is available at https://github.com/morelife2017/alphalora.

## introduction llms have shown impressive performance on various nlp tasks @xcite @xcite @xcite . however, due to the increasing size of modern llms, significant computational resources are required for full fine-tuning. to address this issue, researchers are increasingly focusing on peft methods to reduce training costs, such as adaptertuning @xcite and @xcite . despite their training efficiency, the performance of peft methods in fine-tuning llms is † corresponding author. to address the limitation, recent studies seek to combine lora and moe by adding multiple lora modules @xcite @xcite . the moe structure in llms typically consists of multiple feed-forward "sub-networks", or "experts", that are trained to handle different types of inputs or tasks @xcite . moe structures are designed to dynamically activate only a subset of experts for each input, significantly scaling up the number of parameters, while incurring an affordable computational overhead. existing lora-moe methods integrate multiple lora modules into each sublayer of the transformer block and employ different strategies to assign experts to different tokens or tasks @xcite @xcite @xcite @xcite . for example, at the task level, @xcite ; @xcite explore various composition strategies to combine multiple lora experts trained individually on different tasks. at the token level, both experts and input tokens interact with a routing network, resulting in the activation of experts based on the characteristics of the input tokens @xcite . these methods demonstrate superior performance in single-task and multi-task learning compared to standard lora, with the number of experts per layer uniformly distributed, such as three per layer, as shown in however, recent studies @xcite in moe show that employing a large number of experts may be redundant due to representational collapse or learned routing policy overfitting. similarly, @xcite investigate redundancy in parameter-efficient moe. unlike existing lora-moe methods that uniformly allocate experts across all layers, they manually design four architectures with varying group-wise expert allo-cations. specifically, they divide the 32 layers of the llama-2 model @xcite into 4 groups, with the first 8 layers constituting the first group, and so forth. they allocate a varying number of experts to each group (layers within the same group have the same number of experts). while their allocation strategy provides insights into overall architecture design, suggesting that higher layers need more lora experts, research on achieving more effective integration remains in its early stages. to create a theoretically sound allocation strategy aimed at minimizing expert redundancy, our research draws inspiration from the heavy-tailed self-regularization (ht-sr) theory @xcite @xcite . the ht-sr theory examines the properties of heavy-tailed (ht) structures observed in the empirical spectral density (esd) of weight matrices. the application of ht-sr to model selection and layer-wise adaptive training @xcite showcases the theory's effectiveness in assessing both model and layer quality. in this paper, we propose a fine-grained strategy for allocating layer-wise expert numbers, namely, alphalora. according to @xcite , layers with more pronounced ht properties are generally well-trained. following @xcite , we measure the ht characteristics by fitting a power law (pl) distribution to the esd and use the exponent as the metric to measure ht properties. we then use the hill estimator @xcite to calculate pl_alpha_hill. the core idea behind alphalora is to allocate fewer experts to better-trained (more ht) layers, which is indicated by lower pl_alpha_hill values, thus reducing the redundancy in a more theoretically-principled manner. the contributions of our work are summarized as follows: the relative performance of these metrics reveals that the pl_alpha_hill metric outperforms others, corroborating the findings of @xcite that the pl_alpha_hill metric is better for assessing layer training quality. this further demonstrates the correlation between layer expert number and layer training quality. ## method in this section, we introduce alphalora, an expert allocation strategy based on the pl_alpha_hill metric. given a transformer-based model, we first compute the pl_alpha_hill metric value for each layer. §3.1 elaborates on the calculation of layerwise pl_alpha_hill metric. §3.2 gives the process of mapping the layer-wise pl_alpha_hill value to expert number. the overview of our method is illustrated in figure 1 . alphalora measures the layer training quality based on the ht characteristic of the layer esds, which is quantified by the ht metric pl_alpha_hill. for the i-th weight matrix in each layer from a transformer-based model, we first calculate the eigenvalues of its correlation matrix x @xmath0. we then fit a power law (pl) distribution p to the ht part of the esd, taking values within an interval (λ min , λ max ), which is defined as: we refer to its exponent α as pl_alpha, where a lower value means more heavy-tailed. we then use the hill estimator @xcite to calculate pl_alpha_hill as the following: ## experiment we design two experimental settings to examine the performance of alphalora, including direct fine-tuning and instruction-@xmath1. §4.1 compares the expert allocation between alphalora and variants of mola @xcite . §4.2 presents the results for two experimental settings. implementation details can be found in appendix c. ## conclusion in this paper, we apply analytical methods from ht-sr theory to develop a fine-grained allocation strategy for determining the number of experts per layer in the lora-moe architecture. extensive empirical results show that alphalora offers a simple yet effective approach to layer-wise expert allocation. analysis of three widely used transformerbased language models reveals that well-trained layers require fewer lora experts. our theoretically grounded method provides a scalable solution for various models, further reducing redundancy within the lora-moe architecture. in future work, we aim to integrate alphalora with other peft methods and explore its application across different model architectures and domains.