MEL o RA : Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models’ scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential.The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.

## introduction large language models (llms) have emerged as the default paradigm for natural language processing (nlp) @xcite . fine-tuning (ft) is a prevailing way for tailoring llms for specific downstream tasks @xcite . however, as the models' scale and the diversity of the tasks increase, fully ft becomes infeasible. parameterefficient fine-tuning (peft) has been proposed to alleviate memory demands by reducing trainable parameters @xcite and the proposed @xcite . the core idea of melora is to freeze original pretrained weights and train a group of mini loras in parallel with only a small number of parameters. low-rank adaptation (lora) @xcite ) is widely being used due to its minimal additional memory overhead and because it comes without additional inference latency. as illustrated in figure 1 (left), lora uses low-rank matrices (a, b) to approximate the updates of pre-trained weights (w ). as the rank is smaller than the model's hidden dimension, the overall number of trainable parameters of lora is much smaller than full ft. despite the significant computational advantage, low-rank approximation may lead to a substantial performance gap when compared to full ft @xcite . therefore, the following is a critical challenge: how to enable a higher rank variation while preserving the computational advantage? to increase the rank of lora without introducing more trainable parameters, relora @xcite and cola @xcite append multiple loras to pre-trained weights. they progressively merge old lora to pre-train weights and stack new loras during training. in essence, these methods train multiple loras in series. however, there may be overlap between the series of lora modules. therefore, employing the direct sum of multiple lora modules in these methods does not necessarily guarantee an increase in rank. in this work, we propose a simple yet effective method, called mini-ensemble low-rank adapters (melora), that stacks multiple mini loras in parallel, as shown in figure 1 (right). we demonstrate theoretically that melora ensures a higher rank without imposing an additional parameter overhead. we concatenate multiple mini loras simultaneously along the diagonal to construct an equivalent block diagonal lora matrix. each mini lora is therefore independent of the other and the final rank will be the sum of the rank of each mini lora. each mini lora rank just learns the different dimensions of the hidden state. the shape of the trainable weights a and b in mini lora will be much thinner. we conduct extensive experiments across diverse tasks and models to demonstrate the efficacy of melora. evaluations are performed using roberta-base on natural language understanding tasks and llama-2-7b on instruction following tasks. results indicate that melora achieves superior performance while using significantly fewer parameters. for instance, with 36 times fewer trainable parameters of lora, melora outperforms lora on all instruction following datasets. we summarize our contributions as follows: • we propose a new method (melora) on top of lora that makes it achieve a higher rank and better performance with fewer parameters. ## related work full-parameter fine-tuning poses computational challenges with growing model sizes and the proliferation of downstream tasks. in response to these challenges, parameter-efficient fine-tuning (peft) , modifying only a small portion of parameters while leaving the majority of pre-trained model parameters unchanged, has received increasing attention from researchers. numerous studies @xcite @xcite @xcite @xcite discuss key factors such as reduced inference overhead, memory efficiency, and storage optimization. in particular, lora @xcite introduces trainable lowrank matrices to approximate weight updates during fine-tuning. due to its simplicity in implementation without inducing any noticeable latency during inference, it is widely used in many fields. a number of advanced techniques have been proposed that build on the basic principles of lora. these extensions fall into two main categories: adaptive rank and customized update strategies. ## methodology in this section, we introduce the proposed miniensemble low-rank adapters (melora), a novel method that involves concatenating the outputs from several mini lora modules, as illustrated in figure 1 . ## analysis in this section, we analyze two key hyperparameters in melora: the number of mini lo-ras n and the rank of each mini lora r. according to equation 5, the equivalent rank of melora is denoted as n × r. we investigate the effect of the equivalent rank in section 6.1, and analyze n and r separately in section 6.2 and 6.3. ## conclusion in this paper, we have proposed a new parameterefficient fine-tuning method, melora, that stacks multiple mini loras in parallel, with each mini lora rank learning the different dimensions of a hidden state. we have theoretically demonstrated that melora maintains a higher and flexible rank, as well as lower complexity. we have also shown empirically that melora achieves a higher rank and better performance with fewer trainable parameters on multiple datasets.