Tied- L o RA : Enhancing parameter efficiency of L o RA with Weight Tying (2024)

Abstract: We introduce Tied-LoRA, a novel paradigm leveraging weight tying and selective training to enhance the parameter efficiency of Low-rank Adaptation (LoRA). Our exploration encompasses different plausible combinations of parameter training and freezing, coupled with weight tying, aimed at identifying the optimal trade-off between performance and the count of trainable parameters. Across 5 diverse tasks and two foundational language models with different parameter counts, our experiments provide comprehensive insights into the inherent trade-offs between efficiency and performance.Our findings reveal a specific Tied-LoRA configuration that distinguishes itself by showcasing comparable performance to LoRA across multiple tasks while utilizing only a fraction of the parameters employed by the standard LoRA method, particularly at elevated ranks. This underscores the efficacy of Tied-LoRA in achieving impressive results with significantly reduced model complexity.

## introduction large language models (llms) play a crucial role in various natural language processing (nlp) applications due to their proficiency. a significant factor driving their widespread adoption is the ability to fine-tune pretrained llms efficiently for specific downstream tasks. this fine-tuning process allows the creation of specialized language models that excel in specific domains and tasks. despite dealing with smaller training data compared to pretraining, the computational demand for during fine-tuning remains high, especially for large models with billions of parameters. moreover, for llm service providers, it is often necessary to cater to diverse requirements by maintaining distinct customizations for each combination of user and task in the service. for instance, consider a scenario where a language model is employed to assist users in generating content figure 1 : schematic of our tied-lora paradigm, the main low-rank matrices a and b are tied across (indicated by the symbol) all the layers of the base language model. we use the gradient shading to indicate that these parameters can either be trained or frozen. for social media. user x may have preferences for formal language and professional tone, while another user, y, might prefer a more casual and conversational style. additionally, each user may have different tasks, such as composing business emails, translating documents, creating social media captions or drafting blog posts. to serve these varied preferences and tasks simultaneously, the language model needs to be finely tuned for each specific combination of user (x or y) and task (email composition or translation). as the number of users and tasks per user increases, so does the complexity and cost associated with customization. managing and storing the various combinations of customizations for each user-task pair can introduce additional expenses, especially after the initial training phase. the storage and retrieval of these customized models, each tailored to specific user preferences and tasks, contribute to ongoing operational costs. therefore, in addition to efficient utilization of customizable parameters during training, careful consideration must be given to the post-training phase, where the cost of saving and accessing these combinations becomes a significant factor in the overall resource management strategy. this holistic approach is crucial for maintaining optimal performance across diverse user-task combinations while keeping both computational and operational costs in check. in light of these challenges, developing effective customization methods that not only enhance model performance but also reduce the number of training parameters becomes crucial. parameterefficient fine-tuning (peft) emerges as a valuable approach in this context. peft involves refining pretrained models with minimal parameter updates, enabling the creation of specialized models that excel in specific domains and tasks. this streamlined customization process not only optimizes parameter utilization during training but also mitigates the costs associated with managing, storing, and serving diverse customizations post-training. low-rank adaptation (lora) method (hu et al., 2021), stands out as a popular and efficient parameter-efficient fine-tuning (peft) approach, offering a straightforward implementation and the ability to integrate lora weights into the base model post-training. despite its advantages, the expense of lora training becomes more pronounced, particularly with the growing size of base language models. while previous efforts focused on enhancing lora's parameter efficiency through careful low-rank selection, we introduce an alternative approach. in contrast to controlling parameter count through rank, our method incorporates simple weight tying alongside selective training. this novel combination forms the basis for a range of tied-lora configurations, each evaluated for performance across five diverse customization tasks. through this approach, we aim to push the boundaries of parameter-efficient fine-tuning, making advancements in both effectiveness and simplicity. our contributions are threefold: ## method in this section, we introduce tied-lora, a paradigm for parameter-efficient fine-tuning of large language models through low-rank weightupdate approximations, weight-tying and selective training. our framework offers a range of "lora-like" configurations through a series of design choices over selective parameter training and weight tying, including some of the existing peft methodologies available in the literature. specifically, we use weight tying alongside pairs of projection matrices and scaling vectors that can be selectively either trained or frozen. as the low-rank computation path does not introduce any non-linearity, all tied-lora configurations can be merged into the base model weights to preventing additional latency during inference. ## experiments we now turn to evaluating the different configurations possible within our tied-lora paradigm. while lora (vbua) and peft methods can be used to train models for general instruction following @xcite @xcite , we focus our evaluations in a "task customization" perspective, where each model is trained on a specific task and is evaluated on a test set from the same task. ## results for each task we report the metric used, such as rougel @xcite , exact match (em), accuracy and bleu @xcite , and the rank r used. for each model, the table is segmented into two sections: the first section compares the performance of all tied-lora configurations at the same rank where lora (vbua) achieved its optimum score. the second section shows the best performance of achieved by each configuration. in addition to metric scores and rank (r) we also report parameter usage percentage (p%) as a comparison to the parameter count of the best-performing lora configuration. this offers a direct measure of efficiency, showing how each model, especially tl 5 (vb ua ) and tl 6 (vb ua ) , leverages a smaller percentage of parameters compared to the lora (vbua) for achieving its results. we can immediately see that lora (vbua) is the best performing model for both the 2b and 7b base language models on most tasks. this is hardly surprising as it is the most expensive method with respect to trainable parameters. the tl 6 (vb ua ) configuration demonstrates the best overall performance among all tied-lora configurations for both model sizes and is not far behind lora. for our translation task with the llama2 7b base model, tl 6 (vb ua ) out performs lora (vbua) while using 12.5% of the number of parameters. this consistent performance illustrates it effectiveness across diverse model scales and task types. the tl 5 (vb ua ) configuration, while marginally outperformed by tl 6 (vb ua ) , is notable for its parameter efficiency. this method achieves comparable performance to tl 6 (vb ua ) for typical ranks @xmath0, 16, 32, 64, 128, but with a reduced parameter count. both tl 5 (vb ua ) and tl 6 (vb ua ) show robust performance at the same rank where traditional lora (vbua) is optimized (i.e., performed best). this implies that for systems pretuned for lora (vbua) , tl 5 (vb ua ) and tl 6 (vb ua ) can be utilized with the same rank configuration as a "drop-in" replacement. on average, we observe a 1.36 % decline in tl 6 (vb ua ) performance compared to the lora (vbua) model with the llama 2 7b model. this decline, however, is marginally higher at 1.95 % with the 2b model. these findings suggest that the efficiency of tl 6 (vb ua ) configuration may enhance with a larger or more capable base models (such as the llama2 70b model). this hypothesis warrants a future exploration which we leave for future research. ## conclusion & future work in this paper, we introduced the tied-lora paradigm, a novel approach to enhance the parameter efficiency of lora by employing a simple technique of weight-tying and selective training of low-rank matrices. our empirical analysis demonstrates that the tl 6 (vb ua ) configuration achieves performance comparable to lora across various tasks, as language models continue to advance, the tied-lora configurations, with their optimized efficiency, emerge as a promising candidate to replace traditional lora in a broader range of applications. this progression underscores the relevance of tied-lora as a scalable solution in the dynamic landscape of large language model customization. for future research, we plan to delve into the application of tied-lora methods on larger base models. this exploration aims to assess their scala-bility and effectiveness within the broader context of large language models. additionally, we intend to investigate weight tying in other parameterefficient fine-tuning methods such as adapters and prefix tuning, both of which introduce layerspecific parameters.