L o RETTA : Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models (2024)

Abstract: Various parameter-efficient fine-tuning (PEFT) techniques have been proposed to enable computationally efficient fine-tuning while maintaining model performance. However, existing PEFT methods are still limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). To address this challenge, we present LoRETTA, an ultra-parameter-efficient framework that significantly reduces trainable parameters through tensor-train decomposition. Specifically, we propose two methods, named LoRETTA_adp and LoRETTA_rep. The former employs tensorized adapters, offering a high-performance yet lightweight approach for the fine-tuning of LLMs. The latter emphasizes fine-tuning via weight reparameterization with a set of small tensor factors. LoRETTA achieves comparable or better performance than most widely used PEFT methods with up to 100\times fewer parameters on the LLaMA-2-7B models. Furthermore, empirical results demonstrate that the proposed methods exhibit remarkable anti-overfitting capability, effectively improve training efficiency, and enjoy better multi-task learning performance. Plug-and-play loretta library built upon the Huggingface framework and PEFT library are provided.

## introduction the bert and llama families @xcite @xcite , representing the prevailing paradigm of large language models (llms), showcase remarkable task generalization capabilities in diverse applications, from dialogue systems to question-answering, summarization and translation. while llms exhibit proficiency in following instructions and learning task solutions with minimal contextual input, their accuracy can be further enhanced through fine-tuning techniques. since full-model fine-tuning becomes infeasible as the model size of llms grows rapidly, there has been increased interest in parameterefficient fine-tuning (peft) @xcite . peft methods fine-tune llms by modifying only a subset of parameters. the concept was initially explored in @xcite , which proposes the adapters method to inject trainable modules into the transformer encoders. based on this concept, the lora approach @xcite adds low-rank updating matrices on the weights of linear projection layers in the self-attention blocks. these two types of methods achieve similar or even better performance than full-model fine-tuning, but still incur a large number of trainable parameters. taking the llama-2-70b model as an example, lora needs to update over 16 million parameters, which is even more than the total parameters of some bert models. moreover, we observe that both the adapters and lora approaches experience significant overfitting problems, detrimentally affecting their overall performance. in contrast, other methods like prefix tuning @xcite and prompt tuning @xcite introduce trainable tokens to the input or hidden layers of the base model, significantly reducing trainable parameters but potentially sacrificing accuracy, especially in few-shot learning scenarios @xcite . furthermore, @xcite achieves approximately 90% of the full fine-tuning performance with only 200∼800 parameters on a roberta model by exploring the intrinsic dimension, which is far less than the 0.3 million parameters needed in the lora method @xcite . despite lora's ability to outperform full-model fine-tuning, its number of trainable parameters is still too high, motivating our exploration of more economic and efficient high-performance peft approaches. this raises the question: is there a peft approach with ultra-low trainable parameters that still performs on-par or better than full-model fine-tuning? in this paper, we present low-rank economic tensor-train adaptation (loretta), which is tailored for efficient fine-tuning of variously scaled llms with minimal trainable parameters. our approach leverages the tensor-train (tt) format to represent large weight matrices. loretta encompasses two variants: loretta adp and loretta rep . the loretta adp variant embeds tensorized adapters in encoder/decoder layers and performs better than all peft methods under equivalent trainable parameter sizes. the loretta rep variant, our ultra-efficient innovation, requires substantially fewer trainable parameters, occupies less than 1mb of storage, and maintains comparable performance. our contributions are threefold: ## loretta method peft methods can be broadly categorized into three types, the adapters, the reparameterization method, and the prompt-based method @xcite . among them, the reparameterization-based and adapter-based methods are notable for incorporating new structures within the model architecture, thereby introducing a large number of additional trainable parameters. to reduce the size of the injected modules, we introduce our loretta framework, which contains the adapter-based approach loretta adp and the reparameterizationbased approach loretta rep . subsequent sections will delve into the intricacies of the tensorized layer, followed by an in-depth exploration of the loretta adp and loretta rep structures. ## experiment we conduct comprehensive experiments for the performance of loretta on the downstream task for the llms with different scales. specifically, we present the results on both bert-family (roberta-base @xcite and debertabase @xcite ) models and the large-scale llama-2 models @xcite . we first show that loretta frameworks perform on par or better than other peft methods (like bitfit, lora, adapters, and prefix tuning, etc.) with fewer trainable parameters across different model types, sizes, and tasks, especially on the llama-2 models. then, we discuss some observations of the strong ability of loretta in multi-task learning and addressing overfitting issues. further experiments demonstrate that the loretta method can help to reduce the memory storage, training flops, and improve the memory copy efficiency. finally, we also carry out the tensor rank analysis of our approach to show the applicability of loretta with even fewer trainable parameters. all experiments utilize the adamw optimizer @xcite , and similar learning rate and batch size set up for different methods (see appendix a for details). we use nvidia tesla v100-16gb and a100-40gb for experiments. ## conclusion we propose an ultra-parameter-efficient fine-tuning method, named loretta , which outperforms @xcite and quantization @xcite .