STAR : Constraint L o RA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models (2024)

Abstract: Though Large Language Models (LLMs) have demonstrated the powerful capabilities of few-shot learning through prompting methods, supervised training is still necessary for complex reasoning tasks. Because of their extensive parameters and memory consumption, both Parameter-Efficient Fine-Tuning (PEFT) methods and Memory-Efficient Fine-Tuning methods have been proposed for LLMs. Nevertheless, the issue of large annotated data consumption, the aim of Data-Efficient Fine-Tuning, remains unexplored. One obvious way is to combine the PEFT method with active learning. However, the experimental results show that such a combination is not trivial and yields inferior results. Through probe experiments, such observation might be explained by two main reasons: uncertainty gap and poor model calibration. Therefore, in this paper, we propose a novel approach to effectively integrate uncertainty-based active learning and LoRA. Specifically, for the uncertainty gap, we introduce a dynamic uncertainty measurement that combines the uncertainty of the base model and the uncertainty of the full model during the iteration of active learning. For poor model calibration, we incorporate the regularization method during LoRA training to keep the model from being over-confident, and the Monte-Carlo dropout mechanism is employed to enhance the uncertainty estimation. Experimental results show that the proposed approach outperforms existing baseline models on three complex reasoning tasks.

## introduction large language models (llms) @xcite @xcite @xcite have demonstrated the powerful capabilities of zero/few-shot learning with prompting techniques, including in-context learning @xcite and chainof-thought @xcite , where no parameter update is required. however, previous studies @xcite @xcite have shown that further fine-tuning is still crucial for tasks involving complex reasoning such as arithmetic reasoning @xcite and commonsense reasoning @xcite , etc. fine-tuning llms requires updating a large number of parameters, which takes a lot of time and consumes considerable memory. taking llama-7b @xcite as an example, finetuning it on a dataset of 52k instances takes over 12 hours on 4 a100 80g gpus @xcite . therefore, parameter efficient fine-tuning (peft) methods @xcite @xcite @xcite and memory efficient fine-tuning (meft) methods @xcite have been proposed. in addition to updating a vast number of parameters and consuming substantial memory, a neglected factor in llms fine-tuning is the extensive consumption of annotation data. moreover, due to the inherent complexity of tasks, the human annotation resources required for fine-tuning llms are also significant @xcite . therefore, it is important to develop the data-efficient fine-tuning (deft) method for llms. a common practice to improve data efficiency is active learning @xcite , while it has been shown that the peft methods can alleviate the reliance on annotated data to some extent @xcite . a straightforward idea for deft is to combine the peft method with active learning. however, such a combination is not trivial. as shown in figure 1 (a), simply finetuning an llm with lora @xcite under an uncertainty-based active learning framework yields consistently inferior performances compared to passive learning (random selection of data in active learning) on the openbookqa dataset, while as shown in figure 1 (b), fine-tuning llm with full parameter under active learning yields performances better than passive learning. to investigate the mechanism behind this uncommon phenomenon, probe experiments are conducted by investigating the prediction confidence and entropy of the llm with lora during uncertainty-based active learning. based on the experimental results, we deduce two potential reasons for this phenomenon. the first issue is uncertainty gap. to be more specific, the uncertainty calculated for selecting data during active learning comes from the full parameters, while only partial parameters remain tuned during peft. it suggests that the conventional way of calculating uncertainty may not reflect the knowledge required by the peft parameters and therefore undermines the performance of active learning. the second issue is poor model calibration which becomes particularly significant when using the peft method @xcite . it further indicates that the uncertainty calculated in the conventional way is not well-calibrated, and the data selected for active learning becomes suboptimal. to address the aforementioned issues, we propose constraint lora with dynamic active learning (star), a novel approach to effectively integrate uncertainty-based active learning and lora. specifically, for the uncertainty gap, we introduce a dynamic uncertainty measurement that combines the uncertainty of the base model and the uncertainty of the full model during the iteration of active learning. for poor model calibration, we incorporate the regularization method during lora training to keep the model from being overconfident, and the monte-carlo dropout mechanism @xcite is employed to enhance the uncertainty estimation. experimental results show that the proposed approach outperforms existing baseline models on three complex reasoning tasks. the above issues are partially resolved. in conclusion, our contributions are three-fold: ## probing peft on prediction uncertainty in this section, we describe how to design probe experiments to investigate the reason behind the failure of lora combined with al methods. we will first introduce the experiment setup, and then we will talk about how to prob lora under the al framework with prediction confidence and prediction entropy. we also discuss how to conclude from the experimental results. ## methods in this section, we introduce the proposed method star in detail. we will first describe the overall workflow of star, then we will discuss methods to address the uncertainty gap issue and the model calibration issue. finally, we will conclude the proposed method with a pseudocode. ## conclusion in this paper, to improve the data efficiency of large language models (llms) during the finetuning process, we propose a data-efficient parameter tuning method by combining lora with active learning. to address the issue that uncertaintybased active learning fails to combine with lora, we experimentally identify and summarize two possible reasons: uncertainty gap and poor model calibration. to resolve the uncertainty gap issue, we propose a dynamic uncertainty calculation method, and to address poor model calibration, we introduce a regularization-based constraint method. by integrating these two approaches, we partially solve the aforementioned failure issues. extensive experiments show that our proposed method outperforms baseline models on multiple reasoning datasets.