L o RA -Flow: Dynamic L o RA Fusion for Large Language Models in Generative Tasks (2024)

Abstract: LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain, where different learned additional modules represent diverse skills. Combining existing LoRAs to address new tasks can enhance the reusability of learned LoRAs, particularly beneficial for tasks with limited annotated data. Most prior works on LoRA combination primarily rely on task-level weights for each involved LoRA, making different examples and tokens share the same LoRA weights. However, in generative tasks, different tokens may necessitate diverse skills to manage. Taking the Chinese math task as an example, understanding the problem description may depend more on the Chinese LoRA, while the calculation part may rely more on the math LoRA. To this end, we propose LoRA-Flow, which utilizes dynamic weights to adjust the impact of different LoRAs. The weights at each step are determined by a fusion gate with extremely few parameters, which can be learned with only 200 training examples. Experiments across six generative tasks demonstrate that our method consistently outperforms baselines with task-level fusion weights. This underscores the necessity of introducing dynamic fusion weights for LoRA combination.

## introduction large language models (llms) have demonstrated superior performance over previous smaller models across a wide range of tasks @xcite @xcite , thereby extending the applicability of ai systems to numerous real-world scenarios. because of their substantial model size, training all parameters of llms can ## approach q k v self attention softmax w ⨁ " ∆" ! ∆" " weighted sum " # fusion gate feed forward add bias figure 2: left: we use layer-wise fusion gates to facilitate dynamic lora fusion, which project input hidden states of each layer into fusion weights. right: for a certain module, the provided fusion weights are used to aggregate the outputs of different lora modules. since our goal is to leverage the abilities acquired by existing lora modules to address new tasks, we only train the fusion gate with a few examples, while keeping both the model and the lora modules frozen. the number of parameters of the fusion gate is only approximately 0.2% of those in a lora. ## analysis to gain deeper insights into the behavior of lora-flow, we conduct a comprehensive series of analyses from various perspectives. we first estimate the average fusions at different model layers in section 5.1, and then investigate how the fusion weights change across time steps in section 5.2. finally, we provide a specific case in section 5.3. ## conclusion in this work, we propose lora-flow, a dynamic combination method for lora. by assigning dynamic weights to different lora modules based on the current context, lora-flow can outperform representative baselines with static task-level fusion weights on six complex generative tasks.