Instance-Level Dynamic L o RA s Composition for Cross-Task Generalization (2024)

Abstract: Large language models perform well on tasks that have undergone fine-tuning of instructions, but their performance on completely unseen tasks is often less than ideal. To overcome the challenge of cross-task generalization, task-level LoRAs combination is proposed, which does not require training a model for new tasks. Instead, it learns the LoRA modules combination weights based on a small number of samples to form the task model. However, task-level LoRAs combination only utilizes a few task modules due to its reliance on the weight enumeration method, and it also ignores the specificity between different instances. Therefore, we proposed an instance-level LoRAs composition for cross-task generalization, which selects appropriate multiple task LoRA modules for each input instance and dynamically determines the composition weights. Our experiments on publicly available datasets show that our method outperforms the typical method, LoraHub, in 16 out of 27 tasks. We release the source code at https://github.com/noname822/iLoraComp.git

## introduction currently, large language models (llms) demonstrate remarkable zero-shot learning capabilities on tasks that have undergone instruction tuning @xcite @xcite . however, numerous studies have revealed that when encountering novel tasks outside their training distribution, these models often fail to exhibit satisfactory performance @xcite . exploring strategies to enhance the cross-task generalization abilities of these massive language models, enabling them to adapt swiftly and accurately to diverse new tasks, has emerged as a pressing challenge that demands attention. ## conclusion our work introduces two key enhancements to the lorahub framework. the first is a method that indexes models trained on datasets based on their semantic centroids, improving lorahub's precision at the task level. the second is instance-level adaptation, which leverages the unique characteristics of individual instances to raise the performance ceiling of the lorahub approach. these complementary strategies work together to enhance the model's cross-task generalization capabilities. 9 limitation increased computational cost. our method incurs a higher computational cost than lorahub, mainly because we train weights for each individual instance during the lora group weights training phase. this means that our approach will require computational resources proportional to the number of instances, multiplied by the cost of lo-rahub's training. application scenario limitation. our method is not universally cost-effective. in scenarios where a task involves a limited number of questions, employing our method may not be the most economical choice. for tasks without any instances, zeroshot learning would be a more appropriate and efficient approach.