Multimodal Instruction Tuning with Conditional Mixture of L o RA (2024)

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in diverse tasks across different domains, with an increasing focus on improving their zero-shot generalization capabilities for unseen multimodal tasks. Multimodal instruction tuning has emerged as a successful strategy for achieving zero-shot generalization by fine-tuning pre-trained models on diverse multimodal tasks through instructions. As MLLMs grow in complexity and size, the need for parameter-efficient fine-tuning methods like Low-Rank Adaption (LoRA), which fine-tunes with a minimal set of parameters, becomes essential. However, applying LoRA in multimodal instruction tuning presents the challenge of task interference, which leads to performance degradation, especially when dealing with a broad array of multimodal tasks. To address this, this paper introduces a novel approach that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA (MixLoRA). It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance, aiming to mitigate task interference. Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks, demonstrating its efficacy and adaptability in diverse multimodal tasks.

## introduction the advent of multimodal large language models (mllms) @xcite @xcite have revolutionized the field of artificial intelligence, demonstrating remarkable capabilities in processing and integrating information from various modalities, notably text and image. a key focus in advancing mllms is to enhance zero-shot generalization to novel multimodal tasks. in this pursuit, multimodal instruction tuning, which fine-tunes pre-trained models with diverse, instruction-based multimodal tasks, has demonstrated its efficacy in facilitating zero-shot generalization to unseen multimodal problems @xcite @xcite . concurrently, the growing complexity and scale of mllms have spurred the development of various parameter-efficient fine-tuning (peft) techniques @xcite @xcite @xcite . among these, low-rank adaption (lora) @xcite has emerged as a powerful peft method that finetunes large pre-trained models by updating a small amount of injected adaption parameters. however, in multimodal instruction tuning, the effectiveness of conventional peft methods like lora diminishes due to their reliance on adjusting a limited portion of shared parameters to simultaneously accommodate diverse tasks, leading to task interference -a problem well-studied in multi-task learning @xcite @xcite , but insufficiently investigated in the context of parameter-efficient multimodal instruction tuning. the diverse nature of multimodal tasks significantly increases the risk of task interference. for instance, using the same limited set of adaptation parameters for distinct tasks like ocr and domain-specific classification can cause conflicting updates, potentially leading to suboptimal performance. our research seeks to explore and address task interference in parameter-efficient multimodal instruction tuning. specifically, we aim to answer two critical research questions: (1) does task interference exist in parameter-efficient multimodal instruction tuning? (2) how can we effectively mitigate this issue for robust and versatile performance across various multimodal tasks? to answer the first question, we investigate the task-interference issue in parameter-efficient multimodal instruction tuning from the perspective of gradient direction @xcite in section 3.2. our observations highlight notable task interference in this context, underscoring the necessity for more effective adaptation strategies to ensure robust and versatile performance across diverse multimodal tasks. in response to our second question, this paper proposes a novel multimodal instruction tuning framework -conditional mixture-of-lora (mixlora), designed to mitigate the task interference issue. as shown in figure 1 , unlike conventional lora which uses shared low-rank adaptation matrices a and b across all tasks and instances, mixlora dynamically constructs lowrank adaptation matrices a and b tailored to each input instance, by selecting their decomposition factors from two collections. mixlora introduces a dynamic factor selection mechanism, incorporating two independent factor selection (ifs) routers and a conditional factor selection (cfs) router. the two ifs routers independently select appropriate factors to dynamically construct lora a and b matrices tailored to each input. the cfs router further refines the selection for lora b based on the factors chosen for lora a, ensuring that the factors selections for lora a and b are not only tailed to input but also cohesively aligned. to validate the effectiveness of mixlora, we conduct extensive experiments on mme @xcite , a comprehensive multimodal evaluation benchmark, and seven additional multimodal evaluation datasets that focus on various capabilities. experimental results demonstrate that mixlora, with its dynamic factor selection approach, consistently outperforms lora across various multimodal tasks when using the same number of ranks and remains competitive or superior even against lora with a higher rank number. this effectiveness is attributed to the dynamic factor selection mechanism and its ability to generalize to unseen tasks through adaptive factor activation, underscoring the potential of mixlora to generalize and perform effectively on unseen multimodal tasks. our contributions are summarized as follows: (1) we empirically investigate and demonstrate the existence of task interference in parameter-efficient multimodal instruction tuning. (2) we propose the conditional mixture-of-lora (mixlora) framework, aimed at alleviating task interference by dynamically constructing low-rank adaptation matrices for various inputs. (3) comprehensive experiments demonstrate the effectiveness of mixlora, outperforming lora across various unseen multimodal tasks at equal or even higher ranks. ## related work multimodal instruction tuning instruction tuning @xcite significantly improves the generalization of large language models to unseen tasks based on natural language instructions @xcite . with the advent of multimodal large language models, the scope of instruction tuning has expanded to encompass multimodal and vision tasks, facilitated by the development of diverse multimodal instruction datasets, including both machine-generated @xcite @xcite @xcite and human-annotated @xcite . recently, vision-flan @xcite stands out as a comprehensive human-annotated visual instruction tuning dataset, covering a wide range of 187 tasks, making it ideal for our training. parameter-efficient fine-tuning (peft) parameter-efficient fine-tuning (peft) @xcite @xcite @xcite strategies have become key in efficiently adapting large pre-trained models to various downstream tasks with minimal parameter adjustments. these methods typically involve freezing most of the pre-trained model while fine-tuning a small fraction of the parameters to facilitate the adaptation process. among these, lora @xcite demonstrates competitive trade-offs between performance and parameter efficiency, making it widely adopted. peft methods typically utilize shared adaptation parameters across diverse tasks or train task-specific adapters. however, when applied to multimodal instruction tuning, which requires simultaneous adaptation to diverse instruction tasks, peft can encounter task interference, highlighting the need for more adaptable and versatile peft methods to adeptly handle the complexities of multimodal instruction tuning. task interference task interference @xcite is a notable challenge in multi-task learning, where simultaneous training on multiple tasks can lead to performance decline due to conflicting gradients among tasks @xcite @xcite . to mitigate task interference in multi-task learning, researchers have explored various strategies, including dynamic adjustment of task loss contributions @xcite @xcite and parameter partitioning @xcite @xcite . despite the established understanding of task interference in multi-task learning, its presence and implications in instruction tuning, particularly in multimodal contexts, remain under-explored. given the intrinsic complexity and diversity of multimodal instruction-based tasks, substantial task interference is likely to exist in multimodal instructiontuning scenarios. our research delves into this area, specifically investigating task interference within parameter-efficient multimodal instruction tuning. ## conditional mixture-of-lora inspired by the concept of mixture-of-experts @xcite , we propose conditional mixture-of-lora (mixlora) which leverages low-rank decomposition factors as dynamically chosen experts to construct tailored decomposition matrices a and b for specific input instances. mixlora facilitates dynamic processing pathways for varying input instances, thereby enhancing the efficacy in handling diverse and complex multimodal instruction tasks. the core of conditional mixture-of-lora lies in representing the weight adjustment matrices ∆w from equation 1 via tensor decomposition: leveraging the concept that ∆w can be expressed as the sum of outer products of low-rank decomposition factors a i and b i , mixlora introduces a dynamic factor selection module. this module dynamically constructs unique ∆w for specific inputs by selecting r appropriate factors from an expanded pool of decomposition factors {a e } e @xmath0, {b e } e @xmath1, @xmath2, as shown in fig. 1 (b). here, the number of factors e is much larger than the rank r. following lora (hu et al., 2021), we use a random gaussian initialization for {a e } e @xmath3. ## results and discussion comparison with lora we first present a detailed comparison between mixlora and the conventional lora, focusing on their performance in mme and 7 other multimodal tasks, as detailed in increase the number of rank we investigate the impact of increasing the rank number while keeping the number of factors constant. as shown in ## conclusion we introduce conditional mixture-of-lora, an innovative strategy that dynamically constructs low-rank adaptation matrices specific to individual inputs, to mitigate task interference during parameter-efficient multimodal instruction tuning. comprehensive experiments across a variety of multimodal datasets have demonstrated the efficacy of mixlora, showcasing an enhanced performance on unseen multimodal tasks compared to conventional lora and demonstrating its effectiveness in mitigating task interference.