An Empirical Study on Parameter-Efficient Fine-Tuning for M ulti M odal Large Language Models (2024)

Abstract: Multimodal Large Language Models (MLLMs) fine-tuned with multimodal instruction-following data have demonstrated formidable capabilities in multimodal tasks. However, fine-tuning all parameters of MLLMs has become challenging due to the rapid growth of the overall model’s parameters. To address this issue, we study Parameter-Efficient Fine-Tuning (PEFT) methods for MLLMs. We aim to identify effective methods for enhancing performance in scenarios where only a limited number of parameters are trained. This paper conducts empirical studies that employ four widely used PEFT methods to fine-tune the LLM component of open-source MLLMs. We present a comprehensive analysis that encompasses various aspects, including the impact of PEFT methods on various models, parameters and location of PEFT module, fine-tuning data scale, model stability based on PEFT method, MLLM’s generalization, and hallucination. We evaluated four PEFT methods on seven datasets from two different categories, unseen and seen datasets. Across all experiments, we show that the adapter is the best-performing PEFT method in various aspects. At the same time, fine-tuning the connector layers leads to improved performance in most MLLMs.

## introduction in recent years, the landscape of multimodal learning has been transformed by the emergence of multimodal large language models (mllms), such as llava @xcite , minigpt4 @xcite , and gpt4-vision @xcite . mllms have showcased impressive competency across a spectrum of multimodal benchmarks @xcite @xcite thanks to the integrated architecture of pre-trained visual encoders, connector layers, and llms. this * equal contribution. in contrast to standard llms, mllms introduce additional modules: visual encoder and connector layers. during the fine-tuning process, unimodal llms only receive text features while mllms get multimodal inputs, such that connector layers are also fine-tuned, not just fine-tuning the llm. therefore, it is crucial to reassess the performance of fine-tuning mllms using various peft methods, exploring the impact of connector fine-tuning on the model's performance in downstream tasks, and examining peft's effects on model stability, generalization, and hallucination. in this paper we address these issues by conducting comprehensive studies on three representative mllms containing connector layers: llava-1.5 (7b, 13b) @xcite , sharegptv4 (7b) @xcite , and qwen-vl-chat (7b) @xcite . our study looks at various issues related to peft methods. specifically, we design our study to address the following questions: (1) is it necessary to fine-tune the connector when fine-tuning mllms via vari-1 10057 ous peft methods on unseen and seen datasets? our key findings can be summarized as follows: our contributions can be summarized as follows: ## peft methods figure 1 illustrates the architecture of mllms and the location of various peft modules. in our experiments, all considered mllms consist of three components: a visual encoder, connector layers, and a llm. the structure of the connector layers may vary depending on the specific mllm. peft methods can be classified into three categories, from which we select four methods: (1) reparametrization-based tuning: lora, ia3 (2) adapter-based tuning: adapter. (3) prompt-based tuning: prefix-tuning. lora. we integrate the low-rank strategy proposed by @xcite to adjust the network weights, facilitating the model's handling of complex tasks with an efficient parameter footprint. the original pre-trained weight matrix w 0 ∈ r d×k is updated through low-rank decomposition using 2 10058 train freeze visual encoder / connector should we tune or freeze? how to place it for better performance? do we need more trainable parameters? any connection between generalization and various peft? any relationship between hallucination and different peft? which peft is more efficient? can various peft affect the stability of the model? ## conclusion we conducted an extensive investigation on four peft methods applied to mllms across different multimodal tasks. by fine-tuning different mllms in a uniform way and conducting thorough hyperparameter optimization, we benchmarked the performance of these methods. our findings indicate that adapter excels in accuracy, stability, generalization, and producing fewer hallucinations. additionally, we found that fine-tuning the connector layers of mllms simultaneously does not always yield better results. finally, comprehensive ablation studies were performed to understand the contributions of the location of peft modules, learning rate settings, and the size of training data on peft performance.