M ^2 PT : Multimodal Prompt Tuning for Zero-shot Instruction Learning (2024)

Abstract: Multimodal Large Language Models (MLLMs) demonstrate remarkable performance across a wide range of domains, with increasing emphasis on enhancing their zero-shot generalization capabilities for unseen tasks across various modalities. Instruction tuning has emerged as an effective strategy for achieving zero-shot generalization by finetuning pretrained models on diverse multimodal tasks. As the scale of MLLMs continues to grow, parameter-efficient finetuning becomes increasingly critical. However, most existing parameter-efficient approaches focus only on single modalities and often overlook the multimodal characteristics during finetuning. In this work, we introduce a novel Multimodal Prompt Tuning (M ^2 PT) approach for efficient instruction tuning of MLLMs. M ^2 PT effectively integrates visual and textual prompts into the vision encoder and language processor respectively during finetuning, facilitating the extraction and alignment of features across modalities. Empirical results on various multimodal evaluation datasets demonstrate the superior performance of our approach compared to several state-of-the-art baselines. A comprehensive set of ablation studies validates the effectiveness of our prompt design and the efficiency of our approach.

## introduction figure 1 : comparison of m 2 pt and several peft methods, including lora @xcite , ptum @xcite and vpt @xcite , on multimodal tasks. our approach exhibits superior performance across a range of benchmarks. despite considerable advancements, finetuning mllms for specific domain knowledge poses significant challenges. as the scale and complexity of these models increase, the training overhead for downstream tasks grows exponentially @xcite @xcite . these escalating demands render the current tuning schemes for mllms obsolete and unsustainable, impeding their widespread utility. a promising solution is the utilization of parameter-efficient finetuning (peft) approaches @xcite @xcite @xcite , which have been widely applied and achieved notable success in natural language processing @xcite and computer vision tasks @xcite @xcite . however, most existing peft approaches only focus on single modality tuning while overlooking multimodal instructing learning. a primary challenge is the intricate process of finetuning data of multiple modalities within a cohesive model, extracting and aligning feature representations across modalities. additionally, there is a pressing need to enhance the zero-shot generalization capabilities for unseen multimodal tasks while minimizing training costs. in light of this, we present a novel multimodal prompt tuning (m 2 pt) approach with efficient and effective mllm adaptation for zero-shot instruction learning. m 2 pt demonstrates competitive performance across a wide spectrum of tasks (see fig. 1 ) while tuning 0.09% of overall parameters. specifically, we introduce two sets of soft prompts: visual prompts and textual prompts, which are prepended to the visual and instruction inputs, respectively. the learned embeddings of the visual prompts are projected into the embedding space of the textual prompts. the cross-modality interactions between the two sets of prompts are enforced during instruction tuning, facilitating the alignment and learning of the feature representation between the two modalities. in this way, m 2 pt provides explicit guidance and clear directives through instruction tuning, enabling models to understand context and reduce ambiguity in zero-shot settings. to effectively assess our method, we conduct comprehensive experiments to evaluate its performance. in §4.2, we demonstrate that m 2 pt surpasses several state-of-the-art peft techniques while tuning only 0.09% of the trainable parameters. we further conduct activation analysis to show the effectiveness of the learned prompts during the attention computation. in §5, we provide various ablation analysis on the impact of model components, prompt length, prompt location, and data volumes in detail. furthermore, we conduct case studies to better understand the advantage of m 2 pt and its limitations. we hope this work offers valuable insights into related fields. we summarize the main contributions as follows: ## related work multimodal large language models. mllms @xcite @xcite @xcite ) integrate multimodal information (e.g., audio, image, video), extending beyond the textual semantic information processed by conventional large language models (llms). a general structure of mllms includes three main components @xcite : a pretrained modality encoder to encode multimodality data, a pre-trained llm to reason encoded multimodal data and perform generation tasks, and a connection layer to project modality information into tokens. during the standard full finetuning process @xcite , a substantial amount of weights within all intermediate layers and the pretrained llm are continuously updated. conceptually different, our approach can elegantly fine-tune the model with adjusting a minimum of weights. ## analysis and discussion attention activation pattern analysis. following common practice @xcite , we extract and discuss the activation maps from the attention block of mllms, and investigate the influence of visual and textual prompts in fig. 3 . we randomly select two samples from the mme dataset and visualize their corresponding activation maps of the attention block in the last layers of both visual encoder (fig. 3(a) ) and llm (fig. 3(b) ). to analyze the impact of textual and visual prompts to the frozen components, we categorize them, according to llava's model structure, into textual prompts, system text tokens, visual prompts, image tokens and instructions. several findings can be observed. first, in the llm attention activation map, we observe that the token regions corresponding to textual prompts exhibit elevated activation levels, indicating their significant role in shaping the model's responses. the activation levels of visual prompts within the llm, while comparatively lower, remain notable relative to most other regions. this suggests a secondary yet substantial role in multimodal inference. second, in the activation maps from visual encoder, the activation levels associated with visual prompts are noticeably higher than those of most other tokens, underscoring the critical role of visual prompts in processing visual information during tuning. these observations support that visual prompts effectively interact with the textual prompts, enhancing the alignment between modalities and thereby improving the model's performance on the zero-shot instruction learning. our component analysis study below further strengthens this claim quantitatively. impact of different components. to investigate the impact of different components in m 2 pt across various datasets (i.e., visual prompts, textual prompts, and interaction layer), we conducted component ablation experiments in fig. 4 by removing each component at a time from m 2 pt 10/20 . the results demonstrate that the model performance drops when any of the trainable prompts is removed, which aligns with our expectations. moreover, the model performance also decreases without the trainable interaction layer, indicating the importance of this layer. furthermore, we observe that the importance of different components varies across the tasks. for example, textual prompts play the most important role in cifar-10 and pope, while visual prompts lift the performance most on mme. once again, it is worth noting that combining the multimodal prompts with the interaction layer in m 2 pt leads to the best performance. impact of prompt length. in m 2 pt, prompt length is the only hyperparameter that needs to be tuned. to further analyze the impact of different prompt lengths on model performance, we conduct a comprehensive study on the lengths of visual and textual prompts on the vision-flan dataset to better understand their properties. following common practice @xcite @xcite , we use the grid search to span ## conclusion we introduce m 2 pt, a novel framework in multimodal prompt tuning for zero-shot instruction learning. our framework offers several advantages: i) it introduces visual and textual prompts elegantly into vision encoder and llm, respectively, enabling fast and accurate multimodal adaptation; ii) it synergizes modalities by cross-modality interaction, enjoying coherent integration from multimodal perspectives; and iii) it significantly reduces the number of trainable parameters compared to conventional finetuning methods, while maintaining robust performance across zero-shot tasks. as a whole, we conclude that the outcomes elucidated in this paper impart essential understandings and necessitate further exploration within this realm.