Prompt Tuning for Unified Multimodal Pretrained Models (2023)

Abstract: Prompt tuning has become a new paradigm for model tuning and it has demonstrated success in natural language pretraining and even vision pretraining. The parameter-efficient prompt tuning methods that optimize soft embeddings while keeping the pretrained model frozen demonstrate advantages in low computation costs and almost lossless performance. In this work, we explore the transfer of prompt tuning to multimodal pretrained models. Specifically, we implement prompt tuning to a unified sequence-to-sequence pretrained model by adding a sequence of learnable embeddings to each layer and finetuning the pretrained model on downstream task with only the learnable embeddings being optimized. Experimental results on a series of multimodal understanding and generation tasks demonstrate that our method OFA-PT can achieve comparable performance with finetuning across a series of multimodal generation and understanding tasks. Additionally, it significantly outperforms the unified multimodal pretrained model with other parameter-efficient tuning methods, e.g., Adapter, BitFit. etc. Besides, in comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks. We further figure out that experimental factors, including prompt length, prompt depth, and reparameteratization, have great impacts on the model performance, and thus we empirically provide a recommendation for the setups of prompt tuning.

## introduction recent years have witnessed the great success of large-scale pretraining based on large models and big data in natural language processing (nlp) @xcite @xcite @xcite and computer vision @xcite @xcite . inspired by the success of bertlike models @xcite , researchers have found that pretraining can level up the downstream performance of cross-modal representation learning algorithms by a large margin @xcite @xcite @xcite . following this line of research, unified multimodal pretrained models have gradually attracted much attention, and very recently, a series of such models based on the sequence-to-sequence learning framework have unified both cross-modal understanding and generation tasks and even achieved state-of-the-art performance @xcite @xcite @xcite . furthermore, note that the scale of unified multimodal pretrained models has been growing rapidly, showing a similar trend of developments in large language models @xcite @xcite . despite the great success of large-scale pretrained models across multiple domains, training such models requires a large amount of computation costs. the conventional finetuning is though effective in gaining high performance yet suffers from low training efficiency, especially when the pretrained model is of large scale in model size. there is a strong necessity for parameter-efficient transfer learning methods in the applications of large-scale foundation models. the most popular method in this field is prompt tuning @xcite , which demonstrates success in natural language processing @xcite @xcite @xcite and computer vision @xcite @xcite . in comparison with finetuning, prompt tuning only tunes pretrained models by a trivial amount of parameters (e.g., 1%). prompt tuning freezes most parameters of the pretrained model and only tunes several prompt embeddings, as well as the output layer if necessary. recent advances have shown that prompt tuning can help pretrained models achieve comparable performance with finetuning across different downstream tasks, including natural language understanding and generation, image classification, etc. however, the studies on the parameter-efficient transfer methods for multimodal pretrained models, especially the unified multimodal pretrained models, are still scarce. furthermore, along with the trend of model scaling in unified multimodal pretrained models, how to tune such models cost-effectively should be a significant topic of research in multimodal pretraining. this work fills in the void and takes the lead to explore prompt tuning for the unified multimodal pretrained models. we propose ofa-pt, an implementation of prompt tuning based on the recently open-sourced unified multimodal pretrained model ofa @xcite . to be more specific, in the stage of downstream transfer, we insert a sequence of learnable embeddings to each layer of the encoder and decoder, and only tune those embeddings while keeping the parameters of the pretrained model frozen. for the rest of the setups, we use the same finetuning procedures, which transform data to the format for sequence-to-sequence learning and train the model with maximum likelihood estimation for optimization. in comparison with finetuning, the number of tunable parameters (~1% of the total) for prompt tuning is much smaller than that of finetuning, leading to fewer computation costs, e.g., memory. through extensive experiments we observe that the parameter-efficient prompt tuning is able to help the pretrained model achieve comparable performance with finetuning across 4 multimodal downstream tasks, spanning from understanding to generation. to analyze the differences between finetuning and prompt tuning, we follow the assumption that prompt tuning with most parameters in the pretrained model frozen should induce model robustness. we experiment on the tuning methods with adversarial attack and observe phenomena consistent with the hypothesis. to take a step further, this study delves into the implementation details and investigate whether experimental factors, e.g., the prompt length, prompt depth, and reparameterization, could saliently influence the final downstream performance. we find that in general a longer prompt length (longer than 20 tokens) is a preferable choice, and our experiments show that 64 should be favored in most cases as a longer prompt sequence will not only increase the computation costs but also incur performance degradation. also, we show that reparameterizaton with additional trainable parameters cannot introduce significant improvements in downstream performance. ## method this section introduces the details of our proposed method. it provides the detailed implementation of prompt tuning on a unified multimodal pretrained model. the overall framework is illustrated in figure 1 . ## experiments to validate the effectiveness of prompt tuning for multimodal pretrained models, we conduct experiments on 5 cross-modal tasks. specifically, we experiment on cross-modal generation tasks, including referring expression comprehension and image captioning, and cross-modal understanding tasks, including visual entailment, image captioning, and visual question answering (vqa). we use the commonly used base-size and large-size models for the experiments, whose sizes are around 180m and 470m respectively. we provide more details about the experimental setups in the appendix a.1. ## related work in this section, we include the review of multimodal pretraining as well as prompt tuning. ## conclusion in this work, we explore prompt tuning for unified multimodal pretrained models. specifically, we propose ofa-pt, which is an implementation of prefix tuning, a simple but effective prompt tuning method, on the recently open-sourced sota model ofa. through extensive experiments, we demonstrate that the unfiied multimodal pretrained model with the parameter-efficient prompt tuning can achieve comparable performance with the finetuned model, but with fewer parameters to tune (e.g., 1%), and prompt tuning can surpass other light-weight tuning methods, e.g., adapter and bit-fit. through our analysis, we figure out a significant advantage of prompt tuning about its robustness against adversarial attack. furthermore, we provide a comprehensive analysis about the influence of prompt tuning setups, including the prompt length, prompt depth, and reparameterization. potentially prompt tuning can be an alternative to finetuning, but still, there are some salient limitations in this method, e.g., slow convergence and training instabilities. we hope that future studies in this field can alleviate the aforementioned problems and thus promote the application of prompt tuning.