Modular and Parameter-Efficient Multimodal Fusion with Prompting (2022)

Abstract: Recent research has made impressive progress in large-scale multimodal pre-training. In the context of the rapid growth of model size, it is necessary to seek efficient and flexible methods other than finetuning. In this paper, we propose to use prompt vectors to align the modalities. Our method achieves comparable performance to several other multimodal fusion methods in low-resource settings. We further show that our method is modular and parameter-efficient for processing tasks involving two or more data modalities.

## introduction the success of large-scale pretrained language models (plms; @xcite ; @xcite ; @xcite ; @xcite ) and image encoders @xcite has stimulated a surge of pretrained multimodal models @xcite @xcite that align text with data in other modalities. the fast-growing number of parameters in the pretrained models encourages researchers to create more data-and parameter-efficient methods than finetuning @xcite @xcite @xcite . recently, prompting -concatenating manually designed prompt phrases @xcite @xcite or trained embedding vectors @xcite to the text input of plms -has become an important research direction. following this trend, @xcite introduce frozen, successfully extending plms into few-shot learners (i.e., models that perform well with only a handful of data) for multimodal tasks, by pretraining a vision encoder whose outputs are prompts fed to the plm. strongly on low-resource visual question answering through gpt3-style @xcite priming (in-context learning). frozen consists of two components: a vision encoder (ve) (in their case, nf-resnet-50 @xcite ) and an off-theshelf plm like gpt3. when pretraining frozen, the plm takes the image representations extracted by ve as prompts, to generate captions describing the input image. plm parameters are fixed and ve is pretrained from scratch. the success of frozen shows the potential of prompting-based systems for solving multimodal tasks @xcite @xcite . one inherent discrepancy between frozen and prompting for nlp tasks (li and liang, 2021; @xcite is that the prompt vectors in frozen represent part of the input, the image: they are image features extracted by ve. in contrast, prompt vectors in nlp are agnostic to the input texts: they are trainable parameters of the plm embedding layer to be optimized during training. recall that the plm in frozen is fixed when pretraining ve. this implies that ve's trainable parameters serve two quite distinct purposes: (i) ex-tract high quality image representations; (ii) align the image and text representation spaces. we investigate the efficacy of disentangling the functionality of ve. concretely, we fix the parameters of plm and ve, and allocate extra free parameters for learning the alignment between spaces of different modalities when conducting a multimodal task; this is achieved by introducing additional prompt vectors. as a result, ve can dedicate itself to extract high quality image representations. we hypothesize that disentanglement has two benefits. first, higher modularity is achieved compared to frozen because ve is freed from the objective of aligning modalities. higher modularity brings higher flexibility, which is not applicable in systems like frozen: we can easily change the type of ve, e.g., replacing a cnn with a transformer; adding extra modalities like speech data is made possible as well. our architecture meets the desideratum stated by @xcite : it should be possible to modularly add modalities to an existing multimodal system. second, higher parameter efficiency is achieved by fixing the encoders of different modalities during training; the prompt vectors are the only module to be trained for aligning the representation spaces. we present promptfuse, a prompting-based approach extending plms to multimodal tasks in a modular and efficient manner. our contributions: (i) we show that the prompting paradigm of utilizing plms @xcite effectively strengthens plms with the ability of processing data in modalities besides text. with only ≈15k trainable parameters, promptfuse performs comparably to several multimodal fusion methods in low-resource regimes. (ii) we further propose blindprompt, which enforces that the prompt vectors solely focus on task-specific information and is therefore less prone to overfitting. ## related work prompting is a more data-and parameter-efficient method of using pretrained language models (plms; @xcite al. (2019); yang et al. (2019); brown et al. (2020); raffel et al. (2020) ) than finetuning @xcite . concretely, @xcite , @xcite , @xcite , le scao and rush (2021), and @xcite show that prompting outperforms finetuning in many nlp tasks when annotations are limited, i.e., in few-shot learning. li and liang (2021) introduce prefix-tuning, only updating the prompt vectors, keeping the plm fixed. @xcite introduce prompt-tuning -a simple form of prefix-tuning -achieving performance comparable to finetuning when scaling up the number of parameters in plms. as large plms remain unchanged during prefix-and prompt-tuning, high parameter-efficiency is achieved. multimodal pretraining. the success of plms and pretrained image encoders @xcite encourage fast developments of multimodal pretraining, e.g., large-scale neural networks that align texts with data in other modalities like image @xcite @xcite @xcite , video @xcite and speech @xcite . prompting methods for multimodal models were recently devised. @xcite learn continuous prompt vectors rather than natural language descriptions to model visual concepts. @xcite mark image regions as prompts, adapting pretrained vision-language models to downstream tasks. in frozen, for a fixed plm, @xcite pretrain a ve with image captioning where image representations from the ve are used as prompt vectors. the ve in frozen needs to achieve two objectives: extracting high quality image representations and properly aligning image/text spaces. in this work, we show that disentangling the two functionalities -instead of pretraining a ve like frozen, we utilize pretrained ve as feature extractor and train prompt vectors to fuse the modalities -results in a more modular and efficient multimodal system. ## prompting as multimodal fusing we propose to decompose the functionality of ve in frozen into: (i) providing high quality image representations to the plm; (ii) aligning the image and text spaces for a multimodal task. achieving (i) is straightforward -we leverage off-the-shelf pretrained image encoders, e.g., vision transformer (vit; @xcite ). we align the two representation spaces by prompt-tuning @xcite , i.e., by introducing prompt vectors. concretely, we randomly initialize n trainable vectors in the embedding layer of plm. when processing downstream multimodal tasks, we finetune the prompt vectors but fix plm and ve. figure 1 our method promptfuse. having very few trainable parameters, promptfuse is well suited for lowresource regimes. we design a special attention mask for the plm encoder, shown in figure 2 . while the attention of input data remains fully visible, we enforce prompt vectors to only access each other but be blind to the input data. we refer to this variant of promptfuse as blindprompt. blindprompt fuses data in all modalities using the prompt vectors in self-attention layers. this further emphasizes that prompt vectors should be focusing on the alignment between modalities rather than on specifics of the content of a modality. as a result, blindprompt is more robust to spurious statistical cues @xcite . for example, given a picture that dogs run after a man, overfitting systems tend to answer "poodles" in response to the question "what do dogs chase?". we experiment with visual question answering (vqav2; @xcite ), for which un-derstanding both image and language is necessary when answering a question about an image. vqav2 consists of 443,757 samples, categorized into three types: number, yes/no, and other. we simulate low-resource regimes by sampling 128 and 512 shots of training data. we show that promptfuse and blindprompt are less prone to overfitting in low-resource scenarios than baseline methods, in which the model tends to place extra emphasis on samples of the majority answer type yes/no but pays less attention to other. this is because the two answering words of yes/no have much higher frequency in the text corpus than the answers of the open-ended questions, i.e., other. we train the models for two epochs on the full dataset and 100 epochs on the sampled lowresource datasets. for prompting, we set the prompt length n to 20, and appendix §a shows an ablation study. similar to @xcite , we empirically found that a large learning rate leads to better prompting performance. so we use learning rate 5e-1 for prompting; learning rate 5e-4 is used in all other experiments. batch size is 32 and the adam optimizer (kingma and ba, 2015) is used. ## experiments: three modalities disentangling functionality of the modality data encoder, e.g., ve, makes promptfuse and blind-prompt more modular than frozen. applying our methods to tasks involving more than two modalities is straightforward. in contrast, frozen incurs the high cost of pretraining encoders for new modalities. we experiment on the sarcasm detection dataset mustard @xcite with video, audio, and text data. foot_1 setup. to process video, we first use open-face @xcite to sample important frames containing human faces. next, vit is leveraged to extract visual representations from each frame. we then average visual representations of all frames to represent the video. to process audio, we use librosa @xcite to remove background noise and convert audio to waveform with a sampling rate of 16,000 hz. we then use pretrained wav2vec2 @xcite to encode the waveform and apply the same averaging strategy as for video. bart is used as our plm. we use a verbalizer of true/false in this experiment. we adopt the speaker-dependent setup in mus-tard: 334 training and 356 testing samples. we compare promptfuse, blindprompt, and finetune for 8, 32, and 64 shots. note that finetune uses 180m trainable parameters in the vision and audio encoders. we also conduct an experiment training on the full dataset for 5 epochs. the remaining setup is the same as §4.1. ## conclusion we propose promptfuse and blindprompt as methods for aligning different modalities in a modular and parameter-efficient manner. we show that prompting, which requires only a few trainable parameters, performs comparably to several multimodal fusion methods in low-resource scenarios. the high modularity property of prompting supports -by avoiding the need to finetune large pretrained models -flexible addition of modalities at low cost.