MAP : Low-data Regime Multimodal Learning with Adapter-based Pre-training and Prompting (2023)

Abstract: Pretrained vision-language (VL) models have shown impressive results on various multi-modal downstream tasks recently. Many of the benchmark models build on pretrained causal language models (LMs), leveraging the original few-shot learning and generalization capability of the LMs trained with large text corpora. However, these models are often gigantic and require large-scale image and text data with high computational cost to train. This paper introduces a moderate-size model called MAP for efficient VL transfer learning through adapter-based pretraining and prompting. We aim to answer the question of how much we can complete through VL pretraining within the low-data regime while maximizing efficiency in transferring knowledge of a moderate-size frozen LM. Our experiments demonstrate that MAP achieves substantially better zero-shot and few-shot performance on downstream VL tasks with only 10% the size of pretraining data and a 30x lighter pretrained LM backbone compared to Frozen. MAP also outperforms fully trained models of comparable size at retaining its transfer learning ability when the amount of training data reduces.

## introduction recent vision-language models commonly leverage pre-trained language models (lms) on various multimodal tasks. it is crucial for them to retain the original generation capability of the lms while efficiently incorporating the knowledge from new modalities. a line of work has shown impressive generalization and transfer ability of visionlanguage models that build on large causal decoderonly lms @xcite @xcite . while powerful, these gpt-style pretrained lms request high computing machines for deployment. compared to decoder-only lms, @xcite shows that encoder-decoder model introduces an inductive bias that decouples multimodal feature encoding from generation, yielding improved performance on downstream tasks. recent research by @xcite also demonstrates that vl models pretrained with a moderate-size encoder-decoder lm backbone can be strong fewshot learners. however, in these approaches, the parameters of the language model were entirely updated while learning vision inputs @xcite @xcite , or involving more task-specific data during multitask pretraining or fine-tuning @xcite . ## problem statement despite that larger models are significantly more powerful following the scaling laws @xcite , we aim to answer the following key questions: i) how can we maximize the efficiency in transferring knowledge of a moderate-size frozen lm to a multimodal setting? ii) how much can we achieve on few-shot learning if we limit the size of data and trainable parameters in vl pretraining? ## method this section describes map in details. our approach is to maximize the knowledge transfer of a moderate-size lm to vl learning through adapterbased pretraining and prompting. ## evaluation and results to answer the questions that we raised in section 3, we evaluate map on the aforementioned five downstream tasks in zero-shot and few-shot settings. from our preliminary experiments, jointly updating the nfresnet vision encoder and adapter layers performs slightly better in pretraining than updating adapters only. we therefore applied such settings in all models pretrained using coco combined with three versions of vg regioncaption pairs, denoting as map tiny , map small and map base . transfer efficiency to evaluate map's efficiency on transferring knowledge from a frozen lm to a multimodal setting, we compare map against frozen on vqav2 and ok-vqa. as shown in figure 3 , overall, map achieves better zero-shot and few-shot performance on both tasks. map tiny is able to outperform frozen on zero-shot vqav2, zero-shot and four-shot ok-vqa even with only 10% the size of pretraining data (0.3m v.s. 3.0m) and a 30× lighter lm backbone (224m v.s. 7b). ## conclusion and future work we present an end-to-end moderate-size vl model, which surpasses frozen and comparable-size fully trained baselines on few-shot learning over multiple image understanding tasks, while requiring much less training data and fewer parameters during pretraining. we expect to investigate the core transfer ability of pretrained vl models from a perspective beyond scaling. we propose open questions of whether data can be overloaded in pretraining and how can we use pretraining data more efficiently and wisely.