MAPL : Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2023)

Abstract: Large pre-trained models have proved to be remarkable zero- and (prompt-based) few-shot learners in unimodal vision and language tasks. We propose MAPL, a simple and parameter-efficient method that reuses frozen pre-trained unimodal models and leverages their strong generalization capabilities in multimodal vision-language (VL) settings. MAPL learns a lightweight mapping between the representation spaces of unimodal models using aligned image-text data, and can generalize to unseen VL tasks from just a few in-context examples. The small number of trainable parameters makes MAPL effective at low-data and in-domain learning. Moreover, MAPL’s modularity enables easy extension to other pre-trained models. Extensive experiments on several visual question answering and image captioning benchmarks show that MAPL achieves superior or competitive performance compared to similar methods while training orders of magnitude fewer parameters. MAPL can be trained in just a few hours using modest computational resources and public datasets. We release our code and pre-trained model weights at https://github.com/oscmansan/mapl .

## introduction over the past few years, natural language processing and computer vision have witnessed impressive progress in learning models capable of transferring to unseen tasks or benchmarks @xcite @xcite . recently referred to as foundation models @xcite , these can be adapted to a wide range of unimodal vision and language tasks without any additional training. in this work, we study reusing such powerful unimodal foundation models for multimodal visionlanguage (vl) downstream tasks. in particular, we propose to connect a vision encoder, such as * denotes equal contribution. clip @xcite , to an autoregressive language model (lm), such as gpt @xcite @xcite , with minimal additional training on multimodal data. our goal is to obtain a single vl model that can leverage the in-context learning abilities @xcite of the pre-trained lm to generalize to unseen vl tasks from just a few examples. one challenge in connecting vision encoders with lms is aligning the visual and textual representation spaces. recent works have approached this by adapting the lm to visual representations, either by fine-tuning the entire lm @xcite or training adapter layers @xcite . these systems are computationally expensive to train as they have a large number of learnable parameters (hundreds of millions to a few billions) and use large-scale multimodal training data. on the other hand, frozen @xcite keeps the lm frozen, thus learning ∼10× less parameters than the above methods. however, it requires training a visual encoder from scratch, which is also computationally expensive. differently, we aim to reuse large pre-trained unimodal models while keeping them completely frozen and free of adapter layers. we present mapl (multimodal adaptation of pre-trained vision and language models), a simple and parameter-efficient vl model capable of tackling unseen vl tasks. mapl learns a lightweight mapping between the representation spaces of pretrained unimodal models. mapl has orders of magnitude fewer parameters than previous methods @xcite and can be trained in just a few hours. moreover, mapl's modularity makes it general-purpose and easily extensible to newer and/or better pre-trained models. we evaluate mapl on various image captioning and visual question answering (vqa) benchmarks and compare with frozen @xcite in a controlled setup. mapl significantly outperforms frozen and achieves competitive performance compared to other methods @xcite trained on comparably sized data. we further investigate the parameter efficiency of mapl by training on only 1% of multimodal data (thousands of examples); we call this setting low-data learning. we also study in-domain learning: training on image-text pairs from the same domain as the downstream task domains. we train mapl directly on 100% and 1% of in-domain data for each downstream task, without first pretraining on large-scale domain-agnostic data. thus, we train specialized versions of mapl for each downstream domain. such low-data and in-domain learning are particularly useful when it is difficult to pre-train on large-scale domain-agnostic data. we found mapl to be more effective than frozen trained under the same settings. to summarize, our contributions are: 1) we introduce mapl, a parameter-efficient method capable of tackling unseen vl tasks, which can be trained using only modest computational resources and public datasets; 2) we conduct extensive experiments spanning various image captioning and vqa benchmarks, demonstrating mapl achieves superior or competitive performance compared to similar methods while training orders of magnitude fewer parameters; and 3) we further investigate the parameter-efficiency of mapl in two settings: low-data and in-domain. our experiments show that mapl is more effective than the considered methods in both settings. ## related work fine-tuning based vl methods. a popular family of vl methods are based on the pre-training + fine-tuning paradigm. these methods are either encoder-only @xcite @xcite @xcite or encoder-decoder methods @xcite @xcite and use transformer-based architectures. these transformers are first pre-trained on domain-agnostic image-text pairs (e.g., conceptual captions @xcite ) using selfsupervised objectives, and then fine-tuned for each downstream task (e.g., vqa, image captioning). more recent models that are designed specifically for the task of image captioning use large pretrained lms (e.g., gpt-2 @xcite ) and fine-tune these models with image-caption pairs @xcite @xcite . while all these approaches yield state-of-the-art performance for the tasks they are fine-tuned on, the learned model weights are highly specialized for a single task and cannot transfer to new tasks with zero or few examples. differently, mapl reuses the same set of weights for all downstream tasks without any additional training. few-shot learning based vl methods. most similar to mapl are methods that tackle unseen vl tasks in a zero/few-shot manner, by leveraging the in-context learning abilities of large pre-trained lms (e.g., @xcite . these methods connect a vision encoder with a pre-trained lm to tackle vl tasks. some methods @xcite achieve this connection by finetuning the entire lm on image-text data, while others only train adapter layers inserted into the lm @xcite . the vision encoder is pre-trained and kept frozen in both cases. concurrent work flamingo @xcite pushes this idea even further by scaling up the amount of training data and the lm size. while inserting adapter layers requires training fewer parameters compared to fine-tuning the entire lm, the number of trainable parameters is @xmath0; in contrast, mapl only has 3.4m trainable parameters. additionally, inserting adapter layers is not straightforward since it requires modifying the computational graph of the lm; mapl only adds an external mapping network, which is easier to incorporate on top of pre-trained models. on the other hand, frozen @xcite keeps the pretrained lm frozen and instead trains a vision encoder from scratch. this approach does not scale well with larger vision encoders (sec. 4.5). mapl keeps both the vision encoder and the lm frozen (thus further reducing the number of trainable parameters) and only learns a lightweight mapping network to connect both frozen models. similar to mapl, concurrent work limber @xcite also proposes to connect a frozen vision encoder with a frozen lm but using a linear mapping, which is not as parameter-and compute-efficient as mapl (sec. 4.5). mapping networks. mapl trains a mapping network to align the visual and textual representations of the visual encoder and the lm, respectively. the architecture of our mapping network has some similarities with that in clipcap @xcite and the perceiver resampler in flamingo (alayrac lm embedder vision encoder mapping network lm self-attention lm tokenizer a dog catching a frisbee. a dog catching a frisbee . a dog catching a frisbee . pre-trained and frozen no parameters trained from scratch 3 method mapl is a vision-language (vl) multimodal model capable of generating text from a combination of visual and textual inputs. our model builds on top of pre-trained vision-only and language-only models and leverages their strong generalization capabilities (e.g., zero-shot transfer, in-context learning) to tackle unseen vl tasks. mapl is agnostic to the choice of these pre-trained unimodal models as long as they show such capabilities (sec. 4.5). concretely, mapl maps the image representations from a vision encoder's output embedding space to a lm's token embedding space, so that the lm can be conditioned both on visual and textual information. to this end, we train a mapping network with an image captioning objective (sec. 3.1, 3.2), while keeping the weights of the vision encoder and the lm frozen. once the mapping network is trained, mapl can be prompted with a few examples of unseen vl tasks and predict the response via text generation (sec. 3.3). the overall model architecture is depicted in figure 1 . ## conclusion we introduce mapl, a simple and parameterefficient method to repurpose pre-trained and frozen unimodal models for multimodal tasks. our experiments demonstrate that mapl achieves superior or competitive performance compared to similar methods on several vl benchmarks while training orders of magnitude fewer parameters. importantly, we also show that mapl is effective in the low-data and in-domain settings thanks to its reduced number of trainable parameters. we leave as future work exploring training on a weighted mixture of image-text datasets, evaluating on more downstream tasks such as nlvr2 @xcite and visual dialog @xcite , and investigating the use of masked lms @xcite with mapl.