M eta VL : Transferring In-Context Learning Ability From Language Models to Vision-Language Models (2023)

Abstract: Large-scale language models have shown the ability to adapt to a new task via conditioning on a few demonstrations (i.e., in-context learning). However, in the vision-language domain, most large-scale pre-trained vision-language (VL) models do not possess the ability to conduct in-context learning. How can we enable in-context learning for VL models? In this paper, we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to the VL domain? Specifically, we first meta-trains a language model to perform in-context learning on NLP tasks (as in MetaICL); then we transfer this model to perform VL tasks by attaching a visual encoder. Our experiments suggest that indeed in-context learning ability can be transferred cross modalities: our model considerably improves the in-context learning capability on VL tasks and can even compensate for the size of the model significantly. On VQA, OK-VQA, and GQA, our method could outperform the baseline model while having ~20 times fewer parameters.

## introduction pre-trained language models have shown impressive performance on a range of tasks by learning from large-scale text corpus @xcite @xcite . recent studies find that some of these language models can be used to perform in-context learning out-of-the-box, i.e., adapting to a task by conditioning on a few demonstrations in context without any gradient update @xcite , which is highly desirable. in vl modeling, in-context learning is less explored and only a handful of models are proposed * equal contribution to perform in-context learning mainly by limiting the amount of deviation of a pretrained largescale language model from the language space and translating visual inputs to language embedding space. they either require a large capacity @xcite or a giant corpus consisting of in-context learning examples @xcite @xcite . in this work, we explore whether we could enable in-context learning in vl tasks without resorting to extreme scale-up. we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to the vl domain? to elaborate, not every language model exhibits excellent in-context learning ability; recent studies @xcite show that one could explicitly train language models to perform in-context learning, by training the model on multiple tasks with in-context few-shot examples, a process that resembles meta-learning. thus, an intriguing query arises: when a language model is first meta-trained to perform in-context learning, can it be transferred to perform in-context learning for vl tasks better? a remarkable observation in our study is the utilization of a meta-trained language model as the transformer encoder-decoder and the mapping of visual features to the language embedding space. this innovative approach led to the development of our proposed vl model (we name it metavl). impressively, our experimental results demonstrate that metavl surpasses the baseline model's performance, even when metavl is designed to be 20 times smaller in size. this study makes three main contributions: 1) to the best of our knowledge, this is the first attempt to transfer the meta-learning knowledge for in-context learning from single-modality to multimodality. 2) we propose a vl model, metavl 1 , which outperforms the baseline in in-context learn-ing while having a much smaller model size. 3) through extensive experiments on vqa, gqa and ok-vqa, we demonstrate the in-context learning capability of metavl and analyze its components. ## related work in-context learning in vl. frozen @xcite is the first attempt for incontext learning in multimodality by leveraging a frozen gpt-like language model as the language backbone and mapping visual features to the language embedding space. frozen sheds light on the feasibility of benefiting from the frozen lms in vl modeling to learn a new task from a few examples in context. magma @xcite is another encoder-decoder architecture for vl pretraining which showed that adding adaptor blocks between the frozen language model layers could further improve the performance for vl tasks in a few-shot scenario. other recent works @xcite @xcite follow the similar principle as the previous works to tackle in-context learning in vl modeling and achieve superior results by leveraging extremely large-scale models. in this paper, we study a problem overlooked in prior work: we delve into the possibility of enabling in-context learning for vl tasks without relying on extensive scalability. our focus lies in exploring the hypothesis: is it feasible to transfer the in-context learning capability from the language domain to the vl domain? meta-learning in language modeling largescale language models have shown the capability to be trained on a new task if properly prompted with in-context examples, i.e., in-context learning. in this learning strategy, the language model is asked to generate the desired output, e.g., an answer in the question-answering task, which is prompted by a few data examples along with their corresponding supervision sampled from the training split, and the language model learns the task in context without performing any gradient updates. although such training is highly data-efficient, its performance is far behind supervised fine-tuning. therefore, inspired by @xcite @xcite , metaicl @xcite proposes training the model for in-context learning as a kind of meta-learning. metaicl meta-trained a gpt language model on a diverse set of natural language figure 1 : the training steps of metavl including metatraining the language encoder-decoder (above) and mapping the visual features into the language embedding space while keeping the meta-trained language encoderdecoder frozen (below). ## approach in this section, we first explain the existing metatraining procedure for language modeling and then introduce our proposed method for in-context learning in vl. meta-training in language modeling. metaicl has shown that a language model that is metatrained on a diverse set of tasks in an in-context learning setup is a strong few-shot learner. to metatrain an auto-regressive language model, in each iteration, a meta-learning task is randomly chosen from a collection of diverse meta-training language tasks, and k + 1 data-label examples are randomly sampled from its training split. then, the model is supervised by the concatenation of (x 1 , y 1 , x 2 , y 2 , ..., x k+1 ) which will be fed as a single input to the model for predicting the label (y k+1 ) as the training objective, i.e., the metatraining step aims to maximize: during inference, the same in-context setup (k examples from the training) are sampled from a target dataset to be used as the (x 1 , y 1 )(x 2 , y 2 ) • ••, (x k , y k )(x) and given to the model to predict the label y. the meta-trained language model trained on a diverse set of natural language datasets has shown good performance for an unseen task when few data are given in context @xcite . metavl -a vl method with meta-learning knowledge for in-context learning. metavl has three main submodels including a meta-trained encoder-decoder and is being trained using prefix language modeling (prefixlm) @xcite . in the following, we discuss each submodel in detail. visual encoder and visual prefix. the visual encoder is defined as a function v e (x) that takes an image of x and outputs visual features. we extract the feature grid before the pooling layer n × d v where n is the number of feature maps and d v is the feature size of the visual encoder. then, the output features can be viewed as a sequence of n visual tokens representing the image. the visual encoder is followed by the visual prefix module that is defined as v p (x) ∈ d v × d l which maps the visual features to language embedding space. this module is seeking to properly project the visual tokens into language tokens. during the vl training, the parameters of both of these modules are trainable and are learned with different learning rates by back-propagation guided by the frozen language model. language encoder-decoder the meta-trained language encoder-decoder is used as the lm backbone and is frozen during the vl training process so the meta-trained language model preserves its few-shot capabilities. the language encoder encodes the text into text tokens represented by t 1 , t 2 , ..., t m . then, given the multimodal tokens (image and text) as @xmath0, v 2 , ..., v n , t 1 , t 2 , ..., t m the decoder is trained to reconstruct the corresponding text with a standard language modeling objective to maximize the following likelihood: after the vl training, for learning a new vl task in-context, given a few examples from a new task with a new format, we concatenate k sampled datalabel pairs from the training split along with one data from the val/test split to construct the prompt and feed it to the model for predicting the desired output. the entire process is visualized in fig. 1 . ## conclusion we investigate the feasibility of transferring metalearning knowledge for in-context learning from resource-rich single modality to multimodality. we have shown that by leveraging a meta-trained language model in a vl model, we can transfer the ability of "learning to learn" in context to vl and it results in a strong vl few-shot leaner. with extensive experiments on three common vl datasets, we have shown that the in-context learning performance of metavl is superior compared with the baseline even when the size of our model is 20 times smaller.