Meta-Learning Makes a Better Multimodal Few-shot Learner
Abstract
Multimodal few-shot learning is challenging due to the large domain gap between vision and language modalities.
As an effort to bridge this gap, we introduce a metalearning approach for multimodal few-shot learning, to leverage its strong ability of accruing knowledge across tasks.
The full model is based on frozen foundation vision and language models to use their already learned capacity.
To translate the visual features into the latent space of the language model, we introduce a lightweight meta-mapper, acting as a meta-learner.
By updating only the parameters of the meta-mapper, our model learns to quickly adapt to unseen samples with only a few gradient updates.
Unlike prior multimodal few-shot learners, which need a hand-engineered task induction, our model is able to induce the task in a completely data-driven manner.
The experiments on recent multimodal few-shot benchmarks demonstrate that our meta-learning approach yields better multimodal few-shot learners while being computationally more efficient compared to its counterparts.
Introduction
Learning quickly from a few observations in a multimodal environment is an integral part of human intelligence [1, 2].
Yet, it is quite challenging for current foundation vision and language models to deal with a limited labeled space, when performing multimodal few-shot learning [3, 4].
The challenges arise from the fact that current vision-only and language-only models are trained separately on different datasets, by optimizing different objectives, which results in inconsistent latent representations.
The Frozen model [3] is the first multimodal few-shot learner, trying to bridge the gap between these models, by taking inspiration from language models [5] to perform in-context learning.
This requires prompting of the language model with a hand-engineered task description, followed by a few demonstrations of the task.
While being a good approach for simpler tasks, like binary decisions, it is not optimal to hand-engineer the prompt each time [6], especially when it comes to more complex multimodal tasks which involve reasoning [3, 4].
Meta-learning or learning to learn [1, 7, 8] comes as a natural solution to any few-shot settings.
Notably, it can be deployed to accrue shared knowledge from related tasks and rapidly learn new tasks by observing only limited labeled data.
While it has been extensively studied in unimodal settings, particularly for few-shot image classification [9, 10, 11, 12, 13], meta-learning remains almost unexplored for multimodal few-shot settings.
We hypothesize that empowering a multimodal few-shot learner with the ability to do meta-learning would assist in building internal representations broadly suitable for many tasks, while inducing the task in a data-driven manner, which could fill in the gap between the different foundation models.
Motivated by this, we define a novel multimodal few-shot meta-learning approach, illustrated on
Figure 1.
Instead of training foundation models from scratch, our architecture, shown in Figure 2, adopts pre-trained vision encoder [14] and language model [5], which are kept entirely frozen during training, in order to not distort their learned parameters [15, 16, 17, 18].
Another major reason for exploring how to smartly re-use frozen models is the huge computational burden that they create during training, and their dependency on large-scale datasets.
The multimodal bridge between the frozen backbones is defined as a light-weight meta-mapper and built entirely from self-attention layers.
By updating only the meta-mapper during meta-training, the model learns to map the visual features into a visual prefix corresponding to the latent space of the language model.
This diminishes the need for fixed task inductions, since the model is able to accrue shared meta-knowledge from related tasks and induce the task for the query samples in a data-driven manner.
To summarize, our contributions are as follows: ( i )
We introduce meta-learning to perform multimodal few-shot learning, which enables fast adaptation and efficient learning of multimodal few-shot tasks.
( ii ) We present a multimodal few-shot meta-learner, which bridges frozen vision encoder with a language model by using a light-weight meta-mapper, aiming to meta-learn a learnable visual prefix.
( iii )
We design a new multimodal meta-learning setting and experimentally demonstrate on existing multimodal few-shot benchmarks, that our model yields a strong performance, while being computationally more efficient.
Methodology
Our objective is to train a model that can learn and quickly adapt to new multimodal tasks with limited labeled data.
First, we define the multimodal meta-learning setting, then we explain our architecture in details, and finally how it used during training and inference time.
2.1 Problem Formulation
Different from standard supervised learning, in meta-learning settings we are dealing with a collection of meta-datasets split into disjoint partitions, namely, meta-training and meta-test.
Both partitions consist of meta-datasets D i , each one containing a pair of separate inner-train set D tri or a support set and inner-test set D tsi , or a query set, meaning { ( D tr 1 , D ts 1 ) , . . .
( D trn , D tsn ) } .
Each pair ( D tri , D tsi ) is referred to as a meta-task T i , following [11].
When considering a k -shot, N -way setting, a single support set D tri consists of k labeled samples for each of the N -ways, where N is the number of object categories.
This means that D tri = { ( x i 1 , y i 1 ) . . .
( x ik , y ik ) } , where x ij represents the image and y ij represents the caption.
The query set D tsi is similarly defined, but has more samples than the support set, as they are needed for the inner-optimization step, as described in the next sections.
2.2 Model Architecture
Vision encoder The vision encoder is defined as a function v ϕ , with fixed parameters ϕ ∈ R d v .
The input is a raw image x
and the outputs are the extracted visual features x 1 , . . .
x n = v ϕ ( x ) .
Meta-mapper To map the visual features x 1 , . . .
x n into the language space, we use a set of l learnable parameters p i ∈ R d e , namely the visual prefix for the language model.
We prepend these learnable parameters to the visual features, and we view it as an ordered set of elements: [ p 1 . . .
p l , x 1 , . . .
x n ] .
Then we employ multi-head self-attention to simultaneously encode the whole set [19], i.e., MetaMap θ ( Q, K, V ) =
σ ( QK T ) ∗ V .
The pairwise dot-product QK T measures the similarity amongst features, and is used as feature weighting computed through an activation function σ .
The output of the meta-mapper are the learned parameters i.e. the visual prefix p ∗ 1 . . .
p ∗ k , meaning that p ∗ 1 . . .
p ∗ l = MetaMap
θ
([ p 1 . . .
p l , x 1 , . . .
x n ]) .
Language model The language model uses an embedding function g ψ to embed each generated token into a word embedding t i , followed by a Transformer decoder defined as a function g ω , to perform the text generation.
During the meta-training, the language model receives the visual prefix p ∗ 1 . . .
p ∗ k concatenated with the token embeddings t 1 , . . .
t m , and outputs the next token conditioned on the prefix: t i +1 =
g ω ([ p ∗ 1 . . .
p ∗ l , t 1 , . . .
t i ]) , i < m , in an autoregressive manner.
2.3 Meta-Training & Inference
First, for simplicity, we assume that our full architecture described in 2.2 is defined as a function f θ , which receives an image x as input and produces y as output.
The loss function, optimized per task during training, is a cross-entropy loss L T i ( f θ ) , defined as:
[Equation]
When adapting to a new task T i , the trainable model parameters θ become task-specific parameters, namely ˆ θ i .
These task-specific parameters are computed with n gradient-step updates, with the following rule for one gradient update:
ˆ θ i
= θ − α ∇ θ L T
i ( f θ ) , which is the inner-loop update and α is the hyperparameter for the step size.
Next, the model meta-parameters θ are optimized for the performance of f ˆ θ i , using the query set D tsi and the task-specific parameters ˆ θ i as initialization of the model:
[Equation] akin to [11] but in a multimodal setting.
The meta-optimization across all tasks T i is performed using stochastic gradient descent update rule, as follows:
θ ← θ − β ∇ θ (cid:80) x j , y j ∼D tsi L T i ( f ˆ θi ) .
During inference, or the meta-test stage in meta-learning parlance, we are given new multimodal few-shot tasks with previously unseen objects.
The support set is used for fast adaptation of the meta-parameters θ to the new task, followed by measuring the performance on the query set. The
[Table] [Table] generation of the answer for each query sample is done in an open-ended autoregressive manner, by using top- k nucleus sampling [20] for sampling words from the language model, conditioned on the context.
To obtain the final performance, we take the average of the accuracy over all query samples from all meta-test tasks.
Experiments
We conduct systematic experiments to evaluate how a meta-learned model performs in multimodal few-shot settings.
In particular, we test the ability of fast adaptation as a main characteristic of meta-learning [11] by quantifying the fast binding of visual concepts and words and visual-question answering with limited examples.
Settings To design a meta-learning setting for multimodal few-shot learning, the datasets have to be structured into sequences of tasks, as explained in 2.1.
In practise, any dataset can be suited for few-shot meta-learning, as long as there is an available object information based on which the tasks can be constructed.
For meta-training, we use the COCO2017 captioning dataset [21] and restructure it to construct tasks in N -way, k -shot manner based on the N object categories present in the support set.
Then, for meta-test we use the four datasets introduced in [3], namely,
Real-Name and Open-Ended miniImageNet; and Real-Fast and Fast-VQA.
This is an example for cross-domain few-shot learning setting.
We also consider in-domain few-shot setting, where the meta-training and meta-test partitions are entirely derived from the mentioned multimodal few-shot datasets [3].
Additionally we experiment with two different training procedures: the proposed episodic meta-learning and a standard mini-batched, non-episodic one.
Implementation Details
The vision encoder is implemented as CLIP [14] with the Vision Transformer (ViT/B-32)
[22] as a backbone model, yielding visual features of size 512.
The language model is implemented as GPT-2 [23], with word embeddings of size 768.
The meta-mapper is initialized following Xavier uniform initialization [24].
All images are normalized and have size of 224 × 224 .
Captions are tokenized with Byte-Level Byte-Pair-Encoding (BPE) [25] tokenizer and zero-padded if needed to obtain a squared batch for training.
We use attention mask, to mask the padded positions in the sequence to avoid attending to the padding tokens.
For the meta-learning specific hyperparameters, we empirically determined to have five gradient-update steps with a learning rate of 0.01. The meta-update is performed with AdamW [26] with a meta-learning rate of 0.001 and 4 tasks in each meta-batch. The model is trained end-to-end on one NVIDIA GTX 1080Ti
GPU, in less than 2 hours, which shows the benefit of the light-weight framework.
The total number of trainable parameters of our model is less than two million, which is much lower than Frozen.
3.1 Results & Discussion
Fast binding of visual concepts to words The experiments on Real-Name and Open-Ended miniImageNet measure to what extent the multimodal meta-learner is able to bind visual concepts to words.
Table 1 shows the 2-way and 5-way accuracy in 1 and 5 shots on both datasets.
From the tables, we observe that our multimodal meta-learner is able to largely outperform Frozen [3], even without using a fixed task induction.
This shows the advantage of having a meta-learned visual prefix, in contrast to just reshaping the vision encoder output as a prefix to language models.
Specifically, the meta-learned prefix is able to collect shared meta-knowledge from related instances in the tasks, which is useful for narrowing down the search space in a learnable manner, instead of using a hand-engineered task induction.
We believe that the open-ended approach is more promising due to its flexibility in reasoning about visual concepts, instead of relying on pre-defined closed set of concepts.
However, due to the magnitudes larger search space in text generation, compared to the one of conventional classifiers, it is still not possible to compete with their performance.
Therefore, we use results from ANIL [12] as a reference upper bound to our approach.
VQA with limited labeled context The aim of the experiments on the Real-Fast and Fast-VQA 2-ways benchmark is to evaluate the abilities of the multimodal meta-learner to answer more complex questions about the objects in the image.
There is an implicit testing of the binding of visual concepts and words, since the query samples are designed in such a way to contain both categories from the support set in the query image, while the question is addressed to one of them.
As we observe from the results in Table 2 with different number of shots, our multimodal meta-learner achieves improvements over Frozen [3], showing once more the benefit of the meta-knowledge and the ability to fast adapt to new tasks.
Qualitative results In Figure 3, we show a examples of query images with the questions and answers at inference time. The capability of the multimodal meta-learner to bind visual concepts to words is apparent: the model is able to connect the visual concepts in the image not only to dalmatian as stated in the ground-truth, but also to the word barking. This observation suggests that the model can leverage visual concepts, not necessarily represented by the ground-truth sentence, which are still accurate and in many cases even better capture the image contents.  Conclusion
We present a novel meta-learning approach for multimodal few-shot learning.
Particularly, we introduce a light-weight meta-mapper which acts as a bridge between frozen vision and language models, and is trained in a meta-learning manner.
The meta-mapper accrues shared meta-knowledge from related tasks into a learnable visual prefix, which is used to induce the task and steer the language model into generating relevant outputs.
Our experiments verify the effectiveness of our multimodal few-shot meta-learner by outperforming the Frozen baseline on several multimodal few-shot benchmarks, without using a hand-engineered task induction. Acknowledgement
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the
Netherlands Ministry of Economic Affairs and Climate Policy.
Appendix A Related Works
Large-scale vision and language models Large-scale language models emerged since the introduction of Transformers [27] and the attention mechanism [28] to successfully deal with long-range dependencies in sequences.
The field has seen significant progress over the last years [29, 23, 5], which also initiated the development of similar strategies for vision [14, 30, 22] and multimodal models [31, 32, 18, 15, 4].
Unlike these approaches, which are trained with specific objectives and tasks, our work focuses on unifying them through meta-learning with a single objective for text generation [33].
Similarly as [3], our approach is modular and can accommodate any separately pre-trained vision encoder and pre-trained language model.
In few-shot scenarios, these large models are steered into producing a desired output by using the idea of prompting [5, 3].
Standard prompting prepends fixed task instructions and a few examples as prompt and then generates the output from the language model.
Instead of optimizing over a fixed set of examples, prefix tuning [6, 34] aims to optimize the instruction as learnable embeddings.
Motivated by this, and by the need to develop an efficient way of bridging large models, we adopt a learnable visual prefix, which in our case is meta-learned.
Meta-learning for few-shot image classification tasks Meta-learning for few-shot learning [10, 35, 11, 9, 36] addresses the fundamental challenge of generalizing across tasks with limited labelled data.
Meta-learning approaches for few-shot learning acquire inductive biases and adopt them for individual tasks in different ways [37].
Existing meta-learning algorithms are typically categorized as follows:
( i ) metric-based, focusing on learning a common embedding space and deriving prototypes as metaknowledge [9, 36, 38] ( ii ) memory-based, using an external memory module as meta-knowledge to quickly adapt to new tasks [39, 40, 41, 42], and ( iii ) optimization-based, aiming to learn a good model initialization across tasks as meta-knowledge, which can be used to efficiently adapt to new tasks [10, 11, 11, 37].
Our approach is positioned in this last category, as it is modality-agnostic and offers greater flexibility.
Multimodal few-shot learning Few-shot learning by combining both vision and language, has only emerged recently with the introduction of the multimodal few-shot learner Frozen [3].
In particular, Frozen is based on the idea of in-context few-shot prompting of language models, meaning that the prompt consists of task induction and interleaved sequences of images and their captions, representing the context.
This in-context few-shot prompting paradigm is considered as one of the possible approaches to deal with few-shot learning scenarios in language models [3, 5, 4].
A recently proposed model [4], is following similar multimodal few-shot learning settings.
However, they are training a vision-language model of 70 billion parameters, which is proven to be successful due to its scale and the amount of pre-training data.
Our goals highly differ since our model aims to handle limited labeled space during training and to adapt to new tasks.
In particular, we define optimization-based meta-learning steps [10, 11, 11, 37], by using the context samples to fine-tune the meta-mapper and evaluate on the query samples.
B Vision encoder details
For the pre-trained vision encoder we adopt CLIP [14], due to its already proven performance and large web-scale multimodal pre-training.
CLIP is considered as a multimodal architecture, as it consists of 1) a vision encoder, which can be a ResNet [43] or a Vision Transformer (ViT) [22] and 2) a text encoder implemented as a Transformer [23].
The pre-training is done in a contrastive manner, on a large dataset of 400 million pairs of image-caption, with the aim is to minimize the distance of same images and captions pairs in the embedding space and to maximize the distance between different image and captions.
In this work, we use the vision encoder stream with a base ViT backbone, comprised of 12 layers, 512-dimensions wide, each one with 12 attention heads.
The size of the input images is 224 ×and are split into image patches, each one with dimensions 32 × 32 , yielding 49 flattened patches and one leading special token.
We keep the backbone entirely frozen and use the special token as a visual encoding, since it holistically represents the image.
Require: p ( T ) : distribution over N-way, k-shot tasks
Require: θ ← random initialization 1: while not done do 2:
Sample a batch of tasks T
i ∼ p ( T ) , 3: for all T i do 4:
D tri , D tsi ← T i 5:
Evaluate ∇ θ L T i ( f θ ) using D tri .
6: for i = 1 to n do ▷ n is number of gradient steps 7:
Compute adapted parameters ˆ θ i with a gradient-descent step ˆ θ i
= θ − α ∇ θ L T
i ( f θ ) .
8: end for 9:
Use adapted parameters ˆ θ i and D tsi for meta-optimization.
10: end for 11:
Update meta-parameters θ across all tasks T i with θ ← θ − β ∇ θ (cid:80) x j , y j ∼T i L T i ( f ˆ θi ) .
12: end while Require: p ( T ) : distribution over N-way, k-shot tasks Require:
θ ← meta-learned parameters in the meta-training stage 1: while not done do 2:
Sample a task T i ∼ p ( T ) , 3: D tri , D tsi ← T i ▷ support set and query accordingly 4: Evaluate ∇ θ L T i ( f θ ) using D tri ▷ L T i ( f θ ) is the cross-entropy loss 5: for i
= 1 to n do ▷ n is number of gradient steps 6:
Compute adapted parameters ˆ θ i with a gradient-descent step ˆ θ i
= θ − α ∇ θ L T
i ( f θ ) .
7: end for 8:
Use adapted parameters ˆ θ i and D tsi for computing the final accuracy 9: end while
C Language model details
The pre-trained language model that we employ is GPT-2 [23], particularly the small version with 117M parameters.
Its architecture is following a Transformer decoder [27] with 12 layers and word embedding dimensions of 768 .
The model is pre-trained on a very large corpus of English data in a self-supervised fashion with standard language modelling objective.
Since the model performs best at what it was pre-trained for, which is generating text from a given prompt in an autoregressive manner, we also employ it in a similar fashion.
In particular, we use the word embedding layer to transform each word token into a continuous word embedding, and the full stack of Transformer decoder layers to parameterize the probability distribution over the vocabulary word tokens.
To obtain the next word token we sample from the probability distribution over the vocabulary with top- k nucleus sampling technique as in [20].
To build a more efficient architecture, the language model is kept entirely frozen, same as the vision stream.
D Multimodal meta-learning details
To design a meta-learning setting for the multimodal few-shot learning, we re-purpose an image captioning dataset, with available meta-data about the object categories, to fit the meta-learning criteria [10].
In particular, we use either COCO2017 captioning dataset [21] to obtain cross-domain experimental setup, or the multimodal few-shot datasets [3] for the standard in-domain meta-learning setup.
The partitioning into meta-training and meta-test tasks is illustrated in Figure 4.
We start by splitting the full dataset into task partitions according to the object categories in the images, in the scope of their own meta-training and meta-test partitions.
The sampling of tasks for both stages is straightforward due to the provided object information and the captions targeted to those objects.
Note that the samples in the query set during meta-training should be at least 15 per category, following [11], since the optimization of the meta-parameters is done based on those samples.
The detailed optimization process in the meta-training stage is described in Algorithm 1.
Similarly, the adaptation stage using the meta-test partitions is described in Algorithm 2.