C ROSS -M ODAL
R ETRIEVAL A UGMENTATION
FOR
M ULTI -M ODAL C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing.
Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA).
First, we train a novel alignment model for embedding images and captions in the same space, which achieves state-of-the-art image-caption retrieval performance w.r.t.
similar methods. Second, we show that retrievalaugmented multi-modal transformers using the trained alignment model signiﬁcantly improve results on VQA over strong baselines. reporting state-of-the-art performance.
We further conduct extensive experiments to establish the promise of this approach, and examine novel applications for inference time such as hotswapping indices.
I NTRODUCTION
Neural networks augmented with non-parametric retrieval components have recently shown impressive results in NLP (Khandelwal et al., 2019; Guu et al., 2020; Lewis et al., 2020; Izacard & Grave, 2020).
In this work, we train a state-of-the-art image-caption alignment model and utilize it in various retrieval-augmented multi-modal transformer architectures, achieving state-of-the-art performance on visual question answering (VQA) signiﬁcant improvement over the baselines, including the winner of the VQA 2.0 2020 challenge.
Retrieval components are promising because they allow for easy revision and expansion of their memory, as compared to their parametric, pre-training counterparts.
They provide more interpretability, as well as direct factual consistency with trusted knowledge sources.
In the multi-modal setting, retrieval augmentation allows for leveraging the strengths of text-based models—as evidenced by the strong performance of BERT-based models in vision-and-language (Lu et al., 2019;
Li et al., 2019b; Kiela et al., 2019)—via cross-modal translation from images to text.
Being able to seamlessly “hot swap” knowledge sources without the need for re-training the model affords a unique scalability not typically seen in the traditional deep learning literature.
Nearest neighbor methods are known to be strong baselines in the vision and language domain (Devlin et al., 2015).
We introduce a simple yet effective novel dense cross-modal alignment architecture called
DXR (Dense X-modal Retriever).
DXR achieves state-of-the-art performance on both COCO (Chen et al., 2015) and Flickr30k (Young et al., 2014) image-caption retrieval, with respect to similar methods.
We subsequently use DXR to augment several multi-modal transformer architectures with a retrieval component.
We show that retrieval augmentation yields impressive results for a variety of well-known multi-modal transformer architectures, ranging from VisualBERT (Li et al., 2019b) and ViLBERT (Lu et al.
, 2019)—which use bounding-box features—to Movie+MCAN (Jiang et al., 2020)—which uses grid features.
We name our overall method XTRA, for X-modal Transformer Retrieval Augmentation.
Speciﬁcally, our contributions are as follows:
• We introduce a novel image-caption retrieval architecture, DXR, that achieves state-of-theart performance on COCO and Flickr30k, with respect to similar methods.
• We introduce a new retrieval-augmented multi-modal transformer architecture, XTRA, that achieves state-of-the-art performance signiﬁcant improvement on VQA over the baselines.
To our knowledge, this is the ﬁrst work to showcase the promise of hybrid parametric and non-parametric models for the vision and language domain.
• We conduct extensive experiments to shed light on this novel approach.
We explore different datasets for training the alignment model, as well as the effect of in-domain versus out-of-domain retrieval indices, the index size and inference time applications.
Our experiments show that our proposed method signiﬁcantly improves over a variety of strong multi-modal baselines, and demonstrates superior results over pre-training.
R ELATED W ORK
Cross-Modal Retrieval Prior work in cross-modal retrieval can be divided into two primary categories: (i) methods that use grid-features and/or vector representations of the embedding space, and (ii) methods that use detection features, sequence representations, or share information between the two modalities for computing the similarity metric.
The ﬁrst category consists of methods such as
RRF (Liu et al., 2017) and DPC (Zheng et al., 2017) which use two convolutional network branches for image and text.
CMPM by Zhang & Lu (2018) introduced a pre-trained image backbone with a
Bi-directional LSTM to learn image and text embeddings.
The most relevant work in this category is VSE++ (Faghri et al., 2017), which focuses on hard negative mining and ranking loss.
The second category generally exploits the use of detection features, which enforces an additional complexity.
Methods such as TERN (Messina et al., 2020b), TERAN (Messina et al., 2020a), SAEM (Wu et al., 2019) and MMCA (Wei et al., 2020), use transformer modules to obtain modality-speciﬁc embeddings.
TERAN, as well as SCAN (Lee et al., 2018), utilize sequence similarities.
SCO (Huang et al., 2018) and VSRN (Li et al., 2019a) learn, in addition to image-text alignment, to generate the caption from the image embedding.
MMCA, as well as CAMP (Wang et al., 2019), fuses image and text information to obtain the ﬁnal embeddings.
Other methods, such as Unicoder-VL (Li et al., 2020a),
Oscar (Li et al., 2020b) and UNITER (Chen et al., 2020) learn to align between image and text by using positive and negative tuples of images and captions.
While these models perform well, they suffer from high computational complexity as we discuss in Sec. 3.
External Knowledge Source Methods
The use of an external knowledge source (KS) has gained much attention in the ﬁeld of natural language processing (NLP), such as the work of Verga et al.
Our work is inspired by that of Lewis et al.
(2020), which introduced RAG, a generic approach for a variety of downstream NLP tasks, that uses a learned retriever (DPR by Karpukhin et al.
to augment the inputs by marginalizing across several retrieved phrases retrieved from
Wikipedia.
In the multi-modal domain, previous efforts have focused on building different types of
KS, such as the work of Zhu et al.
(2014), Chen et al.
(2013), Divvala et al.
(2014), Sadeghi et al.
(2015) and Zhu et al.
(2015), which use web information for the construction of the KS.
Methods that use an external KS for a downstream task use a structured KS, such as the work of Narasimhan et al.
(2018), Narasimhan & Schwing (2018), Wang et al.
Wang et al.
(2018) and Zhu et al.
Zhu et al.
(2017) introduced an iterative method for VQA tasks.
Marino et al.
introduced OK-VQA, a novel VQA dataset that requires the use of an external KS.
Fan et al.
(2020) applied a KS to multi-modal dialogue.
In our work, we focus on a more natural KS, such as images and captions, which better reﬂect the data generated in newspapers and social media.
Multi-modal Classiﬁcation In this work, we investigate the potential advantages of using an external KS for the popular and challenging VQA domain, a multi-modal classiﬁcation task.
Current methods for VQA use pre-training on different datasets in order to gain better performance.
In our experiments, we show performance for common methods such as VisualBERT (Li et al., 2019b), which concatenates the text and image modalities as an input to a pre-trained BERT (Devlin et al., 2018) model.
ViLBERT (Lu et al., 2019) fuses text and image modalities using co-attentional transformer layers.
Other methods such as Pythia (Jiang et al., 2018), VLBERT (Su et al., 2019) and
MMBT (Kiela et al., 2019) can beneﬁt from our method, as well as more recent work such as
Oscar (Li et al., 2020b) and UNITER (Chen et al., 2020), which use the alignment task for pretraining their models.
The currently SOTA In this paper, we choose to show our performance on the two common VisualBERT and ViLBERT models, and the winner of the VQA 2.0 2020 challange,
Movie+MCAN (Jiang et al., 2020) , which uses grid features instead of detection features, a modulated convolutional bottleneck for the image backbone, and MCAN (Yu et al., 2019) for fusion. A [Figure] Ablackandwhitedoglaysnexttoafrisbee
Theblackandwhitedogstandsnearapersonholdingafrisbee
Ablackandwhitedogwithafrisbeebyitsfeet
Ablackandwhitedogsittingnexttoafrisbee Adoglookingatamanholdingafrisbeewithanotherdoglayingdown [Figure] similar method was introduced by Nguyen et al.
Our method is also applicable to methods such as Pythia (Jiang et al., 2018) and MMBT (Kiela et al., 2019).
M ETHOD
Our methodology is composed of two disjoint parts: (i) for a given external knowledge source K , consisting of m modalities, we train a model ( i .
e ., the Retriever) to align between the different modalities.
(ii)
Given a knowledge source K and an alignment model, we train a downstream model ( i .
e ., the Reader) by augmenting its inputs with extra data from K .
3. C ROSS - MODAL A LIGNMENT
Let K be a knowledge source consisting of m modalities, where each sample s i
= ( s
i , . . .
, s mi ) ∈ K is a tuple of m elements, corresponding to different modalities.
Our alignment model encompasses m encoders E m , each composed of a feature-extraction module F m , projection layer P m , shared
Transformer encoding layer T with attention pooling, and a normalization layer N :
[Equation]
From this point, we will consider the two-modality case of images and captions as illustrated in
Fig.
For text and image feature extractors, F 1 and F 2 , we use a pre-trained BERT masked language model Devlin et al.
(2018), and a pre-trained ResNet152 CNN backbone on ImageNet, respectively.
The images are represented with convolutional grid features, chosen for robustness and speed, and these are ﬂattened across the spatial dimension.
The projection layers P m project each modality to a constant dimension d .
The projected sequences are then forwarded to a shared Transformerencoding layer, and aggregated by an attention pooling layer, resulting in a vector representation for each modality.
Finally, we normalize each vector using an L2 normalizer, projecting all embeddings to the unit-sphere.
Following Faghri et al.
(2017), we only normalize the text embeddings because of image-caption imbalance (see Sec. 4.1).
We train our dense cross-modal retriever (DXR) using a contrastive loss, speciﬁcally using an inbatch hinge penalty with hard negatives (Faghri et al., 2017).
Given a batch, consisting of b samples, s 1 . . .
s b , for each sample s i , let s 1 i
and s
i be the positive pairs and s 1 i and s 2 j (cid:54) =
i the negative pairs.
We compute the pair-wise similarity between the two modalities, using a dot product, denoted by π ( s 1 i , s 2 j ) =
(cid:104) s 1 i , s 2 j (cid:105) .
The hard-negative in-batch hinge loss is then deﬁned as:
[Equation]
where s 1 i (cid:48) and s 2 i (cid:48) are the hardest samples inside the batch, and α is the margin constant.
3. I NDEXING AND R ETRIEVING
Following Lewis et al.
(2020), we use FAISS (Johnson et al., 2017) as our indexer platform for fast
KNN queries.
Given a knowledge source K , we construct an index by computing the embeddings [Figure] of each sample in K using alignment model (the Retriever ), which can be trained on any arbitrary knowledge source.
We introduce with two variants: we either construct separate indices
I m K for each of the modalities; or we construct one joint index
I K that encompasses all modalities and where a KNN query will return a mixed modality result.
Fig.
2 illustrates the two independent features of the alignment model and external knowledge source.
The retrieval process then consists of input query q , encoder E m and indexer I K
(or I m K ).
I K takes as an input an embedding query e q =
E m ( q ) and k , and returns the k -nearest indices i 1 . . .
i k , corresponding to the k -nearest embeddings.
We then index data from K , resulting in m retrieval sets r m = ( r m 1 . . .
r mn m ) , one for each modality, each consisting of varying number of samples n m , where mi =1 n m =
k .
When using I m K , a single modality m is returned, resulting in r m =
( r m 1 . . .
r mk ) :
For simplicity, we deﬁne the retriever by R ( q, E m , I K , k ) :
= { r 1 , . . .
r m } .
3. E ND - TO -E ND F USION
Let M be any multi-modal reader model, applied to a speciﬁc downstream task that takes as an input x = ( x 1 , . . .
, x m ) consisting of m modalities and outputs prediction y .
The method augments the input x by concatenating the retrieved samples to their corresponding input modalities, resulting in the augmented input x (cid:48) :
[Equation]
The resulting end-to-end training of model M is then deﬁned by some loss function L , minimizing L ( M ( x (cid:48) ) , y ) , with the same hyperparameters as in the non-retrieval augmented case.
Fig.illustrates the complete model.
3. T IME -C OMPLEXITY
As introduces in Sec. 2, we consider two types of retrievers, (i) methods such as ours, that use Maximum Inner Product Search (MIPS), where each modality is computed independently, and (ii) methods that have entangled computation of similarity between the different modalities, e .
g .they cannot compute an independent embedding.
Assuming a knowledge-source of size N , and a forward-pass with O (1) time-complexity.
In type (i), the embeddings of the entire knowledge source need to be computed only once, as well as the embeddings of the query sample.
In our experiments, we use FAISS with “Hierarchical Navigable Small World” search, and as shown by Johnson et al. (2017), this searching method takes O ( AD [log N ] v ) , where A and v are constants, and D is the degree of the graph. Therefore, the total time complexity of retrieving is O ( AD [log N ] v ) .
On the other hand, methods of type (ii) must compute pairwise similarities between a query sample, and all samples in the dataset, resulting in a O ( N ) run time for searching the most similar sample.
As a results, type (ii) methods are not applicable to our end-to-end fusion pipeline, where for each sample we query, a O ( N ) is applied, which results in an inefﬁcient and non-scalable method.
On the other hand, our method does not impose signiﬁcant overhead.
In this section, we describe the two experimental settings of the alignment model and the end-toend downstream task training and evaluation.
All models and experiments are implemented and performed with the MMF library (Singh et al., 2020a).
4. D ATASETS
We use three common datasets for training and evaluating retrieval and VQA tasks.
Flickr30K (Young et al., 2014) is composed of 30,000 images, with 5 captions each.
Following Karpathy & Fei-Fei (2015), we use 1000 images for validation and 1000 images for testing.
COCO (Chen et al., 2015) is a well-known dataset that contains 120,000 images, with 5 captions each.
We use the splits from Karpathy & Fei-Fei (2015) as well, resulting in 80K images for training, 5K images for validation and 5K images for testing.
Following Faghri et al.
(2017), we add an additional 30K images for training, and uses the same 1K and 5K splits.
Conceptual Captions (Sharma et al., 2018) is a dataset that contains image-caption pairs, composed of 3M samples for training and 100K for validation, which we use to test our retrieval model.
The proposed datasets differ in two major axes (i) Size - The largest knowledge-source we use is
CC, which contains 3M image-caption pairs, while COCO and Flickr30K are smaller in one and two orders of magnitude, respectively.
(ii) Domain gap - As shown in (Singh et al., 2020b), CC datasets differs in both visual and textual domain from the VQA task, while COCO has the best domain match in both.
Flickr30K datasets, on the other hand, is very similar to COCO, but suffers from short number of samples in an order of magnitude compared to COCO.
4. C ROSS -M ODAL R ETRIEVAL
In the cross-modal retrieval task, we deal with two modalities: images and captions.
Bi-directional retrieval is evaluated, denoted as Text
→ Image and Image → Text, where the left-hand-side indicates the query and the other indicates the retrieved domain.
For fair comparison, we only report results for methods that use grid-features and vector representations, as noted in Sec 3.2 and 3.4.
For a full comparison with other previous methods, please see Appendix A. Models are trained for 100K iterations with a warm-up of 2k iterations, batch size of 256, and Adam optimizer with learning-rate of 0 .
0001 where the (pre-trained unimodal) feature encoder’s learning-rate is multiplied by 0 .
The hinge-loss margin hyperparameter m is set to 0 .
4. D OWNSTREAM T ASKS
After training the alignment models for each dataset—Flickr30K, COCO and CC—we build indices for each, as deﬁned in Sec 3.2.
Note that for COCO, we only use the training set for indexing, while for Flickr30K and CC, we use the entire set of train/val/test.
This is done for fair comparison with the
VQA task, which uses the COCO training-set images for training.
Our experiments focus on VQA as the downstream task, however we note that extension to other multi-modal tasks is straightforward.
The inputs of the VQA task are image and text tuples, and it is presented as a classiﬁcation problem over a set of answers.
In VQA, we observe that information regarding the content of the image, such as the amount, color and location of objects is very correlated with the question and answer.
Therefore, captions serve as good auxiliary information, while similar/retrieved images (e.g., to which the question does not directly refer) are less informative.
For that reason, we use the separate indices variant, retrieving text captions from images to yield a cross-modal image to text translation.
We experiment with all three datasets, evaluating different training and inference variants.
R ESULTS 5. C ROSS -M ODAL R ETRIEVAL
Tab. 1 and 2 show retrieval results on Flickr30K and COCO, comparing similar methods that use grid-features and vector representations for the embedding space.
Reported numbers correspond to
Recall-at-1/5/10 on the test-sets.
As can be seen, our method signiﬁcantly outperforms previous [Table] [Table]
Additional methods can be found in Appendix A.
[Table]
work in both datasets.
We refer to Appendix A for a comparison with additional methods.
While
CC is not commonly used in the retrieval literature, we use it for our downstream task.
Using DXR, we obtain the following results for CC: R@1: 25 .
1 R@5: 50 . 1 and R@10: 61 . 9 for Text → Image, and R@1: 25 .
4 R@5: 50 . 9 and R@10: 61 . 8 for Image → Text.
The alignment model trained on CC is used for training in the downstream VQA task.
We notice that performance degrades as the dataset size increases, which could affect the downstream task since we query from the entire dataset.
5. V ISUAL Q UESTION A NSWERING
Our main results show performance on the VQA val-set , experimenting with three common
VQA methods: VisualBERT (Li et al., 2019b), ViLBERT (Lu et al., 2019)
[Table]
and the currently stateof-the-art winner of the VQA 2.0 challenge, Movie+MCAN (Nguyen et al., 2020), each along with three different knowledge sources (COCO, CC and Flickr30K).
Following Jiang et al.
(2020), we use the val-set to assist in our exhaustive ablation studies, however we report our ﬁnal SOTA result on the VQA test-std split.
Tab. 3 summarizes four different training settings: (i) vanilla [Figure]
[Figure]
We see that using the ﬁve ground truth (GT) COCO captions as additional data (bottom row of
Tab. 3), sets a soft upper bound for our approach.
On one hand, the GT captions contain relevant information about the content of the image; on the other hand, other captions from the knowledge source may additionally serve as rich, useful descriptions.
We also see that our method signiﬁcantly increases performance across all baselines, even with respect to pre-training.
This suggests that our method serves as a good alternative for pre-training.
Our best model, Movie+MCAN+XTRA-10C, outperform Movie+MCAN on VQA2.0 test-std , Our best model sets a new state-of-the-art on VQA2.0 test-std , using Movie+MCAN+XTRA-10C, obtaining a score of 73.12 (with single model performance).
5. H OT S WAP
Our method is devised such that querying and retrieving from the knowledge source is independent of the downstream model, enabling the swap of the alignment model and/or knowledge source during inference.
This affords interesting explorations.
We describe two forms of “hot swapping”: (i) the entire knowledge source with its trained alignment model are replaced with a new one and corresponding alignment model – we refer to this as “out-of-domain”; (ii) the knowledge source used for retrieving is swapped, but the alignment model remains the same as was originally trained with the downstream model.
In this case, we build a new retriever for the new knowledge source, using the original alignment model – we call this “in-domain”.
“in/out-of-domain” refers to the alignment domain with which the downstream model was trained.
Fig.
3 illustrates the two cases.
In Fig.
4 we show different inference results for hot swapping.
All models in this experiment are trained using 10 retrieved cations.
The title of each graph represents the trained model, followed by the trained knowledge source and the knowledge source to which we swap.
In addition, we show inference results for training with the swapped knowledge source, e .
g .
training with CC knowledge source and alignment model from scratch, using 10 retrievals.
As can be seen, “in-domain” hot [Figure] [Figure] swapping performance is signiﬁcantly higher than “out-of-domain”.
We hypothesize that the reader model has learned an implicit structure of the alignment space.
Surprisingly, when training with
COCO as the knowledge source, “in-domain” hot swapping performs similarly, for the same amount of trained retrievals (10), as training with an alternative knowledge source and alignment model.
On the other hand, we observe that this suffers from a decrease in generalization due to different amounts of retrieval during inference-time.
In the other direction, hot swapping to COCO from
CC or Flickr30K does not result in the same performance as training with COCO as the knowledge source and alignment model, yet, performance and generalization do not degrade.
Qualitative results of “in-domain” hot swapping are presented in Fig 5.
As can be observed, novel information such as the fact that the image shows a “cobblestone street” is retrieved from CC without having to train the alignment model on that source.
5. A BLATION S TUDY
In this study, we explore the use of different amounts of retrieval during training and inference, as well as doing inference without retrieving - which we name unplugged .
We further explore the relationship between pre-training and XTRA.
Number of Retrievals We experiment with different amounts of retrieved captions during training and inference.
In Fig 6 (a), we show the performance of our method when training with different amounts of retrieval, and different knowledge sources.
As can be observed, training with 10 captions and COCO as the knowledge source results in the best performance.
In Fig 6 (b), we show the inference performance for models trained using 10 retrievals.
In addition, we show the inference performance of the same model, trained with random amounts of retrieval, between 1 and 20, on the
COCO dataset (COCO 20R-C).
With this, the best performance is given when we inference with the same amount of trained retrievals, and this then degrades as the number of retrievals differ from how the model was trained.
We also see that training with varying number of retrievals achieves better generalization to different amounts of retrievals during inference, as can be seen in Fig 6 (b), COCO 20R-C, where performance is maintained up to 60 retrievals during inference.
Unplugged Performance One interesting observation we make is the ability to ”unplug” the knowledge source by not retrieving during inference-time.
Tab.
4 shows a noticeable decrease in performance, indicating the dependency of the reader on the retrieved data during training.
When training with COCO as the knowledge source, introducing captions that are very related to the input images is biasing the model to depend on the retrieved captions.
For CC and Flickr30K, the domain [Table] gap between the downstream task and the knowledge source lessens this gap in unplugged performance.
Surprisingly, while ViLBERT performance is generally better than Visual BERT, using our method, the opposite is true when unplugging the knowledge source.
The use of a retrieval mechanism over external knowledge sources raises inriguing questions, e.g.: 1) Is augmentation better than pre-training?; and 2) Can pre-training help the external knowledge source?
To address these questions, we experimented with two different pre-training datasets commonly used for VQA - COCO and CC. Tab.suggests that for the COCO dataset, using our method is signiﬁcantly better than pre-training alone, while using pre-training followed by XTRA causes the performance to vary with respect to the reader architecture (e.g., pre-training helps XTRA with ViLBERT, but not with VisualBERT).
Tab. 3 also shows that ﬁne-tuning our method after pre-training on the same knowledge source yields better performance over pre-training across all knowledge sources and architectures.
C ONCLUSION
In this work, we presented a novel approach that proposes the use of external knowledge sources in multi-modal prediction models with transformer architectures.
We trained a state-of-the-art alignment model, DXR, for performing retrieval over external knowledge sources.
We showed that our method XTRA yields gains in performance when using an in-domain knowledge source on VQA.
We conducted a variety of experiments to show the sensitivity and effects of the used knowledge source with various choices in hyperparameters which shed further light on the different aspects of the model.
Future research and applications of our method include interpretability of retrieved data and predictions for veriﬁcation processes, the demonstration of increased information security by hot-swapping, and unplugged versions of models and new architectures that take advantage of out-of-domain knowledge source.
We hope that our approach inspires further work in the direction of hybrid parametric non-parametric models for multi-modal problems
A R ETRIEVAL
Tab. 6 shows a complete comparison of the different alignment methods in the cross-modal alignment literature.
The top part corresponds to methods which use vector representations, grid-features, and do not share information between the modality branches.
The bottom part shows the rest of the methods.