Cross-Modal Retrieval Augmentation for Multi-Modal Classification (2021)

Abstract: Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r.t. similar methods. Second, we show that retrieval-augmented multi-modal transformers using the trained alignment model improve results on VQA over strong baselines. We further conduct extensive experiments to establish the promise of this approach, and examine novel applications for inference time such as hot-swapping indices.

## introduction neural networks augmented with non-parametric retrieval components have recently shown impressive results in nlp @xcite @xcite . in this work, we introduce a novel imagecaption alignment model architecture and utilize it in various retrieval-augmented multi-modal transformer models, achieving substantial improvements over strong baselines. retrieval components are promising because they allow for easy revision and expansion of their memory, as compared to their parametric counterparts. they provide more interpretability, as well as better factual consistency with trusted knowledge sources @xcite . in the multimodal setting, retrieval augmentation allows for leveraging the strengths of text-based models-as evidenced by the strong performance of bertbased models in vision-and-language @xcite @xcite -via cross-modal translation from images to text. being able to seamlessly "hot swap" knowledge sources without the need for re-training the model affords a unique scalability not typically seen in the traditional deep learning literature. nearest neighbor methods are known to be strong baselines in the vision and language domain @xcite . our contributions are as follows. we introduce a simple, yet effective, novel cross-modal alignment architecture called dxr (dense x-modal retriever). dxr achieves a substantial increase in performance on both coco @xcite and flickr30k @xcite image-caption retrieval, with respect to similar methods. we subsequently use dxr as a retrieval component augmenting several multi-modal transformer architectures. we show that retrieval augmentation yields impressive results irrespective of the exact input strategy, with good performs on vqa for retrieval-augmented versions of well-known multi-modal transformer architectures, from vi-sualbert @xcite and vilbert @xcite -which use bounding-box featuresto movie+mcan @xcite )-which uses grid features. we name our overall method xtra, for x-modal transformer retrieval augmentation. we conduct extensive experiments on various datasets to shed light on xtra's performance and explore the effect of in-domain versus out-of-domain retrieval, index size and inference time applications. our experiments show that xtra outperforms parametric-only pre-training techniques that have access to the same data. to our knowledge, this is the first work to showcase the promise of hybrid parametric and non-parametric models for the vision and language domain. the first category consists of methods such as rrf @xcite and dpc @xcite which use two network branches, for image and text. cmpm @xcite introduced a bi-directional lstm to learn image and text embeddings. the most relevant work in this category is vse++ @xcite , which focuses on hard negative mining and a ranking loss. recently, two methods that use substantial amounts of data were proposed, clip @xcite which uses 0.4 billion image-text pairs, and align @xcite which uses 1.8 billion noisy imagetext pairs. both methods use a dual encoder that produced and embedding vector for each modality. for fair comparison and reproducibility, we train on and compare against methods that use the open source coco and flickr30k datasets. the second category generally exploits the use of detection features, which enforces an additional complexity. methods such as tern @xcite , teran @xcite , saem @xcite and mmca @xcite , use transformer modules to obtain modality-specific embeddings. teran, as well as @xcite , utilize sequence similarities. sco @xcite and vsrn @xcite learn, in addition to image-text alignment, to generate the caption from the image embedding. mmca, as well as camp @xcite , fuses image and text information to obtain the final embeddings. visualsparta @xcite uses fragment-level interaction to compute similarity scores. other methods, such as unicoder-vl @xcite , oscar @xcite and uniter @xcite are trained for multimodal alignment as a pre-training task. while these models perform well, they suffer from high computational complexity as we discuss in sec. 3.4. ## method our methodology is composed of two disjoint parts: (i) for a given external knowledge source k, consisting of m modalities, we train a model (the retriever) to align between the different modalities. ## experiments in this section, we describe the two experimental settings of the alignment model and the end-toend downstream task training and evaluation. all models and experiments are implemented and performed with the mmf library @xcite .