M u RAG : Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text (2022)

Abstract: While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs. Recently, retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated world knowledge into language generation by leveraging an external non-parametric index and have demonstrated impressive performance with constrained model sizes. However, these methods are restricted to retrieving only textual knowledge, neglecting the ubiquitous amount of knowledge in other modalities like images – much of which contains information not covered by any text. To address this limitation, we propose the first Multimodal Retrieval-Augmented Transformer (MuRAG), which accesses an external non-parametric multimodal memory to augment language generation. MuRAG is pre-trained with a mixture of large-scale image-text and text-only corpora using a joint contrastive and generative loss. We perform experiments on two different datasets that require retrieving and reasoning over both images and text to answer a given query: WebQA, and MultimodalQA. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20% absolute on both datasets and under both distractor and full-wiki settings.

## introduction pre-trained language models like gpt-3 @xcite , palm @xcite , etc have been shown to capture a massive amount of world knowledge implicitly in their parameters. however, using such large models incurs an extremely high computation cost. as an alternative to a singular monolithic transformer, retrievalaugmented architectures like knn-lm @xcite , realm @xcite , rag @xcite , fid @xcite , and retro @xcite have been proposed to decouple world knowledge from the model's parameters. more specifically, these models are trained to access an external memory to enhance the model's predictions. such retrieval-augmented architectures have multiple beneficial properties including: decreased model size @xcite , better attribution/explanation for model predictions @xcite , and adaptability to new information without retraining @xcite . however, previous retrieval-augmented models are limited to memories that contain only text or structured data and hence cannot make use of the massive amount of multimodal knowledge available on the webmuch of which contains information only available in non-text modalities. figure 1 , shows several information-seeking queries that require retrieving and reasoning over visual knowledge. here, a user first poses a question such as "what can be found on the white house balconies at christmas". the system then retrieves relevant items from its memory, for exam-ple, the first image of figure 1 with the caption "white house during christmas", which it uses to produce the answer "wreaths and garlands". existing text retrieval-augmented models would struggle with such queries because, in many cases, they would simply not have access to the answer as some knowledge does not exist in text form. that, coupled with the abundance of multimodal knowledge that exists, leads to the conclusion that retrievalaugmented models should ultimately be developed to retrieve and reason over multiple modalities. in this paper, we are specifically interested in endowing pre-trained language models with a nonparametric multimodal memory containing images, text, or image-text pairs. to accomplish this, we first combine pre-trained t5 @xcite and vit @xcite ) models to build a backbone encoder (figure 3 ), which encodes image-text pairs, image-only, and text-only inputs into a multimodal representation. murag uses the backbone encoder to embed items into an external memory as well as queries to retrieve multimodal knowledge from that memory. these retrievals then augment a language model to generate more visually-grounded outputs. we pre-train murag with a mixture of image-text and text-only datasets including laion @xcite , conceptual-caption @xcite , vqa @xcite and probably-asked-questions (paq) @xcite . more specifically, we reformulate these datasets in a retrieve-and-predict format. here, the model's input is an image along with a text prompt. the model then retrieves from a memory containing captions and passages, which it uses to generate a target token sequence. the model is trained with both a contrastive and a gen-erative loss; this teaches the model to discriminate relevant from irrelevant memory entries, and guides the model to leverage the multimodal knowledge into generation. unlike the pre-training stage, during finetuning figure 2 the model's input is a question, and the memory contains a collection of captioned images and text snippets. we fine-tune murag on the downstream datasets with a contrastive and generative loss similar to pre-training. to avoid excessive computation cost, we develop a two-stage training pipeline to first train with small in-batch memory, and then with a statically encoded and indexed large global memory. our experiments show that murag achieves state-of-the-art performance on two different openmultimodal-qa datasets, both of which require retrieving images and text from a large corpus to answer factoid questions: webqa @xcite and multimodalqa @xcite . on both datasets, we outperform sophisticated baselines @xcite @xcite by 10-20% accuracy under both distractor (from 40+ candidates) and full-wiki settings (from 1m candidates). we also perform a comprehensive study to ablate different components of the pre-training to see their contributions. these empirical results demonstrate the effectiveness of our proposed models to integrate multimodal knowledge into pre-trained generation models and pave the way to unified retrieval-augmented frameworks. multimodal transformers multimodal transformers have demonstrated strong performances in learning cross-modal representation that are generally beneficial on downstream vision and language tasks, such as image-text retrieval (karpathy and fei-fei, 2015), image captioning @xcite , and vqa @xcite . these methods typically learn a joint transformer model on top of unimodal visual and textual backbones, via fusing deep features from each modality. the early version of multimodal transformers @xcite @xcite ) usually learns a transformer on pre-extracted unimodal features for contextualization, which makes it impossible to adjust those unimodal features to the target tasks. recently, simvlm @xcite and coca @xcite proposed end-to-end training for both deep multimodal transformers and unimodal featurization networks and demonstrated strong performance in both multimodal and unimodal downstream tasks. the multimodal memory encoder of murag is broadly similar to simvlm and coca, but has a different focus to encode and retrieve multimodal knowledge (i.e. images and texts) to augment language generation models. ## examples we list more examples in figure 6 and figure 7 . as can be seen, in the first example, the model is grounded on the oracle image-text pair to make the correct prediction. however, in the second example, though the model retrieves the wrong imagetext pair, it is able to make the correct prediction of 'the angel is holding a dead body'. we conjecture that the model utilizes textual clues to make the prediction rather than grounding on the image itself. such shortcut learning is concerning and needs to be addressed through better learning algorithms. figure 6 : examples: we demonstrate model retrieval vs. groundtruth and model answer vs. reference. ## conclusion in this paper, we build the first visually-grounded language generator capable of retrieving multimodal knowledge from a large-scale corpus. our experiments show the promise of this approach, as it outperforms existing baselines by a large margin. at the same time, the performance on knowledgeseeking queries that require reasoning over images is still significantly lower than the performance on queries requiring only text. this indicates that there is still ample room for further improvements and we hope our study can motivate more research on better multimodal retrieval-augmented models.