End-to-end Knowledge Retrieval with Multi-modal Queries (2023)

Abstract: We investigate knowledge retrieval with multi-modal queries, i.e. queries containing information split across image and text inputs, a challenging task that differs from previous work on cross-modal retrieval. We curate a new dataset called ReMuQ for benchmarking progress on this task. ReMuQ requires a system to retrieve knowledge from a large corpus by integrating contents from both text and image queries. We introduce a retriever model “ReViz” that can directly process input text and images to retrieve relevant knowledge in an end-to-end fashion without being dependent on intermediate modules such as object detectors or caption generators. We introduce a new pretraining task that is effective for learning knowledge retrieval with multimodal queries and also improves performance on downstream tasks. We demonstrate superior performance in retrieval on two datasets (ReMuQ and OK-VQA) under zero-shot settings as well as further improvements when finetuned on these datasets.

## introduction humans recall, retrieve, and communicate information using many indirect hints and cues. for instance, if we want to explain the concept of a "leopard" but have forgotten the name, we can relate the concept to a picture of a tiger and say "it is an animal that looks like this, but has spots instead of stripes". similarly, when children learn to draw a new shape like an oval, teachers often prompt them by showing a circle, but saying "make the circle stretched-out". this method of learning new concepts from visual aids and language descriptions is a common way of reinforcing existing knowledge and allowing learners to explore and retrieve new concepts @xcite . we propose a task for vision-language models to retrieve knowledge with multi-modal queries, i.e. queries in which hints about the information to be retrieved are split across image and text inputs. figure 1 contains an example of this task, where the image shows the empire state building in new york city. if we retrieve knowledge using only the image, is it likely that the retrieved information (k1) will be related to the empire state building. however, k1 is insufficient to answer the question. on the other hand, if we retrieve knowledge using only the question, then the information retrieved (k2) is likely to be related to the tallest building in all cities (and not restricted to new york city). k2 by itself is also insufficient to answer the question. this example shows that the combined query containing both image and text (question) is necessary for retrieving relevant knowledge (k3). we introduce a new benchmark and dataset called remuq (retrieval with multimodal queries) to train and evaluate models to retrieve the answer from a corpus given multimodal (vision + language) queries. to create multimodal queries, we start with the webqa @xcite dataset as a source -webqa contains images annotated with questions and answers. we select questions from webqa where the answer includes both an image and text. we then remove any image information from text and combine the image and the augmented text to form a new multimodal query. we also construct a large retrieval corpus consisting of answer options of all questions as the source of knowledge for this task. this task requires integrating the contents from both modalities and retrieve knowledge -in this paper we denote such a system as a "vl-retriever". existing vl-retrievers @xcite @xcite typically follow a two-step process to retrieve knowledge: (1) converting the image into captions or keywords, appending them to the text query, and (2) using a text-retriever system to retrieve the knowledge. however, this approach can result in a loss of important information figure 1 : an illustration of the multimodal retrieval task from the remuq dataset. the image shows the empire state building and the question asks if it is the tallest building in "the city". neither the image nor the question explicitly mentions that "the city" is new york. the challenge therefore is to use the cues in the image and question to retrieve relevant information and answer the question. in this illustration we show the retrieved knowledge using only the image (k1), only the question (k2), or both image and question (k3). only k3 can be used to answer the question correctly. from the image, such as context and background. additionally, using a caption generation model trained on a particular domain does not transfer well to other domains in real-world applications. to address these issues, we propose an end-toend vl-retriever that has the potential to leverage the entire image, rather than just object categories, keywords, and captions. we call this model reviz, a retriever model for "reading and vizualizing" the query. as part of reviz, we use a vision transformer-based model, vilt @xcite , to directly encode the image from raw pixels with context inputs, and we employ bert @xcite as the knowledge encoder to represent the long, free-form text as a knowledge embedding. reviz differs from previous retrieval models in two main ways. first, it does not require an extra cross-modal translator (e.g., a captioning model) or object detector to represent the images. second, its end-to-end design allows for the flexible retraining of each submodule of the model, which can mitigate potential issues caused by domain gaps. unlike neural text-retrievers @xcite , the query and knowledge encoders in reviz are of different types of modality (i.e. multimodal transformer and language transformer). the different semantic spaces of the query and knowledge embeddings make alignment between them difficult. to address this, we propose a novel multimodal retrieval pretraining task. to create training data, we construct triplets of (input-image, input-text, outputknowledge) from the wit @xcite dataset which contains encyclopedia-type knowledge from wikipedia. we process the data such that the input image and text have mutually exclusive information. our contributions and findings are listed below. ## related work cross-modal retrieval aims to find information from a different modality than the query; for instance retrieving images from text (text-to-image), text from images (image-to-text) @xcite ), text-to-video and video-totext @xcite @xcite . in contrast, we consider retrieval of knowledge for queries comprised of both modalities (i.e. image and text) together. ## retrieval with multimodal queries in this section, we define the problem statement for knowledge retrieval with multimodal queries and describe the construction of the remuq dataset to assess models performing this task. ## method prior work on vision-language (vl)-retrievers has focused on two-stage methods where the first stage involves feature-extraction using pretrained visual and textual encoders and the second stage learns retrieval using these features. a typical vl-retriever can be expressed as: end-to-end vl-retriever. instead, in this work, we are interested in building an end-to-end vl-retriever, that encodes and selects the knowledge from the corpus using a vl model: we propose reviz, an end-to-end vl-retriever that learns to maximize the multimodal query and knowledge similarity for knowledge retrieval tasks. we introduce its architecture below. ## pretraining task for vl retriever previous work @xcite @xcite suggests that pretraining a retriever on unsupervised task that closely resembles retrieval can greatly improve the downstream tasks performance. we propose a pretraining task called vl-ict, which is inspired by ict @xcite task in nlp domain. ict aims to train text-based information retrieval (ir) system for the open-domain question answering task. to train a model without annotated data, @xcite propose to construct pseudo (question, context) pairs as the training data for ir system. in particular, given a passage p , a random sentence s in the passage is selected as the pseudo question, and the remaining passage p ′ is considered as the relevant context. such a weakly-supervised setting enables large-scale ict pre-training, leveraging any available knowledge base as the training corpus. ## experiments and results datasets. in addition to remuq, we conduct experiments on okvqa to obtain stronger evidence for the efficacy of our method. here, instead of qa task, we use okvqa as a testbed for retrieval task, i.e. to retrieve a relevant knowledge to a question such that it contains the answer span. furthermore, we use two corpora, a small corpus collected from google search api introduced in @xcite , and a large corpus which contains 21m wikipedia knowledge used in @xcite . the statistic of each dataset is given in evaluation metrics. following @xcite ; @xcite , we evaluate the performances of models by precision@k (p@k), recall@k (r@k), and mrr@5. we use similar metrics to evaluate the remuq challenge except that p@1 is used instead of p@5 since remuq has exactly one correct knowledge per query. ## conclusion we study knowledge retrieval with multimodal (vision and language) queries, which, compared with existing retrieval tasks, is more challenging and under-explored. in addition, multimodal-query information retrieval has numerous potential applications, not only in retrieval tasks such as image, text, and video retrieval, but also in question answering, recommendation systems, and personal assistant. the proposed dataset (remuq) is ideally positioned to support the development of such functionalities. we propose an end-to-end vl-retriever model, reviz, which does not rely on any intermediate image to text translation modules. a novel weakly-supervised task (vl-ict) is proposed to enable large-scale pre-training. extensive evaluations on remuq and ok-vqa datasets demonstrate that reviz exhibits strong performance amongst all retrieval models in both zero-shot and fine-tuning scenarios. our proposed dataset and model provide a foundation for future work which could potentially lead to new findings and innovative applications in multimodal-query information retrieval.