Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering (2024)

Abstract: The Knowledge-Aware Visual Question Answering about Entity task aims to disambiguate entities using textual and visual information, as well as knowledge. It usually relies on two independent steps, information retrieval then reading comprehension, that do not benefit each other. Retrieval Augmented Generation (RAG) offers a solution by using generated answers as feedback for retrieval training. RAG usually relies solely on pseudo-relevant passages retrieved from external knowledge bases which can lead to ineffective answer generation. In this work, we propose a multi-level information RAG approach that enhances answer generation through entity retrieval and query expansion. We formulate a joint-training RAG loss such that answer generation is conditioned on both entity and passage retrievals. We show through experiments new state-of-the-art performance on the VIQuAE KB-VQA benchmark and demonstrate that our approach can help retrieve more actual relevant knowledge to generate accurate answers.

## introduction knowledge-based visual question answering (kb-vqa) has recently gained significant attention as a challenging yet promising task for evaluating systems' capabilities to deeply understand both visual and textual information in order to answer multimodal queries. in contrast to standard vqa tasks, kb-vqa further extends the challenge by requiring access to external knowledge sources (e.g., knowledge bases, knowledge graphs), as image and text queries do not explicitly carry the required knowledge for accurate answers. recently, large language models (llms) e.g., palm @xcite , llama @xcite , and large multimodal models (lmms) e.g., git @xcite , blip-2 @xcite , pali @xcite , and beit-3 @xcite achieved impressive results in vision-language tasks thanks to their billions of parameters trained on largescale corpora, endowing them with huge memorization capabilities for solving any downstream tasks @xcite . however, they still struggle to successfully address the challenges related to knowledge-intensive tasks (e.g. kb-vqa), facing problems such as hallucinations and outdated parametric knowledge @xcite . indeed, solving kb-vqa requires addressing two tasks, namely retrieval and reading comprehension: relevant information (passages, documents) is first retrieved from a knowledge base (kb) and then answers are extracted/generated. decoupling these two steps during training accounts for the main drawback as the retriever lacks feedback from the reader. retrieval training is often performed using pseudo-labels indicating if the retrieved items are relevant for answering queries. for example, a common approach for passage retrieval consists in training dual encoders using pseudo-relevant passages where relevance is assessed based on string matching between passages and answers. obviously, such heuristics yield noisy supervision retrieval signals that do not necessarily help answer questions. retrieval augmented generation (rag) @xcite has emerged as a new paradigm where generated answers serve as training signals for the retriever, circumventing the need for exhaustive memorization within model parameters and addressing the limitation of training the retriever and the reader in a decoupled way. in this work, we rely on the rag framework to tackle the knowledge-aware visual question answering about entities (kvqae) task @xcite , which involves answering visual questions related to named entities specified in a knowledge base. applying rag to solve the kvqae task involves retrieving relevant passages used as query context during answer generation. however, the single-step retrieval em-ployed in standard rag may produce irrelevant passages misleading the generation process. in such a context, we found that determining which entity a question is related to plays a crucial role in generating accurate answers. we thus propose a multi-level information rag (mirag) approach that performs retrieval at different levels of granularity as follows: 1) given an initial question, we perform entity retrieval obtaining coarser-grain information which might enhance the subsequent finer-grain retrieval. 2) using entity information, we perform query entity expansion on the initial question before applying passage retrieval offering more relevant context that improves the generator's ability to provide accurate and informative answers. to avoid propagating irrelevant information during the iterative retrieval process, we formulate a rag loss to train all the components in a seamless end-to-end way, such that answer generation is conditioned on both entity and passage retrieval. ## related work in recent years, retrieval-augmented models have made substantial progress, driven by the necessity to exploit world knowledge for tasks like visual question answering (vqa) and open-domain question answering. beyond the implicit knowledge captured in pretrained language models (plms) parameters, the retrieval augmented (ra) learning paradigm has shown how plms can greatly benefit from external knowledge augmentations, thus alleviating problems such as hallucinations @xcite . ra learning principle consists in optimizing through backpropagation standard language modeling objectives combined with a retrieval step that learns providing useful knowledge during training, inference, and finetuning on downstream tasks. regarding the opendomain question answering (open-qa) task, early work such as orca @xcite proposed an end-to-end approach that jointly trains retriever and reader models using inverse cloze task (ict). similarly, realm @xcite is pretrained on masked language modeling (mlm) including a differentiable retrieval step that augments an encoder with external knowledge. these examples have paved the way for more recent retrieval augmented generation approaches that employ generative models for reading comprehension (answer generation) instead of span-based answer extractors. for example, @xcite proposed an iterative process where the cross-attention scores computed for answer generation are distilled as feedback to train the retriever. @xcite introduced a differentiable training method for ra-open-qa allowing information fusion from multiple retrieved documents during answer generation. @xcite addressed equally pseudo-relevant passages using an additional guide retriever during training that leverages the target outputs to retrieve the actual relevant passages. in contrast, @xcite proposed a single model trained endto-end which integrates retriever and reranker models as internal passage-wise attention mechanisms within a transformer architecture. while neural retrievers require large training datasets to achieve good performance, @xcite explored selfsupervised pretraining by recurring spans across passages in a document to create pseudo examples. in contrast, @xcite proposed to filter noisy training examples using confidence scores on the relevance labels, by measuring the connection between query-answer examples and items in a knowledge base. furthermore, @xcite explored with atlas, a pre-trained ra language model, several objectives (e.g. likelihood distillation, attention distillation) on qa and fact-checking downstream tasks in the few-shot setting. other approaches dynamically perform retrieval showing that the retrieved items are not systematically beneficial for the generator. self-rag @xcite for example generates reflection special tokens used during inference for retrieval decision making while flare @xcite employs next sentence generation token probability as a threshold to trigger the retrieval process. multimodal rag. @xcite tackled the kb-vqa task, which requires retrieval of external knowledge to answer, and proposed ra-vqa, a joint training scheme that integrates differentiable multimodal passage retrieval with answer generation, enabling end-to-end training. alternatively, reveal @xcite , extended the rag architecture with a memory module to encode multimodal world knowledge, which helps retrieve relevant entries to answer queries. additionally, they proposed an attentive fusion layer enabling seamless training of the retriever and the generator. finally, @xcite extended ra-vqa @xcite with fine-grained late-interaction multi-modal retrieval (flmr), an efficient retrieval method that increases the interaction between query and document representations using multi-dimensional embeddings. these approaches perform knowledge retrieval at the passage level only. we explore in this work how leveraging entity-level information through retrieval and query expansion may enhance rag training for kb-vqa tasks. ## method our mirag proposed approach relies on the rag framework to solve the kvqae task, which involves end-to-end training of the retriever and reader components. figure 1 illustrates the main steps of our approach. the core idea is to perform retrieval at different levels of granularity (entity and passage) given a multimodal query with both textual and visual inputs. this multi-level retrieval strategy progressively refines the query before generating answers using a generator model. retrieval is initially conducted at a coarse "entity" level to identify a set of candidate entities relevant to the query. subsequently, these retrieved entities are prepended to the query before conducting the retrieval again at a finer "passage" level. adding entities to queries provides a better understanding of the context of the query, making it easier to select more relevant passages and providing additional context for the answer generator. formally, we aim to learn the probability p(a|q, z, e) of generating an answer a conditioned on the query q, a retrieved passage z, and its corresponding retrieved entity e. ## conclusion we proposed in this paper multi-level information retrieval augmented generation (mirag), a fully integrated approach for end-to-end training of rag systems augmented with retrieved items at different levels of granularity. mirag combines entity and passage retrieval using query expansion that helps provide useful knowledge for answer generation. our experiments demonstrate that our training scheme, which conditions answer generation training on both entity and passage retrievals, increases its ability to retrieve more relevant knowledge to solve the kb-vqa task.