E cho S ight: Advancing Visual-Language Models with W iki Knowledge (2024)

Abstract: Knowledge-based Visual Question Answering (KVQA) tasks require answering questions about images using extensive background knowledge. Despite significant advancements, generative models often struggle with these tasks due to the limited integration of external knowledge. In this paper, we introduce **EchoSight**, a novel multimodal Retrieval-Augmented Generation (RAG) framework that enables large language models (LLMs) to answer visual questions requiring fine-grained encyclopedic knowledge. To strive for high-performing retrieval, EchoSight first searches wiki articles by using visual-only information, subsequently, these candidate articles are further reranked according to their relevance to the combined text-image query. This approach significantly improves the integration of multimodal knowledge, leading to enhanced retrieval outcomes and more accurate VQA responses. Our experimental results on the E-VQA and InfoSeek datasets demonstrate that EchoSight establishes new state-of-the-art results in knowledge-based VQA, achieving an accuracy of 41.8% on E-VQA and 31.3% on InfoSeek.

## introduction visual question answering (vqa) addresses the challenge of enabling machines to understand and respond to questions about visual content, typically images or videos. broadly, this task can be divided into two categories: standard vqa @xcite with questions that can be answered directly from the visual content, for example, counting objects, identifying colors, or recognizing simple actions, which rely solely on commonsense and information present in the image; and knowledge-based vqa @xcite @xcite requiring additional context or external knowledge, such as historical facts, de-tailed object properties, or specific situational contexts not evident in the visual content. addressing these two types of questions presents different challenges for vqa systems. questions that draw answers directly from visual content demand robust image understanding capabilities, encompassing tasks such as object detection, scene recognition, and spatial reasoning. conversely, questions requiring external knowledge call for additional mechanisms to access and integrate information from external sources. in this paper, we focus on the latter type of visual question answering, by building a retrieval-augmented multimodal system, that enables searching an external knowledge base for more nuanced understanding and accurate responses. despite the recent accomplishments in developing visual-language models (vlms) @xcite @xcite , knowledge-based vqa remains challenging. this complexity primarily stems from two aspects. (i) existing vlms struggle to adequately encode all essential knowledge, due to its limited model capacity, and infrequent inclusion of encyclopedic, long-tail information in their training data @xcite . (ii) the visual component of the questions often provides limited help in addressing the queries, as establishing a meaningful connection between entity knowledge and visual attributes can be difficult. for example, an image of a church alone barely reveal information about its construction date. in this paper, we introduce echosight, a novel retrieval-augmented vision-language system designed for knowledge-based visual question answering. echosight employs a dual-stage search mechanism that integrates a retrieval-and-reranking process with the retrieval augmented generation (rag) paradigm. initially, the system performs a visual-only retrieval from an external knowledge base, to effectively narrow the knowledge search the naranjo de bulnes is a limestone peak in spain. … this mountain was first scaled via the north face (by the route now known as the vía pidal) on 5 august 1904 by pedro pidal. ## method this section starts with the problem formulation of retrieval-augmented vqa (sec. 2.1), followed by detailing the retrieval-and-reranking module in echosight (sec. 2.2), and finally the answer generation module (sec. 2.3). ## conclusion in this paper, we introduced echosight, a novel retrieval-augmented vision language system designed to address the challenges of knowledgebased visual question answering (vqa). our approach enhances the retrieval capabilities of multimodal models through a two-stage process: initial visual-only retrieval followed by a multimodal reranking stage. this methodology significantly improves the alignment between visual and textual information, leading to more accurate and contextually relevant answers. experimentally, we have conducted thorough ablation studies to demonstrate the effectiveness of our proposed components. while comparing to existing state-of-the-art approaches on the encyclopedic vqa and infoseek datasets, echosight demonstrates significant performance improvement, with an accuracy of 41.8% on e-vqa and 31.3% on infoseek. the success of echosight highlights the importance of efficient retrieval processes and the integration of multimodal information in enhancing the performance of large language models (llms) in knowledgebased vqa tasks.