MM -Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering (2023)

Abstract: Thanks to the strong reasoning capabilities of Large Language Models (LLMs), recent approaches to knowledge-based visual question answering (KVQA) utilize LLMs with a global caption of an input image to answer a question. However, these approaches may miss key visual information that is not captured by the caption. Moreover, they cannot fully utilize the visual information required to answer the question. To address these issues, we introduce a new framework called Multi-Modal Knowledge-Aware Reasoner (MM-Reasoner) for KVQA. MM-Reasoner first utilizes a set of vision APIs, such as dense captioners, object detectors, and OCR, to extract detailed information from the image in textual format. Then, it prompts an LLM to extract query-specific knowledge from the extracted textual information to provide a rich representation that contains external knowledge, commonsense, explicit supporting facts, and rationales required for reasoning. Finally, the knowledge, query, and visual input are used to fine-tune a Vision-Language Model (VLM). At test time, MM-Reasoner uses the potential answers predicted by the VLM to iteratively update and optimize the prompt, refining its answer. Empirical studies show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets.

## introduction the knowledge-based visual question answering (kvqa) task @xcite requires not only visual information from the image, such as object attributes and visual relationship information, but also external knowledge, commonsense, rationales, and supporting facts for reasoning and predicting the correct answer. therefore, a typical kvqa model consists of a knowledge retrieval module and an answer module. the traditional knowledge retrieval module usually retrieves knowledge from sources such as wikipedia, knowledge graphs, and web search @xcite . more recently, large language models (llms) such as gpt-3 are used to produce related knowledge @xcite . the latter approach is preferred since traditional knowledge retrieval often introduces irrelevant information to the question @xcite . after obtaining the required knowledge, the answer module takes the visual input or a global caption of the image, query, and knowledge to predict an answer. the answer module is either a transformer encoderdecoder architecture with trainable parameters @xcite or more recently, a frozen llm such as gpt-3 @xcite . however, there are several key challenges to the above approaches. first, when using an llm, the input needs to be text, but the recent approach of feeding the global caption of the input image into llms @xcite ) may miss key visual information. second, the conceptual diversity of visual questions often necessitates multimodal reasoning with the integration of various external information beyond the image itself. these diverse types of knowledge include: commonsense, knowledge of concepts represented visually, external knowledge, e.g., marco van basten is a former football player born in the netherlands., insights about the world's physics, e.g., mountain areas have a lower temperature than other areas, etc. third, recent approaches cannot fully utilize visual information in their answer module since they use an llm to predict an answer. these approaches often struggle to answer visual questions that require fine-grained visual information, e.g., spatial relationships and object attributes, or knowledge of concepts represented visually, e.g., what is the pattern of the table cloth called? to answer a diverse set of questions, a kvqa model requires (1) vision expertise, e.g., celebrity recognition, optical character recognition (ocr), and object detection (2) the reasoning capacity and world knowledge of the recent llms, and (3) the ability of vision-language models (vlms) to generate rich joint representations of vision and language. however, current knowledge-based vqa models cannot provide all these capabilities within a unified framework. to address these challenges, we introduce a new framework, called multi-modal knowledge-aware reasoner (mm-reasoner), for knowledge-based vqa (figure 1 ). the mm-reasoner first leverages a rich set of vision apis/expertise such as dense captioner, object detector, tag detector, ocr, people detector and celebrity recognizer, to obtain a detailed textual representation of the visual input required to produce more accurate answers (figure 2 ). then, the mm-reasoner leverages an llm to extract query-specific knowledge from the textual representations in a self-supervised way via prompt designing, which requires only a few in-context examples. the llm can provide a rich representation that contains external knowledge, commonsense, explicit supporting facts, and rationales required for reasoning. finally, the extracted knowledge, the question, and the regional visual information of the input image are used to fine-tune a vlm. the vlm can fully utilize fine-grained visual information such as object regions, visual relationships/attributes, and visual spatial information supplementary to the textual representation to answer the question (see figure 3 ). at test time, mm-reasoner uses candidate answers predicted by the vlm to iteratively update and optimize the prompt to refine its final answer. this allows our framework to effectively exchange knowledge and visual information between the llm and the vlm. mm-reasoner is an integrative and composable framework that works with a variety of existing llms and vlms. since these llms and vlms have already been trained on large-scale datasets, the fine-tuning cost of mm-reasoner is much lower than that of existing models. we have conducted extensive empirical studies on mm-reasoner using several datasets. the results show mm-reasoner achieves state-of-the-art accuracy on the kvqa task, achieving 60.8% accuracy on the ok-vqa dataset, 60.2% on the a-okvqa dataset, and 40.0% in the few-shot setting on the ok-vqa dataset. the project page is available at https://github.com/microsoft/ i-code/tree/main/mm-reasoner . in summary, the key contributions of this paper are as follows: our work is related to several recent studies in the area of multi-modal learning and llms. knowledge-based vqa. in revive @xcite , the authors proposed to first employ an object detector to locate the objects, and then use the cropped bounding-box proposals to retrieve various types of external knowledge. finally, they merged this knowledge with the regional visual features into a transformer to predict an answer. in @xcite , the authors introduced a captioning model called promptcap, which is designed to generate question-aware captions using a gpt-3 prompt and question-answer pairs from existing vqa datasets. however, to generate the training examples, the promptcap still relies on global captions of the image. additionally, the visual features are not effectively utilized in the final answering model, as the answer module is a frozen llm. in contrast, our prompt design enables our framework to extract query-specific knowledge from the visual input, avoiding the extraction of unrelated information. additionally, mm-reasoner harnesses the capabilities of vlms such as flamingo @xcite and i-code @xcite to incorporate visual features in the answer module. in @xcite , the authors proposed an end-to-end model, called retrieval-augmented visual language model (reveal), that learns to encode multimodal world knowledge into a memory and utilize it for addressing visual questions that demand extensive knowledge. reveal is composed of four modules: the memory, the encoder, the retriever, and the generator. the memory encodes different types of multimodal knowledge via a shared encoder. the retriever finds the most relevant information, and the generator uses that information with the input question to predict the final answer. the model is pre-trained on a large amount of data and then fine-tuned for specific tasks. however, reveal does not leverage the reasoning capabilities of recent llms. ## multi-modal knowledge-aware reasoner in this section, we present the multi-modal knowledge-aware reasoner framework. as figure 1 shows, mm-reasoner is composed of three components: a set of vision apis, an llm, and a vlm. it is worth mentioning that mm-reasoner is an integrative, modular, and composable framework that works with a variety of existing vision apis, llms and vlms. these components work together to extract intrinsic information from the image and query, extrinsic information from external knowledge, and conduct visual-textual reasoning to produce the answer. in the following, we explain these components in more details. ## experiments this section covers the explanation of datasets and baseline models, followed by a discussion of experimental results and implementation details. ## conclusion mm-reasoner is a new, unified framework for kvqa that combines vision apis, the capabilities of llms, and vlms. experiments showed mm-reasoner is an effective framework for kvqa.