Filling the Image Information Gap for VQA : Prompting Large Language Models to Proactively Ask Questions (2023)

Abstract: Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge not only in natural language tasks, but also in some vision-language tasks such as open-domain knowledge-based visual question answering (OK-VQA). As images are invisible to LLMs, researchers convert images to text to engage LLMs into the visual question reasoning procedure. This leads to discrepancies between images and their textual representations presented to LLMs, which consequently impedes final reasoning performance. To fill the information gap and better leverage the reasoning capability, we design a framework that enables LLMs to proactively ask relevant questions to unveil more details in the image, along with filters for refining the generated information. We validate our idea on OK-VQA and A-OKVQA. Our method continuously boosts the performance of baselines methods by an average gain of 2.15% on OK-VQA, and achieves consistent improvements across different LLMs.

## introduction large-scale language models (llms) have exhibited impressive capabilities in terms of their world knowledge and reasoning abilities, leading to remarkable achievements in various natural language processing (nlp) tasks such as commonsense reasoning @xcite and open-domain question answering @xcite . building upon the success in the realm of text, recent research has explored the utilization of pre-trained llms in vision-language (vl) tasks. these studies have shown promising performance, especially for knowledge-intensive tasks such as knowledgebased visual question answering (vqa) (marino et al., 2019; schwenk et al., 2022), where both image understanding and external knowledge are imperative for answering open-ended questions. the key challenge of leveraging llms in vl tasks is to bridge the gap between images and text, i.e., enabling llms to understand images. to address this challenge, two approaches have been investigated. the first approach extends the llm with visual perception modules to form a vision-language pre-training (vlp) model @xcite @xcite . despite the high performance, this approach requires training on large-scale image-text pairs, which can be com-putationally expensive, particularly for llms with massive parameters such as @xcite and palm @xcite . the second approach transforms images into textual representations, which are then used as prompts for the llms @xcite @xcite . this training-free approach is more cost-effective and enables llms to be utilized in a more flexible paradigm, allowing for easy adaptation to different tasks. however, as discussed in the works of @xcite and @xcite , general image captions may lack the subtle information required to answer visual questions. to resolve this, @xcite compromise captions with image tags, and @xcite propose incorporating questions into the caption generation process. despite their successes, it remains impractical to reveal every subtle detail necessary to answer visual questions in a single concise description. as illustrated in figure 1 , the captions fail to spot the "area being stepped over" as the "water", resulting in hallucinated answers. two primary concerns exist regarding existing image-to-text conversion approaches for vqa: (1) the converted textual descriptions might be insufficient to solve the visual questions, or could contain misleading content; (2) existing methods convert images to text as a preprocessing step of the input. this one-off conversion is a lossy compression of the conveyed information, and does fully provoke the reasoning ability of llms. in this paper, we present a framework where llms proactively interact with vision-language models (vlms) to gather specific information of interest, as depicted in figure 1 . this interaction is aimed at automatically seeking and regaining details that may be omitted during the image-totext conversion. to enhance the informativeness of the generated questions and the correctness of their corresponding answers, we design a refinement module to summarize only the useful information for generating the final answers. we validate our approach on ok-vqa and a-okvqa datasets and conduct experiments across different llms. our contributions are as follows: ## method our overall framework, as illustrated in figure 2 , comprises three key modules: inquiry, refinement and answering. given the image caption and the question, the inquiry module prompts an llm to generate new questions that seek for missing image information required to answer the original question, and uses a vlm to answer them based on the image ( §3.1). then, we adopt a refinement module to summarize information from the generated questions and answers, filtering and extracting the relevant and useful information from them ( §3.2). finally, the answering module prompts the llm to predict the final answer to the original question with the augmented image information ( §3.3). before delving into the details of our method, we shall declare some important notations. we use i, c, q to denote the image, its caption, and the original question, respectively. the caption c is generated by the image captioning model m c : ## experiments we validate our method on open-domain knowledge-base vqa, and conduct experiments on ok-vqa and a-okvqa. implementation details are described in the following sections. ## case study we conduct comprehensive analysis of the cases, and observe that our method exhibits the following capabilities: 1) to unveil a better level of detail as stated in the §3.1; 2) to rectify inaccurate caption information; and 3) to minimize the ambiguity of provided information, e.g., captions and candidates. cases are demonstrated in figure 3 . we compared the results of baselines, ptomptcap and prophet, with and without applying our method. ## conclusion in this paper, we focus on open-domain knowledgebased vqa and propose a model agnostic framework that successfully unveils missing detail during the image-to-text transformation. our method acquires the ability to rectify inaccurate information generated by captioning models, and the ability to minimize the ambiguity of the converted textual information for further improvements. our method can be applied upon existing baselines, and achieves average gains of 2.15% on ok-vqa and 1.25% on a-okvqa over the baselines. ablation studies show that our method attain consistent improvements across different llms.