Learning to Ask Denotative and Connotative Questions for Knowledge-based VQA (2024)

Abstract: Large language models (LLMs) have attracted increasing attention due to its prominent performance on various tasks. Recent works seek to leverage LLMs on knowledge-based visual question answering (VQA) tasks which require common sense knowledge to answer the question about an image, since LLMs have obtained rich knowledge from large-scale training. Several methods have proposed to leverage frozen LLMs by converting visual information to textual prompts. However, how to efficiently exploit the knowledge of LLMs and bridge the disconnects between visual information and language models remain open problems. In this paper, we propose to let LLMs learn to ask (L2A) informative questions to collect essential visual information. We introduce the concepts of denotation and connotation to promote image and question understanding and provide a clear guidance with respect to the objective of question generation. In this way, the model can better capture the associations between different concepts, as well as efficiently collect both explicit information and implicit relevant information that contribute to the final answer. The experiments demonstrate that our proposed method achieves consistent performance on various knowledge-based VQA datasets.

## introduction large language models (llms) @xcite @xcite have shown remarkable potential on various natural language tasks due to their rich knowledge and reasoning abilities obtained by large-scale pretraining. recent works have also explored to leverage llms on vision-language tasks, especially knowledgeintensive tasks such as knowledge-based visual question answering (vqa) @xcite , which aims to answer questions about images that require commonsense knowledge to answer. question: what is the weather like today? ## method our proposed l2a approach motivates llms to learn to ask informative questions that collect essential information for answering visual questions. an overview of our proposed method is shown in figure 2. it consists of three steps: prompt generation, information collection and information integration, which mainly involves the interaction between a inquirer (i.e., llm) and a respondent (i.e., base vqa model). the prompt generation module aims to provide clear guidance for the inquirer to ask informative questions, by introducing the clarified concepts of denotation and connotation. during the information collection stage, under the guidance of the prompt, the inquirer generates both denotative and connotative questions to collect essential infor-mation. the respondent generates the corresponding answers based on the visual contents. then we evaluate the obtained question-answer pairs by their contribution to the final answer, which implies their relevance to the target question. finally, we leverage the implicit knowledge and reasoning ability of llm to integrate the collected information, hence obtain more reliable answers to the knowledge-intensive questions. ## conclusion in this paper we leverage the rich knowledge of llms for knowledge-intensive vqa tasks. we propose a learn to ask (l2a) architecture that inspires the llm to generate questions to collect essential visual information for answering the target questions. we introduce the concepts of denotation and connotation and propose an efficient prompting strategy to guide the question generation with clear objective, which involves both explicit and implicitly related information. our proposed method can effectively generate high-quality questions and efficiently collect required information without expensive training or annotations. experiment results show that our proposed l2a approach achieves promising performance on knowledge-based vqa datasets without requiring in-context examples.