Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge (2023)

Abstract: The open-ended Visual Question Answering (VQA) task requires AI models to jointly reason over visual and natural language inputs using world knowledge. Recently, pre-trained Language Models (PLM) such as GPT-3 have been applied to the task and shown to be powerful world knowledge sources. However, these methods suffer from low knowledge coverage caused by PLM bias – the tendency to generate certain tokens over other tokens regardless of prompt changes, and high dependency on the PLM quality – only models using GPT-3 can achieve the best result. To address the aforementioned challenges, we propose RASO: a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. Rather than following the de facto standard to train a multi-modal model that directly generates the VQA answer, {pasted macro ‘MODEL’}name first adopts PLM to generate all the possible answers, and then trains a lightweight answer selection model for the correct answer. As proved in our analysis, RASO expands the knowledge coverage from in-domain training data by a large margin. We provide extensive experimentation and show the effectiveness of our pipeline by advancing the state-of-the-art by 4.1% on OK-VQA, without additional computation cost.

## introduction open-ended visual question answering (vqa), that requires answering a question based on an image, has received much attention in machine learning research in the past decade @xcite . knowledge-based vqa @xcite is a variant of vqa, where models have to use external knowledge that is not present in the image to generate the answer. it is a more challenging problem as it requires joint reasoning over visual and natural language inputs using world knowledge. for example, in figure 1 , the vqa model needs to conduct multiple levels of inference: to detect the objects in the image (e.g. laptops, whiteboard, etc), to retrieve external world knowledge (e.g, university is an institution and has lecture rooms, lecture rooms have laptops, stairs, and whiteboard, etc), and combine the important visual parts with retrieved knowledge to induce the final answer (e.g. university). in this paper, we focus on improving the important step of external knowledge retrieval. a common procedure of previous vqa methods @xcite is to retrieve with knowledge graphs from diverse knowledge bases (e.g. wikipedia @xcite , conceptnet @xcite , etc.), with the results being input to an answer generation model. however, the retrieved knowledge could be noisy, irrelevant, and redundant, and therefore lead to mismatches that limit the vqa performance. motivated by the development of large-scale plms such as gpt-3 @xcite ) that obtain state-of-the-art (sota) performance in most nlp tasks including text generation @xcite , more recent approaches pica @xcite and kat @xcite propose to re- ## method our method consists of two steps: answer choices generation and answer selection. the overview of the proposed model is shown in figures 2 and 3 ## ablation studies we perform qualitative and quantitative analysis on the answer selection results to better understand whether the expanded external knowledge coverage benefits the end-task vqa much. as illustrated in tables 4 to 6 , we investigate the impact of various inputs on the answer selection results, with answer choices representing the retrieved knowledge. cot rationale impact from the experiments results in tables 4 to 6 where we compare the settings: "w/cot" and "w/o cot", input with cot rationales consistently boosts the answer selection performance of kat, unifiedqa, and clipcap. however, this conclusion fails for iterative prompting -adding cot hurts the performance of iterplm when we use gpt-j and codex. we believe this can result from the difference in cot qualities, and different pre-training methods and data. ## conclusion in this paper, we propose raso: a new vqa pipeline following a generate-then-select strategy guided by world knowledge. raso proposes a new prompting method that largely increases the ex-ternal knowledge coverage by a margin of more than 20% compared to previous approaches on the ok-vqa benchmark. our pipeline achieves the new sota 58.5% on the end-task performance , encouraging avenues for future studies.