A Simple Baseline for Knowledge-Based Visual Question Answering (2023)

Abstract: This paper is on the problem of Knowledge-Based Visual Question Answering (KB-VQA). Recent works have emphasized the significance of incorporating both explicit (through external databases) and implicit (through LLMs) knowledge to answer questions requiring external knowledge effectively. A common limitation of such approaches is that they consist of relatively complicated pipelines and often heavily rely on accessing GPT-3 API. Our main contribution in this paper is to propose a much simpler and readily reproducible pipeline which, in a nutshell, is based on efficient in-context learning by prompting LLaMA (1 and 2) using question-informative captions as contextual information. Contrary to recent approaches, our method is training-free, does not require access to external databases or APIs, and yet achieves state-of-the-art accuracy on the OK-VQA and A-OK-VQA datasets. Finally, we perform several ablation studies to understand important aspects of our method. Our code is publicly available at https://github.com/alexandrosXe/ASimple-Baseline-For-Knowledge-Based-VQA

## introduction knowledge-based vqa (kb-vqa) is a recently introduced vqa task @xcite @xcite where the image alone is not sufficient to answer the given question, but effective utilization of external knowledge resources is additionally required. to solve such a task, a model would need not only strong visual perception but also reasoning capabilities while also being able to effectively incorporate world knowledge from external kbs (e.g. wikipedia, etc) and llms. systems capable of answering general and diverse questions about the visual world find a wide range of applications: from personal assistants to aids for the visually impaired and robotics 1 . recently, several works on kb-vqa @xcite have emphasized the significance of incorporating both explicit and implicit knowledge. however, such approaches usually require complicated pipelines. firstly, a kb (e.g. wikidata) covering world knowledge needs to be maintained and used for knowledge retrieval which is time-consuming and very sensitive to noise. secondly, powerful llms such as gpt-3 @xcite or opt-175b @xcite are leveraged due to the huge amount of implicit knowledge stored in their parameters and their powerful reasoning capabilities through few-shot in-context learning. however, the computational or even actual monetary cost (e.g. cost for api access) associated with accessing such models renders them unaffordable for many researchers. thirdly, it is crucial to train a fusion mechanism that can effectively reason by combining the retrieved explicit and implicit knowledge. ## methodology while explicit knowledge retrieval focuses on semantic matching between an image and knowledge entries, it lacks implicit commonsense knowledge (e.g. lemons are sour) which can be found in llms @xcite . llms are critical in extracting implicit knowledge due to the vast amount of implicit information embedded in their parameters, and their powerful reasoning capacity through few-shot in-context learning. different from pre-vious work @xcite @xcite we leverage the open-source llm llama-13b @xcite instead of gpt-3 as an implicit language knowledge base and treat vqa as an open-ended text generation task. our method builds upon the pipeline of pica, which is the pioneering work that utilizes gpt-3 for few-shot in-context learning in order to address the kb-vqa task. gpt-3 is a decoder-only autoregressive llm of 175b parameters, trained on a diverse range of data sources, including common crawl, webtexts, books, and wikipedia @xcite . during inference, in-context few-shot learning involves formulating a novel downstream task as a text sequence generation task using the frozen gpt-3 model. when provided with a testing input x, the target y is predicted based on a formatted prompt p(h, c, e, c, x). in this prompt, h represents a prompt head or instruction that describes the task, while @xmath0, e 2 , ..., e n } represents a set of n in-context examples (shots), where e @xmath1, y i ) represents an input-target pair of the task, where x i and y i are the input and target, respectively. these pairs are constructed manually or sampled from the training set. @xmath2, c 2 , ..., c n } represents a set of generic image captions describing each x i since images cannot be inputted to gpt-3. the caption for the test input is labeled as c. the target y is denoted as a text sequence consisting of l tokens, expressed as @xmath3, y 2 , ..., y l ). at each decoding step t, the following conditions apply: in order to utilize any llm for the knowledgebased vqa task, the crucial step is to design suitable prompts. when given a question q i and an image v i as inputs, the vqa task's objective is to predict the corresponding answer a i . however, since llms do not inherently comprehend images, it becomes necessary to convert the image into a caption c i using a pre-existing captioning model. while sota pretrained captioning models have demonstrated impressive performance, they are primarily optimized to generate generic image captions. unfortunately, these captions often fail to capture all the specific details required to accurately answer a given question about the image. in this work, instead of generic captions, we generate question-guided informative image captions using the plug-and-play vqa (pnpvqa) framework @xcite which identifies the most re-lated image patches to the question with a saliency map-based interpretability technique and generates captions from these patches only. for each image-question pair, we first generate 50 question-guided informative image captions from the image v i using pnpvqa. we then employ blip's @xcite text encoder to encode all the image captions and blip's image encoder to encode the image v i . we rank the image captions per image v i according to their cosine similarity with the image v i and keep the top-m most similar captions c i per example. after extracting the top-m most similar captions per image v i we construct a carefully designed text prompt consisting of a general instruction sentence, the captions c, the question, the test input's captions c, and a set of context-question-answer triplets (shots) taken from the training dataset that are semantically most similar to the current image-question pair (see fig. 1 ). then this text prompt is passed to a frozen llama-13b model and in-context few-shot learning is performed in order to obtain its output as a promising answer candidate to the current imagequestion pair. ## experimental results comparative results on ok-vqa: ## ablation studies we conduct several ablations on ok-vqa to better understand the key components of our method. ## conclusions we proposed a simple yet effective baseline for kb-vqa. our training-free method is based on in-context few-shot learning of the open-source llama using question-informative captions. we show that this is sufficient to achieve sota results on the widely used ok-vqa and a-ok-vqa datasets.