Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts (2023)

Abstract: Visual question answering (VQA) is the task of answering questions about an image. The task assumes an understanding of both the image and the question to provide a natural language answer. VQA has gained popularity in recent years due to its potential applications in a wide range of fields, including robotics, education, and healthcare. In this paper, we focus on knowledge-augmented VQA, where answering the question requires commonsense knowledge, world knowledge, and reasoning about ideas and concepts not present in the image. We propose a multimodal framework that uses language guidance (LG) in the form of rationales, image captions, scene graphs, etc to answer questions more accurately. We benchmark our method on the multi-choice question-answering task of the A-OKVQA, Science-QA, VSR, and IconQA datasets using CLIP and BLIP models. We show that the use of language guidance is a simple but powerful and effective strategy for visual question answering. Our language guidance improves the performance of CLIP by 7.6% and BLIP-2 by 4.8% in the challenging A-OKVQA dataset. We also observe consistent improvement in performance on the Science-QA, VSR, and IconQA datasets when using the proposed language guidances. The implementation of LG-VQA is publicly available at https://github.com/declare-lab/LG-VQA.

## introduction visual understanding is one of the most complex tasks in artificial intelligence. among the many challenges associated with it, image question answering has been formulated as a task that tests the ability of a system to understand the elements of an image in a way similar to how humans interact with images. this task involves creating models that can accurately answer questions based on the content of an image. while significant progress has been made in image question answering @xcite , most of the existing approaches focus solely on analyzing the visual features associated with the image or the linguistic content or representation of the question (and possibly candidate answers), without utilizing additional sources of guidance. however, incorporating external guidance into these models has the potential to improve their performance and enhance their understanding of visual content. in this paper, we propose a multimodal framework, lg-vqa that leverages language guidance to improve the accuracy of image questionanswering systems. language guidance is sourced from elements such as image captions, scene graphs, and rationales created in response to questions. these sources serve to enrich textual instructions with valuable knowledge. our method is evaluated on the multi-choice a-okvqa dataset @xcite . we choose a-okvqa specifically because the questions in this dataset require a broad base of commonsense and world knowledge to answer. we also evaluate our method on the challenging scienceqa @xcite , visual semantic reasoning (vsr) @xcite , and iconqa @xcite datasets. with the advent of large-scale multimodal pre-training @xcite , the performance in the commonly used vqa and vqa v2 datasets @xcite has saturated. the datasets considered in this paper provide a more challenging test bed for vqa where factual, commonsense, physical, scientific, and visual knowledge are required, going beyond the usual image recognition, attribute detection tasks that are generally found in the vqa, vqa v2 datasets. previously shah et al. (2019); wu et al. (2017); zheng et al. (2021) have used concrete knowledge bases such dbpedia, freebase, or named entity knowledge graphs in vqa. we instead show that simple language guidance without using knowledge graphs is also a powerful technique for improving vqa performance in various challenging datasets. we benchmark our approach with the clip @xcite and blip-2 @xcite models and show significant improvements over those, demonstrating that incorporating language guidance can be an effective strategy for visual question answering. ## background we use the following two strong multi-modal models in our framework: clip @xcite is a model for learning transferable visual representations from natural language supervision. clip is trained on 400 million (image, text) pairs to learn a joint embedding space where image representations and their corresponding text representations are close to each other. we use the clip model with the vit-l/14 image encoder @xcite and the gpt-2 text encoder @xcite . clip encodes the image and the text separately through the corresponding encoders. then, a normalized dot product between the encoded vectors provides the image-text matching score. we denote the image and text encoder as i clip and t clip . given an image img and text instance txt, the matching score is computed as follows: in the first stage of blip-2 pre-training, the q-former learns to extract visual representations from the image that is most relevant to the text using image-text contrastive learning, and imagegrounded text generation tasks. in the second stage, the trained q-former is used to extract a set of query embeddings about the image. the query embeddings are converted into the input embedding space of the llms through a linear projection. the projected vector functions as the soft visual prompts that condition the llm to generate the desired text. blip-2 uses paired (image, text) data for both stages of pre-training. we denote the image encoder and q-former of blip2 as i blip and q blip . the image-text matching score is computed as follows: we use the image-text matching methodology and the (visual prompt, prefix) conditioned text generation in various stages of our framework. ## conclusion in this work, we presented a simple method for visual question answering with language guidance. our language guidance consists of rationale and explanations constrained upon the image and the question, image captions, scene graphs, and description of the objects present in the image. we propose an effective strategy of fusing the language guidance with the pre-trained multi-modal clip and blip-2 models. our language guidance improves the performance of clip by 7.6% and blip-2 by 4.8% in the challenging a-okvqa dataset. we also observe consistent improvement in performance with guidance on the scienceqa, vsr, and iconqa datasets.