Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models (2024)

Abstract: In the realm of multimodal tasks, Visual Question Answering (VQA) plays a crucial role by addressing natural language questions grounded in visual content. Knowledge-Based Visual Question Answering (KBVQA) advances this concept by adding external knowledge along with images to respond to questions. We introduce an approach for KBVQA, augmenting the existing vision-language transformer encoder-decoder (OFA) model . Our main contribution involves enhancing questions by incorporating relevant external knowledge extracted from knowledge graphs, using a dynamic triple extraction

## introduction the domain of knowledge-based visual question answering (kbvqa) not only utilizes visual information extracted from images, such as object attributes and visual relationships but also integrates supporting facts to facilitate accurate reasoning and answer prediction. motivation: recently, large language models (llms) like @xcite have garnered attention for their human-like understanding of both images and language, enabling them to tackle kbvqa questions very effectively. however, these llms come with a significant drawback: their immense size (around a trillion parameters) poses challenges for offline usage. additionally, they struggle with user-centric data, such as questions related to named entities within an image. for instance, consider questions like who is the person in the middle of the image? or what is the age of the person shown in the image? in such cases, a model should provide specific answers, such as the person's name or age, rather than generic responses like man or i can't guess the age. this could also limit the performance of many iot applications where real-time user-centric data plays a crucial role. to solve this problem previous works in kbvqa @xcite @xcite used a fixed number of triples from knowledge graphs as additional sources of information to answer the question. nevertheless, using a fixed number of triples for all questions may lead to either inadequate information or unnecessary noise, potentially resulting in inaccurate predictions. e.g. in figure 1 , who is to the right of r.madhavan? these questions can be answered from image features alone and when additional knowledge is given it introduces noise which of-ten confuses the model and subsequently leads to incorrect predictions. similarly, some questions require more triples to reach the correct answer, but providing a fixed number of triples can limit the model's reasoning capabilities due to insufficient information. our approach: to address this, we propose a dynamic triple filtering module capable of retrieving a variable number of triples from knowledge graphs as context to answer the questions. we use an established vision language transformer encoderdecoder (ofa) @xcite model which takes an image, question, and filtered triples as input to predict the desired answer. ## our approach our approach follows a two-stage process to determine the answer for a given question. let a be the set of potential answers, i be the set of images, i be the input image, q be the set of questions, and q be the input question. a * represents the predicted answer where a * ∈ a, and θ represents the learnable parameters of the model. then the predicted answer where p (a|q, i; θ) represents the probability of an answer given a question and the image. p (a|q, i; θ) is computed in two stages, namely, the triple filtering stage and the prediction stage. triple filtering stage: given a question q and an image i, we retrieve a set of triples t * ⊂ t using an iterative retrieval mechanism, where t is the whole set of triples in the knowledge graph. we integrate multi-hop triples, where we specifically focus on utilizing 2-hop triples for contextual information. prediction stage: then we compute the prob-ability of an answer given question, image and relevant triples as: ## experimental setup & results in this section, we explain the results of kvqa and cric-vqa datasets. ## generalisation capability we demonstrate our model's generalization capability by fine-tuning it on the fvqa dataset following pretraining on the kvqa dataset. the primary challenge with the fvqa dataset is the absence of object labels within the dataset itself. so extracting image-relevant triples directly from the knowledge graph by matching the head or tail entity is not possible. we fine-tuned the clip model @xcite to get image-relevant triples. we have ## relevance of knowledge in the context of mllms given the extensive training of mllms on vast datasets, it's natural to assume that external knowledge might not be essential for using them in tasks like kb-vqa. recently, there have been extensive discussions about whether multimodal large language models (mllms), trained on large datasets, can answer kb-vqa questions based solely on their internal knowledge or if external information is necessary. in this section, we illustrate that relying solely on the implicit knowledge within mllms is insufficient for addressing such questions. additionally, we'll demonstrate the effectiveness of our knowledge retrieval method by evaluating its performance with a contemporary vision language model. ## qualitative analysis we'll illustrate how integrating knowledge boosts the ofa model's predictive power while efficiently filtering noise enhances its reasoning capability. we choose some samples as shown in in the final example, regardless of whether knowledge triples are supplied or not, the model produces incorrect answers. this question falls under the spatial and multi-hop categories, requiring the model to make inferences based on both image features and external knowledge. one possible explanation for the inaccurate predictions could be the model's insufficient training to a broad range of questions within these complex categories, hindering its ability to reason effectively in such scenarios. due to space constraints, we provide these and some more examples in ## conclusion we presented a novel approach for kbvqa, utilizing a dynamic triple-filtering module to extract external context from knowledge graphs. our method outperforms the sota on three different kbvqa datasets, achieving an average improvement of 4.75%. we demonstrated that providing the model with a varying number of triples during pre-training or fine-tuning enhances its reasoning capabilities compared to a fixed number of triples. we also showcased the generalization capability of our approach by achieving sota performance on a small dataset using a model trained on a completely different domain. furthermore, we demonstrated that large mllms also require external knowledge for accurate responses. finally, we also demonstrated the effectiveness of our approach on the latest mllm llava, emphasizing that providing a dynamic number of triples improves accuracy by 20% as compared to static ones. the key insight is that dynamically determining the number of relevant triples in the context eliminates noise, resulting in more precise predictions. we also discuss some future explorations in appendix k.