MAGIC - VQA : Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering (2025)

Abstract: Visual Question Answering (VQA) necessitates models to reason effectively across visual and textual modalities. However, existing Large Vision-Language Models (LVLMs) often fall short in achieving human-like reasoning due to a lack of integrated commonsense knowledge, limiting their robustness and accuracy in real-world scenarios where both explicit facts and implicit understanding are crucial. To address this challenge, we present MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge, a novel framework designed to enhance multimodal inference by integrating commonsense reasoning. MAGIC-VQA introduces a three-stage process: (1) Explicit Commonsense Knowledge Retrieval from external knowledge graphs, (2) By-Type Commonsense Knowledge Post-Processing to refine contextual relevance, and (3) Implicit Commonsense Knowledge Augmentation using a heterogeneous graph processed by a Graph Neural Network (GNN). These stages collectively enable nuanced, context-aware reasoning without extensive pre-training or intricate prompt tuning.Our MAGIC-VQA significantly improves comprehensive benchmark datasets, surpassing existing models in tasks requiring advanced commonsense reasoning. MAGIC-VQA establishes a robust pathway for integrating commonsense knowledge into VQA, bridging the gap between vision-language inputs and high-level reasoning for improved reliability and contextual accuracy.

## introduction visual question answering (vqa) @xcite @xcite is a complex task requiring models to understand the interaction between visual inputs and textual queries. in recent years, large vision-language models (lvlms) @xcite @xcite @xcite have made substantial progress in vqa through extensive pre-training on massive image-text datasets and instruction tuning. these models excel at object-level visual recognition and semantic understanding, capturing attributes such as spatial relationships and contextual details. nevertheless, lvlms often face challenges on questions requiring commonsense reasoning, particularly those hinging on implicit contextual cues or everyday world knowledge @xcite @xcite foot_0 . to overcome this limitation and improve the performance on , different methods have been explored. for example, multimodal retrieval-augmented generation leverages dense retrieval to inject external multimodal information into the generation process, thereby enhancing the factual grounding of lvlms @xcite . multimodal prompt tuning harnesses the model's innate commonsense knowledge by carefully crafting prompts that combine visual and textual cues from representative samples, guiding lvlms to leverage their internal reasoning for context-rich answers @xcite . however, static prompt design usually lacks the dynamic adaptability required for novel scenarios, resulting in limited generalization to unseen or diverse inputs. additionally, graph-based approaches utilize graph neural networks (gnns) to incorporate structured commonsense knowledge @xcite , which surpasses the limitations of purely parametric lvlms, enabling models to capture explicit and implicit knowledge connections via structured graphs. however, a key missing component in existing works is the effective integration of commonsense knowledge with lvlms while addressing their inherent shortcomings. prior approaches either rely on static retrieval that indiscriminately injects inputunaware noisy knowledge or graph-based augmentation that overlooks the dynamic interplay between external and innate knowledge. our work seeks to fill this gap by proposing a unified framework that systematically combines dynamic, contextu-ally aligned commonsense integration with structured graph-based reasoning to robustly filter and incorporate relevant commonsense knowledge. in this paper, we introduce magic-vqa, a novel framework designed to enhance vqa models by effectively integrating commonsense knowledge with lvlms. magic-vqa is built upon a three-stage process that not only improves reasoning capabilities but also mitigates the complexity associated with large-scale pre-training and inefficient prompt-based approaches. first, explicit commonsense knowledge integration extracts relevant knowledge triples from external sources, establishing a reliable reasoning foundation. secondly, by-type commonsense knowledge postprocessing refines these triples based on inputspecific needs, ensuring contextual relevance. finally, implicit commonsense knowledge augmentation constructs a heterogeneous multimodal graph processed by a gnn to capture intricate relationships, providing structured reasoning beyond what lvlms alone can infer. by integrating explicit and implicit commonsense knowledge on top of lvlms, magic-vqa addresses both the limitations of previous approaches and the missing component in existing works. our main contributions are as follows: ## magic-vqa magic-vqa employs a three-stage process to integrate commonsense knowledge into lvlms, as in figure 1 . (1) explicit commonsense knowledge retrieval extracts relevant triples from an external knowledge graph. ## 250m large vision-language models ## qualitative analysis figure 6 compares magic-vqa with gpt4o and qwen2vl, demonstrating how our framework effectively integrates both explicit and implicit commonsense knowledge for enhanced visual question answering. as illustrated in figure 6a , while gpt4o struggles to deduce the complete answer (big buff ale) to input query, magic-vqa successfully incorporate contextual knowledge, such as "person x owns the tap sells beer" and "beverage dispenser used as a beer tap in a bar," linking beer consumption, tap functionality, and beverage machines with the input question to arrive at the correct answer. in figure 6b , while qwen2vl incorrectly identifies the colony as north carolina, our magic-vqa addresses this limitation by integrating explicit image-based commonsense knowledge about virginia's location, historical turnpikes, and wildlife, correctly concluding that the answer is virginia. confidence scores derived from implicit commonsense knowledge further reinforce the evidence for the final accurate prediction. additional case studies demonstrating the characteristics of each dataset and the benefit from each commonsense knowledge injection component are shown in appendix b. ## conclusion this paper introduced magic-vqa, a novel framework integrating commonsense knowledge into vqa to address the limitations of existing lvlms. magic-vqa's threestage process-knowledge retrieval, commonsense post-processing, and gnn-based augmentation-enables nuanced reasoning without extensive pre-training or complex prompt tuning. evaluations on scienceqa, textvqa, and mmmu demonstrate significant improvements in tasks requiring advanced reasoning. this framework establishes a robust approach for bridging raw visual inputs with high-level reasoning, offering scalable enhancements for vqa. we hope this work inspires further research into structured commonsense reasoning for complex multimodal challenges.