Modality-Aware Integration with Large Language Models for Knowledge-Based Visual Question Answering (2024)

Abstract: Knowledge-based visual question answering (KVQA) has been extensively studied to answer visual questions with external knowledge, e.g., knowledge graphs (KGs). While several attempts have been proposed to leverage large language models (LLMs) as an implicit knowledge source, it remains challenging since LLMs may generate hallucinations. Moreover, multiple knowledge sources, e.g., images, KGs and LLMs, cannot be readily aligned for complex scenarios. To tackle these, we present a novel modality-aware integration with LLMs for KVQA ( \texttt{MAIL} ). It carefully leverages multimodal knowledge for both image understanding and knowledge reasoning. Specifically, (i) we propose a two-stage prompting strategy with LLMs to densely embody the image into a *scene graph* with detailed visual features; (ii) We construct a coupled *concept graph* by linking the mentioned entities with external facts. (iii) A tailored pseudo-siamese graph medium fusion is designed for sufficient multimodal fusion. We utilize the shared mentioned entities in two graphs as mediums to bridge a tight inter-modal exchange, while maximally preserving insightful intra-modal learning by constraining the fusion within mediums. Extensive experiments show the superiority of \texttt{MAIL} .

## introduction knowledge-based visual question answering (kvqa) aims to provide appropriate answers for questions about images based on external knowledge @xcite , such as knowledge graphs (kgs) @xcite . it has various applications, especially for assisting the visually impaired users @xcite , yet still, remains a challenging task that requires complex reasoning capability across different data modalities @xcite ). * corresponding author † equal contribution. recently, several studies have explored using large language models (llms) as supplementary knowledge bases and reasoning tools for kvqa @xcite @xcite ; according to how they fuse the knowledge, they can be broadly categorized into direct prompting and modality-agnostic approaches, shown in figure 1 (a) and (b), respectively. the former directly prompts the question and the corresponding image caption to llms for answers @xcite . the latter leverages llms to generate candidate answers with supporting evidence and simply combines both question and the external knowledge embedding, e.g., wikidata @xcite , for reasoning at the final stage @xcite . while the above methods have employed llms in various ways for kvqa, we argue that they have not fully leveraged the knowledge from llms and lack the cross-modal reasoning ability, potentially resulting in sub-optimal performance for complex vqa scenarios. (i) llms could incorrectly answer questions or provide unreliable evidence for reasoning. on the one hand, direct prompting to llms may struggle to identify the right answer for many complex or domain-specific questions, due to the lack of domain knowledge @xcite . on the other hand, llms may be prone to generating hallucination @xcite and producing misleading evidence in support of candidate answers. (ii) integrating multimodal knowledge in a modality-agnostic manner can be sub-optimal. specifically, existing methods simply concatenate different modal representations, e.g., questions, captions, tags, and external knowledge, for reasoning. this design lacks the necessary cross-modal exchange to enrich the semantics of entities, limiting the final reasoning performance. for example, to correctly answer the question in figure 1 , the model is required to infer the season based on a cross-modal understanding of the inputs, such as the "keep warm" purpose of "coat" and the "spring blooming" feature of "sakura". in this work, we study the following research question: how can we effectively leverage the knowledge from llms to enhance the comprehensive understanding and reasoning of the images and questions in kvqa? answering this question is nontrivial due to the following challenges. (i) it is hard to properly incorporate the knowledge from llms. llms may generate hallucinations when dealing with requests that are not covered in their training corpus. simply prompting them may generate noisy and irrelevant responses. (ii) semantic alignment of multiple knowledge sources is challenging. given image captions, object/region features, external knowledge from kgs, and implicit knowledge from llms, appropriately aligning relevant semantic information in different modalities cannot be readily achieved. to tackle these challenges, we present a novel modality-aware framework to effectively integrate llms for kvqa in figure 1 (c ), dubbed mail. specifically, (i) we propose a two-stage prompting strategy to maximally leverage the knowledge from llms for image understanding. we initialize a dense caption by prompting a visual llm, e.g., visual chatgpt @xcite and minigpt-4 @xcite . to depict the detailed visual scenes in the caption, we construct a scene graph by defining twelve condensed relations and prompting the llm to extract spatial and object features accordingly in the form of triples, e.g., (sakura, at_location, tree). (ii) we integrate the external knowledge from kgs to form a coupled concept graph, where the mentioned entities in scene graphs are linked with real-world assertions and facts to facilitate knowledgeable reasoning, such as (coat, used_for, keep warm) and (sakura, typle_of, spring blooming). (iii) a tailored pseudo-siamese graph medium fusion is designed for effective multimodal graph fusion. inspired by the success of pseudo-siamese network in measuring the similarity of two correlative inputs @xcite , we extend it to graphs to process intra-modal information. it consists of two graph attention networks with the same architecture but different weights. in each sub-encoder, we concentrate on one modality and design a tailored context-aware propagation. this guides our model to attentively prioritize the most valuable entities subject to the particular question. then we leverage the shared mentioned entities in both coupled graphs as mediums to bridge the cross-modal interaction. the model continuously exchanges their embeddings between two modalities, bringing sufficient complementary knowledge to the other modality respectively. it merely allows inter-modal exchanging by constraining it within the mediums. in general, mail effectively enhances a tight intermodal fusion while maximally preserving the insightful intra-modal information for each modality. our major contributions are summarized below: ## methodology in this section, we introduce the detailed rationale of our proposed framework. an illustration of mail is shown in figure 2 . we first carefully leverage the knowledge from llms for coupled graph construction. then, we formulate the pseudo-siamese graph medium fusion (ps-gmf). through an effective integration of two tailored training objectives, we jointly optimize the model for accurate prediction. ## experiments in this section, we conduct a variety of experiments to demonstrate the effectiveness of our proposed mail. we aim to answer four research questions: ## conclusions we present mail, a modality-aware integration with large language models for knowledge-based visual question answering. we formally define a novel multimodal learning paradigm for comprehensive cross-modal reasoning among multiple knowledge sources. the knowledge from llms is effectively leveraged via a carefully designed prompting for coupled graph construction, i.e., scene graph and concept graph. then we integrate various multimodal information with a tailored pseudo-siamese graph medium fusion. it effectively enhances a tight inter-modal interaction and maximally preserves insightful intra-modal processing. mail achieves superiority on two benchmark datasets while requiring 2∼ 4× faster inferential time than the existing state-of-the-art baselines.