Query-Driven Multimodal G raph RAG : Dynamic Local Knowledge Graph Construction for Online Reasoning (2025)

Abstract: An increasing adoption of Large Language Models (LLMs) in complex reasoning tasks necessitates their interpretability and reliability. Recent advances to that end include retrieval-augmented generation (RAG) and knowledge graph-enhanced RAG (GraphRAG), whereas they are constrained by static knowledge bases and ineffective multimodal data integration. In response, we propose a Query-Driven Multimodal GraphRAG framework that dynamically constructs local knowledge graphs tailored to query semantics. Our approach 1) derives graph patterns from query semantics to guide knowledge extraction, 2) employs a multi-path retrieval strategy to pinpoint core knowledge, and 3) supplements missing multimodal information ad hoc. Experimental results on the MultimodalQA and WebQA datasets demonstrate that our framework achieves the state-of-the-art performance among unsupervised competitors, particularly excelling in cross-modal understanding of complex queries.

## introduction the integration of knowledge graphs (kgs) with llms (llms) through retrieval-augmented generation (rag), known as graphrag, has emerged as a promising paradigm for enhancing factual accuracy and reasoning capabilities @xcite . while traditional rag systems leverage unstructured text retrieval, recent graphrag approaches attempt to combine structured knowledge graphs (kgs) with llms to capture complex entity relationships and enable multihop reasoning @xcite . to do that, the existing methods construct kgs typically in a bottom-up, information-driven fashion @xcite , which begins by identifying available data sources and integrating them , and document data for online reasoning. answering the query requires identifying key details from a table to identify the theme person when colton dixon performed 'piano man'. then, this information must be corroborated with an image to determine his emotional expression. this highlights the difficulty in building knowledge graphs that can fully cover all aspects of a query without knowing the exact question in advance. such inefficiencies motivate us to pursue a more dynamic and query-driven approach for kg construction. in this paper, we draw insight from cognitive science. that is, human reasoning often employs top-down, goal-driven attention to prioritize task-relevant information, so as to filter and integrate the most pertinent knowledge @xcite . to instantiate this insight, we propose an unsupervised query-driven multimodal graphrag framework that promotes efficient llm-kg integration through three key components: 1) dynamic graph pattern construction: by analyzing query semantics, our framework derives entity-relationship patterns to guide knowledge extraction, ensuring focused and selective retrieval. 2) multimodal selective attention: we implement a multi-path filtering strategy to identify the most relevant information across textual, visual, and tabular modalities. 3) iterative knowledge refinement: our framework iteratively supplements missing information through cross-modal reasoning, creating a comprehensive and adaptive knowledge representation. contributions of this paper are as follows: ## related work rag has emerged as a key paradigm for addressing hallucination issues in llms by integrating external knowledge bases into the generative process. traditional rag systems retrieve relevant documents based on user queries and use them as context for llms to generate grounded responses @xcite . while effective, these systems frequently assume that relevant information is localized within specific text regions, which fails to capture cross-document or long-range contextual dependencies @xcite . moreover, their reliance on text matching for retrieval commonly results in fragmented results, limiting their effectiveness for multi-hop reasoning tasks @xcite . the integration of kgs with rag, known as graphrag, addresses many limitations of traditional rag by leveraging structured knowledge representations. unlike conventional rag, which operates on raw text, graphrag constructs kgs from documents, explicitly modeling semantic relationships between entities to support multi-hop reasoning and improve interpretability @xcite . recent advancements in graphrag have focused on enhancing both retrieval and reasoning capabilities: (1) graph-based retrieval: methods like @xcite propose entity graphs and community summaries to improve diversity and comprehensiveness in large-scale text processing. (2) query optimization: approaches such as @xcite integrate historical query-based kgs to preserve structural relationships and optimize retrieval. (3) efficient architectures: ligh-trag @xcite introduces a dual-layer retrieval system that incorporates kgs into text indexing, balancing efficiency and retrieval quality. (4) reasoning enhancement: the kg-rag framework @xcite employs the chain of explorations (coe) algorithm to combine structured kgs with llms, improving performance on long-text and multi-hop reasoning tasks. despite these advancements, a critical challenge remains in effectively integrating multimodal data into kgs. most graphrag frameworks focus primarily on textual knowledge, overlooking the structured incorporation of images, tables, and other modalities. while recent efforts like murag @xcite have begun exploring multimodal rag, seamless fusion of heterogeneous data at the kg level remains an open has poster acted in released on tell me that you ... ben piazza the bad news bears 1970 nightwing 1979 released on image depicts half a woman's face. image depicts half a woman's face. round 0 round 1 -goal-can kg answer q? content: {kg} q: {question} answer: [yes/no] visual enhancement tabular mining ben piazza the bad news bears mined image depicts half a woman's face. image depicts half a woman's face. graph expansion 1979 the concorde ... the bad news bears 1985 discovered round n tell me that you ... ben piazza acted in the bad news bears 1970 nightwing 1979 released on image depicts half a woman's face. image depicts half a woman's face. 1985 the concorde... no yes acted in material missing. he is smiling to indicate he is happy, what does the person who was the theme for the episode where colton dixon chose "piano man" do? evidence thinking imgs: txt1:piano man ... txt2:drama critic ... [docs 1] music man [docs 2] mr. music [docs 3] piano man piano man is a song written and performed by american... image understanding failure. txt3:the music... table: extract all triplet information from materials. extract triplets from materials based on pattern graph. song person television episode emotion i'm in the mood for love piano man billy jole miranda grosvenor billy jole image piano man colton dixon vector retrieval rag retrievaled query & library extract triplet graphrag evidence extract triplet by pattern graph ours evidence he is smiling he is smiling semantic retrieval character match visual evidence intergration stage ii-phase 2: stage iii：rterative knowledge refinement multimodal knowledge base problem. this limitation is particularly evident in complex queries requiring cross-modal reasoning, where existing methods struggle to synthesize information from diverse sources. the lack of dynamic kg construction methods further exacerbates this issue, as static kgs cannot adapt to new knowledge or evolving query contexts. ## experiments to evaluate the effectiveness, we conduct experiments on two multimodal multihop questionanswering datasets: multimodalqa @xcite and webqa @xcite , employing evaluation metrics of em and f1 for mul-timodalqa, and qa-fl, qa-acc, and qa-f1 for webqa. we use bge-m3 @xcite ) for text embeddings and clip @xcite for image embeddings. the qwen 2.5 72b model @xcite handles graph construction, while llava 34b @xcite and gemini-1.5-flash @xcite ) manage crossmodal reasoning. all experiments are conducted on nvidia a100 gpus with 80gb memory. we compare our method with two categories of approaches: (1) supervised: ar @xcite , id @xcite , mgt @xcite , mmhqa-icl @xcite , preasm-large @xcite , hpopro @xcite , vlp-x101fpn @xcite , vlp-vinvl @xcite , murag @xcite ; and (2) unsupervised: vicuna-7b @xcite , llama2chat-13b @xcite , openchat-v2-w-13b (wang et al., 2024), moqagpt @xcite , binder @xcite , ofa-cap (wang et al., 2022), and promptcap @xcite . the results of the comparison methods are taken from the original papers. ## conclusion this paper proposes a new approach to enhance llm performance in complex multimodal question answering tasks. the proposed query-driven local multimodal kg construction approach is inspired by cognitive science modeling of goal-directed user behaviors and their selective attention. our approach realizes efficient multimodal knowledge acquisition and integration through three key components, including graph pattern construction, multipath retrieval, and dynamic reasoning. experimental results on two challenging datasets, multi-modalqa and webqa, demonstrate that our proposal is on a par with the state-of-the-art supervised methods while outperforming its unsupervised competitors by considerable margins.