Multimodal Reasoning with Multimodal Knowledge Graph (2024)

Abstract: Multimodal reasoning with large language models (LLMs) often suffers from hallucinations and the presence of deficient or outdated knowledge within LLMs. Some approaches have sought to mitigate these issues by employing textual knowledge graphs, but their singular modality of knowledge limits comprehensive cross-modal understanding. In this paper, we propose the Multimodal Reasoning with Multimodal Knowledge Graph (MR-MKG) method, which leverages multimodal knowledge graphs (MMKGs) to learn rich and semantic knowledge across modalities, significantly enhancing the multimodal reasoning capabilities of LLMs. In particular, a relation graph attention network is utilized for encoding MMKGs and a cross-modal alignment module is designed for optimizing image-text alignment. A MMKG-grounded dataset is constructed to equip LLMs with initial expertise in multimodal reasoning through pretraining. Remarkably, MR-MKG achieves superior performance while training on only a small fraction of parameters, approximately 2.25% of the LLM’s parameter size. Experimental results on multimodal question answering and multimodal analogy reasoning tasks demonstrate that our MR-MKG method outperforms previous state-of-the-art models.

## introduction recently, large language models (llms) @xcite have demonstrated their superiority and robustness across a variety of nlp tasks @xcite @xcite . to further unlock the potential of llms, researchers @xcite @xcite have attempted to endow them with multimodal reasoning capabilities, as exemplified by visual llms like blip-2 @xcite , minigpt-4 @xcite , llava @xcite ., * corresponding author. [generated answer] purple, gold subclass of kobe bryant black mamba los angeles lakers charlotte hornets draft team team alias snake image of … llm [generated answer] yellow [question] what are the representative colors of the team that the basketball player known as the "black mamba" played for? [prompt] (kobe bryant, alias, black mamba), (kobe bryant, team, los angeles lakers) what are the representative colors of the team that the basketball player known as the "black mamba" played for? llm kobe bryant black mamba los angeles lakers charlotte hornets 2023), etc. although these models have made significant strides in enabling reasoning with both images and text, they are still prone to hallucinations @xcite , often caused by inadequate or outdated information. fine-tuning large language models (llms) to update their knowledge base is often a timeconsuming and costly process. an alternative strategy, as suggested by @xcite , involves leveraging knowledge graphs (kgs) as a means to directly augment llms with the requisite knowledge. although recent efforts @xcite @xcite have focused on employing textual kgs, their singular modality limits llms' ability to process and reason with multimodal information (as illustrated in figure 1a ). this limitation leads us to consider the use of multimodal knowledge graphs (mmkgs) instead of textual kgs (see figure 1b ). in this paper, we propose the multimodal reasoning with multimodal knowledge graphs (mr-mkg) method, designed to expand the mul-timodal knowledge of llms by learning from mmkgs. in particular, mr-mkg first encodes the retrieved mmkg using a relation graph attention network (rgat) @xcite , which generates knowledge node embeddings that are able to capture complex graph structures. then, knowledge and visual adapter layers are designed to bridge the cross-modal gap, mapping both knowledge nodes and visual embeddings to word embedding of llms, respectively. finally, embeddings of knowledge nodes, image and text are concatenated to form the prompt and subsequently forwarded to llms to provide guidance and instructions. in addition, we introduce a novel cross-modal alignment module to optimize the image-text alignment through a matching task within mmkgs. to equip the model with initial expertise in multimodal reasoning, we first pretrain mr-mkg on a customized mmkg-grounded dataset, which is constructed by matching each vqa @xcite instance with a corresponding mmkg, derived from the scene graph of its image and containing essential knowledge for answering questions. to thoroughly evaluate our mr-mkg method, we conduct comprehensive experiments on multimodal question answering @xcite and multimodal analogy reasoning @xcite tasks, spanning various llm sizes and training configurations. the experimental results confirm that mr-mkg effectively processes and utilizes knowledge from mmkgs for multimodal reasoning, outperforms previous state-of-the-art models with a 1.95% increase in accuracy and a 10.4% improvement in the hits@1 metric. importantly, mr-mkg freezes both llm and the visual encoder, with only a small fraction of the parameters, approximately 2.25% of the llm's parameter size, being updated. in summary, our main contributions are three-fold: ## method in this section, we begin with an overview of mr-mkg, followed by a detailed description on its architectural design and training approach. ## conclusion in this study, we addressed the challenge of enhancing the multimodal reasoning capabilities of llms through the use of multimodal knowledge graphs. our proposed approach, termed mr-mkg, is designed to empower llms with advanced multimodal reasoning skills by harnessing the rich knowledge (image, text and knowledge triplets) contained in mmkgs. comprehensive experiments on multimodal question answering and multimodal analogy reasoning tasks demonstrated the effectiveness of our mr-mkg approach, achieving the new state-of-the-art results in these tasks. furthermore, we also conducted a series of ablation studies, analytical examinations, and case studies to provide additional evidences of effectiveness.