Multi-modal Concept Alignment Pre-training for Generative Medical Visual Question Answering (2024)

Abstract: Medical Visual Question Answering (Med-VQA) seeks to accurately respond to queries regarding medical images, a task particularly challenging for open-ended questions. This study unveils the Multi-modal Concept Alignment Pre-training (MMCAP) approach for generative Med-VQA, leveraging a knowledge graph sourced from medical image-caption datasets and the Unified Medical Language System. MMCAP advances the fusion of visual and textual medical knowledge via a graph attention network and a transformer decoder. Additionally, it incorporates a Type Conditional Prompt in the fine-tuning phase, markedly boosting the accuracy and relevance of answers to open-ended questions. Our tests on benchmark datasets illustrate MMCAP’s superiority over existing methods, demonstrating its high efficiency in data-limited settings and effective knowledge-image alignment capability.

## introduction medical visual question answering (med-vqa) aims to provide accurate answers to questions about medical images, playing a crucial role in enhancing intelligent clinical services @xcite . despite the growth in datasets spanning various medical conditions and anatomical areas @xcite , med-vqa encounters significant challenges with open-ended questions due to their complexity and the deep medical knowledge required. traditional approaches @xcite @xcite to med-vqa have largely treated it as a classification task, drawing on predefined set of answers from training datasets, and focus on addressing data scarcity and improving cross-modal reasoning. however, these approaches struggle with the complexity of openended questions, which necessitate comprehension * corresponding author: junwen duan of medical knowledge, covering areas such as the treatment of a particular disease (see figure 1 ) and overlook the importance of integrating external medical knowledge, a gap evident when models fail to address deeper, more nuanced questions. recent developments in language models have shown promise in processing and generating complex text, highlighting a new direction for tackling med-vqa challenges @xcite @xcite . the key challenge now is to effectively combine these advanced language models with medical images, particularly for questions that require medical knowledge. structuring medical concepts and their interrelations through knowledge graphs emerges as a potent strategy to enhance question answering capabilities @xcite . to address these challenges, this study regards med-vqa as a generative task, introducing a novel approach known as multi-modal concept alignment pre-training (mmcap). by utilizing a knowledge graph, derived from extensive medical image-caption datasets @xcite and the unified medical language system @xcite , mmcap achieves the alignment of images with the knowledge embedded in language model and endows the model with comprehensive image understanding capabilities. specifically, this approach is further refined using a graph attention network @xcite and a transformer decoder @xcite , facilitating the encoding of medical knowledge and its interaction with visual features. a crucial aspect of mmcap's fine-tuning phase is the type conditional prompt, which adjusts model responsiveness to the question's nature, enhancing accuracy and contextual relevance in responses. experimental results on two benchmark med-vqa datasets @xcite demonstrate the superior performance of mmcap. it surpasses advanced methods and achieve significant improvements in addressing open-ended questions. further analysis demonstrates its high efficiency in utilizing limited data resources and effective knowledge-image alignment capability. ## method our methodology consists of two main phases: pre-training and fine-tuning. initially, we focus on aligning images and external medical knowledge with textual information through generative pre-training. this phase enhances the model's understanding of medical contexts. the fine-tuning phase then adapts the pre-trained model to the med-vqa task, leveraging the learned capability to answer medical questions accurately. ## conclusion in this work, we redefined med-vqa as a generative task and design a multi-modal concept alignment pre-training (mmcap) method based on the specifics and shortcomings of current methods. with a constructed medical knowledge graph and a knowledge alignment pre-training method, mmcap surpasses existing methods and achieves significant improvements on open-ended questions. further analysis demonstrates its high efficiency in utilizing limited data resources and effective knowledge-image alignment capability.