ML e VLM : Improve Multi-level Progressive Capabilities based on Multimodal Large Language Model for Medical Visual Question Answering (2024)

Abstract: Medical visual question answering (MVQA) requires in-depth understanding of medical images and questions to provide reliable answers. We summarize multi-level progressive capabilities that models need to focus on in MVQA: recognition, details, diagnosis, knowledge, and reasoning. Existing MVQA models tend to ignore the above capabilities due to unspecific data and plain architecture. To address these issues, this paper proposes Multi-level Visual Language Model (MLeVLM) for MVQA. On the data side, we construct a high-quality multi-level instruction dataset MLe-VQA via GPT-4, which covers multi-level questions and answers as well as reasoning processes from visual clues to semantic cognition. On the architecture side, we propose a multi-level feature alignment module, including attention-based token selector and context merger, which can efficiently align features at different levels from visual to semantic. To better evaluate the model’s capabilities, we manually construct a multi-level MVQA evaluation benchmark named MLe-Bench. Extensive experiments demonstrate the effectiveness of our constructed multi-level instruction dataset and the multi-level feature alignment module. It also proves that MLeVLM outperforms existing medical multimodal large language models.

## introduction medical visual question answering (mvqa) is an interdisciplinary problem that combines cv and nlp which requires deep understanding of medical images and questions to provide reliable answers. recently, lots of works have constructed professional mvqa datasets @xcite @xcite , 2021). however, limited by the number of parameters, the past vqa models usually focus on a specific dataset and cannot cover as many scenarios as possible. with the rapid development of large language models, multimodal large language models (mllms) have become an important solution for solving vqa tasks @xcite @xcite , due to the abundant number of parameters, large amount of pre-trained contents and excellent text generation capabilities. similarly, by directly substituting the training data, these general domain methods have been directly transferred to the medical domain @xcite @xcite . more challenging than the general domain vqa is that when doctors conduct mvqa in real scenarios @xcite , they first need to focus on the key parts in the image, and further consider the detailed information afterward. based on the visual clues acquired, they make a diagnosis of the disease, and then further answer knowledge-based questions such as how to treat it. we summarize these progressive capabilities in the following five aspects: recognition, details, diagnosis, knowledge and reasoning. they cover a cascade of capabilities from low-level vision to high-level semantics, simulating the thought process of a doctor in answering a medical question. as is illustrated in figure 1 , for a complex semantic question ( "how to treat the disease shown in the image?"), obtaining its answer is not straightforward, but involv-ing a sequence of implicit complex reasoning or medical knowledge. that is, recognize the objects ("what are the main anatomical structures visible in this image?"), extract local details ("what is the appearance of the heart?"), diagnose based on acquired visual clues ("based on this image, what diseases may the patient suffer from?"), consider knowledge of specific diseases (" how to treat dilated cardiomyopathy?"), and finally integrate the information obtained from the above-mentioned processes to form a reasoning process. however, existing mvqa models do not have the multi-level progressive capabilities and suffer from the following two problems: 1) unspecific data. the public training data @xcite used by the model is often derived from paper crawlers and lacks instruction data for specific construction, especially for multi-level progressive capabilities. 2) plain architecture. existing multimodal large language models usually align visual features and text features through simple linear layers @xcite . since different capabilities focus on different modal features, it is difficult to effectively align multimodal feature with a simple linear layer. therefore, in this paper, we propose multi-level visual language model for mvqa (mlevlm), which aims to improve the multi-level capabilities in medical visual question answering. on the data side, we first carefully construct a dataset containing 60k high-quality multi-level instructions called mle-vqa. the dataset covers multi-level progressive capabilities: recognition, details, diagnosis, knowledge and reasoning. on the architecture side, we propose a multi-level feature alignment module to effectively align features from visual to semantic. specifically, it contains an attention-based token selector that focuses on important areas and a context merger that fuses visual details. to better evaluate the model's capabilities of different levels, we manually construct a multilevel evaluation benchmark named mle-bench. experimental results on the benchmark demonstrate the effectiveness of our constructed multilevel instruction dataset and the multi-level feature alignment module. it also proves that mlevlm outperforms existing medical mllms. our contributions are as follows: ## model architecture existing multi-modal large language models usually align visual features and text features through simple linear layers. to effectively align features of different levels from visual to semantic in medical vqa, we design a multi-level feature alignment (mlfa) module. specifically, we add attentionbased token selector and context merger to the widely used visual encoder and text encoder to improve the model's focus on key areas and visual details. we then use two projectors to map the lowlevel visual information and high-level semantic information into the large language model. the model architecture is shown in figure 3 . ## qualitative analysis case study on different levels. to compare the capabilities between mlevlm and existing medical mllms, we conduct case study on questions of different levels. plot (a) of figure 4 shows the models' answers to the detail-based questions. it can be seen that both llava-med and mlevlm generate correct answers, while med-flamingo has errors in its answers. plot (b) of figure 4 shows the models' answers to the knowledge-based questions. mlevlm can effectively give the treatment suggestions, while the other models can't. more case studies in the appendix g.2 demonstrate that our model can have greater multi-level capabilities. case study on reasoning capability. we also conduct the case study on reasoning capability. plot (c) of figure 4 shows the reasoning capability of the models. it can be seen that llava-med and med-flamingo have the correct answers but lack the correct reasoning process. mlevlm gives a detailed and reliable progressive reasoning process while ensuring the correct answer. more examples in the appendix g.2 can demonstrate the strong reasoning capability of mlevlm compared to the existing medical mllms. ## conclusion in this paper, we summarize the multi-level progressive capabilities of medical visual question answering based on real scenarios for the first time. we propose mlevlm, optimized in terms of both data and architecture. extensive experiments based on our manually constructed multi-level evaluation benchmark, mlebench, demonstrate that mlevlm is the best performing medical mllm at present. meanwhile, case studies and human evaluation experiments demonstrate that mlevlm exhibits great progressive reasoning capability.