MMXU : A Multi-Modal and Multi- X -ray Understanding Dataset for Disease Progression (2025)

Abstract: Large vision-language models (LVLMs) have shown great promise in medical applications, particularly in visual question answering (MedVQA) and diagnosis from medical images. However, existing datasets and models often fail to consider critical aspects of medical diagnostics, such as the integration of historical records and the analysis of disease progression over time. In this paper, we introduce MMXU (Multimodal and MultiX-ray Understanding), a novel dataset for MedVQA that focuses on identifying changes in specific regions between two patient visits. Unlike previous datasets that primarily address single-image questions, MMXU enables multi-image questions, incorporating both current and historical patient data. We demonstrate the limitations of current LVLMs in identifying disease progression on MMXU- test , even those that perform well on traditional benchmarks. To address this, we propose a MedRecord-Augmented Generation (MAG) approach, incorporating both global and regional historical records.Our experiments show that integrating historical records significantly enhances diagnostic accuracy by at least 20%, bridging the gap between current LVLMs and human expert performance. Additionally, we fine-tune models with MAG on MMXU- dev , which demonstrates notable improvements. We hope this work could illuminate the avenue of advancing the use of LVLMs in medical diagnostics by emphasizing the importance of historical context in interpreting medical images.Our dataset is released at github.

## introduction stemming from the ever-increasing number of parameters and large-scale training corpus, large vision-language models (lvlms) @xcite @xcite overall figure 1 : the upper part of the figure presents an example from our constructed dataset, which includes two images and a question regarding their changes. the lower part illustrates the performance of human experts and several models on three types of disease progression questions, emphasizing a notable gap between the models and human experts. although current medical lvlms have shown strong performance in downstream tasks such as medical visual question answering (medvqa) @xcite @xcite and medical report generation @xcite @xcite on public benchmarks, they are still limited in responding to basic visual questions (i.e., those involving a single image and a brief description @xcite ). in real-world scenarios, diagnosis often requires physicians to integrate both case history and current evidence @xcite . case history serves as an invaluable source of evidence, encompassing factors such as previous medical conditions, treatments, and patient demographics @xcite . identifying the differences between symptoms, clinical signs, and diagnostic outcomes across the timeline is key to accurate diagnosis. based on these considerations, we ask: can lvlms enhance the diagnostic process by identifying these critical factors? in this paper, we introduce mmxu (multimodal and multix-ray understanding), leveraging the rich resources of patient electronic medical records (emrs) from the mimic-cxr dataset. as shown in figure 1 , unlike previous works @xcite , which focus solely on the current image or are restricted to simple questions, this benchmark is specifically designed to inquire about differences in specific regions between a patient's two visits. mmxu contains two splits: test and dev. mmxu-test consists of 3,000 entries from 1,201 patients and 2,469 studies, while mmxu-dev contains 118k qa pairs involving 114k images. we evaluate a range of open-source and closed-source models and conduct a user study to assess human performance. the results indicate that lvlms struggle to identify differences between two visits, even the lvlm that demonstrates notable performance on other medical benchmarks exhibits a significant performance gap-nearly 40%-when compared to human performance. recognizing the limitations of current lvlms in such scenarios, we take steps toward bridging this gap. since physicians often rely on patient history, previous treatments, and other contextual information during the diagnostic process, we propose a novel approach, medrecord-augmented generation (mag) to facilitate lvlms. we introduce two types of records as contextual information for diagnosis: global records (e.g., patient demographics and overall medical report) and regional records (e.g., specific regional details and diagnostic results). our experiments show that incorporating global or regional medical records leads to a significant accuracy improvement of at least 20% for several well-known models. acknowledging the in-adequate training on such tasks, we apply the mag method for fine-tuning the mmxu-dev dataset and substantiate the effectiveness of both the dataset and the mag approach in enhancing overall performance. in summary, this paper presents three key contributions. first, we introduce mmxu, the first multi-image medvqa dataset designed to investigate differences in specific regions of cxr images between a patient's two visits, addressing the gap between current medvqa benchmarks and realworld clinical scenarios. second, our evaluation results show that current large vision-language models (lvlms), including proprietary models such as gpt-4o, face significant challenges in identifying disease progression between two visits. third, we propose the medrecord-augmented generation (mag) method to enhance the diagnosis of x-rays over time by leveraging the patient's global and regional historical records. experimental results demonstrate the effectiveness of mag, underscoring the potential of incorporating contextual information in diagnostic processes. medvqa dataset on radiology medical visual question answering (medvqa) datasets play a pivotal role in advancing ai-driven clinical decisionmaking. vqa-rad @xcite , as an early pioneering work, introduces a meticulously curated dataset for radiology images, featuring cliniciangenerated questions and corresponding answers tailored to clinically relevant tasks. slake @xcite stands out as a large, bilingual dataset, enriched with extensive semantic annotations and spanning a wide range of radiological modalities. mimic-cxr (johnson et al., 2019) provides a vast collection of 371,920 chest x-rays from 65,079 patients, serving as the foundation for numerous subsequent studies. the comparison of these datasets is shown in ## dataset construction in this section, we outline the pipeline of constructing mmxu, as shown in figure 2 . the process starts with the chest imagenome dataset @xcite , which includes the silver_dataset section containing annotations for 243,310 images from 63,945 patients. these annotations cover bounding boxes for 29 anatomical regions, along with corresponding region-level report phrases, labeled attributes, and relationships. the entire method consists of four distinct phases: (1) comparative sentences extraction ( §3.1), (2) comparative targets selection ( §3.2), (3) qa pairs generation ( §3.3), and (4) post-processing ( §3.4). ## benchmark results to ensure the professionalism and accuracy of mmxu-test benchmark, we recruited a panel of 5 board-certified chest radiology experts to assess it. following that, we evaluated the performance of several prominent open-source and closed-source lvlms capable of supporting multi-image vqa on the mmxu-test benchmark. since all the questions from our benchmark are single-choice, we use accuracy as the metric. ## medrecord-augmented generation through prior experiments, we pinpointed that current lvlms face significant challenges in identifying disease progression. to address this challenge, we propose a novel approach, medrecord-augmented generation (mag). in routine diagnostic practice, physicians often rely on a patient's historical records to inform their analysis and diagnosis of current conditions. to replicate this process, we integrate historical records directly into the prompt as contextual information. our study investigates the effectiveness of global reports derived from historical images, alongside regional reports related to the specified questions, which we categorize as global and regional historical records. ## conclusion in this paper, we introduce mmxu, a dataset designed for multi-modal and multi-x-ray understanding in medvqa. first, we propose a benchmark, mmxu-test, and invite five chest x-ray experts to evaluate the performance. then, we conduct evaluations using several well-known opensource and closed-source large vision-language models (lvlms) that support multi-image vqa. the experimental results indicate that even the best-performing models exhibit a significant performance gap-nearly 40%-compared to human experts. to bridge this gap, we propose the mag method, which leverages historical records to en-hance the understanding of disease progression. we further evaluate its performance both without fine-tuning and with fine-tuning on mmxu-dev and the experiment confirm the effectiveness.