X ray GPT : Chest Radiographs Summarization using Large Medical Vision-Language Models (2024)

Abstract: The latest breakthroughs in large language models (LLMs) and vision-language models (VLMs) have showcased promising capabilities toward performing a wide range of tasks. Such models are typically trained on massive datasets comprising billions of image-text pairs with diverse tasks. However, their performance on task-specific domains, such as radiology, is still under-explored. While few works have recently explored LLMs-based conversational medical models, they mainly focus on text-based analysis. In this paper, we introduce XrayGPT, a conversational medical vision-language (VLMs) model that can analyze and answer open-ended questions about chest radiographs. Specifically, we align both medical visual encoder with a fine-tuned LLM to possess visual conversation abilities, grounded in an understanding of radiographs and medical knowledge. For improved alignment of chest radiograph data, we generate ~217k interactive and high-quality summaries from free-text radiology reports. Extensive experiments are conducted to validate the merits of XrayGPT. To conduct an expert evaluation, certified medical doctors evaluated the output of our XrayGPT on a test subset and the results reveal that more than 70% of the responses are scientifically accurate, with an average score of 4/5. We hope our simple and effective method establishes a solid baseline, facilitating future research toward automated analysis and summarization of chest radiographs. Code, models, and instruction sets will be publicly released.

## introduction the large-scale vision-language models have emerged as a transformative area of research at the intersection of computer vision and natural language processing, enabling machines to understand and generate information from both visual and textual modalities. these models represent a significant advancement in the field, bridging the gap between visual perception and language comprehension, and have demonstrated remarkable capabilities across various tasks, including but not limited to image captioning @xcite , visual question answering @xcite , and visual commonsense reasoning @xcite . training these models requires vast amounts of image and text data, enabling them to learn rich representations that capture the intricacies of both modalities. additionally, fine-tuning can be employed using task-specific data to better align the models with specific end tasks and user preferences. recently, bard and gpt-4 have demonstrated impressive capabilities in various tasks, raising excitement within the research community and industry. however, it is important to note that the models of bard and gpt-4 are not currently available as open-source, limiting access to their underlying architecture and implementation details. recently, mini-gpt @xcite demonstrated a range of impressive capabilities by aligning both vision and language models. it excels at generating contextual descriptions based on the given image. however, it is not as effective in medical scenarios due to the significant differences between medical image-text pairs and general web content. adopting vision-text pre-training in the medical domain is a challenging task because of two factors: (1) lack of data, mini-gpt has trained the projection layer on a dataset of 5m image-text pairs, while the total number of publicly available medical images and reports is orders of magnitude below. (2) different modalities and domains, while mini-gpt may involve distinguishing between broad categories like "person" and "car" the distinctions within medical domains are much more subtle and fine-grained. for instance, differentiating between terms like "pneumonia" and "pleural effusion" requires more precision by capturing and aligning relevant medical domain knowledge. chest radiographs are vital for clinical decisionmaking as they offer essential diagnostic and prognostic insights about the patients. text summarization tasks can partially address this challenge by providing meaningful information and summaries based on the given radiology reports. in our approach, we go beyond traditional summarization techniques by providing concise summaries that highlight the key findings and the impression based on the x-ray. additionally, our model allows for interactive engagement, enabling users to ask followup questions based on the provided answers. we argue that based on the visual and large language models, the majority of knowledge acquired during the pertaining stage of these models requires a domain-specific high-quality instruction set derived from task-specific data to achieve promising results. the main contributions of our work are:-• we generate interactive and clean summaries ( 217k) from free-text radiology reports of two datasets: mimic-cxr @xcite and openi @xcite . these summaries serve to enhance the performance of our xraygpt by fine-tuning the linear transformation layer on high-quality data. ## method xraygpt is an innovative conversational medical vision-language model designed for analyzing chest radiographs. our approach draws inspiration from the design of vision-language models in general, but with a specific focus on the medical domain. due to the limited availability of medical image-summary pairs, we adopt a similar methodology by building upon a pre-trained medical vision encoder (vlm) and medical large language model (llm), as our foundation. the fine-tuning process involves aligning both modalities using high-quality image-summary pairs through a simple transformation layer. this alignment enables xraygpt to possess the capability of generating insightful summaries about chest radiographs. ## conclusion to conclude, we introduce xraygpt, an innovative medical vision-language model that combines both vision-language modalities to summarize and answer inquiries regarding chest radiographs. by aligning both modalities and leveraging our proposed interactive summaries derived from free-text radiology reports, xraygpt demonstrates exceptional visual conversation abilities grounded in a deep understanding of chest radiographs.