Gla- AI 4 B io M ed at RRG 24: Visual Instruction-tuned Adaptation for Radiology Report Generation (2024)

Abstract: This paper introduces a radiology-focused visual language model designed to generate radiology reports from chest X-rays. Building on previous findings that large language models can acquire multimodal capabilities when aligned with pretrained vision encoders, we demonstrate similar potential with chest X-ray images. The model combines an image encoder (CLIP) with a fine-tuned large language model (LLM) based on the Vicuna-7B architecture. The training process involves a two-stage approach: initial alignment of chest X-ray features with the LLM, followed by fine-tuning for radiology report generation. The study highlights the importance of generating both FINDINGS and IMPRESSIONS sections in radiology reports and evaluates the model’s performance using various metrics, achieving notable accuracy in generating high-quality medical reports. The research also addresses the need for domain-specific fine-tuning to capture the intricate details necessary for accurate medical interpretations and reports.

## introduction radiology reports constitute the primary medium through which radiologists convey the findings and conclusions derived from radiography, such as chest x-rays. these reports play a pivotal role in the diagnostic and therapeutic processes across a wide range of diseases, emphasizing their significance in contemporary medical practice @xcite . structured to enhance clarity and efficacy in medical communication, radiology reports primarily feature findings and impressions sections @xcite . the findings section details the critical observations of the radiologist on the image, while the impressions section summarizes the conclusions and recommendations of the radiologist. these sections collectively ensure that radiology reports are indispensable in diagnostic and therapeutic decision-making, combining image analysis and clinical insight. ## related work nowadays, exemplified by open-source projects such as llava @xcite , the effectiveness of self-supervised vision-language models (vlms) using parallel data has been demonstrated in different research domains. these vlms, when instruction-tuned with multimodal inputs, align well with human intentions and perform robustly in various downstream tasks, including converting images to text @xcite . however, the unique characteristics of biomedical image-text pairs significantly differ from those in general domains. biomedical images often contain subtle and complex features that require precise interpretation, while the corresponding text must convey highly specific medical information @xcite . in biomedical settings, vlms designed for general domains often fail to meet these specialized needs, as they lack the ability to accurately interpret medical data and generate relevant clinical descriptions @xcite . this discrepancy underscores the urgent need for domain-specific fine-tuning. by tailoring vlms to the distinct demands of the biomedical field, such fine-tuning can enhance their ability to capture and convey the intricate details necessary for accurate medical interpretations and reports. recent advancements have been made in adapting general-purpose foundation models for medical applications, particularly in radiology. the med-flamingo @xcite , an extension of the openflamingo framework @xcite , leverages images and captions from medical textbooks to enhance few-shot visual questionanswering capabilities. similarly, the med-palm m, developed by @xcite , fine-tuned the palm-e model @xcite using comprehensive biomedical datasets. llava-med, proposed by @xcite , modifies the llava @xcite framework with image-text pairings and multimodal instructions from pubmed data. additionally, the elixr model, developed by @xcite , integrates the supcon cxr encoder @xcite with the palm 2-s model @xcite to support classification, semantic search, question answering, and quality assurance. finally, the radiology-gpt, created by @xcite , utilizes radiology reports from mimic-cxr @xcite to facilitate the generation of findings-to-impression text, based on the alpaca instruction-tuning framework @xcite . historically, research in radiology report generation has varied, with some studies focusing exclusively on either the findings or the impressions sections @xcite 2023 ) covered all settings, demonstrating that the choice of sections significantly influences reported performance metrics, complicating comparative evaluations across different study designs. however, these existing models have limitations. most notably, they are typically designed to process single images and often fall short in generating reports that match the depth and detail of those written by human radiologists. additionally, they do not fully replicate the workflow of medical professionals, who often reference multiple images to enhance report accuracy. our work addresses these gaps by developing a model capable of handling multiple images simultaneously and generating comprehensive radiology reports. this approach aims to more closely mimic the process used by medical professionals, thereby improving the accuracy and quality of the generated reports. ## methodology in our study, we follow the observations from llava-med @xcite , suggesting superior performance when initiating with a language-only pretrained llm rather than a multimodal-trained base. our model architecture incorporates an image encoder and a learnable adapter placed atop the image outputs, mirroring the llava-1.5 model design @xcite . we adopt an auto-regressive language modelling approach using cross-entropy loss @xcite and align hyperparameters with those from llava-1.5, including a joint tuning phase for the llm and adapter @xcite . in alignment with llava-1.5 protocols, we initially pretrain the adapter alone for one epoch, followed by a full training cycle lasting three epochs, employing low-rank adaptation of large language models techniques (lora) @xcite for efficient parameter tuning. ## results we report the performance of our two proposed models over five evaluation metrics in ## discussion performance disparities observed between the findings and impressions sections of the test results can be attributed to several factors. firstly, the impression and findings sections address distinct medical purposes, resulting in performance disparities. the findings section offers an objective description of symptoms, while the impressions are oriented towards diagnostic conclusions. the variability in word count between these sections also affects the complexity of model inference, as reflected in the lexical evaluation scores. additionally, significant discrepancies in the medical evaluation metrics highlight a varied distribution of diseases within the test set. this heterogeneity could impact the generalisability and accuracy of the model. furthermore, our analysis indicates that the performance may be compromised in multi-image inference scenarios where it does not account for superfluous images. such factors are essential to consider when assessing the diagnostic accuracy and reliability of the model in clinical settings. enhancing the ability of model to differentiate between relevant and superfluous images could significantly improve diagnostic accuracy. furthermore, exploring domain-specific adaptations and fine-tuning strategies tailored to the unique characteristics of medical data could further enhance model performance. incorporating temporal dynamics into the model to capture changes 2 https://vilmedic.app/misc/bionlp24/leaderboard over time and developing more sophisticated frameworks for generating multi-modal radiology reports are other promising avenues for future research. these advancements are expected to enhance both the practicality and accuracy of our model within clinical scenarios. ## conclusion in this work, we have developed a vision-language model capable of processing multiple images. through visual instruction tuning, we achieved alignment between two modalities and further finetuning for specific downstream tasks. notably, our system attained a commendable fourth-place standing across four diverse test datasets at the rrg24 at bionlp 2024 workshop @xcite , substantiating the practicality of vision-language models within specialized medical tasks. moving forward, we intend to conduct in-depth research into more sophisticated methods for generating multi-modal radiology reports. this will involve incorporating temporal dynamics and developing frameworks specifically focused on text generation. such advancements are expected to enhance both the practicality and accuracy of our model within the clinical scenario.