SICAR at RRG 2024: GPU Poor’s Guide to Radiology Report Generation (2024)

Abstract: Radiology report generation (RRG) aims to create free-text radiology reports from clinical imaging. Our solution employs a lightweight multimodal language model (MLLM) enhanced with a two-stage post-processing strategy, utilizing a Large Language Model (LLM) to boost diagnostic accuracy and ensure patient safety. We introduce the “First, Do No Harm” SafetyNet, which incorporates Xraydar, an advanced X-ray classification model, to cross-verify the model outputs and specifically address false negatives from the MLLM. This comprehensive approach combines the efficiency of lightweight models with the robustness of thorough post-processing techniques, offering a reliable solution for radiology report generation. Our system achieved fourth place on the F1-Radgraph metric for findings generation in the Radiology Report Generation Shared Task (RRG24).

## introduction radiology is indispensable in healthcare, offering non-invasive methods to diagnose and monitor medical conditions. central to this practice are radiology reports, which provide detailed interpretations of medical images crucial for clinical decision-making @xcite . however, writing these reports is a meticulous process that demands significant domain expertise @xcite . radiologists must manually review images and formulate descriptive narratives, a task that is 1 https://stanford-aimi.github.io/rrg24/ not only time-consuming but also susceptible to variability and errors, potentially affecting patient care and outcomes @xcite . one of the primary challenges in radiology report writing is the sheer volume of imaging studies that radiologists must interpret @xcite . with the increasing use of imaging modalities such as computed tomography (ct), magnetic resonance imaging (mri), and xray, radiologists are facing a growing workload that exceeds their capacity to provide timely and accurate reports @xcite . this challenge is further compounded by the rising demand for imaging services due to an aging population and the increasing prevalence of chronic diseases. another imperative issue in radiology report generation is the variability in report quality and consistency @xcite . different radiologists may interpret the same set of images differently, leading to inconsistencies in the information provided in the reports. this variability can stem from differences in writing styles, experience levels, and individual biases, all of which can have significant implications for patient care @xcite @xcite . inconsistencies in reports may lead to missed diagnoses or incorrect treatment decisions, underscoring the importance of standardized and automated approaches to report generation. to address these challenges, automated systems have the potential to enhance the efficiency and accuracy of radiology report generation @xcite @xcite . these systems can reduce the time and effort required by radiologists while standardizing reporting practices to ensure consistency and relevance in reports. moreover, automation can help address the increasing workload and demand for imaging services. as large language models (llms) have become widely available, numerous studies have explored the development of multimodal llms (mllms) capable of natively processing additional modalities, such as images @xcite . although there have been significant advancements in the development of mllms for various tasks @xcite , none have specifically focused on lightweight models for the medical domain. local deployment is critical as many hospitals are concerned that uploading images to the cloud for ai processing may violate privacy laws such as the general data protection regulation (gdpr) in europe or the personal data protection act (pdpa) in thailand. addressing this issue is essential to ensure that patients can receive enhanced medical services while maintaining their privacy. additionally, most hospitals in developing countries are gpuconstrained and lack access to high-end gpus which are typically required for deployment. therefore, it is imperative to develop lightweight models capable of performing inference on-premise using consumer-grade gpus. motivated by these challenges, we investigate various architectures with a focus on identifying models that offer the optimal cost-to-performance ratio for local deployment. for the purposes of this study, we concentrate on the task of findings generation. our contributions are summarized as follows: ## conclusion we present our approach to the radiology report generation task in the bionlp 2024 shared task. this study investigates various training configurations and data mixtures to develop lightweight models for generating radiology reports from chest x-ray images. our findings demonstrate that even a smaller model, such as the phi-2 language model, can perform comparably to larger models in the report generation task. additionally, incorporating post-processing techniques significantly enhances the quality of the reports and ensures patient safety. this is particularly crucial for hospitals in resourceconstrained settings. by focusing on models that can be fine-tuned on a single a100 gpu and operated on-premises with a consumer-grade gpu, we address privacy concerns and improve the accessibility of this technology.