SERPENT - VLM : Self-Refining Radiology Report Generation Using Vision Language Models (2024)

Abstract: Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports. Existing methods often hallucinate details in text-based reports that don’t accurately reflect the image content. To mitigate this, we introduce a novel strategy, SERPENT-VLM (SElf Refining Radiology RePort GENeraTion using Vision Language Models), which improves the R2Gen task by integrating a self-refining mechanism into the MLLM framework. We employ a unique self-supervised loss that leverages similarity between pooled image representations and the contextual representations of the generated radiological text, alongside the standard Causal Language Modeling objective, to refine image-text representations. This allows the model to scrutinize and align the generated text through dynamic interaction between a given image and the generated text, therefore reducing hallucination and continuously enhancing nuanced report generation. SERPENT-VLM outperforms existing baselines such as LlaVA-Med, BiomedGPT, etc., achieving SoTA performance on the IU X-ray and Radiology Objects in COntext (ROCO) datasets, and also proves to be robust against noisy images. A qualitative case study emphasizes the significant advancements towards more sophisticated MLLM frameworks for R2Gen, opening paths for further research into self-supervised refinement in the medical imaging domain.

## introduction radiology report generation (r2gen) serves as a crucial link between medical imaging and natural language processing, to automate the interpretation of radiological images into comprehensive text reports. this task requires models to learn longrange dependencies effectively while generating * equal contribution. the report, a challenge that remains largely unmet in current systems. the primary goal of r2gen is to generate accurate and comprehensive medical reports from radiological imagery, an essential step toward enhancing diagnostic accuracy and efficiency. prevailing methods @xcite @xcite @xcite in r2gen often rely on (1) large datasets for pre-training to impart domain-specific knowledge, and (2) typically utilizing computeintensive encoder-decoder architectures for finetuning. these approaches are fraught with drawbacks, such as omission of minor yet clinically significant details @xcite @xcite and the persistent issue of hallucination as seen in fig. 1 , where generated reports from llava-med and biomedgpt wrongly include details not present in the images. minimizing hallucinations in radiology report generation is crucial since these inaccuracies can lead to misdiagnoses, directly impacting patient treatment plans and outcomes. moreover, reducing hallucinations ensures the reliability and trustworthiness of automated reports, which is vital for maintaining clinical credibility and facilitating effective patient care. therefore, the limitations pertaining to existing approaches underscore the necessity for a more refined approach for accurate medical diagnosis, addressing the critical gaps in r2gen. in this paper, we introduce a streamlined pipeline, serpent-vlm, which begins by processing a given x-ray image by passing it through a visual encoder and mapping it to a vector representation in a high-dimensional space. this process facilitates a nuanced understanding of the medical imagery. the encoded image, alongside a report generation prompt, is then passed as inputs to a large language model (llm) for text generation. we employ a cross-entropy loss for the causal language modeling objective and introduce a novel self-refining objective that leverages the pooled ground truth llava-med biomedgpt serpent-vlm the lungs are hyperexpanded . cardiomediastinal silhouette is within normal limits . no pleural effusion focal airspace opacities or pneumothorax . no free subdiaphragmatic air . heart size and mediastinal contours are within normal limits . pulmonary vascularity is within normal limits . no focal consolidation pleural effusion or pneumothorax identified . visualized osseous structures of the thorax are without acute abnormality . clear and well-defined lung field. ribs are well-defined, no obvious fractures or dislocations. the costophrenic angles are sharp, and there are no pleural effusions. however, there appears to be a small, faint, and irregular shadow in the lower left lung field, which could be indicative of a small nodule or mass. lung fields are clear, with welldefined contours. ribs show no fractures or dislocations, appearing intact. costophrenic angles maintain sharpness, indicating absence of pleural effusions. a slight haziness is noted in the right upper lung field, potentially indicating early signs of infiltrative process. the cardiomediastinal silhouette and pulmonary vasculature are within normal limits in size. the lungs are clear of focal airspace disease, pneumothorax, or pleural effusion. there are no acute bony findings. clear, well-defined diaphragm. slight irregularity, lower right lung lobe, potential small nodule, early pulmonary issue. normal heart shadow, no cardiomegaly. normal-sized aorta, no pneumothorax, no pleural effusion. intact ribs, vertebral column, no fractures, dislocations. centrally located mediastinum, no masses, lymphadenopathy. normal lung vascular markings, no atelectasis, pneumonia. clear, well-defined diaphragm and lung fields, except for faint linearity in the middle right lung, suggesting early fibrotic changes. the heart shadow is normal, excluding cardiomegaly, while the aorta is of normal size with no evidence of pneumothorax or pleural effusion. bones, including ribs and vertebral column, show no fractures or dislocations, despite minimal displacement in the right clavicle. the heart size and mediastinal contours are within normal limits . the lungs are clear without focal airspace opacity pleural effusion or pneumothorax . there is no visible pneumothorax . there is no evidence of acute bony abnormality image representation and the generated report's contextual representation. this allows for tuning the network without compromising inference latency, while significantly improving performance evaluated using metrics such as bleu, rouge l , bertscore. the contributions of our work are summarized as follows: this marks a substantial advancement in the field of r2gen, setting new benchmarks for accuracy, efficiency, and robustness. the remainder of the paper is organized as follows: we begin by delving into the literature review in section 2, focusing on current and past state-of-the-art (sota) methodologies in the domain of radiological report generation. section 3 discusses the proposed strategy for the self-refining fine-tuning our approach. the datasets, baselines, experimental setups, and ablation studies are detailed in section 4. finally, we conclude with a summary of our findings in section 5. we now discuss the proposed methodology in the subsequent section. ## experiments and evaluation we now discuss the details corresponding to the experiments and ablation studies carried out and enumerate the observations. ## summary and conclusion in this paper, we propose serpent-vlm, an innovative method for producing detailed and accurate radiology reports from chest x-rays without hallucinations. the process utilizes a frozen visual encoder to transform x-ray images into a high-dimensional space, which a large language model (llm) then uses to generate initial reports. these reports undergo further refinement through a novel combination of self-refining loss and causal language modeling loss, significantly surpassing existing methods as detailed in section 4. our experiments in section 4 and supplementary materials, confirm the effectiveness of our self-refining approach, even with distorted noisy images. our future works involve the extension of our method to other medical imaging types, such as mris and ct scans, and to incorporate diagnostic radreports to enhance report accuracy further.