The More, The Better? A Critical Study of Multimodal Context in Radiology Report Summarization (2025)

Abstract: The Impression section of a radiology report summarizes critical findings of a radiology report and thus plays a crucial role in communication between radiologists and physicians. Research on radiology report summarization mostly focuses on generating the Impression section by summarizing information from the Findings section, which typically details the radiologist’s observations in the radiology images. Recent work start to explore how to incorporate radiology images as input to multimodal summarization models, with the assumption that it can improve generated summary quality, as it contains richer information. However, the real effectiveness of radiology images remains unclear. To answer this, we conduct a thorough analysis to understand whether current multimodal models can utilize radiology images in summarizing Findings section. Our analysis reveals that current multimodal models often fail to effectively utilize radiology images. For example, masking the image input leads to minimal or no performance drop. Expert annotation study shows that radiology images are unnecessary when they write the Impression section.

## introduction radiology report acts as a bridge of communication between radiologists and physicians @xcite . having reliable radiology reports could improve communication, enhance patient care, and facilitate research and data analysis in the field of radiology @xcite . a standard radiology report usually contains reason for the exam, comparison (with any available previous exams), findings, and im-pression sections @xcite . specifically, findings section, describes what the radiologist observes in the image(s), and impression section, summarizes important findings and possible causes (differential diagnosis) @xcite are the two most critical sections in the radiology report analysis. earlier work on radiology report summarization aims to generate impression given the findings section as input @xcite @xcite . more recently, researchers @xcite @xcite explored the usage of radiology images as additional context, aiming to improve the generated impression, under the assumption that radiology images could provide richer information to generate a more accurate im-pression section. however, the effectiveness of incorporating radiology images in radiology report summarization remains inconclusive. in this paper, we critically examine the effectiveness of using radiology images in radiology report summarization. our key research question is: to what extent does the radiology image contribute to generating more accurate and informative im-pression section? to this end, we systematically perform controlled experiments with several multimodal summarization models. we first observe that text-only models often outperform multimodal models but masking image inputs lead to minimal or no performance drop, raising concerns about whether visual information is genuinely utilized by multimodal summarization models. we then explore two-stage fine-tuning strategies aimed at encouraging image utilization and introduce an exclusive set, by removing sentences describing critical diagnosis from the text input, enforcing the reliance of radiology images. finally, we conduct an expert annotation study to assess the actual necessity of ra- ## two-stage fine-tuning: encourage image utilization most multimodal models consist of three main components: an image encoder, a language model, and a projection layer that projects visual embeddings ## do medical experts need image to write impression section? to understand the utilization of text and vision modalities by medical practitioners, we conducted an annotation study with the help of two medical experts with over 20 years of experience. we provided 100 samples from mimic-cxr and chexpert (50 randomly selected samples from each dataset) to two annotators. specifically, we provide x-ray images, findings, and impression section, and asked whether it is possible to write a complete impression section from the findings section only, without using the corresponding radiology image. we perform a medical expert annotation study on mimic-cxr and chexpert datasets only, as openi dataset is relatively smaller compared to mimic-cxr and chexpert. it might not reflect the real characteristics of radiology report datasets in a real clinical setting. the annotation interface and guidelines can be found in appendix b. results our annotation study result is presented in our annotators pointed out that one critical information missing from mimic-cxr and chexpert datasets is imaging request, which typically indicates specific clinical questions or reasons for the exam. this context is essential in a real-world clinical setting as it guides radiologists to focus on specific concerns when interpreting radiology images or writing reports. future work on constructing a radiology report summarization dataset could include this information to guide the summarization model in generating a more accurate impression section. ## discussion multimodal radiology summarization is ill-defined in a clinical setting, radiologists first write the findings section based on a radiology image and write the impression section based on the findings section. due to the nature of how the impression section is written, radiology image is often unnecessary when writing the im-pression section. this is confirmed by our medical annotators (section 5). therefore, we argue that a multimodal radiology report summarization task is ill-defined, as in most examples, findings section (text input) have provided enough information to generate an accurate impression section. as we have shown that x-ray images are not required to write the impression section in general, future work on multimodal radiology report summarization should focus on identifying cases that require additional input, other than the findings section. we believe this insight may extend to other multimodal tasks as well, particularly those where multimodal inputs are added to tasks that are originally designed as single modality task, resulting in limited contribution from the additional modality. in such a case, the role of each modality should be carefully evaluated before assuming its necessity and effectiveness in model design, to avoid modality collapse @xcite . multimodal models learn shortcuts and completely ignore one modality from the medical expert annotation study, we confirm that multimodal inputs (findings section and radiology image) in radiology report summarization dataset are highly redundant, where the images are not needed to generate the impression section. these training data causes the model to heavily rely on text input, and ignore image input completely. this is shown by our ablation experiments, where masking the image leads to minimal or no performance changes, indicating that the model relies on text cues in generating the impression section. ## conclusion & future work in this work, we critically examined the effectiveness of radiology images in multimodal radiology report summarization. through extensive model analysis, input ablations, and medical expert annotations, our results highlight the need to reassess how multimodal models utilize different modalities, and how multimodal tasks are formulated, along with the necessity of multimodal inputs. future work should focus on reassessing multimodal task definition, the modality imbalanced problem in datasets, and involving expert annotation to better reflect and align with real-world clinical decisionmaking.