UTSA - NLP at R ad S um23: Multi-modal Retrieval-Based Chest X -Ray Report Summarization (2023)

Abstract: Radiology report summarization aims to automatically provide concise summaries of radiology findings, reducing time and errors in manual summaries. However, current methods solely summarize the text, which overlooks critical details in the images. Unfortunately, directly using the images in a multimodal model is difficult. Multimodal models are susceptible to overfitting due to their increased capacity, and modalities tend to overfit and generalize at different rates. Thus, we propose a novel retrieval-based approach that uses image similarities to generate additional text features. We further employ few-shot with chain-of-thought and ensemble techniques to boost performance. Overall, our method achieves state-of-the-art performance in the F1RadGraph score, which measures the factual correctness of summaries. We rank second place in both MIMIC-CXR and MIMIC-III hidden tests among 11 teams.

## introduction radiology reports are written documents that record and interpret the results of radiological exams. a radiology report usually consists of a findings section, which contains multiple sentences presenting detailed observations and discoveries made by the radiologist regarding the images acquired from the examination, and an impression section that summarizes the findings with major observations, conclusions or diagnosis @xcite . however, manually summarizing radiology findings into impressions is a time-consuming and error-prone process @xcite . hence, developing tools to augment (i.e., help) radiologist is important. the resulting systems have the potential to significantly improve the efficiency of clinical communication and accelerate the radiology workflow @xcite . * these authors contributed equally to this work. current radiology report summarization methods have generally focused on summarizing radiology reports only from the report findings. @xcite developed a model for automatically summarizing radiology findings into natural language impression statements. more recently, some studies used pre-trained language models such as bart @xcite or pega-sus @xcite for radiology report summarization @xcite . @xcite effectively used a domain adaptation module, an ensemble module, and text normalization heuristics to achieve better summarization results. however, the initial radiology findings may be inaccurate and overlook certain image information. therefore, it is important to explore the use of multimodal frameworks, i.e., methods that incorporate the image, for radiology report summarization. multimodal training faces significant constraints, including uncertainty regarding the necessity of visual-linguistic reasoning and comprehension in completing tasks involving images and text @xcite . training multimodal models has been recognized as a challenging task because it requires the model to comprehensively understand the content of images and bridge the gap between the image and its description @xcite . multimodal models are also susceptible to overfitting due to their increased capacity, and modalities tend to overfit and generalize at different rates @xcite . to overcome the limitations of traditional multimodal training and radiology report summarization, we employ a multimodal encoder where the final model uses the text alone using similarity matching. overall, this leads to a more accurate and comprehensive representation of findings than using the images directly. we use the joint information to fine-tune pre-trained t5-based models and leverage few-shot with chain-of-thought and model ensembles to further bolster our model's performance. ## method the core of our approach involves two stages shown in figure 1 . in the first stage, we retrieve the most similar radiology report based on chest x-ray images and findings from a medical corpus using a pre-trained multi-modal encoder. we then prepend the top k most similar report's findings and impressions to the input example's findings. in the second stage, a text-only model generates the impression based on the new inputs. this approach is inspired by previous work @xcite ; @xcite that treated medical report generation as an image-text retrieval task and utilized a large retrieval corpus to provide sufficient coverage of potential diagnoses of an input chest x-ray image @xcite . by incorporating information from similar radiology reports, our approach aims to generate a more precise impression. ## experiments and results in this section, we describe the data, evaluation metrics, and report results. ## results ## error analysis based on our error analysis, we observe that our model is less accurate at generating negative impressions for radiology reports. positive impressions typically indicate the presence of certain symptoms, whereas negative impressions suggest the absence of symptoms with limited contextual information. however, negative impressions share the same meaning across different anatomies, despite varying input forms. possibly due to the negative impression's simplistic format, our model struggles to capture anatomy and observation semantics meaning, leading to a higher likelihood of predicting "no acute cardiopulmonary process." negative impression example: "findings: no focal consolidation is seen. there is no pleural effusion or pneumothorax. cardiac and mediastinal silhouettes are stable with possible slight decrease in right paratracheal prominence." for example, the ground truth states "no radiographic findings to suggest pneumonia" while our model generates "no acute cardiopulmonary process" as its prediction. the f1radgraph score confirms that there is no similarity between the two. this problem affects both general and clinical domain models, indicating their difficulty in accurately recognizing "pneumonia". negative impression example: "findings: an endotracheal tube, ... there is no change in left lower lobe opacity. there is no large pleural effusion, or pneumothorax. the cardiac silhouette remains moderately enlarged, mediastinal contours are notable for calcification of the aortic arch." when comparing the clinical and general domain ensemble models, we notice that for the groundtruth of "mild residual retrocardiac opacification remains, pneumonia vs. atelectasis", the clinical model's prediction is "unchanged left lower lobe opacity, likely atelectasis, though pneumonia not excluded" (f1radgraph is 30.76), while the general model predicts "no change in left lower lobe opacity" (f1radgraph is 0). this shows the clinical model captures clinically specific semantics. ## conclusion this paper presents our system at task 1b: radiology report summarization of the bionlp 2023 shared task. we build our system on the basis of a multimodal encoder and pre-trained t5-based models. we employ similarity matching to deal with multimodal inputs and generate a more precise description of the given images and findings. additionally, we use few-shot with chain-of-thought and ensemble techniques to further enhance the performance of our approach. as a result, our proposed approach achieved second place in both mimic-cxr and mimic-iii hidden tests with f1radgraph scores of 42.86 and 36.31, respectively. furthermore, we ranked first in the open test with f1radgraph scores of 51.84 on mimic-cxr and 35.25 on mimic-iii, out of 11 teams.