KU - DMIS - MSRA at R ad S um23: Pre-trained Vision-Language Model for Radiology Report Summarization (2023)

Abstract: In this paper, we introduce CheXOFA, a new pre-trained vision-language model (VLM) for the chest X-ray domain. Our model is initially pre-trained on various multimodal datasets within the general domain before being transferred to the chest X-ray domain. Following a prominent VLM, we unify various domain-specific tasks into a simple sequence-to-sequence schema. It enables the model to effectively learn the required knowledge and skills from limited resources in the domain. Demonstrating superior performance on the benchmark datasets provided by the BioNLP shared task (Delbrouck et al., 2023), our model benefits from its training across multiple tasks and domains. With subtle techniques including ensemble and factual calibration, our system achieves first place on the RadSum23 leaderboard for the hidden test set.

## introduction chest radiography is a widely used imaging modality for assessing the thorax and diagnosing cardiopulmonary conditions. however, there is a significant shortage of clinical doctors in several under-resourced regions, delaying diagnosis and treatment and reducing the quality of care. developing an automated system for analyzing radiographs can improve radiologist workflow efficiency and expand healthcare services to these regions. to promote research in this direction, the bionlp 2023 workshop opens a new shared task called radsum23 @xcite for the radiology report summarization. 1 in this task, participants are asked to build a model that takes a findings section as input, which is a generic radiology report of a given x-ray image, and then outputs an impression section, which is a summary of key observations in the given report. high-resolution x-ray images can be used as input along with findings sections. the radiology report summarization task aims to effectively distill complex clinical observations from chest x-ray images into concise and coherent summaries. in this paper, we introduce chexofa (one for all tasks with chest x-ray), a novel visionlanguage model designed for the chest x-ray domain. specifically, we initialize our model using ofa @xcite , a transformer model pre-trained with a unified sequence-to-sequence schema on diverse uni-or cross-modal tasks such as image classification, language modeling, and image captioning. we further train the model to generate full-text reports from the chest x-ray image using the mimic-cxr dataset @xcite . then, we fine-tune the model on the radiology report summarization task. when summarizing the report, our model jointly encodes visual information from the chest x-ray image with the full-text report, taking advantage of its multimodal nature. additionally, we employ subtle techniques such as task-specific ensemble @xcite and factual calibration to further improve the model performance. our experiments demonstrate that our proposed methods largely enhance model performances on two test sets of the shared task. on the official leaderboard foot_0 for mimic-cxr hidden test set, our system ranked first place, achieving the state-ofthe-art performances in most evaluation metrics. especially, it surpasses the second-best model by 2.3 and 2.9 in bleu and f1-chexbert score. in the ablation study, we demonstrate how much each method contributes to the improvement. ## chexofa we propose a novel vision-language model, chex-ofa, specifically designed for the radiology report summarization. we first initialize our model parameters with ofa @xcite , which has been shown to be effective in the general domain. then, we pre-train and fine-tune the model with the various tasks in the medical imaging domain. in addition, we newly introduce a factual calibration technique to further improve the model performance. ## conclusion we proposed a pre-trained vlm, chexofa, for the chest x-ray domain. we showed that pre-traing with the report generation task improves the downstream task, report summarization. taking advantage of its multimodal nature, we improved the performance by jointly encoding visual and linguistic features. furthermore, we explored subtle techniques such as ensemble and factual calibration to improve the model performance. our experimental results demonstrated that proposed methods benefit the summarization performance. our model ranked first on the hidden test set in radsum23 shared task. we showed promising results about the domain-specific vlm in the chest x-ray tasks. we hope that our method can shed light on automating radiology report generation. the iitp(institute for information communications technology planning evaluation). it was also supported by the national research foundation of korea, south korea (nrf-2023r1a2c3004176) and a grant of the korea health technology rd project through the korea health industry development institute (khidi), funded by the ministry of health welfare, republic of korea (grant number: hr20c0021).