K now L ab at R ad S um23: comparing pre-trained language models in radiology report summarization (2023)

Abstract: This paper presents our contribution to the RadSum23 shared task organized as part of the BioNLP 2023. We compared state-of-the-art generative language models in generating high-quality summaries from radiology reports. A two-stage fine-tuning approach was introduced for utilizing knowledge learnt from different datasets. We evaluated the performance of our method using a variety of metrics, including BLEU, ROUGE, bertscore, CheXbert, and RadGraph. Our results revealed the potentials of different models in summarizing radiology reports and demonstrated the effectiveness of the two-stage fine-tuning approach. We also discussed the limitations and future directions of our work, highlighting the need for better understanding the architecture design’s effect and optimal way of fine-tuning accordingly in automatic clinical summarizations.

## introduction summarization of radiology reports is a useful tool for both doctors and patients. it allows doctors to quickly prioritize and extract essential information from long documents, saving time and improving patient outcomes. this is especially beneficial in medical settings, where time is often limited and there may be a large volume of documents to review. the rapid development of natural language processing (nlp) techniques, particularly large generative language models such as gpt @xcite , have significantly advanced the field of automatic text summarization. however, there is still much to explore when it comes to generating summaries for specific domains or tasks. for example, generating radiology reports is a challenging task that requires specialized knowledge and language expertise. radiology reports are typically written in a specific format and include complex medical terminology, which makes it difficult for traditional summarization techniques to produce accurate and comprehensive summaries. in the past, several methods have been proposed for radiology report summarization. @xcite developed a method for generating summaries of radiology reports using an attentionbased neural network. @xcite explored the problem using an augmented pointergenerator model resulted in high overlap with human-generated references. based on this work, @xcite then built an ontologyaware pointer-generator, which led to improved summarization quality. furthermore, @xcite proposed vilmedic, which is a replicable pipeline that can reproduce the latest results in various medical tasks using multimodal data resources (images and texts) for generating radiology reports. this paper investigates transfer learning methodologies for adapting large generative language models to the specific domain of radiology report generation. our main contribution involves a two-stage fine-tuning procedure on a large corpus of radiology reports to handle the specialized language and structure of these reports. we conducted experiments using two clinical datasets: mimic-iii, which includes various kinds of radiology reports, and mimic-cxr, which includes only chest xray radiology reports. the work and results are presented as part of the radsum23 shared task at bionlp 2023 @xcite . ## task description and dataset the goal of this research is to generate a summary ("impression" section in findings: the patient is status post intubation. note is made of degenerative disc disease involving multiple levels of c-spine, however, there is no evidence of fracture of the cervical spine. again note is made of multiple skull base fractures, as noted on the prior head ct. note is made of multiple hemorrhagic contusions in posterior fossa, as noted in the prior head ct. the lung apices are unremarkable. note is made of small amount of deep tissue emphysema posterior to the clavicles. impression: multiple skull fractures and hemorrhagic contusions as noted on the prior head ct. no evidence of c-spine fractures. djd of the c-spine. small amount of distal air posterior to the clavicles. the information has been communicated with ed physicians in person. two datasets are used in this challenge: the mimic-iii and mimic-cxr radiology report summarization datasets @xcite . mimic-iii radiology reports contains freetext radiology reports from the beth israel deaconess medical center. it contains radiology reports from various modality-anatomy including head, abdomen chest, etc. mimic-cxr radiology reports, which is a subset of mimic-iv dataset, that only contains chest radiology reports. in terms of the evaluation, the test-sets has been split into two by the organizers. the details of data description can be found in ## methods we applied pre-trained language models that are specific for generative tasks. the models are finetuned for the task of radiology report summarization. specifically, we fine-tuned two different types of pre-trained models -bart @xcite and t5 @xcite . by fine-tuning these models on a large corpus of radiology reports, we aim to generate high-quality summaries that capture the most important information in the reports. t5, short for text-to-text transfer transformer, was proposed by google and utilizes the standard transformer architecture pre-trained on denoising, where spans of text are replaced with a drop token. t5 was trained using a "text-to-text" approach, which means it can perform a wide range of tasks by converting input text into output text, such as language translation, summarization, question answering, and even programming tasks. t5 is also known for its ability to perform well with few-shot learning, meaning it can quickly adapt to new tasks with just a few examples. bart and t5 are both transformer-based architectures with subtle differences in their layer configurations. t5 is trained as a causal language model (i.e., predicting next tokens), while bart is trained on a masked language modeling objective. their performances vary depending on the specific tasks. to optimize the performance and generalizability of our approach, we employed a two-stage fine-tuning process using two different datasets -mimic-iii and mimic-cxr. in the first stage, we fully fine-tuned the pre-trained model on the mimic-iii dataset using the typical fine-tuning approach. this involved training the model on the dataset to adapt it to the specific task of radiology report summarization. in the second stage, we performed another round of fine-tuning on the mimic-cxr dataset by freezing the last two layers in the encoder and decoder. it is assumed that freezing the last two layers can help prevent the model from overfitting the training data. lower layers tend to learn more generalizable feature representations, while higher layers may be more prone to overfitting on the specific patterns of the training data. by freezing the last two layers, we can limit the degree of fitting to the training data, enabling the model to generalize better on new, unseen data. overall, the two-stage fine-tuning process is a key component of our approach, allowing us to leverage the strengths of both datasets and optimize the performance of the model on the task of radiology report summarization. ## experiments and results for the mimic-cxr experiments, we performed a second round of fine-tuning on the models that were previously fine-tuned on mimic-iii. the baseline model in in summary, our experimental results demonstrated that bart and t5 models have different strengths and may perform better on different datasets or evaluation metrics. this may due to various factors, such as the size and complexity of the dataset, the quality of the training data, etc. there is considerable potential for future improvements to our approach. firstly, our results suggest that freezing the last two layers of the models may yield some advantages, but this finding requires further investigation to fully understand the effects of this approach. secondly, we acknowl- ## conclusion this paper presented our investigation into transfer learning for the radiology report summarization task. specifically, we conducted a comprehensive evaluation of different pre-trained language models which were built by deploying encoder-decoder architectures. we introduced a two-stage fine tuning methodology which involved bart, t5, and their variants in the biomedical domain, and compared their performances on the summarization task. we also evaluated the effectiveness of our approach using various metrics, such as bleu, rouge, bertscore, chexbert, and radgraph, and demonstrated that our method achieved high performance on the summarization task. in fact, our approach was ranked third on the leaderboard of the radsum23 mimic-cxr hidden test set, highlighting the competitiveness of our approach compared to other methods.