Optum at MEDIQA 2021: Abstractive Summarization of Radiology Reports using simple BART Finetuning (2021)

Abstract: This paper describes experiments undertaken and their results as part of the BioNLP MEDIQA 2021 challenge. We participated in Task 3: Radiology Report Summarization. Multiple runs were submitted for evaluation, from solutions leveraging transfer learning from pre-trained transformer models, which were then fine tuned on a subset of MIMIC-CXR, for abstractive report summarization. The task was evaluated using ROUGE and our best performing system obtained a ROUGE-2 score of 0.392.

## introduction a bionlp 2021 shared task, the mediqa challenge aims to attract research efforts in nlu across three summarization tasks in the medical domain: multi-answer summarization, and radiology report summarization. we participated in the radiology report summarization and offer experiments and results. a radiology report describes an exam and patient information resulting from trained clinicians(radiologists) interpreting imaging studies during routine clinical care @xcite . the primary purpose of the report is for radiologists to communicate imaging results to ordering physicians @xcite . a standard report will consist of a background section which will contain details of the patient and describe the examination undertaken, a findings section, in which the radiologist has dictated the initial results into the report, and an impression section. the impression section consists of a concise summarization of the most relevant details from the exam based on the dictated findings. although guidelines for the practice of generating radiology reports are outlined by the american college of radiology (acr), there is flexibility in the document in the usage of terms for describing findings and where they are documented. this can lead to referring physicians focusing on just the impressions section of the document @xcite . additionally, the process of writing the impressions from the dictation of the findings is time-consuming and repetitive. in this work we propose experiments to automate the generation of the impressions section from the findings of the radiology report, accelerating the radiology workflow and improving the efficiency of clinical communications. experiments were performed implementing sequence to sequence models with encoder-decoder architecture like bart @xcite , pegasus @xcite , and t5 @xcite . these models were then further fine-tuned on a subset of mimic-cxr dataset @xcite , to generate abstractive summaries from the findings section of the report. mimic-cxr is de-identified and protected health information (phi) removed, large publicly available dataset of chest radiographs in dicom format with free-text radiology reports. a subset of mimic-cxr and indiana datasets foot_0 used for validation carried out using standard rouge @xcite metrics. ## related work initial efforts on summarization were mainly focused on extractive summarization. extractive summarization is the process involving extraction of noteworthy words from the text to form a summary. @xcite the advent of neural network models enabled abstractive summarization, which involves producing new words to convey the meaning of the text. this involves rephrasing the text in a shorter and more succinct form using similar but not the exact words used in the main text. @xcite proposed an rnn based approach to not only achieve state-of-the-art results in extractive summarization but also enable this model to be trained on abstractive summaries. @xcite described an attention-based summarization approach where an encoder and a generator model are jointly trained on article pairs. their work builds on attention-based encoders that are used in neural machine translation @xcite ). @xcite build on the previous work on abstractive summarization to create length constrained summaries and summaries concentrated on particular entities and subjects in the text. @xcite used intra-temporal attention to produce state-of-the-art results on cnn/daily mail dataset. the work on summarizing radiology reports started with the extraction of information from the text @xcite . for instance, @xcite proposed using clinical language understanding of a radiology report to extract named entities. a bidirectional lstm architecture was used to achieve this. @xcite describes one of the first attempts at automatic summarization of radiology reports. this work describes an encoder-decoder architecture. both the encoder and decoder sides are made of bidirectional lstms using the attention framework @xcite . with the advent of transformers, pretraining based language generation has been the norm in summarization. @xcite and @xcite used bert @xcite , a pre-trained transformer model on extractive summarization, and achieved state of the art results. @xcite proposed an approach to content selection for abstractive text summarization in clinical notes. @xcite presented a general framework and a training strategy to improve the factual correctness of neural abstractive summarization models for radiology reports. in this work, we fine-tune a pre-trained bart architecture @xcite for the radiology report summarization task. ## task description & dataset the objective of this task is to generate summary of a given radiology report. the training data for the mediqa 2021 radiology report summarization shared task is extracted from a subset from the mimic-cxr dataset @xcite . the training set contains around 91,544 examples of radiology reports and the corresponding summaries. each example contains three fields; findings field contains the original human-written radiology findings text, impression contains the human-written radiology impression text and background contains background information of the study in text format. one can use both the findings and the background fields to generate the summary. there are two development sets that come from two different institutes. the first development set from mimic-cxr contains around 2000 examples. there is another development set that also contains 2000 examples from the indiana university radiology report dataset @xcite . in all our experiments, we first trained our model on the training set and tested on the validation set. for the actual task submissions, we trained our models by combining training set and both the development sets. ## method & results our proposed method leverages pretrained summarization models. we finetuned three types of pretrained models for the radiology report summarization; bart @xcite , t5 @xcite and pegasus @xcite . we used huggingface transformers @xcite ) library for finetuning. bart: developed by facebook, bart is a denoising autoencoder. since it uses the standard transformer-based neural machine translation architecture, it is a generalization of both bert and gpt3 @xcite . for pretraining, it was trained by shuffling the order of sentence (an extension of next sentence prediction) and text infilling (an extension of the language masking). during text infilling, random spans of text are replaced by masked tokens. the job of the model during training is to recreate this span. due to its flexible transformer architecture, the inputs to the encoder do not need to be aligned with the outputs of the decoder. this enables the bart model to be trained on a variety of tasks such as token masking, token deletion, sentence permutation, document rotation, etc. since bart has an autoregressive decoder, it is better suited for sequence generation tasks such as summarization. ## conclusion in this paper, we present all our experiments of fine-tuning pre-trained models for radiology report summarization. our experiments demonstrate how an encoder-decoder architecture like bart, which achieved state-of-the-art results in text generation tasks outperforms other architectures in this particular task. our methods proved effective on the summarization task and were ranked third on the leaderboard.