nav-nlp at R ad S um23: Abstractive Summarization of Radiology Reports using BART Finetuning (2023)

Abstract: This paper describes the experiments undertaken and their results as part of the BioNLP 2023 workshop. We took part in Task 1B: Radiology Report Summarization. Multiple runs were submitted for evaluation from solutions utilizing transfer learning from pre-trained transformer models, which were then fine-tuned on MIMIC-III dataset, for abstractive report summarization.

## introduction bionlp 2023 shared task, aims to attract researchers and students working on nlp across three summarization tasks in the bio-medical domain: problem list summarization, radiology report summarization and lay summarization of biomedical research articles. we took part in radiology report summarization and performed experiments. documents that summarize and record radiological examinations are known as radiology reports. three elements typically make up a radiology report: (1) a background section that contains general information about the patient and the exam, (2) a findings section that includes examination data, and (3) an impression section that summarizes the findings in relation to the background. usually, medical professionals go through the finding section and pull out the valuable findings and try to summarize the whole radiology to make clinical decisions. but this manual process of summarization is time-consuming and inefficient. so, in this work we propose experiments to automate the generation of the impressions section from the findings of the radiology report, speeding up the radiology * corresponding author workflow and improving the efficiency of clinical communications. a key feature of this task is that radiology reports used for training and evaluation are collected from various sources, e.g., training instances are sampled from the mimic-iii database. experiments were performed implementing sequence to sequence models with encoder-decoder architecture like bart, t5. these models were then further fine-tuned on a portion of the mimic-iii dataset to produce abstract summaries of the report's conclusions from the findings section. a portion of the mimic-iii is used for validation purpose and standard rougel metric is used for evaluation purpose. ## related work summarization tasks were initially just focused on extractive summarization. it is a method of extracting noteworthy words from the text to form a summary. advancement in neural networks models brought up the scope of abstractive summarization, which involves producing new words to convey the meaning of the text. it attempts to capture the meaning of the text and express it in a more concise and coherent form. one of the earliest approaches to abstractive summarization was the use of rule-based systems that relied on handcrafted templates and heuristics to generate summaries. however, these systems were limited in their ability to capture the nuances of natural language and often produced summaries that were either too general or too specific. in recent years, deep learning techniques have revolutionized the field of abstractive summarization, enabling models to learn from large amounts of data and generate summaries that are more accurate and fluent. one of the most influential models in this regard is the encoder-decoder framework with attention mechanism, which was first proposed by @xcite for machine translation and later adapted for summarization by @xcite . the encoder-decoder model involves training a neural network to encode the input text into a fixedlength vector representation, which is then decoded into the summary. the attention mechanism allows the model to focus on different parts of the input text during the decoding process, which improves the quality of the generated summary. since the introduction of the encoder-decoder model, numerous other models have been proposed that build on this basic framework, including transformerbased models like bert @xcite and gpt @xcite . these models have achieved state-of-the-art performance on a range of summarization tasks and have become increasingly popular in both industry and academia. ## task description & dataset the goal of this research is to develop an effective algorithm for summarizing radiology reports. given a radiology report, the algorithm should generate a concise summary of the key findings and conclusions of the study. the performance of the algorithm will be evaluated using standard metrics such as rougel @xcite , bertscore @xcite , and f1-radgraph @xcite . the dataset used in this research is based on the mimic-iii database, which is a publicly available clinical database containing de-identified health data for over 50,000 patients. the dataset includes radiology reports from six different modalities: ct head, ct abdomen, ct chest, mr head, ct spine, ct neck, and five different anatomies: head, abdomen, chest, spine, and neck. the dataset contains a total of 79,779 radiology reports, with varying numbers of reports for each modality/anatomy pair. the dataset is divided into three subsets for training, validation, and testing, with the majority of reports (at least 10,000) in the training subset for each modality/anatomy pair. the performance of the summarization algorithm will be evaluated using standard metrics such as rougel, bertscore, and f1-radgraph, with the test subset of the dataset used for evaluation. the results of the evaluation will be reported in terms of the metrics and compared to state-of-theart summarization algorithms. ## methods & results pretrained summarization models are used in our suggested methodology. we fine-tuned two types of pretrained models for the radiology report summarization; bart and t5. we used hugging face transformers libraries for fine-tuning. bart: bart (bidirectional and auto-regressive transformers) is a pre-trained neural network architecture that is based on the transformer model, which is a popular architecture for natural language processing (nlp) tasks. bart was introduced by @xcite and has since been shown to achieve state-of-the-art results on a variety of nlp tasks, including text summarization.bart consists of two main components: a masked language model (mlm) and a sequence-to-sequence (seq2seq) model. the mlm component is trained to predict masked tokens in an input text, while the seq2seq component is trained to generate a target sequence given an input sequence. the combination of these two components allows bart to be fine-tuned on a range of nlp tasks, including text summarization. in the context of summarization, bart takes an input document and generates a summary by conditioning it on a special token indicating the start of the summary. during fine-tuning, the model is trained to generate a summary that maximizes a similarity metric such as rouge (recall-oriented understudy for gisting evaluation) with respect to a reference summary.bart's ability to generate high-quality summaries can be attributed to its pre-training on large amounts of text data, which allows it to capture the general patterns and structures of natural language. fine-tuning specific summarization tasks further refines the model's ability to generate accurate and concise summaries. in addition to text summarization, bart has also been shown to perform well on other nlp tasks, including machine translation, question answering, and dialogue generation. its flexibility and effectiveness make it a valuable tool for a wide range of nlp applications. t5: t5 (text to text transfer transformer) is a powerful language model that has demonstrated impressive results on a wide range of nlp tasks, including language translation, summarizing, question answering, and text classification, among others. unlike traditional nlp models that are de ## experiments we propose different runs for this task. different runs of this task are: 1. our first proposed method is bart-base. we finetuned the bart-base model using the mimic-iii-train set and chose the best bart-base model by testing with the mimic-iii-validation set and estimating validation loss. we then tested it with mimic-iiitest and mimic-iii-hidden-test sets respectively and the results are shown in the above tables. we have used a batch size of 20 for the training set and 4 for validation set. ## conclusion in this paper, we presented all our experiments of fine-tuning the pre-trained hugging face model for abstractive radiology report summarization. our approach aimed to identify the most effective architecture for this task, and after experimenting with various architectures, we found that a bartlike architecture outperformed the others. we then provided detailed results and analysis of our experiments, which demonstrated the effectiveness of our methods for radiology report summarization. rankings for the mimic-iii-open-test set and mimic-iii-hidden-test set are shown in tables 3 and 4 respectively. our findings suggest that finetuning pre-trained models is a promising approach for improving the performance of nlp models in medical settings, with potential implications for enhancing the efficiency and accuracy of medical record analysis and ultimately improving patient care.