EL i RF - VRAIN at B io NLP Task 1 B : Radiology Report Summarization (2023)

Abstract: This paper presents our system at the Radiology Report Summarization Shared Task-1B of the 22nd BioNLP Workshop 2023. Inspired by the work of the BioBART model, we continuously pre-trained a general domain BART model with biomedical data to adapt it to this specific domain. In the pre-training phase, several pre-training tasks are aggregated to inject linguistic knowledge and increase the abstractivity of the generated summaries. We present the results of our models, and also, we have carried out an additional study on the lengths of the generated summaries, which has provided us with interesting information.

## introduction radiology reports are documents that interpret radiological examinations. usually, a radiology report consists of three sections: (1) a background section that describes general information about the patient and exam, (2) a findings section that presents details of the examination, and (3) an impression section that summarizes the findings against the background. this last section is the most crucial for doctors to make clinical decisions. due to the recent success of self-supervised learning, the focus of text summarization research has exhibited a gradual shift from extractive techniques to abstractive techniques. the bestperforming abstractive models are bart @xcite , t5 @xcite , pegasus @xcite , and gpt-3 @xcite , being all of them transformers @xcite pre-trained self-supervisedly as denoising sequence to sequence autoencoders. this kind of approaches allow to pre-train deep architectures to learn vast amounts of general linguistic knowledge from large corpora, that can be transferred to downstream tasks by means of fine-tuning. almost all of these systems used benchmark datasets compiled from news articles, such as the cnn-dailymail dataset (cnn-dm) @xcite and newsroom @xcite . however, not so many efforts have been carried out in the biomedical domain. language models pre-trained on biomedical corpora may further enhance the performance of current biomedical nlg methods, such biobert @xcite or pubmedbert @xcite . however, there are very few in-domain generative language models for biomedicine. in @xcite , authors proposed a biomedical autoregressive generative language model, biobart, pre-trained on the biomedical corpora. they continuously pre-train bart on pubmed foot_0 abstracts to achieve biomedical domain adaption only using the text-infilling task. the in-domain biobart outperforms bart model and sets strong baselines for several nlg tasks. in the framework of bionlp workshop, some challenges and shared tasks focusing on summarization were created. mediqa 2019 edition focused on question entailment and textual inference and their applications in medical question answering @xcite . mediqa 2021 @xcite promoted research on summarization for consumer health qa and clinical text. in this edition, the winner system @xcite , based on pegasus, employed a domain adaptation strategy by further fine-tuning a small amount of in-domain data to improve generalization and transfer abilities. ## task description the shared task-1b of the 22nd bionlp workshop 2023 @xcite , focuses on the summarization of radiology reports. the task of the summarization of radiology reports can be defined as follows: given a radiology report with findings and background sections, the goal is to generate the impression section. for this shared task, the impressions are only generated from the findings section. shared task 1b was divided into two subtasks. the first one is about generating impressions sections based exclusively on the text report. the second one is summarizing the report based on the text information and the indicators that could be extracted from the attached radiology image. the participants were invited to approach both subtasks but were allowed to participate in one; we chose to participate only in the first subtask. ## pre-training model inspired by the work of the biobart model, we continuously pre-trained a general domain bart model with biomedical data to adapt it to this specific domain. specifically, our starting point was the architecture and weights of the base version of bart foot_1 , publicly available at the repository of huggingface @xcite . for the pre-training phase, we followed the methodology used in the news abstractive summarization models (nas) work @xcite . in nas, several pre-training tasks are aggregated to inject linguistic knowledge during the pre-training stage and to increase the abstractivity of the generated summaries. we chose this methodology because we hypothesize that reference impressions are written in a mostly abstractive way. also, the pre-training method helped to transfer more knowledge to the summarization task, which increased the model's performance in the original work. for pre-training, we chose data that were as similar as possible to radiology reports in terms of vocabulary and grammar. we selected the following mimic datasets available at physionet @xcite : note events in mimic-iii (2 083 180 samples) @xcite , radiology reports in mimic-cxr (128 032) @xcite , and discharge (331 794) and radiology reports (2 321 355) in mimic-iv @xcite . additionally, we included wikipedia articles related to medicine to reinforce the domain vocabulary @xcite . the base version of the bart model is limited to 1024 input tokens; however, most samples exceeded this size. this fact could lead us to lose valuable training data. to overcome this limitation, we split texts into narrower samples as follows: due to infrastructure limitations, we could only train the model for one epoch, which took 12 days with four nvidia rtx 3090 graphic cards. the following hyperparameters were used: 4 samples per device, 256 gradient accumulation steps, a learning rate of 5×10 -5 with a linear scheduler, 1% of the epoch for warm-up, and 32 bits of training precision. for the hyperparameters not mentioned, we have used the default values of the 4.23.1 version of transformers library of huggingface foot_2 . ## models for the task for the downstream task, we obtained three models based on our pre-trained model. the first model (m1) was fine-tuned with the train partition of the shared task. the second one (m2) was fine-tuned with the train and validation partitions. finally, the third one (m3) was fine-tuned with all the partitions with available references: train, validation, and test. for the fine-tuning phase, we did a grid search of certain hyperparameters with raytune @xcite through the huggingface library. we did 20 trials over the following hyperparameters: learning_rate (from 8 × 10 -6 to 4 × 10 -5 ), num_train_epochs (10 or 15), and gradient_accumulation_steps (2, 4, or 8). since we wanted to find which models obtained a more balanced performance among the four metrics of the task (rouge-l, bleu, radgraph-f1, and bertscore-f1), we defined the objective to maximize as the harmonic mean @xcite of these four scores. finally, the three models were fine-tuned during 15 epochs with an nvidia rtx 4090 with the following hyperparameters: 8 samples per device, 4 gradient accumulation steps, and a learning rate of 2.14 × 10 -5 . for the generation of impressions, we used the generate method foot_3 of huggingface. to achieve better performance, we identified certain hyperparameters and performed grid search using the m1 model and the validation partition, specifically: max_length (60, 70, 80, 90, or 100), num_beams (3, 4, 6, 8, or 10), and no_repeat_ngram_size (3, 4, 5, 6, 8, or 10). the bolded values maximized the harmonic mean score; thus, we fixed them to generate impressions with ours models. overall, our models have lower performance than the best systems. in the test partition, our best model (m1) averages a 9% lower performance than the other two groups if bleu is excluded from the count, and 5.3% less when is included, meaning that m1 performs substantially better in bleu than the other systems. comparing our models, m2 seems to perform worse than m1, despite being trained with more data. in the case of hidden-test, our best model (m3) averages 10.7% lower performance than g1 if bleu is excluded from the count and 8.43% if not. comparing the performance of our models, unlike what happened in the test partition, m3 performs better than m1. therefore, the inclusion of the test partition in training resulted in more acknowledgment for the model, probably because of the additional findings types. ## discussion when we observe ## conclusions we presented an approach for radiology report summarization that continuously pre-trains a general domain bart model. this approach focuses on two main aspects: the use of biomedical data to adapt the model to this specific domain and the use of several pre-training tasks designed to inject linguistic knowledge and increase the abstractivity of the generated summaries. after the pre-training phase, we fine-tuned this model with different amounts of data from the shared task. we also presented a study of the relationship between the models performance and the lengths of the generated summaries. we observed that our models condense the main information in the first sentences of the summaries. from the length distribution of the summaries, we found that our models tend to generate summaries with a common length; meanwhile, the reference summaries present more length variability. it seems that this behavior could penalize the performance of our models, especially on those reports with short reference summaries.