Team Converge at P rob S um 2023: Abstractive Text Summarization of Patient Progress Notes (2023)

Abstract: In this paper, we elaborate on our approach for the shared task 1A issued by BioNLP Workshop 2023 titled Problem List Summarization. With an increase in the digitization of health records, a need arises for quick and precise summarization of large amounts of records. With the help of summarization, medical professionals can sieve through multiple records in a short span of time without overlooking any crucial point. We use abstractive text summarization for this task and experiment with multiple state-of-the-art models like Pegasus, BART, and T5, along with various pre-processing and data augmentation techniques to generate summaries from patients’ progress notes. For this task, the metric used was the ROUGE-L score. From our experiments, we conclude that Pegasus is the best-performing model on the dataset, achieving a ROUGE-L F1 score of 0.2744 on the test dataset (3rd rank on the leaderboard).

## introduction text summarization is the process of shortening a corpus of text into a smaller version while retaining all the crucial information present in the original text. there are two types of text summarizations: abstractive text summarization and extractive text summarization. abstractive text summarization identifies the critical points of the data and generates a new summary that captures the data's crux. on the other hand, extractive text summarization generates a summary from the words and phrases present within the original data. text summarization is used in fields like bio-medicine, journalism, finance, etc. one such field in which there has been a rise in the use of text summarization is clinical medicine. manually reading through medical notes, hospital progress notes, and daily care notes written by doctors and nurses can be monotonous and timeconsuming. however, this can be sped up by leveraging automatic text summarization. with the ex-ponential increase in the amount of data that is digitized and readily available, information overload is bound to occur. the automation of summarizing medical notes can help abate the information and cognitive overload faced by medical professionals daily. text summarization can help these people effectively filter through a plethora of data and focus only on the significant points. there will also be a decrease in the inevitable human errors if the entire process is automated. through this paper, we intend to provide a concise summary containing a list of diagnoses and problems for a patient during hospitalization based on the input given in the form of progress notes. in order to increase efficiency and lower diagnostic errors in hospital care, this task intends to stimulate the development of text summarization models for use in diagnostic decision support systems. ## related work work on automatic text summarization started in 1958 when @xcite proposed a method of extracting summaries of scientific literature. this paved the way for extensive research on using natural language processing techniques for text summarization. other important works from early research include @xcite (sentence position and title of the article were used as features for summarizing documents) and @xcite (cue words such as "important" or "crucial" were used in addition to title words and sentence location). @xcite introduced machine learning as a method for text summarization in 1995 by training a naive bayes classifier to extract a summary from a text. subsequently, several machine learning approaches have been proposed that use naive bayes or decision tree classifiers, such as chuang and yang @xcite , which extracted segments of the sentence using special cues, and @xcite , which employed a combination of statistical and linguistic features. in recent years, deep learning methods have been the modus operandi for text summarization, focusing more on generating abstractive summaries. encoder-decoder rnns @xcite and lstm-cnn frameworks @xcite have been shown to perform well on abstractive text summarization tasks. the advent of transformer architecture in @xcite has coincided with a preference for the use of transformer-based methods for summarization. bert @xcite , t5 @xcite , bart @xcite , and pegasus @xcite ) have all been used successfully for abstractive summarization. there has been a growing focus on text summarization in the biomedical domain. research in the field gathered momentum at the turn of the century, with ten biomedical text summarization studies published between @xcite @xcite . most of the early work in this arena used extractive summarization methods, while abstractive summarization has started gaining traction due to the use of highly sophisticated deep-learning models. @xcite proposed biosquash, a query-based extractive summarization system that uses domain-specific ontologies to rank sentences. @xcite proposed biomed-summarizer, a framework that uses deep neural networks and rnns to extract meaningful sentences from biomedical text. approaches that use a combination of extraction and abstraction have also been used @xcite , adams et al. (2021) ). however, transformer-based models have been shown to perform best for abstractive summarization tasks @xcite , @xcite al. (2020), krishna et al. (2020), chintagunta et al. (2021) ). we have conducted a set of experiments on transformer-based models to create abstractive summaries from patient daily care notes. ## dataset description the dataset provided for this task @xcite was sourced from mimic-iii @xcite , a publicly available medical dataset. mimic-iii consists of de-identified ehr data, which is taken from approximately 60,000 hospital icu admissions at beth israel deaconess medical center in boston, massachusetts. the data for this task was sourced from the mimic-iii dataset. the training set has 765 samples, and the test set has 237 samples. the data contains five features: file id, subjective sections, objective sections, assessment, and summary. assessment input consists of notes taken by a doctor with details about the patient and their diagnosis. objective sections consist of detailed information regarding a patient's medication dosage and vitals. the subjective sections contain general observations regarding the patient's progress. the summary feature concisely represents all this information and is the ground truth for this task. ## methodology we train the below-mentioned models on the training data. the hyperparameters for each model are mentioned in its respective section. the results obtained on the test data are mentioned in ## conclusion in this paper, we compared the performance of four models (pegasus, bart, t5-base, and t5small) on the task of summarizing patient daily care notes. our experiments concluded that using only the assessment feature yielded better results on the rouge-l metric. we also observed an improvement in model performance by using preprocessing techniques like stop word removal and lemmatization. overall, pegasus outperformed other models yielding a score of 0.2744 on the test set. in the future, we plan to augment the data using data that is more similar to the task dataset. we also plan to implement an ensemble of the summarization models that we have used.