ISIKS umm at B io L ay S umm Task 1: BART -based Summarization System Enhanced with Bio-Entity Labels (2023)

Abstract: Communicating scientific research to the general public is an essential yet challenging task. Lay summaries, which provide a simplified version of research findings, can bridge the gapbetween scientific knowledge and public understanding. The BioLaySumm task (Goldsack et al., 2023) is a shared task that seeks to automate this process by generating lay summaries from biomedical articles. Two different datasets that have been created from curating two biomedical journals (PLOS and eLife) are provided by the task organizers. As a participant in this shared task, we developed a system to generate a lay summary from an article’s abstract and main text.

## introduction in this paper, we present our participation in the biolaysumm task by proposing an approach that combines the power of the bart baseline model with the use of stanza libraries to detect technical words. our approach addresses the challenge of removing technical terms, such as bio-entities, from biomedical scientific reviews to improve their accessibility and understandability. the presence of technical terms in scientific articles hinders comprehension by non-experts and limits the accessibility of scientific research. these specialized terms, including bio-entities like proteins and genes, can be complex and unfamiliar to readers outside the domain. by accurately detecting and removing these technical terms, we can generate lay summaries that are more comprehensible to a wider audience. our proposed approach integrates the precise detection of technical words using stanza libraries with the powerful text summarization capabilities of the bart model. by combining these strengths, we aim to provide concise and accessible summaries that facilitate the dissemination of scientific knowledge beyond the scientific community. through extensive experiments and evaluations, we have achieved promising results on the biolay-summ task. our approach has the potential to enhance the accessibility of biomedical research, enabling non-experts to understand and engage with scientific advancements in the field. ## related work automatic text summarization has been extensively studied in various domains @xcite with significant advancements achieved in recent years. deep learning models, particularly transformerbased architectures, have shown remarkable performance in generating high-quality summaries @xcite . in the biomedical domain, several approaches have been proposed to address the challenges specific to biomedical text summarization. one prominent model is biobert @xcite , a pre-trained language representation model based on bert (bidirectional encoder representations from transformers). biobert has been widely adopted in various natural language processing tasks, including text summarization, and has shown improved performance in biomedical text mining. by leveraging a large biomedical corpus during pre-training, biobert captures domain-specific knowledge and terminology, making it effective for summarizing biomedical articles. another notable approach is the use of bart (bidirectional and auto-regressive transformer) models in biomedical text summarization @xcite . bart is a pre-trained sequence-tosequence model that has achieved state-of-the-art performance in various natural language processing tasks. researchers have proposed bart-based models which have demonstrated significant improvements in summarization performance within the biomedical domain. these models leverage the powerful representation capabilities of bart and have the potential to facilitate biomedical research by providing accurate and concise summaries of scientific articles. while the existing approaches @xcite @xcite have made substantial progress in biomedical text summarization, there is still a need for further development to offer high-quality lay summaries in the biomedical domain. the challenges include effectively handling technical terminology, such as bio-entities, and ensuring the accessibility of scientific research to the general public. ## methods the technical terms such as protein entities in the article, are one of the aspects that make the article hard to be comprehend by general audience, who do not have much biology background. we assume that these technical terms are embedded in text as named entities. following this assumption, we aim to detect the named entities and replace them by further explanations before feeding them to the bart model, which also lacks of technical terms in the biomedical domain. for this aim, since the performance of stanza's biomedical ner models reported to be higher than the performances of the biobert models and scispacy models previously @xcite , we utilized the stanza library to detect technical terms. we believe that this aspect will include specific domain knowledge in the summarization process and improve the accuracy of our lay summaries for biomedical articles. model 2: this approach used the original abstract text from the article as a summary and combined it with the summary generated by our bart-based model, which was trained on the preprocessed full text of the article. for model 2, we utilized the bart baseline model which has a maximum input length of 1024 characters. since the articles are usually longer than the model's input length limit, we divided the article into sections with 1024 characters each, and fed them into the model one by one. the output from the model was then concatenated to form the complete summary. in the post-processing step, we applied text cleaning techniques to remove irrelevant information such as parentheses. we also used the stanza library in the same way that has been utilized for model 1 to identify and tag unknown words in the text. the performance of both models was evaluated using the rouge @xcite , bertscore @xcite and bartscore @xcite metrics. the results show that model 2 outperforms model 1 in all metrics. ## experimental setup we conducted all experiments for the biolaysumm task on a google colab platform, using a premium account to access gpu hardware acceleration. specifically, we utilized an nvidia tesla k80 gpu, which has 12gb of memory, to accelerate the training and inference processes of our bart-based summarization model. to train our model, we used the pytorch deep learning framework and the hugging face transformers library. we fine-tuned the bart model on the training datasets provided for the task, consisting of biomedical research articles from plos and elife journals. during training, we set the batch size to 4 and the maximum sequence length to 512, which we found to be optimal for our model and hardware configuration. we trained the model for a total of 10 epochs, using the adam optimizer with a learning rate of 3e-5 and a weight decay of 0.01. for evaluation, we used the official evaluation script provided by the bionlp workshop organizers, which calculates rouge @xcite scores to measure the quality of the generated lay summaries. we evaluated our model on the test datasets for both the plos and elife journals, and reported the average performance across both datasets. overall, the hardware and software parameters we utilized in our experiments allowed us to effectively train and evaluate our bart-based summarization model for the biolaysumm task, achieving competitive results in terms of rouge scores. ## results and discussion we have evaluated our bart-based summarization model on the test data sets for both the plos and elife journals. table presents our performances on both datasets. the results show that our model achieved averaged on both data sets, rouge-1, rouge-2, and rouge-l scores of 0.379, 0.070, and 0.353, respectively, which are promising in the biolaysumm task. our experiments have shown that fine-tuning the bart model on the provided biomedical research article data sets is effective in generating high-quality lay summaries. the utilization of the gpu hardware acceleration provided by the google colab platform allowed us to train our model in a reasonable amount of time and with sufficient resources. additionally, we observed that pre/postprocessing steps using biomedical named entity recognizers for the identification of technical terms (protein names, cell-types, chemicals, and so on so forth) improved the quality of the generated summaries. despite achieving promising results, our experiments have also revealed several challenges associated with the biolaysumm task. one major challenge is to generate summaries that accurately captured the main ideas and concepts of the original article, while also presenting the information in a simplified and accessible manner for a lay audience. another challenge is to deal with complex scientific terminology, which can be difficult to translate into lay language without losing important details. ## conclusion in this paper, we presented our participation to task 1 of biolaysumm shared task, whose aim is to generate high-quality lay summarizes from biomedical articles of different journals. our participation includes a bart-based summarization system, whose experiments showed that fine-tuning the bart model on the provided datasets, combined with pre-processing techniques such as identifying named entities, which has been assumed to be domain-specific and technical terms, resulted in higher quality lay summaries. despite the challenges associated with the biolaysumm task, we achieved competitive results, which demonstrated the potential of our model for making scientific research more accessible to the general public. future work will focus on exploring additional pre/post-processing techniques, investigating alternative approaches to fine-tuning the bart model, and addressing the challenges of generating accurate and accessible lay summaries for complex scientific articles.