B io L ay_ AK _ SS at B io L ay S umm: Domain Adaptation by Two-Stage Fine-Tuning of Large Language Models used for Biomedical Lay Summary Generation (2024)

Abstract: Lay summarization is essential but challenging, as it simplifies scientific information for non-experts and keeps them updated with the latest scientific knowledge. In our participation in the Shared Task: Lay Summarization of Biomedical Research Articles @ BioNLP Workshop (Goldsack et al., 2024), ACL 2024, we conducted a comprehensive evaluation on abstractive summarization of biomedical literature using Large Language Models (LLMs) and assessed the performance using ten metrics across three categories: relevance, readability, and factuality, using eLife and PLOS datasets provided by the organizers. We developed a two-stage framework for lay summarization of biomedical scientific articles. In the first stage, we generated summaries using BART and PEGASUS LLMs by fine-tuning them on the given datasets. In the second stage, we combined the generated summaries and input them to BioBART, and then fine-tuned it on the same datasets. Our findings show that combining general and domain-specific LLMs enhances performance.

## introduction in today's era, a lot of research is being conducted in the field of biomedical science, resulting in a huge amount of biomedical literature. the vast scientific knowledge poses a challenge for healthcare professionals, researchers, and the non-expert public in staying informed about advancements in the biomedical domain @xcite . making the information accessible and understandable, regardless of their background knowledge, is difficult. manually summarizing long scientific articles requires too much domain-oriented knowledge, effort, and time, especially for lay summarization. first, summarizing and then transforming the summarized information for non-experts is impractical. this problem can be tackled by designing lay summarization systems that bridge the gap between non-experts and experts by modifying intricate scientific knowledge into a clear and condensed form with increased readability. this step will increase scientific literacy and enable decision-making for experts and non-experts. this study's contributions include: ## related work recent studies have showcased the significant potential of large language models (llms) in natural language generation tasks. in addition, the first version of the biolaysum @xcite illustrated the effectiveness of utilizing llms in both summary formation @xcite and data augmentation @xcite . llms exhibit proficiency in perceiving complex relationship patterns due to their training on diverse large-scale datasets across various tasks @xcite . ## methodology in recent times, large language models such as biobart pre-trained on biomedical corpora have achieved enhanced performance for biomedical natural language generation tasks. however, the readability of the generated summaries needs to be improved from the perspective of a non-expert audience. to achieve this aim, a two-stage framework is designed in this work, as shown in figure 1 , to generate lay summaries for complex and lengthy scientific research articles, targeting non-expert audiences. in the first phase of the framework, bart and pegasus general-purpose llms are selected for summary generation by fine-tuning them on the challenge datasets: elife and plos. both models performed well on validation and test sets, indicating their capability to generate high-quality summaries due to their pre-training on multiple tasks and large datasets. as these transformers have limited input lengths, the first 1024 tokens are used for training because these llms are resourceintensive and time-consuming. specifically, training with starting information proves to be more efficient, as studies indicate that important information is typically presented at the beginning and end of research articles @xcite . bart and pegasus are transformer-based models that excel in text generation and are specifically designed for abstractive summarization. in the initial phase of our approach, we fine-tune these models using article-lay summary pairs from the elife and plos datasets separately. this process involves setting the hyperparameters specified in algorithm 1 outlines the steps for deduplicating sentences in aggregated text. this process removes identical sentences while considering case sensitivity (lower-case), ensuring non-redundant information. in the second and final phase of the framework, the deduplicated text for a correspond- ## experimental setup the experiments in this study are conducted on the google colab pro plus platform, using an nvidia a100 gpu with 40 gb of gpu ram for training and inferencing. appendix a provides insights into the datasets used, while ## conclusion and future scope a two-stage fine-tuning framework combining general-purpose bart and pegasus llms with the biomedical-specific biobart llm showcased satisfactory performance for generating lay summaries of biomedical scientific articles. in the first phase, bart and pegsus were fine-tuned on the training data of elife and plos datasets, while in the second phase, biobart was fine-tuned on the merged and deduplicated lay summary generated in the first phase. all hyperparameters were set using the validation set. this framework achieved promising results for relevance and readability metrics but at the cost of marginally lower performance for factuality metrics. in the future, different combinations of domain-specific llms can be employed, along with language simplification techniques, to generate high-quality lay summaries for non-experts, optimizing scores for relevance, readability, and factuality. metrics. in the future, different combinations of domain-specific llms can be employed, along with language simplification techniques, to generate high-quality lay summaries for non-experts, optimizing scores for relevance, readability, and factuality.