UIUC _ B io NLP at B io L ay S umm: An Extract-then-Summarize Approach Augmented with W ikipedia Knowledge for Biomedical Lay Summarization (2024)

Abstract: As the number of scientific publications is growing at a rapid pace, it is difficult for laypeople to keep track of and understand the latest scientific advances, especially in the biomedical domain. While the summarization of scientific publications has been widely studied, research on summarization targeting laypeople has remained scarce. In this study, considering the lengthy input of biomedical articles, we have developed a lay summarization system through an extract-then-summarize framework with large language models (LLMs) to summarize biomedical articles for laypeople. Using a fine-tuned GPT-3.5 model, our approach achieves the highest overall ranking and demonstrates the best relevance performance in the BioLaySumm 2024 shared task.

## introduction new research in the biomedical field is often reported in the latest scientific articles and plays a crucial role in improving human health and wellbeing. however, the complex terminology and scientific language used in these publications can be challenging to understand for those without extensive knowledge of the field. the biomedical lay summarization task (biolaysumm) @xcite addresses this challenge by creating summaries that are easier to read and understand, specifically tailored for readers unfamiliar with biomedical studies. unlike traditional text summarization, which aims to condense documents into brief summaries, biolaysumm also focuses on using easy-to-understand language. this approach ensures that the summaries are less technical and more accessible, while traditional summarization tasks prioritize capturing the precise scientific terminology used in the original documents. to solve figure 1 : our extract-then-summarize framework for the biomedical lay summarization task. we assess the performance of two models, gpt-3.5 and led, in generating lay summaries. the input of the led model includes the article sections that are ranked for relevance, wikipedia knowledge, and an extractive summary. meanwhile, the gpt-3.5 model is fine-tuned by the extractive summaries. the limited size and scope issue in current lay summarization corpora, @xcite have proposed two novel lay summarization datasets, plos and elife, including biomedical journal articles alongside expert-and author-written lay summaries, which form the basis for the shared task. we submit lay summaries generated by two finetuned llms for this task (one for each dataset). considering the constraints on input size and computational resources (i.e., one gpu with 32 gb memory), using the full length of scientific articles for llm fine-tuning is not feasible. therefore, we adopt an extract-then-summarize approach @xcite (illustrated in figure 1 ), which allows us to reduce the input length while maintaining competitive performance. we fine-tune the longformer encoder-decoder (led) model @xcite and @xcite and explore the effectiveness of combining unsupervised extractive summarization methods with a retrieval-augmented gener-ation (rag) approach @xcite in generating lay summaries. our experimental results on gpt-3.5 achieve the best overall ranking and the highest relevance score in the shared task. ## methods we introduce our methodology for the biomedical lay summarization task (illustrated in figure 1 ), including dataset description, section re-ranking, extractive summarization, rag, gpt-3.5 fine-tuning, and evaluation measures. the detailed experimental settings are reported in appendix a. ## results and analysis we present the evaluation results of our methods in the leaderboard in despite gpt-3.5's better performance over the smaller encoder-decoder models in most evaluation ## discussion and conclusion applying the extract-then-summarize framework to fine-tune gpt-3.5 demonstrates superior performance in the biomedical lay summarization task compared to led-based fine-tuning. the ablation study indicates that incorporating external knowledge during model fine-tuning slightly enhances relevance metrics in the experiments of the ledbase model. however, it negatively impacts factuality scores, similar to the results observed when using extractive summarization and pubmed led large model. none of the components enhance the factuality scores compared to the baseline settings, although there are improvements in relevance and readability scores ( the case study detailed in appendix f reveals that the gpt-3.5 and pubmed led models produce unrelated information when creating lay summaries compared to the gold lay summary. notably, gpt-3.5 produces longer summaries than the pubmed led model despite both models having the same maximum decoding token limit of 512. consequently, while gpt-3.5 includes more relevant sentences that closely match the original summary, it also introduces more irrelevant content. overall, fine-tuning gpt-3.5 with extractive summaries achieves the best overall ranking and highest relevance score in the biolaysumm 2024 shared task, demonstrating the effectiveness of using key sentences from the article for llm finetuning. the pubmed led model, with additional functional components, also shows competitive results compared to gpt-3.5. meanwhile, our findings using the pubmed led model suggest a promising direction for future studies to develop advanced modules that combine extractive summarization and rag to generate lay summaries, especially in improving the relevance scores and enhancing the accessibility of biomedical research.