HGP - NLP at B io L ay S umm: Leveraging L o RA for Lay Summarization of Biomedical Research Articles using S eq2 S eq Transformers (2024)

Abstract: Lay summarization aims to generate summaries of technical articles for non-experts, enabling easy comprehension for a general audience. The technical language used in research often hinders effective communication of scientific knowledge, making it difficult for non-experts to understand. Automatic lay summarization can enhance access to scientific literature, promoting interdisciplinary knowledge sharing and public understanding. This has become especially important for biomedical articles, given the current global need for clear medical information. Large Language Models (LLMs), with their remarkable language understanding capabilities, are ideal for abstractive summarization, helping to make complex information accessible to the public. This paper details our submissions to the BioLaySumm 2024 Shared Task: Lay Summarization of Biomedical Research Articles. We fine-tune and evaluate sequence-to-sequence models like T5 across various training dataset settings and optimization methods such as LoRA for lay summarization. Our submission achieved the 53rd position overall.

## introduction scientific research aims to advance knowledge, but a major challenge is the lack of domain-specific knowledge among the public. technical reports and research articles are often incomprehensible to non-experts, hindering knowledge dissemination. lay summarization addresses this by generating factual, readable summaries of technical texts for non-experts @xcite . in the biomedical field, lay summarization is crucial due to its highly technical content involving complex medical terms and detailed research findings. access to clear medical information is essential for informed health decisions. misunderstandings or a lack of access can lead to poor decisions, increased anxiety, and a general distrust of medical advice @xcite . recent advancements in large language models (llms) @xcite and autoregressive llms @xcite like chatgpt, gemini, mistral, and llama have shown significant potential. with their large-scale pretraining, these models can generate high-quality, contextually relevant summaries that are informative and accessible. however, they are computationally expensive and often require fine-tuning for specific tasks, needing substantial computational resources and energy. sequence-to-sequence models @xcite provide a promising alternative, addressing these computational challenges. with an encoder-decoder structure, these models efficiently handle input and output sequences, making them well-suited for summarization @xcite . they are more computationally efficient than llms, requiring fewer resources while delivering high performance. this paper describes our approach to the biolaysumm 2024 shared task: lay summarization of biomedical research articles @xcite . we fine-tune and evaluate sequence-to-sequence models like t5 for the task of lay summarization (challagundla and peddavenkatagari, 2024) across various training dataset settings and optimization methods such as lora @xcite . our final submission to this task is a lora-based flan-t5-base model @xcite , which secured us the 53rd position on the leaderboard. ## system overview our approach to the task involved fine-tuning sequence-to-sequence transformer-based models @xcite , such as t5, across various training dataset settings and optimization methods like lora on the plos and elife datasets. the models were evaluated based on the readability @xcite , relevance @xcite , and factuality @xcite of the summaries generated. the model demonstrating the best performance across these three evaluation criteria was chosen as our final submission for the task. ## analysis & results the challenge provides two individual datasets of different distributions for training, evaluation, and testing. details about the distribution of the datasets are illustrated in the detailed results of our experiments are presented in ## conclusion we fine-tune and evaluate sequence-to-sequence models like t5 for the task of lay summarization (challagundla and peddavenkatagari, 2024) across various training dataset settings and optimization methods such as lora. we extended our evaluation beyond traditional accuracy metrics to encompass real-world application considerations like relevance @xcite , readability @xcite , and factuality @xcite . our analysis revealed interesting trade-offs of fine-tuning pre-trained models using traditional and parameter-efficient methods like lora @xcite . future work should explore the impact of peft-similar models for lay and abstractive summarization.