LHS 712 EE at B io L ay S umm 2023: Using BART and LED to summarize biomedical research articles (2023)

Abstract: As part of our participation in BioLaySumm 2023, we explored the use of large language models (LLMs) to automatically generate concise and readable summaries of biomedical research articles. We utilized pre-trained LLMs to fine-tune our summarization models on two provided datasets, and adapt them to the shared task within the constraints of training time and computational power. Our final models achieved very high relevance and factuality scores on the test set, and ranked among the top five models in the overall performance.

## introduction biomedical research articles are vital sources of information based on latest scholarly findings in the health domain, ranging from acute and chronic health diagnoses, population health, advances in cellular, molecular, and pharmaceutical technologies, and long-term impacts of recent global pandemics. however, due to the highly technical jargon and professional language used in these articles, it can be challenging for laypersons who didn't receive formal training in biomedical sciences to understand the contents. summarization of biomedical text in layperson-friendly language can provide a solution by automatically generating concise and readable summaries of technical documents @xcite . these lay summaries can be used to communicate the key findings of a study to a broader audience, including patients, their care partners, and members of the general public. text summarization has been extensively studied by researchers in the past. there are two main categories of text summarization techniques -extractive summarization and abstractive summarization @xcite . extractive summaries only include key phrases and sentences directly selected from the original text, while abstractive summaries consist of new, generated sen-tences that summarize the original content @xcite . although abstractive summarization is more challenging and complex than extractive summarization, it has the potential to convey more information and better meet human needs @xcite . in recent years, most of the research on abstractive summarization was inspired by the encoder-decoder architecture of deep neural network models. this includes the use of pre-trained encoders proposed by @xcite and the development of multi-task encoder-decoder models, as suggested by @xcite . as part of our participation in the biolaysumm 2023 shared tasks on lay summarization of biomedical research articles, we investigated the use of large language models (llms) to generate concise, relevant, factual, and easily readable summaries @xcite . we utilized existing llms, including bidirectional and auto-regressive transformer (bart) @xcite and longformer encoder-decoder (led) @xcite , to train our summarization model on the provided datasets. despite having limited time and computational resources, we investigated several approaches to fine-tune the llms for this task. further, we retrained our model over the abstract sections of articles in the plos dataset provided as a reference to generate the technical abstracts for the readability-controlled summarization task. our final summarization model achieved strong performance in relevance and factuality, and was ranked among the top five models for lay summarization of biomedical literature. ## results task 1 the performance of the two submissions for task 1 were not significantly different from one another. task 2 while the final model performed very well on the lay summarization task (task 1), it yielded relatively poor results in technical abstract generation compared to other submissions. the performance of our model significantly improved compared to the first submission ( ## discussion we leveraged the capabilities of two transformer based models -bart and longformer -to create high-quality lay summaries or technical abstracts of biomedical research articles, and ensuring that our approach can handle the challenges associated with processing long, complex text. for task 1, there was no significant improvement in performance after we changed the model from bart to led. the led models were expected to perform better because they can accept longer sentences, enhancing their ability to capture text features. however, due to time and memory constraints, we used fewer epochs and smaller batch sizes to train the led model, which might have resulted in its similar performance to the bart model. in task 2, the model trained for lay summarization did not perform well in generating technical abstracts. after retraining the model on plos abstracts, the model was able to perform as well as the previous model. this implies that adjusting the parameters of the existing model alone does not change the readability of the generated text. on the other hand, retraining with the expected output as a reference can improve the model performance. ## conclusion in our participation in the biolaysumm 2023 shared tasks, we were able to successfully utilize pre-trained large language models -bart and led -to generate lay summaries and technical abstracts. we did so by fine-tuning our summarization models on two different datasets, plos and elife. despite the limited time and computational resources, we were able to develop models that performed well in the relevance and factuality score of the summarizing tasks, and finally ranked as the fifth best models in the overall performance. the biolaysumm tasks showed the potential of lay summarization models in making biomedical research accessible to a broader audience. we believe that the development of these models will continue to play a critical role in advancing healthcare and empowering individuals to make informed decisions about their health.