Saama Technologies at B io L ay S umm: Abstract based fine-tuned models with L o RA (2024)

Abstract: Lay summarization of biomedical research articles is a challenging problem due to their use of technical terms and background knowledge requirements, despite the potential benefits of these research articles to the public. We worked on this problem as participating in BioLaySumm 2024. We experimented with various fine-tuning approaches to generate better lay summaries for biomedical research articles. After several experiments, we built a LoRA model with unsupervised fine-tuning based on the abstracts of the given articles, followed by a post-processing unit to take off repeated sentences. Our model was ranked 3rd overall in the BioLaySumm 2024 leaderboard. We analyzed the different approaches we experimented with and suggested several ideas to improve our model further.

## introduction while many academic publications in the biomedical field can potentially benefit a wide readership including many non-experts, their accessibility is often limited by their use of technical terms and relatively sophisticated expressions. therefore the summarization of biomedical research articles is an interesting and important task that can benefit the general public, and biolaysumm 2024 @xcite aims to solve this question by adopting techniques of nlp. biolaysumm asks participants to suggest models that summarize the biomedical articles based on the plos and elife datasets @xcite composed of original research articles and lay summaries written by experts. in this paper, we explain our approaches to the biolaysumm 2024 in detail. to generate better lay summaries, we experimented with multiple finetuning approaches with lora based on the abstract part of the biomedical research papers. as a result of a series of experimentations, we concluded that our best-performing model is the unsupervised fine-tuned model with lora followed by a postprocessing unit that chops off repeated sentences in the raw predictions. at the end of the competition, our model was ranked 3rd overall in biolaysumm 2024 leaderboard. ## system overview to find the best-performing system for biolay-summ 2024, we experimented with several different systems based on the abstracts of the research articles. in this section, we introduce the systems we experimented including the system we submitted to the leaderboard of biolaysumm 2024. throughout all experiments, we used elife (plos) training data only for model training or prompting 1 https://github.com/tgoldsack1/biolaysumm2024evaluation_scripts to generate summaries for elife (plos) validation/test data. ## conclusion as we participated in biolaysumm 2024, we experimented with different fine-tuning approaches with lora to generate summaries based on the given abstract of a biomedical research article. in particular, we explored unsupervised fine-tuning, supervised fine-tuning, and direct preference optimization, and we concluded that our bestperforming model is the unsupervised fine-tuned model with post-processing to chop off repeated sentences. our model achieved 3rd place overall in the leaderboard of biolaysumm 2024. while our model was successful, it would be interesting to extend our approach to a variety of larger llms or to adopt other schemes to utilize the full article of the research paper instead of the abstracts. potential future researches on analysis on different fine-tuning methodologies and benchmarking on evaluation criteria beyond the current challenge may deepen the understanding of our approach.