W is P er M ed at B io L ay S umm: Adapting Autoregressive Large Language Models for Lay Summarization of Scientific Articles (2024)

Abstract: This paper details the efforts of the WisPerMed team in the BioLaySumm2024 Shared Task on automatic lay summarization in the biomedical domain, aimed at making scientific publications accessible to non-specialists. Large language models (LLMs), specifically the BioMistral and Llama3 models, were fine-tuned and employed to create lay summaries from complex scientific texts. The summarization performance was enhanced through various approaches, including instruction tuning, few-shot learning, and prompt variations tailored to incorporate specific context information. The experiments demonstrated that fine-tuning generally led to the best performance across most evaluated metrics. Few-shot learning notably improved the models’ ability to generate relevant and factually accurate texts, particularly when using a well-crafted prompt. Additionally, a Dynamic Expert Selection (DES) mechanism to optimize the selection of text outputs based on readability and factuality metrics was developed. Out of 54 participants, the WisPerMed team reached the 4th place, measured by readability, factuality, and relevance. Determined by the overall score, our approach improved upon the baseline by approx. 5.5 percentage points and was only approx. 1.5 percentage points behind the first place.

## introduction in the biomedical domain, scientific publications and research play a central role in communicating research findings and results. however, these documents are usually written in complex language and use terminology and technical jargon that can be challenging for lay readers or researchers from different fields to understand @xcite . in this context, lay summarization can be utilized to extract the most relevant information from the original article or publication while also providing supplementary explanations. this often entails incorporating background information that may not be contained within the article itself. in this context, this paper presents the participation of the team wispermed in the biolay-summ2024 shared task @xcite on automatic lay summarization and describes the employed approaches to tackle this challenge. summaries generated by llms, as demonstrated by @xcite , can be of equivalent or superior quality to original references. additionally, instruction tuning is an effective approach for enhancing performance. however, llms face limitations when applied to domain-specific abstractive summarization. key challenges include the quadratic complexity of transformer-based models @xcite concerning input text length, model hallucination, where factually incorrect text is generated, and domain shift from training to test data @xcite . similarly, studies on text simplification @xcite indicate that although general-purpose llms are capable of effectively simplifying clinical reports, they sometimes generate factual inaccuracies and omit crucial information. to adapt llms to a specific domain or task @xcite , it is possible to fine-tune the models, leverage few-shot learning or further pre-train the models on domain data. examples of domainadapted llms for the biomedical domain include the biomistral @xcite and openbi-ollm @xcite model series. the biomistral models are based on the mistral 7b instruct v0.1 @xcite model. they are further pre-trained on the pmc open access subset 1 . openbiollm models are based on the llama3 (ai@meta, 2024) models and were adapted to the biomedical domain through finetuning. ## dataset the dataset @xcite of the shared task @xcite ) contains two collections of scientific journal articles and the corresponding lay summaries, namely plos and elife. plos and elife also include the section headings and keywords of the article. the plos dataset has 24,773 examples in the training split and 1,376 examples in the validation split, whereas the elife dataset is smaller with 4,346 examples in the training split and 241 examples in the validation split. the test split consists of 142 examples for both datasets. lay summaries of the plos dataset were written by the authors of the articles and are approximately 150-200 words long, while elife lay summaries were written by expert editors in correspondence with the authors and are about twice as long. for the remainder of this paper, any reference to the validation or test set will include elife and plos unless otherwise specified. ## evaluation metrics the generated summaries were evaluated across ten metrics that fall into the following categories: relevance, readability, and factuality. relevance was assessed through recall-oriented understudy for gisting evaluation @xcite (rouge-1, rouge-2, rouge-l) and bertscore @xcite . rouge counts the overlapping ngrams in the generated texts and target lay summaries, whereas bertscore uses contextual word embeddings to compare the semantic similarity of the two texts. readability was evaluated using the flesch-kincaid grade level (fkgl) @xcite , dale-chall readability score (dcrs) @xcite , coleman-liau index (cli) @xcite , and learnable evaluation metric for text simplification (lens) @xcite . the fkgl uses sentence lengths and syllable count per word to estimate readability. the dcrs uses a word list to compute the occurrences of words unknown to most 4-th graders and the cli estimates the grade level necessary to comprehend the text. the lens metric is a learnable evaluation metric trained on datasets containing human ratings of simplifications. in this setting, lens measures the simplification of the abstract by the generated text using the target lay summary as a reference. factuality was assessed with alignscore @xcite and summary consistency (summac) @xcite . the alignscore quantifies the degree of alignment between the facts in the summary and the scientific article, while summac also includes consistency. ## methods and experiments this section outlines the methodology employed in the experiments conducted on the specified dataset. ## results the results of the experiments using biom, llama3, and openbio are presented in biom exhibits the highest lens score in the zero-shot setting. however, its relevance and factuality performance are the lowest. few-shot learning resulted in enhanced performance across all metrics except for lens. the persona prompt (biom pers ) led to an improvement in relevance. including the introduction in the prompt (biom intro ) resulted in a reduction in all aspects despite the fact that the model had access to more information from the article itself. in comparison, the prompt with the guide (biom guide ) exhibits minimal enhancements. the optimal few-shot learning for biom occurred with the initial prompt, which achieved the highest readability and factuality in the few-shot setting, excluding the des approach. however, openbio slightly underperformed with this prompt in the few-shot setting, except for the lens score, where it performed best in this setting. the des used all four prompts and outperformed the baseline with improvements in factuality and readability, achieving the best results in the few-shot setting. fine-tuning biom improved relevance and factuality scores, though the lens score decreased slightly, with other readability metrics similar to the few-shot setting. the fine-tuned biom outperformed the baseline in terms of relevance and overall quality. the des approach improved all metrics except for a slight drop in the lens score. in contrast, llama3 underperformed despite being larger. it was less effective at extracting relevant information from full articles and produced lowerquality text in terms of readability, even though its lens score was higher than biom's. additionally, llama3's factuality scores decreased, leading to an overall performance drop compared to the baseline. ## conclusion this paper presents the wispermed team's approaches to automatic lay summarization within the biomedical domain, utilizing a combination of fine-tuning, prompt variations, and dynamic expert selection. among these approaches, fine-tuning emerged as an effective method, leading to the best performance across most metrics. this underscores the importance of task-specific training in optimizing model output for complex summarization tasks. additionally, biom showed strong few-shot learning capabilities, illustrating its robustness and versatility in generating accurate and relevant summaries even without extensive training. as the model adjusts to the factuality and readability of given examples, providing better examples could lead to further enhancements in these aspects. biom reached high factuality, even when provided solely with abstracts as input, suggesting that biom leveraged domain-specific knowledge acquired during pre-training. this indicates that domain adaptation remains an important factor when using llms for lay summarization of scientific articles, as biom outperformed the larger general model llama3. the four prompt variations exhibited differing effects on the evaluation metrics. biom is adept in fulfilling the role of a science communicator (biom pers ), as evidenced by the enhanced relevance. biom intro and biom guide did not significantly enhance the metrics, indicating that the increase in context was not beneficial for all texts. without des, a shorter prompt (biom initial ) yielded the optimal results, suggesting that the model effectively comprehends the task from the provided examples. the des mechanism further refined readability and, in particular, factuality by retrospectively selecting the best text outputs based on evaluation metrics. this highlights the potential of metricdriven selection to improve the quality of lay summaries further. in conclusion, our study demonstrates that finetuning, the use of informed prompt variations, and selection mechanisms can enhance the capability of autoregressive llms to produce lay summaries that are factually accurate, relevant, and readily accessible to non-specialist audiences. this approach fosters broader public engagement with scientific findings, advancing the goal of making biomedical research comprehensible and accessible.