SINAI at B io L ay S umm: Self-Play Fine-Tuning of Large Language Models for Biomedical Lay Summarisation (2024)

Abstract: An effective disclosure of scientific knowledge and advancements to the general public is often hindered by the complexity of the technical language used in research which often results very difficult, if not impossible, for non-experts to understand. In this paper we present the approach developed by the SINAI team as the result of our participation in BioLaySumm shared task hosted by the BioNLP workshop at ACL 2024. Our approach stems from the experimentation we performed in order to test the ability of state-of-the-art pre-trained large language models, namely GPT 3.5, GPT 4 and Llama-3, to tackle this task in a few-shot manner. In order to improve this baseline, we opted for fine-tuning Llama-3 by applying parameter-efficient methodologies. The best performing system which resulted from applying self-play fine tuning method which allows the model to improve while learning to distinguish between its own generations from the previous step from the gold standard summaries. This approach achieved 0.4205 ROUGE-1 score and 0.8583 BERTScore.

## introduction science outreach and scientific advocacy are crucial for the development of the science itself, as most funding still comes from public sources and thus demands public's support. furthermore, science is central to most of the grand challenges of today's society, such as climate change, economic productivity, health and new drug discovery. these factors highlight the relevance of making information about scientific advancements accessible for general public. moreover, it also may help the public make sound and informed choices about issues like participating in a clinical trial or getting a vaccination @xcite . nevertheless, even with an increased online availability of scientific publications, accessing the information from these sources remains a challenging task for non-experts due to the technical language and specific terminology used to write scientific work. one viable solution for addressing the informational requirements of the general public or gatekeepers like journalists are plain language summaries or lay summaries -a format that presents scientific research in an easily understandable manner for non-experts @xcite . however, manual generation of lay summaries is a tedious and costly process that involves contracting expert writers specialised in science outreach. for this reason, the development of effective automatic lay summarisation systems is attracting an increasing amount of attention of the natural language processing (nlp) researchers @xcite . biolaysumm shared task held on the bionlp 2024 workshop at acl brings the community effort to tackle the task of automatic abstractive summarisation of biomedical articles for non-technical audiences by leveraging the extensive work done by the creators of elife @xcite and the public library of science (plos) database in manually composing lay summaries. this paper presents the methodology developed by the sinai team as a part of our participation in the biolaysumm shared task. our experimentation involved comparison between few-shot learning (fsl) of instruction-tuned pre-trained large language models (llms), parameter-efficient tuning of open-source pre-trained llms and self-play fine tuning (spin) methodology which allows the model to improve while learning to distinguish between its own generations from the previous step form the gold standard summaries. the latter mentioned approach resulted to be the highest-scoring submission among all made by our team achieving 0.4205 rouge-1 score and 0.8583 bertscore. the remainder of the paper is structured as follows: section 2 provides a concise description of the data utilized for this task. section 3 details the systems developed by our team for the official evaluation. the details of the evaluation process and the results are presented in section 4. finally, section 5 concludes our work. ## data the organisers of the biolaysumm shared task put at the disposal of the participants two datasets, plos and elife, each of which consisted of biomedical research articles (including their technical abstracts) and their expert-written lay summaries @xcite . as for the dimensions of the data, while plos may be considered large-scale dataset with 24,773 instances for training and 1,376 for validation, elife is a mediumscale dataset comprised of 4,246 training instances and 241 validation instances. one important difference between the two datasets lies in the process of its generation. the public library of science (plos) is a publisher that hosts peer-reviewed journals several of which require authors to submit an 150-200 word long author summary alongside their work. in contrast with this, elife is an open-access journal that started creating plain-language summaries of its research articles in 2021 @xcite . as a result, lay summaries from two datasets differ from each other according to several characteristics, such as length and the extent to which they are abstractive. as can be seen on ## methods this section details the implementation of systems presented by our team for the official evaluation. we tested three approaches to lay summary generation: fsl of instruction-tuned models, parameter efficient fine-tuning of text generation pre-trained models and a novel method of fine-tuning of llms called spin. 1 we used the tokenizer of biomistral-7b model to split the texts into tokens 3.1 few-shot learning fsl of pre-trained llms like gpt-3.5 proved to be a robust approach to the task of lay summarisation during the previous editions of the biolaysumm shared task @xcite . for this reasons we decided to perform experiments with both closed, such as gpt-3.5 @xcite and gpt-4 @xcite and open models, namely llama-3 (ai@meta, 2024). the gpt models were accessed via ope-nai api foot_0 , while the llama-3-8b model was run on 1 nvidia tesla v100-pcie-32gb by making use of transformers python library @xcite . our team adopted 2-shot-prompting approach that introduced one randomly selected example from each of the datasets and injected a mention of the dataset to which each text abstract belonged. thus, our method creates one system for the two datasets. the details regarding the prompt can be found in appendix a. while performing the experiments, we empirically found out that outputs from llama-3 occasionally contained definition of what a lay summary is, repetition of prompt's content and lay summary prefix. in order to remove this noise from the generation, we designed a post-processing procedure based on regular expressions. ## evaluation this section presents the results of the official evaluation campaign that was carried out by the organizers by assessing the predictions made by our system on 142 articles for each of the two datasets. ## conclusions in this study, we explored various methodologies to generate lay summaries of biomedical articles, an important task for improving public accessibility to scientific information. our participation in the bi-olaysumm shared task at the bionlp2024 workshop involved experimenting with fsl, parameterefficient tuning and spin methods. among these, spin fine-tuning demonstrated the highest performance in terms of relevance metrics, achieving a 0.4205 rouge-1 score and 0.8583 bertscore. the evaluation of readability and factuality revealed a trade-off between these two aspects. models that generated more readable texts tended to have lower factuality scores, with the gpt-4 based fsl systems exemplifying this trend. conversely, the fine-tuned llama-3-8b with qlora achieved the best factuality scores while getting fairly good readability scores as well, indicating its potential for producing accurate and readable summaries.