Overview of the B io L ay S umm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles (2024)

Abstract: This paper presents the setup and results of the second edition of the BioLaySumm shared task on the Lay Summarisation of Biomedical Research Articles, hosted at the BioNLP Workshop at ACL 2024. In this task edition, we aim to build on the first edition’s success by further increasing research interest in this important task and encouraging participants to explore novel approaches that will help advance the state-of-the-art. Encouragingly, we found research interest in the task to be high, with this edition of the task attracting a total of 53 participating teams, a significant increase in engagement from the previous edition. Overall, our results show that a broad range of innovative approaches were adopted by task participants, with a predictable shift towards the use of Large Language Models (LLMs).

## introduction lay summarisation describes the task of generating a summary of a technical or specialist text that is suitable for a non-expert audience. to achieve this goal, a good lay summary will typically focus on explaining the relevant background information alongside the significance and findings of an article, while avoiding extensive use of jargon or technical language. as such, lay summaries offer significant benefits in broadening access to technical articles that are of interest to a broad range of audiences. biomedical research publications, containing the latest research on prominent health-related topics, represent a perfect example of such texts. not only are the contents of these articles relevant to other researchers working in the same domain, but often they can also be of interest to researchers in related domains, medical practitioners, and even members of the public (e.g., those affected by an illness/disease being studied). in this scenario, the lay summary is an essential tool in allowing these secondary audiences, who don't possess the expertise required to interpret the full article, to understand its key findings and relevance to their information needs. although lay summaries are required or encouraged by some biomedical publications, they are not universal, leaving a significant amount of inaccessible to lay audiences. furthermore, the burden of writing these summaries is often placed upon the article authors, who are not always adept at effectively communicating their work to a non-technical audience. as such, automatic lay summary generation has significant potential to help both authors and non-expert readers by improving the dissemination of important research. the biolaysumm shared task foot_0 focuses on improving the automatic lay summarization of biomedical research. through this shared task, we aim to encourage the development of novel approaches and increase interest in the research of automatic techniques for disseminating scientific research to broad audiences. in this paper, we present the results of the second edition of the biolaysumm shared task, hosted by the bionlp workshop at acl @xcite in what remains of the paper, we address the task formulation ( §2), datasets ( §3), and evaluation procedure ( §4), before providing a description of overall results and notable insights ( §5), and finall the participating systems ( §6). ## task description as part of the task, participants must develop systems capable of generating a lay summary of biomedical research, given the article's text as input. our competition was hosted using the codabench platform @xcite . as with the previous edition of the task, two separate datasets, plos and elife are used. participants were provided with both training and valida-tion sets, complete with reference lay summaries, alongside a blind test set. final system performance is determined by the performance of participants' systems on the blind test set, which could be obtained by submitting their predicted lay summaries to our codabench competition, where they were automatically evaluated. more information regarding our datasets and evaluation protocol is provided in subsequent sections ( §3 and §4, respectively). we allowed submissions to be generated from either two separate summarisation models (i.e., one trained on each dataset) or a single unified model (i.e., trained on both datasets). participants were required to indicate which approach was taken for each submission, in addition to whether or not they made use of additional training data (i.e., data not provided specifically for the task). participants were also allowed to compete as part of teams, where each team was permitted to make a maximum of 15 test set submissions to the codabench platform. 2 however, teams were required to select only one of their submissions to appear on the final task leaderboard. because of its strong performance in the previous edition of the task, we also choose to keep the same baseline system. specifically, this baseline system consists of two separate bart-base models @xcite , trained independently on the plos and elife datasets. ## datasets the datasets used for the task are based on the previous works of @xcite and @xcite , and are derived from two different biomedical publications: public library of science (plos) and elife. each dataset consists of biomedical research articles paired with expertwritten lay summaries. as described in @xcite , the lay summaries of each dataset also exhibit numerous notable differences in their characteristics, with elife's lay summaries being longer, more abstractive, and more readable than those of plos. furthermore, for plos, lay summaries are author-written, and articles are derived from 5 peer-reviewed journals covering biology, computational biology, genetics, pathogens, and neglected tropical diseases. for elife, lay summaries are written by expert editors (in correspondence with authors), and articles are derived from the peerreviewed elife journal, covering all areas of the life sciences and medicine. for a more detailed analysis of dataset content, readers can refer to @xcite . as with the previous task edition, we collect new test splits for both plos and elife data using more recently published articles from each respective journal. this test data consists of 142 plos articles and 142 elife articles. in utilizing these datasets for our task, we hope to enable the training of abstractive summarisation models that are capable of generating lay summaries for unseen articles covering a wide range of biomedical topics, enabling the significance of new, important publications to be effectively communicated to non-expert audiences. ## evaluation for both subtasks, we evaluate summary quality according to three criteria -relevance, readability, and factuality -where each criterion is composed of one or more automatic metrics: the scores calculated for each metric are the average of those calculated independently for the generated lay summaries of plos and elife. the aim is to maximize the scores for all metrics except for fkgl, dcrs, and cli the readability metrics. for these metrics, the aim is to minimize scores, as lower scores are indicative of greater readability. 3 following the submission deadline for each subtask, an overall ranking is calculated based on the average performance of submissions across all criteria. specifically, we first apply min-max normalization to the scores of each metric (thus establishing a common value range), before averaging across metrics within each criterion to obtain criterion-level scores. 4 note that, for metrics that we minimize (i.e., fkgl, dcrs, and cli) we calculate 1 minus the mix-max normalized value. finally, criterion-level scores are then averaged to obtain an overall score, by which submissions are then ranked. ## task results overall, the final leaderboard of biolaysumm 2024 contains a total of 53 teams, who made a combined total of over 200 submissions. this represents a 165% increase in participation over bio-laysumm 2023, which attracted a total of 20 teams across two subtasks. in this section, we summarize some of the key results and notable trends that were observed among participants. ## submissions out of the 53 participating teams, 14 teams submitted system papers. here, we provide a brief summary of the approaches taken by these teams. uiuc_bionlp @xcite this team produced the top-ranked submission, adopting an extract-then-summarize approach that utilizes tex-trank @xcite for salient sentence extraction, followed by a fine-tuned gpt-3.5-turbo model for summary generation. specifically, their submitted system extracted the top 40 most salient sentences using textrank, and their gpt-bsaed model is fine-tuned on 200 examples. additional experimentation was conducted using various extractive summarization approaches and comparing the number of examples required for effective fine-tuning. furthermore, the team also explored a longformer-based approach that further incorporates retrieved wikipedia data in a retrieval-augmented generation (rag) setup. cytun ai @xcite making the secondranked submission, the methodology of this team surrounds the use of fine-tuned llms. as part of their experimentation, they compare two approaches for handling lengthy input articles: hard truncation and text chunking. additionally, their summary-generation pipeline includes data preprocessing, augmentation, and prompt engineering. saama technologies @xcite this team achieved the third-ranking submission, which surrounded fine-tuning a mistral-7b model foot_7 in an unsupervised fashion using low-ranked adaptation (lora) @xcite , followed by zero-shot summary generation and post-processing to remove redundant sentences. this team also experiments with several other fine-tuning methods, including supervised fine-tuning with lora and direct preference optimization @xcite . @xcite ranking in fourth place, the selected submission of wis-permed utilized a fine-tuned biomistral model, combined with few-shot prompting and a dynamic-expert selection (des) mechanism. specifically, their biomistral model was trained using abstracts and lay summaries of the provided train set; and their proposed des mechanism involved generating several lay summary versions with different prompts for a given input, before selecting the most desirable based on the scores of the references-less readability and factuality metrics used in the task. in additional experiments, they also measured system performance utilizing llama3, as well as that of few-shot and zero-shot model variants. the task organizers selected this team to receive an award for the "most innovative approach". auth @xcite being one of the only teams to utilize external data, this retrieves 300 abstract-lay abstract pairs scraped from the science journal for kids website. 8 they use this retrieved data as in-context examples for gpt-4, which they prompt to augment the provided datasets by rewriting reference summaries with higher readability scores. finally, they use this data to fine-tune to fine-tune biobart @xcite , whilst also experimenting with controllable generation techniques in the form of control tokens prepended to the input article (<@xmath0. eulerian @xcite the team experimented with different combinations of the flan-t5 (chung et al., 2022b) model variations and data selection. they compare the performance of these methods with a preposed "preprocessing over abstract" technique, in which they use a regular expression to remove some abstract information (i.e., anything inside of parentheses, braces and brackets), finding that this outperforms all neural methods tested in terms of relevance and factuality metrics. biolay_ak_ss @xcite focusing largely on data augmentation, this team generated additional summary samples using 2 generalpurpose models: bart @xcite and pegasus @xcite . the augmented dataset was then used to fine-tune a domain-specific biobart model, which was found to improve its overall improved overall performance. hulat-uc3m @xcite again comparing the performance of domainsspecific and general-purpose models, this team experimented with fine-tuning both longformer @xcite and biobart models on the 8 https://sciencejournalforkids.org/ given datasets. additionally, they experiment with extending biobart to utilise longformer-based sparse attention, thus allowing it to process longer inputs. overall, they found that fine-tuning the standard biobart model on each dataset yields the best performance. deakinnlp @xcite this team assessed the performance of both a fine-tuned longformer and gpt-4 (with zero-and few-shot prompting). additional analysis is also conducted surrounding data selection and the performance vs. cost trade-off between select methods. rag-rlrc-laysum @xcite this team developed a retrieval-augmented generation (rag) lay summarisation approach, utilizing multiple knowledge sources (including both source documents and wikipedia). they experiment with llms (gemini and chatgpt) for both summary generation and paraphrasing, in addition to a longformer baseline. lastly, the team also develop a reinforcement learning strategy to fine-tune the readability of generated summaries. sinai @xcite focusing largely on a few-shot setting, this team compared the performance of several popular llms including gpt-3.5, gpt-4, and llama3. further experimentation surrounded the fine-tuning of a smaller llama model (llama3-8b) using both parameter-efficient lora techniques and direct preference optimization @xcite . @xcite this team performed a thorough comparison of several state-of-the-art llms, focusing largely on comparing the readability of generated summaries. further experimentation surrounds summary rewriting, title infusing, k-shot prompting, and lora-based fine-tuning, with their best-performing submission utilizing a combination of these methods and obtaining the best overall readability scores. hgp_nlp @xcite this team finetune and evaluate multiple t5 model variants, also experimenting with lora-based fine-tuning. ## conclusion the second edition of the biolaysumm shared task was hosted by the bionlp workshop@acl 2024. several changes were implemented over the previous edition of the task covering participation rules, evaluation metrics, and ranking protocol. in terms of participant engagement, the task attracted a total of 53 teams, representing a significant growth from the previous edition's 20 teams. our results indicate a drastic shift towards the use of llms for lay summarisation, with a wide range of both domain-specific and general-purpose llms being adopted in various settings across participant submissions.