Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articles (2023)

Abstract: This paper presents the results of the shared task on Lay Summarisation of Biomedical Research Articles (BioLaySumm), hosted at the BioNLP Workshop at ACL 2023. The goal of this shared task is to develop abstractive summarisation models capable of generating “lay summaries” (i.e., summaries that are comprehensible to non-technical audiences) in both a controllable and non-controllable setting. There are two subtasks: 1) Lay Summarisation, where the goal is for participants to build models for lay summary generation only, given the full article text and the corresponding abstract as input; and2) Readability-controlled Summarisation, where the goal is for participants to train models to generate both the technical abstract and the lay summary, given an article’s main text as input. In addition to overall results, we report on the setup and insights from the BioLaySumm shared task, which attracted a total of 20 participating teams across both subtasks.

## introduction biomedical publications report upon the latest research concerning prominent health-related topics, ranging from common illnesses to global pandemics @xcite . accordingly, the content of these publications is of interest to a wide variety of audiences, including researchers, medical professionals, journalists, and even members of the public. however, the highly technical and specialist language used within such articles typically makes it difficult for non-expert audiences to understand their contents. this results in useful knowledge and findings having limited accessibility to the general public @xcite @xcite . abstractive summarisation models can be used to generate a concise summary of an article, capturing its most salient points using words and sentences that do not necessarily appear in the original text of the article. as such, these models have the potential to make highly technical documents accessible to a much wider audience through the generation of "lay summaries" -more readable summaries consisting largely of background information and containing minimal technical terminology @xcite @xcite . the biolaysumm shared task foot_0 focuses on the abstractive summarisation of biomedical articles whilst placing an emphasis on controllability and ensuring comprehensibility for non-expert audiences. through this shared task, we aim to foster increased research interest in lay summarisation (in both controllable and non-controllable settings), enabling further progression for novel model development and high-quality dataset construction. in turn, we hope this will help to broaden the accessibility of technical texts to non-specialist audiences and to drive progress towards more usable and effective abstractive summarisation models for the biomedical domain with the ability to cater to audiences possessing different levels of expertise. in this paper, we present the results of the first biolaysumm shared task, hosted by the bionlp workshop at acl 2023. we cover the task formulation ( §2), datasets ( §3), and evaluation procedure ( §4), before providing a description of the participating systems, overall results, and notable insights ( §5). ## task description the shared task is composed of two separate subtasks, focusing on 1) the generation of summaries more suitable for a lay audience (lay summarisation), and 2) the development of controllable summarisation models capable of catering to audiences with different levels of expertise (readabilitycontrolled summarisation). ## datasets the datasets used within each subtask are based on the previous works of @xcite and @xcite , and are derived from two different biomedical publications: public library of science (plos) and elife. each dataset consists of research articles, their technical abstracts, and their expert-written lay summaries. as detailed in §2, each form of summary within these datasets (i.e., abstract and lay summary) has a different utility in each subtask. the lay summaries of each dataset also exhibit numerous notable differences in their characteristics, with elife's lay summaries being longer, more abstractive, and more readable than those of plos. furthermore, for plos, lay summaries are author-written, and articles are derived from 5 peer-reviewed journals covering biology, computational biology, genetics, pathogens, and neglected tropical diseases. for elife, lay summaries are written by expert editors (in correspondence with authors), and articles are derived from the peer-reviewed elife journal that covers all areas of the life sciences and medicine. for more detailed analysis of dataset content, please refer to @xcite . alternatively, we collect new test splits for both plos and elife data using more recently published articles from each respective journal. the test data for subtask 1 is composed of 142 plos articles and 142 elife articles. the test data for subtask 2 is composed of 142 plos articles (however, these are different from those used in subtask 1). in utilising these datasets for our task, we hope to enable the training of abstractive summarisation models that are capable of generating lay summaries for unseen articles covering a wide range of biomedical topics, enabling the significance of new, important publications to be effectively communicated to non-expert audiences. ## evaluation for both subtasks, we evaluate summary quality according to three criteria -relevance, readability, and factuality -where each criterion is composed of one or more automatic metrics: for subtask 2, the scores presented for each metric are the average of those calculated independently for the generated abstracts and lay summaries. notably, for readability metrics in this subtask, we calculate the absolute difference between the scores of generated summary and target summary pairs (rather than simply using the scores obtained for generated summaries, as in subtask 1). the aim is to maximise the scores for relevance and factuality metrics and minimise the absolute difference scores calculated for readability metrics. following the submission deadline for each subtask, an overall ranking is calculated based on the cumulative rank of the evaluation criteria, where a lower overall ranking equates to better overall performance). to produce a criterion ranking, we apply min-max normalisation to the scores of each metric, before averaging across metrics within each evaluation criterion. ## shared task submissions for both subtasks, we include a baseline system based on bart-base @xcite in order to provide a simple, widely-used benchmark with which submission performance can be compared. for subtask 1, this baseline system is composed of two separate bart models, trained independently on the plos and elife datasets. for subtask 2, the baseline system is a controllable bart model, trained to generate either the abstract or lay summary of an article based on the inclusion of control tokens prepended to the input document ([abstract] and [summary], respectively). participating teams for each subtasks were allowed to make a maximum of 3 submissions in total. ## conclusion the first biolaysumm shared task was hosted at the bionlp workshop @ acl2023 and consisted of two subtasks focusing on lay summari-sation and readability-controlled summarisation, respectively. the task attracted a total of 20 teams, between them making 56 individual submissions across both subtasks. submissions were evaluated according to three general criteria -relevance, readability, and factuality -with each criteria consisting of one or more automatic metrics. the results of both subtasks show that achieving strong performance for all three criteria (relative to other submissions) was particularly rare, attesting to the challenging nature of generating lay summaries for research articles in both controlled and non-controlled settings. furthermore, when also taking into account the relatively strong performance of the bart baseline models (in particular for the factuality component of our evaluation), this suggests that further research effort is required to develop truly usable models that can be reliably deployed in real-world settings. however, as demonstrated by highly-ranked teams mds and marsfield_sds (who obtain first and joint third-ranking submissions for subtask 1, respectively), recent developments in the abilities of both general-purpose and in-domain llms have the potential to offer significant benefits for the automatic generation lay summaries. as such, we expect that utilising such models for summary generation, data augmentation, and evaluation to be promising future directions for lay summarisation.