CSIRO D ata61 Team at B io L ay S umm Task 1: Lay Summarisation of Biomedical Research Articles Using Generative Models (2023)

Abstract: Lay summarisation aims at generating a summary for non-expert audience which allows them to keep updated with latest research in a specific field. Despite the significant advancements made in the field of text summarisation, lay summarisation remains relatively under-explored. We present a comprehensive set of experiments and analysis to investigate the effectiveness of existing pre-trained language models in generating lay summaries. When evaluate our models using a BioNLP Shared Task, BioLaySumm, our submission ranked second for the relevance criteria and third overall among 21 competing teams.

## introduction text summarisation @xcite is a task in natural language processing (nlp) where its goal is to generate a short and informative summary. recent developments of summarisation techniques are supported by advancement in pre-trained language models @xcite @xcite and availability of largescale datasets which include news articles @xcite , scientific publications @xcite and meeting records @xcite . despite the advances in generic text summarisation, lay summarisation for scientific documents remains less explored. that is, there has been less focus on simplifying technical terms in scientific documents, causing the generated summary to have the same readability level as the input text. lay summarisation aims to produce a non-technical summary for technical articles @xcite . for instance, a lay summary for a scientific article should not contain any technical terms and should be readable and understandable to the public. the lay summary is especially important in the biomedical area, where people with no relevant background such as journalists, interdisciplinary researchers, or patients may want to keep updated with recent advances in the field. we participate in the biolaysumm shared task @xcite to investigate methods of generating high-quality lay summaries for scientific articles. we present our empirical findings from three different perspectives: (1) model size; (2) data augmentation techniques; and, (3) input length. our contributions are two-fold: (1) we explore the effectiveness of using pre-trained language models for fine-tuning and data augmentation in lay summarisation, and (2) we investigate the impact of input length on the quality of the generated summary. in the final shared task rankings, foot_0 our approach ranks second for relevance metric and third overall among 21 teams, with a relatively high balance across all metrics. pre-trained lms in summarisation text summarisation has benefited from pre-trained language models such as bert @xcite , bart @xcite , and t5 @xcite . most studies however used pre-training objectives that are not related to summary generation. summarisation-specific pre-trained language models are also studied. @xcite introduced bertsum, which is capable of generating both extractive and abstractive summaries with a document-level encoder built on top of bert. @xcite proposed pegasus, a transformer-based pre-trained language model where summary generation is explicitly included as a pre-training objective. the model is trained to generate abstractive summaries from masked sentences in a document. here, we leverage the pre-trained lms not only to generate lay summary but also to generate additional training data. ## approach backbone models t5 @xcite ) is a transformer-based text-to-text pre-trained language model. different tasks, including translation, summarisation, question answering, and classification, are all converted into a text-to-text format. flan-t5 @xcite is an enhanced version of t5 using instruction fine-tuning. that is, the original t5 model is fine-tuned on tasks phrased as instructions. we use pre-trained flan-t5 models of different sizes (i.e., base, large, and xl), 3 with the same instruction across all experiments: "summarise the following article: [document] summary:". since the model should generate long summaries, we use a beam search decoder during inference-a beam width of four-to generate up to 512 tokens. intermediate-task pretraining due to the relatively small size of elife training set, we also explore the effectiveness of intermediate-task pretraining @xcite @xcite to improve the effectiveness of fine-tuned flan-t5 models. specifically, we first fine-tune flan-t5 on a target task-related data set (e.g., plos training set and xsum @xcite and then on the elife training set. in addition, we employ an off-theshelf pre-trained language model (i.e., gpt-3.5) to generate additional training data, which will be explained in detail in section 5. ## datasets and evaluation metrics there are two lay summarisation datasets used for the shared task, namely elife @xcite and plos @xcite . elife foot_1 is a scientific journal for biomedical and life sciences. publications on elife contain lay summaries written by expert editors based on the journal article to explain the background and key points of a scientific article to the non-expert. plos foot_2 (public library of science) is an openaccess journal, focusing on science and medicine publications where each publication is paired with a lay summary provided by the authors themselves. the shared task organisers provided training and development sets for model development, and two hidden test sets to score submissions. the descriptive statistics of these datasets can be found in we follow the setup of the shared task to evaluate generated summaries across three aspects: relevance (rouge, bertscore), readability (fkgl, dcrs), and factuality (bartscore). the aim is to maximise the scores for relevance and factuality metrics and minimise scores for readability metrics. we provide a short description of these metrics and refer readers to ( goldsack et al., 2023 rouge @xcite assesses the quality of generated summaries by comparing generated summaries to gold summaries and counting the number of overlapping n-grams, word sequences, and word pairs between summaries. bertscore @xcite uses contextual embedding in bert @xcite to compute the similarity score for each token between the source document and generated summary and average them to obtain the overall effectiveness metric of a summarisation model. fkgl (flesch-kincaid grade level) @xcite indicates difficulty in reading a passage in english based on two factors: the average sentence length and the average number of syllables per word. dcrs (dale-chall readability score) @xcite is a score based on occurrence of words unknown to most 4th-grade students (in the us education system). a lower score indicates higher readability. bartscore @xcite defines the evaluation of generated text as a text generation problem that uses bart as its backbone. it utilises the generation probabilities of bart to measure the quality of a sentence. ## conclusions we present our approach to the lay summarisation for scientific documents in bionlp 2023 shared task. our approach utilises gpt-3.5 to generate additional training data and pre-trained flan-t5 in generating a lay summary for scientific documents. our results show that extra data generated from a generative model can boost the effectiveness of a summarisation model to a certain degree, especially in terms of relevance metrics. in addition, we showed that input length impacts the relevance of generated summary, but has no obvious impact on the readability and factuality metrics. future work could focus on developing a summarisation model that optimises readability and factuality of generated summary.