EL i RF - VRAIN at B io L ay S umm: Boosting Lay Summarization Systems Performance with Ranking Models (2024)

Abstract: This paper presents our contribution to the BioLaySumm 2024 shared task of the 23rd BioNLP Workshop. The task is to create a lay summary, given a biomedical research article and its technical summary. As the input to the system could be large, a Longformer Encoder-Decoder (LED) has been used. We continuously pre-trained a general domain LED model with biomedical data to adapt it to this specific domain. In the pre-training phase, several pre-training tasks were aggregated to inject linguistic knowledge and increase the abstractivity of the generated summaries. Since the distribution of samples between the two datasets, eLife and PLOS, is unbalanced, we fine-tuned two models: one for eLife and another for PLOS. To increase the quality of the lay summaries of the system, we developed a regression model that helps us rank the summaries generated by the summarization models. This regression model predicts the quality of the summary in three different aspects: Relevance, Readability, and Factuality. We present the results of our models and a study to measure the ranking capabilities of the regression model.

## introduction nowadays, there is more information than ever at the disposal of the general public. in the specific domain of biomedical research, there is information that would be interesting to non-expert audiences, including journalists or even members of the public, such as what occurred during the recent covid-19 global pandemic @xcite . however, the technical language is a barrier for the non-specialist public that may prevent them from accessing that information @xcite . abstract summarization models should be useful in reducing the gap in understanding information. since the models can generate a concise summary of a given text and capture its most relevant information @xcite @xcite . it is possible to obtain new models that generate summaries adapted to a much wider audience; what is known as lay summary. in a lay summary, the text should contain the main ideas of the article that would be interesting for a non-expert audience, enhancing readability by adding background information and reducing (or avoiding) technical terminology. in this paper, we present the results and analysis of our system in the participation at the biolay-summ @xcite at the 23rd bionlp workshop @xcite . ## task drescription in the 2024 edition, the biolaysumm poses a single shared task, rather than two, as in the previous edition @xcite . the task is to create a lay summary, given a biomedical research article and its technical summary (abstract section of the article). the organization provides a biomedical dataset @xcite that contains biomedical research articles from two sources: elife sciences foot_0 and public library of science (plos) foot_1 . each sample contains the text of the article, the technical summary, and the reference lay summary. the dataset is divided into three partitions: train, val, and test. to measure the performance of the systems, the organizers of the competition selected a set of measures that would help to evaluate the performance in three different aspects: relevance, readability, and factuality. for relevance the following scores were chosen: rouge (1, 2, l) @xcite , bertscore @xcite . to measure the readability aspect: flesch-kincaid grade level (fkgl) @xcite , dale-chall readability score (dcrs) @xcite , coleman-liau index (cli) @xcite , and lens @xcite . finally, to measure factuality, two scores were selected: alignscore @xcite , summac @xcite . ## pre-trainined model for this task, we have used a longformer encoder-decoder (led) @xcite since we were approaching summarizing long texts, such as the case of scientific articles. this lets us increase the amount of information available on the encoder side. we used as a starting point the led base model from ai2 foot_2 , publicly available at the repository of huggingface @xcite , and continuously pre-trained it with in-domain data. for the continual pre-training phase, we followed the training methodology used in the news abstractive summarization models (nas) work @xcite . this methodology combines multiple pre-training tasks to incorporate linguistic knowledge in the pre-training phase and enhance the abstract nature of the produced summaries. incorporating those tasks in continuous pre-training should help the model to transfer knowledge specific to the summarization task and increase the performance of the downstream model after finetuning, just as it did in the original nas work. the data used for continuous pre-training was chosen specifically to adapt the model to the biomedical research domain. we collected text from three different sources: abstracts (technical summaries) from pubmed (national center for biotechnology information (ncbi), 2024) (17m samples), pubmed articles and abstracts from the scientific_papers foot_3 dataset @xcite . also, articles and technical summaries from the dataset train partition used in this competition (elife + plos) (29k). due to infrastructure limitations, we limited the encoder input to work with no more than 4096 tokens. taking into account this restriction, and with the objective of maximizing the amount of data, we split text by lines, using a window of no more than 4000 words. we generated subsamples that contained at least a new line and filled the windows with as many words as possible. the final amount of samples went up to 59m samples. when working with longformers, you have to select which tokens will receive global attention in addition to local attention. in the original work @xcite , the authors recommend setting [cls] token with global attention. however, we hypothesized that adding landmarks across the input with global attention could increase performance. for this reason, we added a special token with global attention (<@xmath0. the number of sentences was not constant but dictated by a minimum number of words of separation @xmath1. thus, the special token was placed at the end of every number of sentences with a total length of at least k words. previous experimentation was carried out to determine the number of words. the best results were obtained with at least @xmath2. the base model was pre-trained for three epochs in our research institute's cluster with 8 nvidia a40 graphic cards with 48gb of vram were used for the process; which took a month. the main hyperparameters are: 128 samples per device, 4 gradient accumulation steps, a learning rate of 5 × 10 -5 with a constant scheduler, gradient checking, and an 8-bit quantified optimizer. ## lay summarization models we developed two different approaches for the competition. in the first approach (m1), the model re-ceives the technical summary and adapts the text and information to a lay summary style. in the second approach (m2), additional text is included beside the technical summary, that was, the introduction and the discussion sections of the article, similar to @xcite . since the distribution of samples is not wellbalanced, we fine-tuned two models per approach: one for elife and another for plos. the four models were fine-tuned for ten epochs each with an nvidia rtx 3090 with 24gb; each approximation took nearly 24 hours. the relevant hyperparameters are: 4 samples per device and a learning rate of 5 × 10 - foot_4 with a linear scheduler. in our tests over validation, m1 outperformed m2 in the overall performance. the detailed results can be seen in ## ranking model in order to increase the quality of the lay summaries of the system, we developed a regression model to rank the summaries generated by the summarization models. this regression model predicts the quality of the summary in three different aspects: relevance, readability, and factuality. ## results for the competition, we sent a total of three submissions. s1 that included lay summaries generated with m1 approach without any kind of ranking. s2 that contained lay summaries generated with m1 and selected with the rank model (10 summaries per sample). additionally, we sent a third submission (s3) that contained summaries from m1 and m2 and selected with the regression model ( 20 summaries per sample). ## discusions the results presented in section 6 raise the benefits and constraints that must be taken into account when combining generation models with ranking models to choose which text will be presented to the end user. regarding the benefits, they are evident. with the ranking models, we can enhance the quality of the summaries presented to the user even though we use the same automatic summarization models. we use the ranking model to choose those summaries that obtained the best ranking scores since those texts will have better quality compared to other summaries generated by the same models. this selection should boost the overall performance of the system in most cases. in relation to the constraints. the ranking model does not generate summaries or make texts better; it just rates summaries generated by the summarization models, and we select the best summaries based on those scores. therefore, if summarization models have a bad performance and/or we can not provide enough variety to choose from, the benefits will be diminished. for this reason, we should combine the ranking models with summarization models that can complement each other depending on the text to summarize and offer variety in the generated summaries. ## conclusions in this work, we have presented our contribution to the biolaysumm 2024 shared task of the 23rd bionlp workshop. we used led models to allow adding more text in the model input. although we started from the same pre-trained model, different fine-tuned models were trained for the two sources of the competition: elife and plos. two different approaches were followed, one with just the technical summary as input, and another with additional text beside the technical summary. our preliminary evaluation showed that the first approach performed better, but the second should be developed further since the larger input context improved the factuality aspect. an additional contribution of our approach is the use of a regression-based ranking model that helped to boost the quality of the final summary by choosing the promising one from a set of summaries generated by the models. the model that obtained the best results in the competition was the one that combined the first approach and the ranking model.