HULAT - UC 3 M at B iolay S umm: Adaptation of B io BART and Longformer models to summarizing biomedical documents (2024)

Abstract: This article presents our submission to the Bio- LaySumm 2024 shared task: Lay Summarization of Biomedical Research Articles. The objective of this task is to generate summaries that are simplified in a concise and less technical way, in order to facilitate comprehension by non-experts users. A pre-trained BioBART model was employed to fine-tune the articles from the two journals, thereby generating two models, one for each journal. The submission achieved the 12th best ranking in the task, attaining a meritorious first place in the Relevance ROUGE-1 metric.

## introduction in the context of the rapidly expanding quantity and complexity of biomedical literature, the ability to effectively and accurately summarise documents has become crucial for researchers and healthcare professionals. in this regard, natural language processing (nlp) technologies have emerged as promising tools to address this need. the objective of biolaysumm 2024 shared task @xcite is the simplification of biomedical research articles playing a vital role in making information more comprehensible to non-experts thus enabling a wider audience to understand and use medical information effectively. concerning generating summaries, there are a number of different approaches that can be employed. one such approach is the extractive model, which involves selecting the most important sentences from the original text and incorporating them directly into the summary. these models were the first to emerge and the most widely used until the abstractive models came onto the scene. these models have the capacity to comprehend the content of the input text and generate summaries that may include new sentences and expressions that are not present in the original text @xcite . the first paper to describe an abstractive summarisation model was @xcite , and from that moment on, they began to gain greater relevance and were used more frequently than the extractive models. in this paper, we will employ abstractive models. in our participation in the biolaysumm 2024 shared task, we utilise existing large language models (llms), such as bio-bart @xcite , which is a biomedical variant of the bart model @xcite , and longformer encoder-decoder (led) @xcite , to train our models for the generation of summaries from the provided articles. the summaries generated by the various models were evaluated in accordance with the metrics provided by the organisers. (rouge(1,2 and l) @xcite , bertscore @xcite , fkgl @xcite , dcrs @xcite , cli @xcite , lens @xcite , align-score @xcite and summac @xcite ). the experiment that yielded the most favourable results was the one that used the bio-bart pre-trained model. this model was used to train two models, one for each of the journals from which the articles were obtained. these models were used to generate the abstracts for each journal. this release achieved excellent results in the relevance metric of the shared task, with the highest score in the rouge-1 metric. however, the readability and factuality metrics yielded less impressive outcomes, resulting in a final ranking of 12th place. nevertheless, this remains a satisfactory performance, as it places the team in the top half of the table of all participants. 2 method ## results and discussions the conversely, an analysis of the readability metrics reveals that the optimal outcome is achieved when the two models are combined. however, when the dalle-chall readability score (dcrs) metric is considered, the led model exhibits significantly superior performance. furthermore, this metric presents an intriguing phenomenon: the results in the biobart model are quite poor, with a score of 1.5 points above our best result. this is a significant drawback for the model in terms of its final score. in contrast to the previous observation regarding relevance, the texts of the journal elife obtain much better results than those of the journal plos. with regard to the factuality metrics, the bio-bart model yielded the most favourable results, with the exception of the elife journal, where the outcomes were considerably less favourable. consequently, the average score was reduced, resulting in the led model, which is more balanced, achieving better results in the alignscore metric. the findings of this study indicate that while the information is well-maintained, as evidenced by the relevance metrics. the plos journal articles contain more accurate information but are more challenging to comprehend. this discrepancy may be attributed to the smaller abstracts (175-220 words), which may have a detrimental impact on the readability metrics. the biobart model is the most effective in terms of relevance, outperforming all the metrics in this category thanks to its specific biomed-ical training. although the combined long-former+biobart model improves readability, it loses accuracy due to the double simplification of the content. on the other hand, the longformer model, although it obtained good results in some metrics, did not stand out in any of them; this could be an effect of having trained a single model with the texts of the two journals. ## conclusions this paper presents our participation in the biolay-summ 2024 shared task, which aimed to generate lay summaries of large biomedical documents. in this task, we trained two different models (led and biobart) from which we generated three different experiments. upon completion of the task, we observed that the best results were obtained by training two biobart models (one for the plos journal articles and another for the elife articles). this is our final submission to the competition, which resulted in a 12th-place finish. our performance was particularly noteworthy in the rouge-1 metric, where we achieved first place, as well as in the relevance metrics. as future work, we would have liked to experiment with other models that we found interesting, particularly trained with medical data, such as medical mt5 @xcite . with respect to the models we have presented, we would like to continue working with them to improve the results in the readability and factuality metrics, in which we have not obtained such good results. we would like to study what happened in generating not adequate summaries by conducting an analysis of errors. we believe that managing specific medical terminology would help to generate more lay-oriented medical terms to ascertain the efficacy of the keyword translation from the original text to the summary. in the event that this process is not executed correctly, due to the inherent complexity of the keywords, an external dataset comprising words from the biomedical field and a translation into simpler expressions could be employed as a preprocessing step for the texts prior to training. see the open-access and collaborative (oac) consumer health vocabulary 3 (chv) as an example of a medical lay-oriented vocabulary.