D eakin NLP at B io L ay S umm: Evaluating Fine-tuning Longformer and GPT -4 Prompting for Biomedical Lay Summarization (2024)

Abstract: This paper presents our approaches for the BioLaySumm 2024 Shared Task. We evaluate two methods for generating lay summaries based on biomedical articles: (1) fine-tuning the Longformer-Encoder-Decoder (LED) model, and (2) zero-shot and few-shot prompting on GPT-4. In the fine-tuning approach, we individually fine-tune the LED model using two datasets: PLOS and eLife. This process is conducted under two different settings: one utilizing 50% of the training dataset, and the other utilizing the entire 100% of the training dataset. We compare the results of both methods with GPT-4 in zero-shot and few-shot prompting. The experiment results demonstrate that fine-tuning with 100% of the training data achieves better performance than prompting with GPT-4. However, under data scarcity circumstances, prompting GPT-4 seems to be a better solution.

## introduction the task of summarization has witnessed the development based on pre-trained language models. more recently, the superiority of large language models (llms) has been demonstrated on a wide range of natural language processing (nlp) tasks @xcite . in the biolaysumm 2024 shared task @xcite , the competition focuses on generating summaries for biomedical research articles that are easily understandable by the general public. these summaries are usually known as "lay summaries". recently, the study of the summarization task using generative models has increased for both general domains @xcite and biomedical text @xcite . additionally, according to @xcite , each article generally has more than 10,000 words. many pre-trained language models have been developed to handle such long text @xcite . in this paper, we implement the longformer-encoder-decoder (led) @xcite as an approach for biolaysumm shared task, as its performance has been demonstrated in @xcite . in this paper, we present a comparison between the performance of the fine-tuned led model on 50% and 100% of the training set. additionally, we evaluate @xcite on zeroshot and few-shot prompting for this shared task. our aim is to investigate how a fine-tuned model and a large language model such as gpt-4 perform in lay summarization biomedical text. this study focuses on three aspects: performance, training time, and computational cost. our contributions are as follows. in the following sections, we briefly analyze the datasets, describe our methods in detail, showcase the experiment settings, and present our results, findings, and conclusion. ## datasets the task is evaluated on two datasets: plos and elife @xcite . both datasets contain biomedical articles and a lay summary manually written for each article. to first understand the evaluation datasets, we proceed tokenizing the input and the output text on two datasets using tokenizer from led model @xcite . we sum-marize the statistics of the plos and elife dataset in dataset article(#tokens) summ.(#tokens) train val test train val plos 9,851 9,924 9,978 263 279 elife 12,321 12,753 11,967 435 445 according to @xcite and ## evaluation metrics in this shared task, the generated summaries are evaluated on three aspects and ten metrics accordingly: the objective of the evaluation is to maximize the relevance, factuality and lens in readability scores and minimize fkgl, dcrs, and cli scores. ## preliminary study due to the fact that each instance in both datasets is lengthy and may contain a large amount of irrelevant information to generate lay summaries, we perform a heuristic evaluation on the validation sets. we are aware that each article has at least an abstract and a conclusion paragraph. we evaluated the abstract, the conclusion part and other parts of each article with the lay summaries on the relevance aspect. the abstracts written by the author of each article contain the most similar information. these abstracts are likely to be used as the base knowledge when creating the lay summaries. additionally, the conclusion parts also achieve competitive scores, which indicates that they have potential to be used as sources to generate lay summaries. ## experiments based on the results of our preliminary study, we first extract the abstract and conclusion paragraph from the original articles. we then perform the fine-tuning process and prompting gpt-4 using the combination of abstract and conclusion from the original articles. ## results in this section, we list our results on the private test set. the scores are retrieved through the codabench page of the shared task and reported in 3 https://platform.openai.com/docs/models/ gpt-4-turbo-and-gpt-4 ## discussion the results demonstrate that traditional fine-tuning can produce summaries with accurate keywords and context rather than prompting. led model also creates less hallucination than llms, because it achieves better factuality scores. however, finetuning is less effective in making the summaries simpler and easier to understand. furthermore, we believe that fine-tuning led model on elife is less efficient than on plos dataset because of the size of elife dataset. furthermore, the text in elife dataset is also longer than plos. therefore, it is likely that led model is not able to capture the keywords and learn enough context on elife. hence, gpt-4's performance is slightly better in this case. ## performance versus cost in this section, we discuss the trade-off between model performance and costs. in our analysis, the costs include training time, computational cost, and prompting cost. we summarize our comparison in through our result analysis and cost-effective study, it demonstrates that gpt-4 prompting cost us more on querying, however it takes less time then fine-tuning and still achieves competitive results. especially, in the situation where we have less training data (such as in elife case), gpt-4 can outperform fine-tuned led model. ## conclusion this paper details our approach to the biolay-summ 2024 shared task, comparing traditional finetuning of the longformer-encoder-decoder (led) model and few-shot prompting with gpt-4 for generating lay summaries of biomedical articles. our results indicate that the fine-tuned led excels on the plos dataset, while gpt-4's few-shot prompting outperforms led on the elife dataset, highlighting gpt-4's advantage in data scarcity scenarios. future work may explore self-evaluation meth-ods and cost-reduction strategies for fine-tuning using parameter-efficient techniques.