MDC at B io L ay S umm Task 1: Evaluating GPT Models for Biomedical Lay Summarization (2023)

Abstract: This paper presents our approach to the BioLaySumm Task 1 shared task, held at the BioNLP 2023 Workshop. The effective communication of scientific knowledge to the general public is often limited by the technical language used in research, making it difficult for non-experts to comprehend. To address this issue, lay summaries can be used to explain research findings to non-experts in an accessible form. We conduct an evaluation of autoregressive language models, both general and specialized for the biomedical domain, to generate lay summaries from biomedical research article abstracts. Our findings demonstrate that a GPT-3.5 model combined with a straightforward few-shot prompt produces lay summaries that achieve significantly relevance and factuality compared to those generated by a fine-tuned BioGPT model. However, the summaries generated by the BioGPT model exhibit better readability. Notably, our submission for the shared task achieved 1st place in the competition.

## introduction effective communication of scientific ideas is essential for sharing research findings with the general public. while scientific publications serve as an important tool for scientists to share their work, they are primarily intended for other researchers. the use of technical language and the required background knowledge makes these articles difficult for the general public to comprehend, limiting accessibility and research impact @xcite . the goal of lay summarization is to generate a concise summary of technical texts without using technical language so it can be understood by non-experts. although certain publishers require authors to provide a lay summary alongside their manuscripts, not all publishers have this requirement, and historical publications are unlikely to include such summaries. therefore, the devel-opment of effective automatic lay summarization becomes crucial in bridging this communication gap @xcite . gpt-style autoregressive language models have demonstrated impressive ability for natural language generation (nlg) tasks including summarization @xcite . these models are pre-trained on large corpora and can leverage knowledge gained from this data to generate text. recent research has rapidly advanced the performance of gpt-style models by substantially scaling the size of these models and their training corpora @xcite , along with instruction tuning and reinforcement learning with human feedback @xcite . researchers have evaluated gpt models on biomedical-specific natural language understanding (nlu) tasks and found that they underperform due to a lack of domain knowledge @xcite . recent research by @xcite aimed to alleviate this shortcoming by proposing biogpt, a model based upon gpt-2 @xcite but specifically trained on biomedical text from pubmed foot_0 . the authors demonstrate that biogpt can significantly outperform gpt-2 on biomedical nlu tasks; however, there is currently no evaluation on biomedical natural language generation (nlg) tasks. therefore, the objective of this work is to evaluate the ability of general-purpose and domainspecific gpt models to generate lay summaries of biomedical research articles as part of the biolay-summ challenge @xcite . this approach raises an issue. the attention mechanism used in many gpt-style models has quadratic memory scaling with the length of a sequence, which limits the maximum length of input text that can be processed. at the time of writing, the average length of a research article exceeded this limit. journal articles already contain author-written abstracts that provide a technical summary of the paper and previous work by @xcite demonstrated the importance of the abstract in lay summarization. therefore, we focused on using only the article abstract as the input and leveraging the domain knowledge of these models to generate lay summaries. we present results for this method for zero-shot and fine-tuned biogpt and gpt-2 models. additional experiments are conducted using gpt-3.5 variants accessed via the openai api and we compare their performance with the smaller -but domain-specific -biogpt model. our findings indicate that a gpt-3.5 model, combined with a straightforward few-shot prompt, achieves the best performance among the evaluated models. this approach secured 1st place in the shared task. ## results and discussion few shot prompting of the instruct model significantly improved its performance and this model achieved the best results for the factuality and relevance metrics. the remarkable increase in performance of the few-shot versus the equivalent zeroshot models demonstrates the large language models' ability for in-context learning. interestingly, the performance of the fine-tuned biogpt large model is not far off the text-davinci-003 despite being 1/100 th the size of the gpt-3.5 model and even outperformed chat and instruct models in their zero-shot setting. this shows the benefit of domain knowledge for domain-specific natural language generation tasks but also alludes to potential performance gains of scaling up the size of domainspecific language models. the chat model performed significantly better than the instruct model when using the zero-shot prompt. this may be because the zero-shot prompt is in a conversational style, which the chat model was trained for, or additional training to the chat model that the instruct model did not receive. we are limited to conjecture as the training details and differences of the chat model from the instruct model are largely unknown as they have not been published by openai. due to the superior performance on the validation set scores, we chose the instruct model for the final test evaluation. the test set scores are listed in it would be valuable to compare the performance of the proposed approaches with other recently open-sourced large language models such as the biomedical-domain biomedlm foot_7 , the scientificdomain galactica @xcite or the general-domain llama @xcite . another interesting future research avenue is available with the longer context models. the emergence of models such as gpt-4 (openai, 2023), palm-2 @xcite and claude foot_8 offers significantly increased context lengths compared to the models assessed in this paper. using these models, it would be possible to input more sections such as the introduction from a paper into the few-shot prompts, potentially leading to substantial enhancements in performance as these sections provide additional background information. furthermore, the performance of the biogpt large model has room for improvement. all hyperparameters were fixed during training of the model due to resource constraints so tuning of the training and generation hyperparameters may benefit model performance. the hyperparameters for all models were also not tuned for a specific dataset and separately tuning these for each dataset may improve summary quality. advanced prompting techniques have been ## conclusion this paper presents an evaluation of generalpurpose and biomedical domain-specific autoregressive language models for generating lay summaries from the abstracts of scientific articles for the biolaysumm shared task at bionlp 2023. our findings demonstrate that the biomedical domainspecific model, biogpt, outperforms generalpurpose gpt-2 models when fine-tuned for lay summary generation. we also explored the effectiveness of zero and few-shot prompting for generating lay summaries using openai's gpt-3.5 models. while the zero-shot performance of these models is worse than the fine-tuned biogpt large model, we discovered that using the text-davinci-003 coupled with a few-shot prompt yielded the best results among all the language models tested. this approach was selected as our submission for the shared task, achieving the overall 1 st place submission and 3 rd place in both relevance and factuality metrics. to further enhance the performance of this approach, we anticipate that advanced prompting methods, evaluation of additional models, and utilization of models with longer context lengths could be beneficial. despite the promising results obtained in this study, it is essential to conduct further research to validate the factuality of the generated lay summaries using human experts before practical application.