Evaluation of C hat GPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers (2023)

Abstract: ChatGPT is a large language model developed by OpenAI. Despite its impressive performance across various tasks, no prior work has investigated its capability in the biomedical domain yet. To this end, this paper aims to evaluate the performance of ChatGPT on various benchmark biomedical tasks, such as relation extraction, document classification, question answering, and summarization. To the best of our knowledge, this is the first work that conducts an extensive evaluation of ChatGPT in the biomedical domain. Interestingly, we find based on our evaluation that in biomedical datasets that have smaller training sets, zero-shot ChatGPT even outperforms the state-of-the-art fine-tuned generative transformer models, such as BioGPT and BioBART. This suggests that ChatGPT’s pre-training on large text corpora makes it quite specialized even in the biomedical domain. Our findings demonstrate that ChatGPT has the potential to be a valuable tool for various tasks in the biomedical domain that lack large annotated data.

## introduction the rapid growth of language models @xcite in the field of natural language processing (nlp) in recent years has led to significant advancements in various domains, including the biomedical domain @xcite . although specialized models (e.g., biobert @xcite , biobart @xcite , biogpt @xcite , etc.) have shown promising results in the biomedical domain, they require fine-tuning using domainspecific datasets. this fine-tuning process can be time-consuming due to the requirement of taskspecific large annotated datasets. in contrast, zeroshot learning enables models to perform tasks without the need for fine-tuning on task-specific datasets. chatgpt, a large language model, has demonstrated impressive zero-shot performance across various tasks @xcite . however, its performance in the biomedical domain remains to be thoroughly investigated. in this regard, this paper presents a comprehensive evaluation of chat-gpt on four key biomedical tasks: relation extraction, question answering, document classification, and summarization. in this paper, our primary objective is to explore the extent to which chatgpt can perform these tasks without fine-tuning and assess its performance by comparing with state-of-the-art generative fine-tuned models, biogpt and biobart. to our best knowledge, this is the first work that evaluated chatgpt on benchmark biomedical datasets. our evaluation of chatgpt can have a profound impact on the biomedical domain that lacks domainspecific datasets by exploring its zero-shot learning capabilities. to ensure the reproducibility of our evaluation and to help facilitate future research, we will release all the chatgpt-generated responses along with our evaluation code here: https:// github.com/tahmedge/chatgpt-eval-biomed . ## related work the effective utilization of transformer-based @xcite nlp models like bert @xcite have also led to significant progress in the biomedical domain @xcite @xcite @xcite in recent years. bert leverages the encoder of the transformer architecture, while gpt leverages the decoder of the transformer. in addition to these models, sequence-tosequence models like bart @xcite that leverage both the encoder and the decoder of transformer have also emerged as a powerful approach in various text generation tasks. it has been observed that domain-specific pretraining of these models on the biomedical text corpora followed by fine-tuning on task-specific ## our methodology for a given test sample x, we prepare a task instruction t and concatenate the text in the test sample with the task instruction to construct the prompt p . then the prompt p is given as input to chat-gpt (gpt-3.5-turbo) to generate the response r. in this paper, we evaluate chatgpt on 4 biomedical tasks across 11 benchmark datasets. below, we describe these tasks, the datasets we use for evaluation, and the prompt p that we construct for each task depending on the respective dataset. to this end, we evaluate chatgpt on various biomedical summarization tasks, such as healthcare question summarization (we used meqsum (abacha and demner-fushman, 2019) and mediqa-qs (abacha et al., 2021) datasets), medical answer summarization (we used mediqa-ans (savery et al., 2020) and mediqa-mas (abacha et al., 2021) datasets), and dialogue summarization (we used the icliniq and healthcaremagic datasets (zeng et al., 2020; mrini et al., 2021) for doctor-patient dialogue summariza-tion to generate short queries for healthcare forums describing patient's medical conditions). we show our prompts for this task in ## experiments since chatgpt is a generative model, we consider two state-of-the-art generative transformers as our baselines. below, we first present these baselines, followed by presenting the results. ## conclusions and future work in this paper, we evaluate chatgpt on 4 benchmark biomedical tasks to observe that in datasets that have large training data, chatgpt performs quite poorly in comparison to the fine-tuned models (biogpt and biobart), whereas it outperforms fine-tuned models on datasets where the training data size is small. these findings suggest that chatgpt can be useful in low-resource biomedical tasks. we also observe that chatgpt is sensitive to prompts, as variations in prompts led to a noticeable difference in results. though in this paper, we mostly evaluate chat-gpt on tasks that require it to generate responses by only analyzing the input text, in the future, we will investigate the performance of chatgpt on more challenging tasks, such as named entity recognition and entity linking @xcite @xcite , as well as problems in information retrieval @xcite @xcite @xcite . we will also explore the ethical implications (bias or privacy concerns) of using chatgpt in the biomedical domain.