Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical Machine Reading Comprehension (2024)

Abstract: Large language models (LLMs) have shown remarkable performance on many tasks in different domains. However, their performance in contextual biomedical machine reading comprehension (MRC) has not been evaluated in depth. In this work, we evaluate GPT on four contextual biomedical MRC benchmarks. We experiment with different conventional prompting techniques as well as introduce our own novel prompting method. To solve some of the retrieval problems inherent to LLMs, we propose a prompting strategy named Implicit Retrieval Augmented Generation (RAG) that alleviates the need for using vector databases to retrieve important chunks in traditional RAG setups. Moreover, we report qualitative assessments on the natural language generation outputs from our approach. The results show that our new prompting technique is able to get the best performance in two out of four datasets and ranks second in rest of them. Experiments show that modern-day LLMs like GPT even in a zero-shot setting can outperform supervised models, leading to new state-of-the-art (SoTA) results on two of the benchmarks.

## introduction machine reading comprehension (mrc) is defined as a task where a system tries to answer a question based on a given context. the context could be anything ranging from a couple of passages to a list of documents. even though much research has been conducted on mrc, several challenges remain when dealing with mrc tasks @xcite , such as the inability to handle long-range dependencies when trying to do reasoning and domain adaptation. recent improvements in large language modeling has alleviated a lot of the aforementioned issues. mrc in the biomedical domain @xcite has always been a key area of research. solving a biomedical mrc task faces various challenges including large intricate in-domain vocabulary, dependency on global knowledge, etc. due to these challenges, there is a wide gap between the performance of conventional methods in the general domain and that of the biomedical domain. although, traditional machine learning models did show some improvement, they have never been any close to human baselines or gold standards. contrary to this preconceived notion, modern-day llms have shown remarkable performance on many biomedical tasks @xcite @xcite . mrc can have different variations in itself. a contextual mrc requires the llms to answer a query solely by relying on a given context. in contrast, a context-free mrc relies on model's embedded knowledge base or any open-source knowledge base, such as wikipedia, to answer a query instead of using only the context provided. some of the datasets corresponding to context-free mrc are @xcite ; @xcite . these datasets are classified under context-free mrc because the given context is not sufficient to answer the questions. there is a definite need to explore the llm's inherent knowledge base or any other source of knowledge to answer these queries. similarly, @xcite ; @xcite ; @xcite comprise of the datasets for contextual mrc. again, these datasets have been categorized under contextual mrc because the queries asked can be correctly answered just by looking at the provided context. there is no need to induce any kind of external knowledge in order to answer these questions. recent llms have attained unprecedented performance in a wide array of natural language processing (nlp) tasks @xcite . although their performance have been evaluated on a multitude of mrc benchmarks in a context-free setting, their performance in a contextual setting has been dataset processbank biomrc mash-qa clicr # qa pairs 150 6250 3493 7184 avg context length 85 255 863 1461 max context length 266 510 2911 3952 ## datasets the four datasets from the biomedical and healthcare domain we choose to explore and analyze the performance of gpt are processbank @xcite , biomrc @xcite , mash-qa @xcite and clicr @xcite . there are a multiple reasons for selecting these four datasets. first, we want to focus on datasets that have not yet been evaluated by modern-day llms like gpt. next, we want to pick up datasets that vary in their statistics and nature. finally, based on our understanding, these 4 datasets covered the majority of research around contextual mrc in the biomedical field. processbank contains descriptions of biological processes as context accompanied by multiplechoice questions. biomrc, an improved version of bioread @xcite is a large-scale cloze-style dataset. it contains abstracts and titles of biomedical articles and the task of any mrc system is to predict the missing entity in a title using the corresponding abstract as context. in biomrc, all the biomedical entities mentioned in the abstract are considered as candidate answers and thus one option needs to be chosen from them. mash-qa is associated with consumer health domain where the answers can consist of sentences from multiple spans of the long context. the candidate answers include every single sentence of the given context. clicr is again a cloze-style dataset. it contains cloze queries from clinical case reports. unlike biomrc, clicr doesn't really have a list of candidate answers with one of them being the correct answer. rather, clicr contains a groundtruth answer set which consists of different lexical and semantic variations of the ground-truth answer, and thus all of them are correct. we use only the test sets of these datasets for all our prompting experiments in a zero-shot setting. we use biomrc lite version of biomrc. the statistics of the four datasets are listed in ## prompting techniques while an exhaustive study of all prior prompting strategies could have been a better experimental setup but due to the cost-prohibitive nature of running large-scale experiments on gpt, we only select the techniques that have shown to perform well in the general domain. along with these strategies, we also introduce a novel prompting method named implicit rag. we elaborate on all of these different prompts and their corresponding templates. there may be slight differences in prompt templates of the same prompting strategy across different datasets in order to adhere to the syntactic and semantic rules of english grammar as well as align with the dataset characteristics. ## results & analysis processbank the results for processbank are shown in ## conclusion in this work, we show that even in a zero-shot setting, gpt surpasses the performance of supervised