Dr C hat GPT tell me what I want to hear: How different prompts impact health answer correctness (2023)

Abstract: This paper investigates the significant impact different prompts have on the behaviour of ChatGPT when used for health information seeking. As people more and more depend on generative large language models (LLMs) like ChatGPT, it is critical to understand model behaviour under different conditions, especially for domains where incorrect answers can have serious consequences such as health. Using the TREC Misinformation dataset, we empirically evaluate ChatGPT to show not just its effectiveness but reveal that knowledge passed in the prompt can bias the model to the detriment of answer correctness. We show this occurs both for retrieve-then-generate pipelines and based on how a user phrases their question as well as the question type. This work has important implications for the development of more robust and transparent question-answering systems based on generative large language models. Prompts, raw result files and manual analysis are made publicly available at https://github.com/ielab/drchatgpt-health_prompting .

## introduction prompt-based generative large language models, such as chatgpt, can be used to answer complex natural language questions, often with impressive effectiveness. where previously, a search engine was the main tool available, now users can choose to use a large language model (llm). while queries to search engines are typically short and ad-hoc, prompts to a llm can be far longer and richer. this can be a double edged sword: on the one hand, far more information is available to the model to generate a good answer; on the other hand, llms suffer from hallucinations @xcite and incorrect, biased or misleading prompt information (e.g., obtained through a retrieved-then-generate pipeline) may derail the model's ability to give a good answer. while in some domains this may not matter, in this paper we investigate a domain where this could have serious consequences -health. in particular, we consider the case of a health consumer (i.e., non professional) using chatgpt to understand whether a treatment x has a positive effect on condition y. our evaluation is done using the trec misinformation track, a resource that contains complex health questions, often pertaining to misinformation regarding the efficacy of treatments, and that are typical of a subset of information seeking tasks undertaken by health consumers. even though the risks of searching for health information online are well documented @xcite @xcite , users still continue to seek health information through online mediums @xcite . research on the effectiveness of using traditional search engines for health answers does exist @xcite , but the parallel for llms does not. this paper aims at investigating this by considering two main experimental conditions and one specific, popular llm, chatgpt, currently being relied upon by end-users: question-only: chatgpt is asked to provide an answer to a health question without further information provided to the model; and evidence-biased: chatgpt is asked to provide an answer to a health question after being provided a prompt with information from a web search result (document) containing information about treatment x and condition y. this provides insights into a typical retrieve-thengenerate setup, whereby a search engine provides results in a prompt to the llm. we investigate cases where the document provided in the prompt either supports the use of the treatment (supporting evidence) or where the document dissuades from the use of the treatment (contrary evidence). given these settings, we can then evaluate our two main research questions: rq1 -question-only effectiveness: how effective is chatgpt at answering complex health information questions typical health consumers would ask? how does question phrasing impact this effectiveness? rq2 -evidence-biased effectiveness: how does biasing chatgpt by prompting with supporting and contrary evidence influence answer correctness? the paper is divided in two parts around these two questions. for question-only effectiveness (rq1), we show that chatgpt is actually quite effective at answering health related questions (accuracy 80%), but this effectiveness hides a systematic bias based on both how the question is phrased and whether the correct answer to the question is a "yes" or "no" that majorly degrades effectiveness. when biasing the prompt with supporting or contrary evidence (rq2), we find that: (1) the evidence often is capable of overturning the model answer about a treatment, showing prompts have real impact on answer correctness; and (2) both supporting and contrary evidence can have a detrimental effect on answer correctness, reducing accuracy to 63%. previous work has shown that engineering prompts has an important impact on the effectiveness of llms like chatgpt @xcite . this paper adds to that understanding by contributing that it is not just the "form" of the prompt that matters (e.g., the clarity of the instructions contained in the prompt, or the stance expressed in the prompt), but also that the correctness of information contained in the prompt can highly influence the quality of the output of the llms. this is important when llms are integrated in a retrieve-then-generate foot_0 pipeline @xcite @xcite , where information related to the question is first identified from a corpus (e.g., the web), and this is then passed to the model via the prompt to inform the model's output. ## rq1 -question-only effectiveness our first research question relates to how effective chatgpt is in answering complex health information questions. measuring this serves two purposes: (i) it answers the basic question of how effective is this llm in health information seeking tasks; and (ii) it provides the baseline effectiveness of the model when relying solely on the question; i.e., without observing any additional knowledge in the prompt. we also consider chatgpt effectiveness on yes vs no type questions and how question phrasing can impact chatgpt. ## rq2 -evidence-biased effectiveness our second research question investigates the impact on answer correctness of biasing chatgpt by prompting with (i) supporting evidence; and ## conclusions we have examined the correctness of chatgpt, a rapidly emerging generative large language model, in answering complex health information questions regarding the effectiveness of a treatment for a condition. we did this in two settings: when only the question is presented to chatgpt (question-only) and when the question is presented along with evidence (evidence-biased); i.e., a web search result retrieved when searching for the question. importantly, we controlled the stance used by the question, and whether the evidence was in supporting or contrary to the treatment. this in turn allowed us to understand the effect of question phrasing and of providing evidence in the prompt. chatgpt answer accuracy was 80% when relying solely on the question being posed, and no additional evidence. however, (i) chatgpt displays a systematic bias based on whether the question has a yes or no answer, (ii) simply rephrasing the question to its negative form strongly degrades effectiveness, and (iii) providing evidence in the prompt degrades answer accuracy, even if the evidence is correct. we specifically highlight the last result: injecting evidence into the prompt, akin to retrieve-then-generate pipeline, can heavily influence the answer -more importantly, it reduced chatgpt accuracy to only 63%. this finding is at odds with the idea of relying on retrieve-then-generate pipelines to reduce hallucinations in models like chatgpt. large language models seem set to be increasingly used for information seeking tasks. if such tasks drive important health related decisions (such as what treatment to take) then it is paramount we better understand these models. this understanding includes the obvious aspect of answer correctness, but also how different prompting can sway the model. an understanding of such aspects both helps mitigate potential harms and helps inform the design of better models. prompts, raw result files and manual analysis are made publicly available at: https://github. com/ielab/drchatgpt-health_prompting .