Evaluating Large Language Models for Health-related Queries with Presuppositions (2024)

Abstract: As corporations rush to integrate large language models (LLMs) it is critical that they provide factually accurate information, that is robust to any presuppositions that a user may express. In this work, we introduce UPHILL, a dataset consisting of health-related queries with varying degrees of presuppositions. Using UPHILL, we evaluate the factual accuracy and consistency of InstructGPT, ChatGPT, GPT-4 and Bing Copilot models. We find that while model responses rarely contradict true health claims (posed as questions), all investigated models fail to challenge false claims. Alarmingly, responses from these models agree with 23-32% of the existing false claims, and 49-55% with novel fabricated claims. As we increase the extent of presupposition in input queries, responses from all models except Bing Copilot agree with the claim considerably more often, regardless of its veracity. Given the moderate factual accuracy, and the inability of models to challenge false assumptions, our work calls for a careful assessment of current LLMs for use in high-stakes scenarios.

## introduction conversational search experiences hold the potential to transform how people consume information online, and such offerings are gaining traction: just in the first month of preview, users conversed with bing copilot over 45 million times @xcite . however, it is crucial that such services provide factually accurate responses (or abstain from answering). this is particularly important for a large number of health-related queries, where providing inaccurate information may cause real-world harm. it is estimated that about 4.5% of all search queries are health-related @xcite , and another study notes that 72% of the surveyed internet users in the united states have searched 1 data and code to replicate our evaluation is available at: flair-iisc.github.io/uphill/ for health-related information online in a given month @xcite . one of the affordances of these llm-powered search experiences is that people can hold conversations, and therefore, describe their requests in detail. while the additional context is likely useful, it also opens up space for (possibly incorrect) assumptions to be expressed as a part of their request. therefore, it becomes important for llmpowered search experiences to be robust to any presuppositions. for instance, when a user inquires chatgpt @xcite if vegetarians are unaffected by covid-19, the model correctly denies any relationship between vegetarianism and covid-19 (figure 1 ). however, if a user requests to write an article about the "fact that vegetarians are unaffected by covid-19", the model response contradicts it's original stance to fulfil the user's request. although chatgpt is marketed to "answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests" @xcite , it is not clear how often deployed models hold up to such bold claims. in this work, we introduce uphill, a benchmark for understanding pressupositions for healthrelated inquiries to llms. uphill builds upon debated health-related claims on the internet that have been fact-checked by experts @xcite , and also comprises novel fabricated claims (which majority of human raters find absurd). we pose these claims as questions for different models, and assess their factual accuracy by computing how often the model responses acknowledge true claims and refute false ones. we find that the factual accuracy for true claims is 66% for instructgpt @xcite , 72% for chatgpt @xcite , 80% for gpt-4 (openai, 2023) and 87% for bing copilot @xcite . problematically, for false claims, it drops to 51% for instructgpt, 64% for chatgpt, 66% for gpt-4 and 63% for bing copilot. even figure 1 : given a health-related claim, we pose queries to the model with increasing levels of presupposition. the models' responses are checked for agreement with the claim using an entailment model. responses are considered accurate if they acknowledge true claims and refute false ones. we also assess if the responses are consistent. for bimedix @xcite , a domain-specific model trained specifically on clinical data, the accuracy on false claims is 64%. further, we query the models with increasing degrees of presuppositions, ranging from neutral or none to strong. we also include two other modalities, wherein we query models for writing assistance but the prompts contain an explicit assumption. the idea is to study how models respond to conflicts between meeting users' requests (say, for writing assistance) and factually responding to presupposed (mis)information. here, we find that for the strongest demands of writing assistance, model responses often agree with false claims to fulfill the writing requests: responses from instruct-gpt agree to 76% of queries with false claims, the corresponding number for chatgpt, gpt-4, bing copilot and bimedix is 62%, 60%, 28% and 39% respectively. this effect is more pronounced for fabricated novel claims, for which chatgpt responses agree to 88% queries, with gpt-4 and bing copilot responses supporting 84% and 71% of them respectively. from our experiments, we observe that instruct-gpt is the least robust to presuppositions, whereas bing copilot is the most robust. since intructgpt is tuned to follow instructions without any safety or alignment procedures, this is perhaps unsurprising, however the extent to which these models can be led to produce and reinforce false information is concerning. we believe that reinforcement learn-ing with human feedback (rlhf) helps chatgpt responses to be more in line with true facts and negate incorrect assumptions, which is also corroborated by recent work @xcite @xcite . gpt-4 performs slightly better than chatgpt, possibly due to bigger model size and additional rlhf training. interestingly, we note that bing copilot responses are not as sensitive to varying doses of presuppositions, perhaps because its responses are anchored in web pages, which remain unaffected. bimedix, an open-source mixture-of-experts model trained specifically on clinical data, performs at par with bing copilot at lower presupposition levels, but is still susceptible to increasing presuppositions. many responses from closed-source models end with a disclaimer suggesting readers to consult with a healthcare professional. while this may serve as a warning and fulfills necessary legal requirement, it remains to be seen how such disclaimers affect the users, if at all. overall, our work points to a large gap in the factual accuracy of the examined models, and shows that models could often be led to generate healthrelated misinformation. in the following sections, we describe our approach to source claims ( §2.1), and generate queries with various presupposition levels ( §2.2). we also assess the viability of using entailment models for evaluation ( §2.3), and discuss both quantitative and qualitative results ( §3). veracity # claims examples true 766 probiotics help with irritable bowel syndrome false 854 vegetarians are unaffected by covid mixture 159 a home pregnancy test can detect testicular cancer fabricated 166 a liquid-only diet is beneficial in managing depression. ## approach consider a set of public-health claims c, comprising true, false and mixed claims represented as c true , c false and c mixed respectively. further, let ℓ ∈ @xmath0, 1, . . . 4} denote the degree of presuppositions (explained in section 2.2), varying from neutral information seeking requests (ℓ = 0) to writing demands containing presuppositions (ℓ = 4). for a given claim @xmath1, the query generator q : c × @xmath2, ℓ), which is then fed to a conversational model m : @xmath3,ℓ = m(q(c, ℓ)). we use instructgpt, chatgpt, gpt-4 and bing copilot as candidates for m. the response is considered factually accurate if it agrees with a true claim and disagrees with a false claim. the model is consistent if it maintains its stance across different levels. we use an entailment model to assess the agreement or disagreement of the response with the claim. ## results and discussion we evaluate four conversational models m: in-structgpt, chatgpt, gpt-4 and bing copilot. we choose these models partly due to their popularity @xcite , but also because it allows us to compare models with different features: chatgpt builds on instructgpt and uses reinforcement learning from human feedback (rlhf) to align its outputs to human preferences @xcite . further, gpt-4 is a larger (and allegedly a mixture-of-experts) model compared to chatgpt and is tuned on additional preference data. lastly, bing copilot uses retrieval augmentation along with gpt-4. in this section, we discuss the results of our evaluation on uphill queries posed to these models. factual accuracy. for neutral informationseeking queries without any presuppositions, we find that the factual accuracy of instructgpt is 55%, chatgpt 63%, gpt-4 67% and bing copilot 69% ( ## related work medical question answering. numerous studies evaluate llms on medical questions spanning professional medical exams (usmle, medqa, medmcqa), medical literature (pubmedqa, mmlu), and consumer queries (liveqa, medica-tionqa, healthsearchqa). med-palms @xcite , gpt-3.5 @xcite and gpt-4 @xcite are shown to have reasonable performance on a subset of these datasets. however, these evaluations of gpt models exclude consumer queries. our work evaluates llms by focusing on health-related claims and goes a step further to study the accuracy and consistency of models when presuppositions are introduced. ## conclusion we quantified the factual accuracy of llms for health-related queries across queries with varying degrees of presuppositions. we experimented with instructgpt, chatgpt, gpt-4 and bingchat and found that while the model responses rarely contradicted true claims, they often (problematically) acknowledged popular and novel false claims. further, the agreement between models' responses and the input claim increased with increasing level of presupposition, regardless of the veracity of the claim. through our study, we noted that instruct-gpt was the most susceptible and bing copilot was the most robust to presuppositions, although the factual accuracies of all the examined models call for a careful reassessment of using llms for such high-stakes scenarios. our results suggest that careful thought should go into designing conversational models, to ensure that information is presented from reliable sources or models abstain from answering when such information cannot be presented with certainty. future research directions could develop methods to identify reliable sources for retrieval, quantify the credibility and certainty of the retrieved content. more research is required to understand how different responses (and ways of presenting evidence) impact users' belief and understanding of the topic.