Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong (2024)

Abstract: Large Language Models (LLMs) are increasingly used for accessing information on the web. Their truthfulness and factuality are thus of great interest. To help users make the right decisions about the information they get, LLMs should not only provide information but also help users fact-check it. We conduct human experiments with 80 crowdworkers to compare language models with search engines (information retrieval systems) at facilitating fact-checking. We prompt LLMs to validate a given claim and provide corresponding explanations. Users reading LLM explanations are significantly more efficient than those using search engines while achieving similar accuracy. However, they over-rely on the LLMs when the explanation is wrong. To reduce over-reliance on LLMs, we ask LLMs to provide contrastive information—explain both why the claim is true and false, and then we present both sides of the explanation to users. This contrastive explanation mitigates users’ over-reliance on LLMs, but cannot significantly outperform search engines. Further, showing both search engine results and LLM explanations offers no complementary benefits compared to search engines alone. Taken together, our study highlights that natural language explanations by LLMs may not be a reliable replacement for reading the retrieved passages, especially in high-stakes settings where over-relying on wrong AI explanations could lead to critical consequences.

## introduction imagine you are told a claim about neptune: "only one spacecraft has visited neptune and it has more than 13 moons." and you want to verify whether it is factual. what would you do-look up relevant pages from search engines or ask chatgpt for its take? this is not just a question of checking a piece of trivia; our information ecosystem depends on people being able to check the veracity of information online. misinformation, whether accidental or deliberate, has the potential to sway public opinion, influence decisions, and erode trust in credible sources @xcite . moreover, the wide adoption of large language models like chatgpt increases the danger of misinformation, both by malicious actors and models generating inadvertent hallucinations @xcite . consequently, verifying the accuracy of information is important. fact-checking claims is a wellestablished task in nlp @xcite . however, automated fact-checkers are far from perfect, and they are only useful when users trust their predictions @xcite . building that trust and providing effective help is crucial: a team without trust leads to suboptimal human-ai team performance while over-trusting wrong ai predictions could lead to catastrophic failures in high-stakes applications. therefore, in real-life applications, we care about the ai-assisted human accuracy of fact-checking, rather than evaluating and improving automated fact-checkers alone @xcite . the two major types of tools for helping human users (many of which are non-expert fact-checkers) are retrieval and explanation @xcite , exemplified by the widely-used web search engines (e.g., google) and generative language models (e.g., chatgpt) respectively. showing retrieved passages to users has long been established as an effective information-seeking tool @xcite . in contrast, the usefulness of generative explanations on fact-checking remains understudied. on the one hand, competent generative models (especially llms) can generate fluent and convincinglooking natural language explanations that not only provide an answer (i.e., whether the claim is true or false), but also elucidate the context and basis of its judgment. on the other hand, these models are prone to hallucinations @xcite claim: only one spacecraft has visited the planet and neptune has more than 13 moons. ## related work we review relevant literature from nlp and hci on fact-checking, explanations, and over-reliance. we also discuss additional related work in section a.3 ## research questions to understand the comparative advantages of retrieval and explanation in human fact verification, we pose the following research questions: chatgpt's supporting and refuting arguments side by side. ## study design overview we present an overview of the study design including the task setup, data used, measured variables, models used, and users involved. ## meta-analysis we analyze several additional factors across all experiments, such as the impact of retrieval recall, time, confidence, and qualitative responses. ## conclusion our human studies highlight the false promise of using natural language explanation produced by chatgpt to aid human fact-checking. humans over-rely on explanations even when they are wrong, making human accuracy worse than showing retrieval or the baseline of not showing any evidence. contrastive explanation mitigates users' over-reliance on wrong explanations, but combining retrieval and explanation does not help further. overall, neither of these two approaches significantly beats the retrieval baseline, highlighting the need for better methods (e.g., adaptive or customized explanations) for combating over-reliance. figure 6 : interface for the contrastive explanation condition. we present chatgpt's explanations for both sides together to encourage more careful thinking. we also highlight all the keywords to ease reading.