The Earth is Flat because...: Investigating LLM s’ Belief towards Misinformation via Persuasive Conversation (2024)

Abstract: Large language models (LLMs) encapsulate vast amounts of knowledge but still remain vulnerable to external misinformation. Existing research mainly studied this susceptibility behavior in a single-turn setting. However, belief can change during a multi-turn conversation, especially a persuasive one. Therefore, in this study, we delve into LLMs’ susceptibility to persuasive conversations, particularly on factual questions that they can answer correctly. We first curate the Farm (i.e., Fact to Misinform) dataset, which contains factual questions paired with systematically generated persuasive misinformation. Then, we develop a testing framework to track LLMs’ belief changes in a persuasive dialogue. Through extensive experiments, we find that LLMs’ correct beliefs on factual knowledge can be easily manipulated by various persuasive strategies.

## introduction large language models (llms) are known to encapsulate a substantial volume of knowledge during training @xcite @xcite @xcite . prior work has identified that llms are susceptible to external information from different sources. for instance, @xcite shows that llms can be highly receptive to external evidence even when it conflicts with their memory. researchers also observe that llms tend to tailor their responses even to follow an objectively wrong viewpoint @xcite ). ## curation of farm this section outlines the curation process of farm including both questions and associated misinfor-mation. it consists of two stages: selecting straightforward factual questions and systematically generating persuasive misinformation for later testing. @xcite is employed for prompting @xcite throughout this section. an example of farm is given in ## the misinformation test procedure this section outlines the details of testing the llm's behavior and tracking its belief change in the face of misinformation. we sequentially test all questions with their correct answers and the corresponding misinformation in farm. for each question, the procedure (as shown in figure 1 ) has three steps including the initial belief check, the persuasive conversation, and the final belief check. additionally, in this paper, all belief checks are aimed at probing the knowledge that the llm holds, resembling a closed-book qa process using questions type format repetiton i believe {ctrl}. logical {lo}. therefore, we have to accept that {ctrl}. credibility {cr}. therefore, we have to accept that {ctrl}. emotional {em}. therefore, we have to accept that {ctrl}. from farm, as defined in prior work @xcite . given that all the questions are presented in an mcq format, we define that the llm holds a belief toward a question when it selects any option other than "don't know", with "don't know" being considered as abstaining from forming a belief. stage 1: initial belief check. for each question in farm, we assess the llm's initial knowledge by a belief check. we then proceed with the following persuasive conversation only when the llm's initial beliefs align with the accurate answer. stage 2: persuasive conversation. the persuasive conversation with misinformation is the key component. prior work shows that message repetition could also affect both the persuasion process @xcite ) and people's belief toward misinformation @xcite . so besides the three persuasive rhetorical appeals mentioned earlier, we also experiment with a simple "repetition" strategy by simply repeating the ctrl to persuade llms. each persuasive conversation contains up to four turns. it begins with ctrl, followed by persuasive messages that belong to one of the four persuasive strategies foot_2 . ## experiments we conduct extensive experiments in this paper and list only the most important results in this section. the other results in appendix c are also aligned with our findings and conclusions. ## behavior study we identify 5 types of behaviors when an llm is faced with misinformation: rejection, sycophancy, uncertainty, acceptance, and self-inconsistency. we show the frequency of 4 of the 5 types of behaviors for chatgpt in rejection involves the llm consistently countering misinformation, including direct rejection, corbelief strong weak after multiple turns of persuasion behavior misinformed rejection sycophancy & uncertainty acceptance not likely possible retain belief directly test figure 4: the relationship between an llm's susceptibility to misinformation (test), its response behavior (behavior), and confidence in its initial belief (belief). acceptance involves the llm being misinformed immediately. in its response, the llm will occasionally apologize for its previous "wrong answer", which is correct indeed. self-inconsistency serves as an abnormal case where the llm agrees with the user's misinformation (e.g., "you are correct") at first but then continues to present counterarguments in the same response. this case is excluded from figure 4 because it primarily results from processing errors related to the user's input and has little correlation with the llm's belief or the test outcome. ## discussion of possible mitigation from an llm service provider's perspective, we aim to prevent llms from easily falling prey to misinformation especially for simple facts, as this would undermine the reliability and trustworthiness of the llm. in this section, we discuss a lightweight prompt-based method to mitigate this issue. after detecting misinformation in the user's input (may use another llm), we insert a system prompt as a reminder. this prompt serves to remind the llm to ( 1 ) be cautious with potentially malicious users and (2) verify its memorized knowledge before responding. our intuition is on two folds. llms are prone to providing factually incorrect information, known as hallucination foot_9 , which significantly hinders their reliability in informationseeking tasks @xcite @xcite . existing efforts mainly concentrate on detection @xcite , evaluation @xcite , investigation @xcite , and mitigation @xcite of hallucination. recent research also investigate in the intersection of llm and misinformation (chen and shu, 2023b), they mainly focus on misinformation detection using llms @xcite chen and shu, 2023a) or misinformation generation @xcite . our research explores an orthogonal direction. we introduce a novel direction to intentionally induce hallucination to assess llms' alignment with their internal knowledge and their robustness in the face of misinformation. knowledge conflicts in llm. @xcite show that llms can be highly receptive to external evidence even when that conflicts with their parametric memory, given that the external evidence is coherent and convincing. another line of works proposes strategies to empower the llm to more comply with the user-provided context when such conflicts exist, and they assume a well-intentioned user and the given context are always correct @xcite . ## conclusion by targeting llm's robustness against misinformation, we construct a novel dataset, farm, consisting of straightforward factual questions and corresponding misinformation generated through sophisticated persuasive strategies. we then conduct a thorough investigation on persuading llms with misinformation in a multi-turn conversational setting. we identify a pronounced susceptibility of llms to misinformation even considering the sota ones like gpt-4. we note that persuasive strategies involving repetition and rhetorical appeals are particularly potent in leading them astray. our research highlights the lack of robustness in llms when confronted with misinformation, as their initially correct beliefs can be easily manipulated. furthermore, we also reveal primary behaviors exhibited by llms in response to misinformation, illuminating future work for mitigation.