Chaos with Keywords: Exposing Large Language Models Sycophancy to Misleading Keywords and Evaluating Defense Strategies (2024)

Abstract: This study explores the sycophantic tendencies of Large Language Models (LLMs), where these models tend to provide answers that match what users want to hear, even if they are not entirely correct. The motivation behind this exploration stems from the common behavior observed in individuals searching the internet for facts with partial or misleading knowledge. Similar to using web search engines, users may recall fragments of misleading keywords and submit them to an LLM, hoping for a comprehensive response. Our empirical analysis of several LLMs shows the potential danger of these models amplifying misinformation when presented with misleading keywords. Additionally, we thoroughly assess four existing hallucination mitigation strategies to reduce LLMs sycophantic behavior. Our experiments demonstrate the effectiveness of these strategies for generating factually correct statements. Furthermore, our analyses delve into knowledge-probing experiments on factual keywords and different categories of sycophancy mitigation.

## introduction recently large language models (llms) @xcite @xcite @xcite have revolutionized natural language processing by achieving human-like performance on various downstream tasks, but understanding their susceptibility to sycophancy has received less attention. sycophancy can be regarded as a type of hallucination in llms and it refers to the model's nature to align their responses to the user's intent in the input, even though it is misleading. this could lead llms to confidently present fabricated information, undermining their reliability @xcite and trustworthiness @xcite . given the increasing integration of llms in realworld applications @xcite @xcite , understanding and addressing the issue of sycophancy becomes crucial. it can potentially result in the generation of misleading or false information @xcite . the consequences can extend beyond mere misinformation, impacting decision-making processes @xcite , perpetuating biases @xcite , and endorsing inaccurate or harmful narratives @xcite . as we rely more on these llms for critical tasks such as information retrieval @xcite , content generation @xcite , and decision support systems @xcite , it becomes imperative to explore their susceptibility to sycophancy and develop strategies to mitigate it. in this work, we first demonstrate that misleading keywords can lead llms to generate factually incorrect statements. consider an individual searching for facts that they vaguely remember, such as lionel messi's connection to the 2014 fifa world cup and the golden boot. to verify their memory, they may ask an llm to generate a factual statement with the keywords "lionel messi, 2014 fifa world cup, golden boot". however, relying on llms to produce factual information based on partial or misleading cues can result in sycophantic behavior-meaning generated responses align with what users want to hear rather than providing accurate facts. figure 1 demonstrates that golden boot keyword misleads multiple llms, resulting in factually incorrect statements like "lionel messi won the golden boot in the 2014 fifa world cup." notably, this behavior persists across distinct domains (mentioned in we then adopt several llm hallucination mitigation strategies to reduce sycophancy in factual statement generation. these include using demonstrative exemplars, adding precautionary statements, and providing additional context through both llm inference and web search. the results demonstrate that all sycophancy mitigation strategies are beneficial in reducing hallucinations, contributing to a more accurate factual statement generation. moreover, we thoroughly explore diverse sycophancy mitigation categories, investigating how the llms modify and correct factually inaccurate statements produced by them. by asking knowledgeprobing questions, we also demonstrate that llms memorize factual information about misleading keywords. next, our analysis of misleading keywords identifies specific types of keywords that are more susceptible to causing sycophancy. in the end, we investigate the behavior of llms when given non-misleading keywords and also test the robustness of the mitigation strategies on these keywords. the key contributions of this paper are: ## related work despite their remarkable capabilities, transformerbased @xcite llms still face challenges that impede their widespread adoption in practical applications. one prominent issue is hallucination in llms, which has garnered significant attention from the research community due to its increasing prominence. recent work @xcite categorizes llm hallucination into three categories: input conflict, context conflict, and factual conflict and emphasizes that the latter has more significant effects on the practical applications of llms. in our work, we address sycophancy which falls under this category. @xcite introduced the concept of sycophancy by showing the behavior of llms to align with user opinion. @xcite , in particular, focused on the opinions embedded within the prompt. their work also presented that sycophantic hallucination increases with model size and suggested that alignment techniques like reinforcement learning (rlhf) @xcite may encourage it to align with user opinions, increasing sycophancy. interestingly, lu and le (2023) report that instruction tuning @xcite significantly increased sycophancy and attribute this observation to the absence of data that does not distinguish between user's opinions and instructions. @xcite show that llms exhibit sycophancy when involved with subjective user opinions or when factual contradictions are expected. existing works have explored how llms exhibit sycophantic behavior when presented with explicit user opinions. however, these works do not investigate the llms' innate tendency to align their responses with misleading cues in the input, even when such cues do not accurately reflect the user's true intent. in our work, we analyze this particular sycophancy exhibited by llms while generating factual statements. we also evaluate the effectiveness of four hallucination mitigation strategies in addressing this sycophantic behavior. ## sycophancy mitigation strategies in this section, we outline the strategies employed to mitigate sycophancy in factual statement generation. we adopt four existing hallucination mitigation strategies. these involve using in-context exemplars @xcite , adding a pre-cautionary statement @xcite , augmenting contextual knowledge from llms @xcite and external sources @xcite . we systematically evaluate these strategies to identify effective approaches for generating accurate and contextually appropriate factual statements. for a comprehensive understanding of our mitigation efforts, please refer to the detailed prompts examples provided in appendix a.4. ## experiments 5.1 experimental prompts to evaluate the performance of large language models in generating factual statements, we conducted experiments in two different settings. first, we used a general prompt for 500 sets of misleading keywords and analyzed the factuality in the model's output. then, we expanded our experiments to incorporate domain-specific prompts for five differ ## conclusion in conclusion, this study addresses the critical issue of llms' sycophantic behavior exhibited in factual statement generation. we conduct a comprehensive analysis involving five different llms on 500 misleading keywords and 200 domain-specific ones. additionally, we evaluate the effectiveness of four strategies to mitigate sycophancy. the analyses contribute valuable insights into the nature of llms' responses to misleading keywords, their knowledge retention capabilities, the challenges posed by misleading keywords, and the effect of using non-misleading keywords. ultimately, the findings presented in this paper aim to contribute to the development of trustworthy and reliable llms.