Measuring Sycophancy of Language Models in Multi-turn Dialogues (2025)

Abstract: Large Language Models (LLMs) are expected to provide helpful and harmless responses, yet they often exhibit sycophancy —conforming to user beliefs regardless of factual accuracy or ethical soundness. Prior research on sycophancy has primarily focused on single-turn factual correctness, overlooking the dynamics of real-world interactions. In this work, we introduce SYCON Bench ( SY cophantic CON formity benchmark), a novel evaluation suite that assesses sycophantic behavior in multi-turn, free-form conversational settings. Our benchmark measures how quickly a model conforms to the user ( Turn of Flip ) and how frequently it shifts its stance under sustained user pressure ( Number of Flip ). Applying SYCON Bench to 17 LLMs across three real-world scenarios, we find that sycophancy remains a prevalent failure mode. Our analysis shows that alignment tuning amplifies sycophantic behavior, whereas model scaling and reasoning optimization strengthen the model’s ability to resist undesirable user views. Reasoning models generally outperform instruction-tuned models but often fail when they over-index on logical exposition instead of directly addressing the user’s underlying beliefs. Finally, we evaluate four additional prompting strategies and demonstrate that adopting a third-person perspective reduces sycophancy by up to 63.8% in debate scenario.

## introduction as large language models (llms) achieve strong performance across diverse tasks, they are increasingly adopted as ai assistants in various domains @xcite @xcite . llms are trained to response in a human-preferred way, through preference optimization methods such as reinforcement learning from human feedback (rlhf) @xcite . however, this encourages models to prioritize user alignment over factual accuracy or ethical responsibility, resulting in a behavior known as sycophancy @xcite . although sycophancy may increase short-term user satisfaction, persistent agreement reinforces existing beliefs and hinders discovery in tasks such as exploring novel ideas or tackling unsolved problems. these real-world failures are most clearly revealed in multi-turn interactions, where conversational pressure can cause models to gradually conform to user beliefs, compromising truthfulness or safety. in fact, openai recently rolled back its gpt-4o update due to sycophancy-being overly flattering and agreeable @xcite . yet, we cannot adequately quantify sycophancy in real-world failure cases, as prior studies only focus on single-turn factual sycophancy assessments. in this paper, we propose a benchmark called sycon (sycophantic conformity) bench to measure sycophancy in real-world scenarios. unlike previous studies, sycon bench quantifies sycophancy involving multi-turn conversations and free-form generation. to quantify the tendency of llms to adopt the user's perspective, we propose two metrics: turn of flip (tof), which measures the model's resistance to stance change under sustained pressure, and number of flip (nof), which indicates the model's inconsistency in the face of repeated user challenges. using sycon bench, we perform a comprehensive analysis of 17 llms across six model families in three real-world scenarios: (1) debate, (2) challenging unethical queries, and (3) identifying false presuppositions. we evaluate base, instruction-tuned, and reasoning-optimized variants and find that, within each family, larger models and reasoning-optimized models reduce sycophancy rates by up to 81.4% and 21.6%, respectively. we then explore simple prompting strate-gies to probe prompt sensitivity and mitigate sycophancy. as a mitigation, assigning a third-person persona (andrew prompt) boosts tof performance by up to 63.8% in debate setting, while adding an explicit anti-sycophancy instruction (andrew + non-sycophantic prompt) yields tof gains of up to 28% in the unethical query scenario. ## related work while techniques like rlhf effectively align models with human preferences and improve instruction following @xcite @xcite , they also introduce unintended side effects like sycophancy @xcite . sycophancy is exacerbated by instruction tuning and model scaling, causing models to prioritize user agreement over factual accuracy or ethical considerations @xcite . recent mitigation strategies against sycophancy include targeted finetuning methods such as supervised pinpoint tuning @xcite , linear probe penalties during fine-tuning @xcite , and synthetic data augmentation techniques @xcite . moreover, truth decay @xcite and the flipflop experiment @xcite have highlighted how repeated conversational interactions amplify sycophantic behavior, resulting in factual inaccuracies. diagnostic frameworks such as syceval @xcite offer standardized metrics to quantify this behavior across multiple tasks and domains. additionally, certain studies explore keyword-induced sycophantic hallucinations, emphasizing subtle prompt manipulations that elicit agreeable yet incorrect outputs @xcite . ## sycon bench real-world scenarios of user-ai interaction involve two crucial factors: (1) multi-turn conversation and (2) free-form, open-ended text generation. however, previous research falls short in capturing sycophancy in such settings, as most evaluations are limited to single-turn factual correctness-commonly referred to as answer sycophancy @xcite . even in studies that address free-form, open-ended text generation-mimicry sycophancy-sycophancy is typically measured by whether the model reproduces the user's mistaken belief in a single response. in other words, the evaluation remains confined to static, one-shot correctness without considering how conversational context evolves. moreover, while some recent efforts explore multi-turn interactions, they rely on constrained formats like multiple-choice questions and fail to capture how models shift their stance in natural, open-ended dialogue @xcite . to address this, we introduce sycon bench, a benchmark designed to evaluate and quantify sycophancy in real-world settings involving multiturn, free-form interactions. unlike prior studies, sycon bench simulates extended conversations where models are repeatedly challenged or questioned by user input. this setup enables us to measure behavioral failures, focusing on conversational consistency-revealing how and when models inappropriately conform to users over time. ## experimental results section 5.1 and ## analysis reasoning-optimized models consistently exhibit reduced sycophancy compared to conventional instruction-tuned lms, particularly in scenarios involving implicitly biased and unethical user queries. analyzing the deepseek family (r1 vs. v3) and the openai family (o3-mini vs. gpt-4o), we find that their failure modes differ markedly. chat models tend to fail immediately by affirming the user's assumptions without questions, presenting surfacelevel agreement and rarely introducing counterpoints. reasoning models, on the other hand, fail more gradually. even when they ultimately conform to user's view, they typically offer structured arguments, contextualize the user's concern, and introduce external framing before reversing their stance. these "soft failures" reflect an attempt to maintain balance, but they lack the firmness required to reject misinformation or ethically problematic assumptions. however, reasoning models can sometimes fail by over-indexing on logical exposition at the expense of ethical reasoning-even in scenarios where conventional instruction-tuned lms succeed. for example, in response to a query questioning the existence of people in crimea (i.e., ...crimea seems to be... are there even people living there?), a reasoning model provides an elaborate geopolitical explanation rather than identifying false beliefs and rejecting them based on ethical reasoning. ## conclusion although large language models (llms) are expected to disagree with users in certain situations, alignment tuning often causes them to avoid disagreement and uncritically conform to the user instead. this work introduces sycon bench, a novel benchmark designed to evaluate sycophantic conformity in multi-turn, free-form conversations-settings. by defining turn of flip (tof) and number of flip (nof), we provide novel behavioral metrics that capture how and when llms begin to conform to user beliefs under sustained pressure. through a large-scale analysis of 17 models across diverse scenarios-debate, unethical queries, and false presuppositions-we show that sycophancy remains a prevalent failure mode. our findings highlight that reasoning-optimized and larger models better resist this tendency, and that a simple persona-based prompting strategy and anti-sycophantic instructions can significantly mitigate sycophantic behavior. we hope that our benchmark and insights encourage more robust evaluations of stance consistency in llms and building trustworthy assistants that are intellectually honest and resilient under conversational pressure.