Challenging the Evaluator: LLM Sycophancy Under User Rebuttal (2025)

Abstract: Large Language Models (LLMs) often exhibit sycophancy , distorting responses to align with user beliefs, notably by readily agreeing with user counterarguments. Paradoxically, LLMs are increasingly adopted as successful evaluative agents for tasks such as grading and adjudicating claims. This research investigates that tension: why do LLMs show sycophancy when challenged in subsequent conversational turns, yet perform well when evaluating conflicting arguments presented simultaneously? We empirically tested these contrasting scenarios by varying key interaction patterns. We find that state-of-the-art models: (1) are more likely to endorse a user’s counterargument when framed as a follow-up from a user, rather than when both responses are presented simultaneously for evaluation; (2) show increased susceptibility to persuasion when the user’s rebuttal includes detailed reasoning, even when the conclusion of the reasoning is incorrect; and (3) are more readily swayed by casually phrased feedback than by formal critiques, even when the casual input lacks justification. Our results highlight the risk of relying on LLMs for judgment tasks without accounting for conversational framing.

## introduction the emergence of large language models (llms), such as chatgpt, has fundamentally reshaped ai, transforming how information is accessed, processed, and applied across diverse domains. llms are sycophantic in conversational scenarios: despite their advancements, llms exhibit sycophancy, a tendency to align responses with user beliefs: in multi-turn conversations, llms are readily persuaded to alter their initial answers in tasks with definitive solutions such as multiple choice and short answer questions (sharma et al., 2024; 1 code and conversation logs are public. @xcite . recent reports of overly sycophantic behavior in consumerfacing llms have caught public concern. for example, therapists have cautioned against relying on ai for mental health, foot_0 and it prompted openai to revert chatgpt to an earlier version. 3 llms seem to be effective in evaluative scenarios: despite this tendency, llms have been successfully adopted as evaluative agents for a variety of tasks. they serve as evaluators of model performance @xcite , for various text qualities such as harmlessness, reliability, and relevance @xcite , and evaluative agents in reinforcement learning from ai feedback (rlaif) @xcite . they are also used in multi-llm systems, such as multi agent debate, where multiple llms evaluate and discuss each other's chain of thought (cot) responses to converge on a final answer @xcite . the two scenarios are similar but evoke different behaviors: we posit that in both scenarios, responding to user feedback in conversation and acting as evaluative agents, llms are engaged in a similar task: determining the most appropriate response from a set of options. however, llms readily defer to user feedback in sequential interactions, even if the feedback is flawed @xcite . conversely, when tasked with evaluating options presented simultaneously, they can more reliably identify the superior response @xcite . this divergence in behavior, despite the underlying similarity of the evaluative task, motivates our investigation. our hypotheses: building on this observed discrepancy, this work seeks to provide a granular understanding of llm behavior when challenged in conversational vs. evaluative/comparative settings (figure 1 ). based on the difference between user- question let's analyze each option step by step ... the answer is a. compare the conflicting responses and decide which is correct. question ... the answer is option a ... the answer is option b 👦 🤖 🤖 i) conversational setting ii) evaluative setting too quick to dismiss option b. but i think that you were hypothesis 1 hypothesis 2 hypothesis 3 let's analyze each option step by step ... the answer is a. answer is b. let's break down the information step by step ... llms often defer to user input when challenged in a follow-up conversational turn, a phenomenon known as sycophancy (left). however, when asked to evaluate identical conflicting responses in an evaluative setting, they frequently identify the correct response (right). this paper investigates the three hypotheses (h 1 , h 2 , h 3 ; discussed in introduction). llm conversational scenario, and llm-as-a-judge evaluative scenario, we examine the following hypotheses: h 1 even when the argument is identical, llms are more likely to choose the argument when it is presented as a user rebuttal challenging the original output, than when both the argument and the original output are presented concurrently for evaluation (i vs ii in fig. 1 ). h 2 inclusion of reasoning in user feedback (the orange text in fig. 1 ) increases probability of llm to accept feedback. h 3 personalized language (e.g. "i think that", "the answer should..."; the blue text in fig. 1 ) commonly used in user feedback amplify sycophantic behavior. we test h 1 by comparing the llm's probability of accepting an argument b as the final answer when it is presented in a follow-up conversation challenging the original response a, versus when both arguments a and b are presented simultaneously for evaluation. we test h 2 by challenging the llm's original response in the second conversational turn with varying levels of reasoning, and measuring the probability that the llm adopts the rebuttal. for h 3 , we similarly challenge the llm's original response using rebuttals written informally. we then compare the llm's probability of accepting the refutation to results from h 2 , to identify which factor-reasoning or personalized language-more strongly influences model concession. we reveal the following: 1. llms more often endorse a conflicting response when framed as a follow-up from a user rather than when both responses are presented simultaneously for evaluation. recent papers also aim to understand the effect of model sycophancy in the second conversational turn. @xcite showed that overall accuracy always decreased when prompting llms with context-free disagreeing prompts. furthermore, @xcite explored the model's average response change when challenged in multi-turn conversation. @xcite investigated sycophancy when llm responses were refuted in a second conversational turn using counterarguments generated by another llm. previous works have quantified sycophancy by measuring the rate at which an llm accepts a user's counterargument. we adopt a similar metric, with specific details provided in §3.5. a key distinction in our work lies in the generation of refutation prompts. @xcite employed response-agnostic refutations, while @xcite and @xcite , complemented them with adversarial responses specifically designed to rebut the initial llm output (e.g., by providing the ground truth answer or the llm's original reasoning to an auxiliary llm tasked with generating a counterargument). our approach differs. we prompt multiple llms on the same question, collect each model's chain-of-thought output, and then sample as refutations those reasoning paths that disagree with each other. this method is intended to create scenarios that more closely translate to benign user-llm interactions where a user might simply offer a genuinely different perspective rather than mount an explicitly adversarial counterargument. ## a framework for quantifying sycophancy in llms this study utilizes an experimental framework (figure 2 ) to investigate llm sycophancy. we first gather a diverse set of multiple choice questions (mcqs) and elicit initial llm responses via zeroshot cot prompting. from these responses, we identify conflicting response pairs, then either construct a (rebuttal) challenge presented to the llm in the subsequent turn, or in a fresh setting, prompt the llm to judge between the original response and the conflicting counterpart. finally, we measure the llm's acceptance to the challenge to analyze how interaction patterns affect sycophantic behavior. all llm calls use greedy decoding to ensure consistency and reproducibility. mentions of step n in subsequent sections refers to the labeled steps shown in figure 2 . ## conclusion and future directions conclusion: this research provides a granular analysis of llm sycophancy in response to secondturn conversational challenges. we find that llms are generally more susceptible to persuasion in multi-turn conversation than in neutral evaluation (llm as a judge) settings, that the depth of reasoning in a challenge incrementally affects persuasion, regardless of the correctness, and, critically, that the stylistic nature of feedback, particularly casual assertiveness, can be a highly effective tool for persuasion, sometimes outweighing detailed reasoning. these insights are crucial for designing robust llm interactions and for users to be aware of the dynamics that can influence ai responses. future directions: a deeper dive into the conversation logs, including sentiment analysis of final responses or analysis of the intermediate reasoning steps when a model decides to accept user rebuttal or stand its ground, would be promising. we already observe a departure from the apologetic tone reported by @xcite in older models. our logs show that llm seldom apologize. instead they warp or discard their original reasoning to match user rebuttal.