Acquiescence Bias in Large Language Models (2025)

Abstract: Acquiescence bias, i.e. the tendency of humans to agree with statements in surveys, independent of their actual beliefs, is well researched and documented. Since Large Language Models (LLMs) have been shown to be very influenceable by relatively small changes in input and are trained on human-generated data, it is reasonable to assume that they could show a similar tendency. We present a study investigating the presence of acquiescence bias in LLMs across different models, tasks, and languages (English, German, and Polish). Our results indicate that, contrary to humans, LLMs display a bias towards answering no, regardless of whether it indicates agreement or disagreement.

## introduction when humans are asked whether they agree with something, they have a bias towards doing so, independent of their actual beliefs. in other words, the question "is the weather good or bad?" can lead to different responses than asking "is the weather good?" the phenomenon, called acquiescence bias, is well documented and has been studied for decades (see section 2). one guideline for good survey design is, therefore, to avoid such yes/no or agree/disagree questions. large language models (llms) have shown to be sensitive towards prompt variations @xcite @xcite @xcite and to replicate a variety of cognitive biases found in humans, e.g. with regard to anchoring @xcite , item order @xcite , and group attribution @xcite . it is therefore somewhat reasonable to assume that llms could also display some form of acquiescence bias, which could have implications on how prompts should be designed and whether or not llms can be used to simulate human responses. in this paper, we evaluate five llms of different sizes (namely llama-3.1-8b-instruct, mistral-small-24b-instruct-2501, gemma-2-27b-it, llama-3.3-70b-instruct, and gpt-4o-2024-08-06) on nine different tasks in three languages (english, german, and polish), to investigate whether they show response patterns resembling acquiescence bias. our results indicate that changing questions into a yes/no format has significant influence on the responses of llms, across tasks and models, and the tested languages. while no clear pattern emerged for german and polish, for english, we found that, unlike humans, llms are biased towards answering no, independent of whether that indicates agreement or disagreement. these findings do not only imply that the design of prompts should be considered very carefully but also that llms are not well suited to simulate human responses to survey questions. ## related work acquiescence bias, i.e. "the tendency to endorse any assertion made in a question, regardless of its content" @xcite , has been found in surveys across multiple domains, from political questions @xcite to assessing personalities @xcite . possible explanations for this bias range from the tendency to present socially acceptable behaviour by not disagreeing with an assumed authority of the questionnaire (see @xcite for a more detailed discussion). according to @xcite , the size of the effect is estimated to be around 10%. while researchers have extensively investigated a variety of cognitive biases found in humans and whether llms replicate them @xcite @xcite , @xcite were the first to explicitly investigate whether llms replicate ac-quiescence bias, alongside with four other human biases relevant in survey design. while they found that llms are indeed sensitive to changes in prompts, they did not find consistent patterns in these changes. despite these initial results, we believed, a deeper analysis of the acquiescence bias in particular was warranted. given the broader scope of the work by @xcite , only 176 questions and 352 responses have been evaluated with regard to acquiescence bias. in this paper, we analyse responses to more than 37,975 question variations. additionally, we address a limitation identified by @xcite , by not only using english corpora but also other languages, namely german and polish foot_0 . lastly, we believed that the way in which the questions were adapted by tjuatja et al. could have potentially added noise that influenced the results. in order to turn the original questions into yes/no questions they added the phrases "don't you agree" or "wouldn't you agree" to all questions (e.g. "on social media, do you think of yourself more as a a. sharer of news b. receiver of news" was turned into "on social media, wouldn't you say you are more of a sharer of news than a receiver of news? a. yes b. no"). such questions are also called negative yes/no questions and carry inherent ambiguity, it is not always clear whether answering them with "yes" indicates agreement or disagreement @xcite . additionally, negative modifier (wouldn't you agree instead of would you agree) also have an influence on human responses @xcite . therefore, we believe they are not well suited to investigate a potential acquiescence bias. ## results the main metric we were interested in originally was the number of positives (the sum of true and false positives), i.e. answers that are equivalent to choosing option a. if llms indeed show response patterns equivalent to acquiescence bias in humans, we would expect to see an increase in positives (irrespective of whether they are true or false positives) when changing from the neutral prompt to the other options. for the disagreement prompt, responses are inverted because a "yes" here indicates disagreement which correlates with a "no" in the original data. therefore, a higher positive rate here means that the model responded with "no" more often, which indicates agreement with the question and is equivalent to option a. across all languages and models, we did see that the different prompts have a significant influence on the responses. however, we did not find that models are systematically biased towards agreeing (detailed results can be found in appendix d). on the contrary, we found that for most cases, the number of positives was significantly reduce in comparison to responses of "a" in the neutral prompt. what we found instead is a bias towards replying "no" independent of whether that indicates disagreement or agreement with the question foot_2 . instead of focusing on positives, we decided to focus the evaluation on the absolute number of responses, particularly "no" responses. most surprisingly was that we see the same pattern for all but one model for the disagree questions in which the increase of no is still consistent, despite the fact that it is logically contradictory to the previous conditions. the results indicate that, for english, the models are not biased towards disagreeing, like humans are biased towards agreeing, but simply are biased towards replying "no", independent of whether that indicates agreement or disagreement. to validate our findings, we conducted mcnemar's tests (within-subjects chi-squared test, @xcite ) for all tasks individually and the overall results. the results of the analysis (see appendix d) show that all conditions significantly influenced the answers overall. even on the individual task level, despite the much smaller number of samples, almost all effects are statistically significant at the threshold of p ă 0.05. while in german and polish the phrasing of the prompt also has a significant influence on the results, no clear pattern could be identified across models or conditions. with regard to accuracy, we see that the influence that the observed bias has on accuracy is heavily based on the type of errors that are most prevalent in the neutral setting. for the lllama-3.3-70b model on the cuad_non-compete dataset, for example, there is a very small number of false positives in the neutral setting (total of 9 or 2%). changing to a condition which has a bias towards no (like the agree prompt), in such cases decreases accuracy. in other settings, where there is a higher number of false positives, e.g. mistral-small-24b on the contract_nli_confidentiality_of_agreement datasets, with a total of 40 false positives (48%), changing to a condition with a bias towards no increases accuracy. ## conclusion our initial hypothesis, that llms could display a bias resembling the acquiescence bias, was not confirmed. while we did find a systematic bias that could be observed across models and tasks (although only in english), interestingly it was also not the opposite, i.e. a bias towards disagreeing, but a bias towards replying "no", independent of whether that indicates agreement or disagreement. we believe that these findings bear importance, not only because they indicate that llms do not replicate response biases found in humans making it questionable whether they can be used to simulate human responses, but also because they shed doubt on the reasoning abilities shown by these models. if the responses would be mainly based on reasoning, we would not expect to see significant differences in the responses between the questions "do you agree sentence x is a definition?", and "do you disagree sentence x is a definition?". however, we found that in both cases models, irrespective of their size, are biased towards answering "no", thereby contradicting previous responses.