Arbiters of Ambivalence: Challenges of using LLM s in No-Consensus tasks (2025)

Abstract: The increasing use of LLMs as substitutes for humans in “aligning” LLMs has raised questions about their ability to replicate human judgments and preferences, especially in ambivalent scenarios where humans disagree. This study examines the biases and limitations of LLMs in three roles: answer generator, judge, and debater. These roles loosely correspond to previously described alignment frameworks: preference alignment (judge) and scalable oversight (debater), with the answer generator reflecting the typical setting with user interactions. We develop a “no-consensus” benchmark by curating examples that encompass a variety of a priori ambivalent scenarios, each presenting two possible stances. Our results show that while LLMs can provide nuanced assessments when generating open-ended answers, they tend to take a stance on no-consensus topics when employed as judges or debaters. These findings underscore the necessity for more sophisticated methods for aligning LLMs without human oversight, highlighting that LLMs cannot fully capture human non-agreement even on topics where humans themselves are divided.

## introduction as large language models (llms) have become more performant, they are increasingly being used to replace human annotators @xcite @xcite @xcite . although works question this trend @xcite @xcite @xcite , it is driven by the increasing demand and cost of human annotations, as well as the improving performance of llms across a wide range of tasks. a prevalent use is for so-called alignment where an llm guides the training of another model by replacing humans with ai in the role of providing preference annotations and feedback @xcite @xcite @xcite . we test whether llms are more or less likely to take a stance when provided with ambivalent scenarios as answer generator, pairwise or pointwise judge, and debater. when generating an answer directly, llms are fairly neutral or nuanced. however, they are much more likely across the board to take a stance when acting as judge (either pairwise or pointwise) or in a debate. two prominent approaches have emerged for performing alignment @xcite : first, aidriven preference alignment @xcite @xcite , where models are trained using rewards derived from llm judges, rather than relying on explicit human feedback. second, alignment with scalable oversight @xcite @xcite , where two ai models (known as llm debaters) engage in a zero-sum game debate to convince each other of their stance, with a third model deciding the outcome based on the debate's transcript. llm debaters have shown promise in domains such as long-context question answering @xcite , mathematical reasoning, and factuality @xcite , and improved their evaluation abilities @xcite . two main arguments are used to justify the widespread adoption of llms in these human annotator-free settings: (i) reported high agreement with human preferences @xcite on tasks like math, writing, reasoning, stem knowledge, coding, and (ii) measured (iii) as debaters, where two llms engage in a multi-turn dialogue to debate an issue, with a third llm reviewing the conversation and determining the outcome. we examine how the models' neutrality varies across these roles on questions where humans disagree. this study has three key motivations: first, existing benchmarks @xcite @xcite evaluating the quality of llm judges or reward models align with the majority of human preferences, overlooking minority viewpoints. these benchmarks typically assess llms on tasks with definitive answers, such as math, or examine disagreements among judges primarily in low-stakes, subjective areas like writing style, response formatting, or refusal formatting @xcite . as a result, they inadequately address the full range of real-world scenarios where llms must operate amidst diverse human preferences, particularly in areas where human disagreement is widely recognized, such as ethics @xcite , textual entailment @xcite @xcite , toxicity and hate speech @xcite @xcite , demographic bias @xcite , cultural norms, personal preferences, and controversial issues. second, assessments of llms' neutrality or bias have primarily focused on their ability to generate responses @xcite @xcite rather than examining their biases when judging different stances in no-consensus sit-uations. third, while debate as a scalable oversight method has demonstrated improved reasoning @xcite and factual accuracy @xcite in tasks with clear answers, it remains uncertain whether this method maintains llms' neutrality in settings characterized by fundamental human disagreement, raising the question of whether debates enhance neutrality or increase bias. our main contributions are as follows: ## related work preference alignment benchmarks like @xcite , or anthropic's hhh data @xcite , which are often used to evaluate llm judges, typically only feature chosen/rejected answer pairs that are determined by majority human ratings. additionally, llm judge benchmarks @xcite @xcite evaluate judges on matching human agreement on tasks that have a correct answer (stem knowledge, math, reasoning) or a small subset of subjective tasks (writing, summarizing). this neither accurately captures the full spectrum of human disagreement nor clarifies which scenarios are inherently ambivalent. while automating evaluation has a long history in nlp and nlg @xcite @xcite @xcite @xcite , recently, researchers have proposed various llm-based approaches whereby ai systems help evaluate and align other ais @xcite @xcite . specifically, llmas-judges @xcite @xcite @xcite @xcite @xcite @xcite and llm debaters @xcite @xcite are leading contenders to provide human-like labels at scale. @xcite , for instance, demonstrate up to 80% agreement with human evaluators, primarily in scenarios where human agreement can be effectively modeled through majority voting or clear preferences. there is a scarcity of research exploring llm judges in contexts characterized by high disagreement, and the existing studies typically involve relatively low-stakes tasks such as preference ranking for summarization, creative writing, or refusals @xcite . in the context of llm debaters, studies of factual (q&a over very long passages), math and reasoning domains where two ai experts can engage in a zero-sum game to convince a judge (human or ai) prevail. some research also examines llm judge's ability to handle disagreement in terms of the adaptability of llm judges as personalized evaluators, but these only assess how well they emulate survey results @xcite or conversational safety ratings of dialogues based on the annotator's identity @xcite . llms as debaters have been studied in the context of factuality @xcite , confirmation biases @xcite and reasoning capabilities @xcite . recently, llms were found to be up to 85% accurate in predicting individuals' future wants based on a simple prompt enriched with hours of interview @xcite (but others find such simulated settings can suffer from a lack of robustness @xcite ). while these signal that llms may be able provide high-quality human-like preferences across diverse scenarios and embed various perspectives @xcite , llms should also reflect the nuances of a collective sample, and we examine llms' capacity to express human-like preferences on tasks where there is no consensus. ## experimental setting there are different "modes" used to let a model annotate queries. the three modes under study herein are shown in figure 2 and described below. ## discussion our examination of llms across the three modes can be compared to the three modes of pluralistic alignment @xcite : overton, steerable, and distributional. as answer generators, llms demonstrate overton alignment by providing balanced and varied responses. as debaters, llms exhibit steerable alignment by crafting arguments for opposing viewpoints and achieving high neutrality scores with pointwise judges, indicating effective steering (appendix a.4). as judges, llms reflect distributional stances, similar to distributional alignment. for example, they often use utilitarian reasoning in moral dilemmas, such as prioritizing saving more lives, and default to "no" in scientific debates with mixed results, like "does cranberry juice help prevent uti?". in political issues of ongoing debate, such as citizenship for children born on us soil to undocumented immigrants or the removal of "in god we trust" from currency, llms rely on historical constitutional precedents. they also take definitive stances on subjective matters, like advocating for watching sports on tv for convenience or promoting vegetarianism based on health and environmental benefits, as shown in appendix e. ## conclusion our study provides significant insights into how llm alignment varies across different modes. we find that while models often exhibit extreme neutrality as answer generators, they do not maintain this neutrality when acting as judges. this suggests that current alignment efforts to generate diverse and nuanced answers @xcite may create only a facade of neutrality, evident primarily when llms are used as answer generators. furthermore, although previous research indicates that debate can enhance llms' evaluative capabilities and lead to more truthful answers, our findings reveal that llm debates do not preserve neutrality on topics, even those they were trained to handle neutrally. this work opens up new avenues for exploring the complexities of model alignment and neutrality, highlighting the need for more robust strategies to ensure consistent neutrality and express ambiguity across various llm modes.