Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answering (2025)

Abstract: Large Language Models (LLMs) have demonstrated strong performance in question answering (QA) tasks. However, Multi-Answer Question Answering (MAQA), where a question may have several valid answers, remains challenging. Traditional QA settings often assume consistency across evidences, but MAQA can involve conflicting answers. Constructing datasets that reflect such conflicts is costly and labor-intensive, while existing benchmarks often rely on synthetic data, restrict the task to yes/no questions, or apply unverified automated annotation. To advance research in this area, we extend the conflict-aware MAQA setting to require models not only to identify all valid answers, but also to detect specific conflicting answer pairs, if any. To support this task, we introduce a novel cost-effective methodology for leveraging fact-checking datasets to construct NATCONFQA, a new benchmark for realistic, conflict-aware MAQA, enriched with detailed conflict labels, for all answer pairs. We evaluate eight high-end LLMs on NATCONFQA, revealing their fragility in handling various types of conflicts and the flawed strategies they employ to resolve them.

## introduction recent advances in large language models (llm) @xcite ) have led to substantial performance improvement in various tasks, including question answering (qa) with one or multiple correct answers @xcite . although previous work on multi-answer qa (maqa) largely assumes that the different answers are mutually consistent and complementary @xcite , realistically, questions can be controversial and lack a definitive answer. in such cases, models 1 natconfqa is publicly available at https://github. com/en555/contraqa . should not only generate a response that incorporates several answers, but also detect the conflicts and communicate them to the reader. for example, when asked "what is the effect of aspartame?", a comprehensive response should aggregate various effects reported in the available sources, while explicitly distinguishing between effects with consensus and those that remain contested or under debate. collecting qa instances with naturally occurring contradictory answers is challenging, as knowledge conflicts are not always prevalent in arbitrary texts. as a result, only a few datasets handle knowledge conflicts while exhibiting some limitations, such as relying on llms to inject misinformation into other reliable texts, or focusing only on yes/no questions (section 6). furthermore, although recent benchmarks aim to evaluate whether the entire response acknowledges the debatable nature of the question @xcite , they do not assess whether the response accurately reflects which answers are subject to disagreements (e.g., whether aspartame increases cancer risk). to foster research on this important challenge and to enable fine-grained evaluation, we create nat-confqa, the first conflict-aware maqa dataset with annotations labeling individual answer pairs that are in conflict. to collect natconfqa, we first leverage standard fact-checking datasets to identify sources with naturally occurring disagreements. then, we ask human annotators to write various qa pairs and to label the relationship between pairs of conflicting answers. finally, we verify the annotations for quality assurance. figure 1 illustrates our general annotation scheme. natcon-fqa is a high-quality dataset that covers yes/no and wh-questions, and includes instances, based on reliable sources, with a mix of conflicting and non-conflicting answers for the same question. we evaluate the performance of eight llms on natconfqa, including open-source and proprietary models, measuring both answer quality and conflict identification. in terms of answer quality, we show that while models generally achieve high precision, they fail to output all correct answers. furthermore, our fine-grained evaluation of conflict identification reveals that models struggle to distinguish between conflicting and non-conflicting answer pairs. further analysis of the model failures reveal insightful error patterns: models evade exposing conflicts by selecting a single answer, erroneously attempting to reconcile contradictory information, or refraining from answering the question altogether. taken together, our work uncovers the behavior of strong llms when confronted with conflicting information, while providing suitable methodologies and data to investigate these challenges in future research. ## conflict-aware multi-answer qa the conflict-aware multi-answer qa task is an extension of the traditional multi-answer qa @xcite task, that considers potential conflicts between the different answers. we extend recent work in conflict-aware qa @xcite , which either focused on binary conflicting answers @xcite or addressed multiple answers without indicating which pairs conflict @xcite . our generalized task formulation supports two or more answers per question and, importantly, pinpoints the answer pairs that are in conflict. given a question 𝑞 and a set of candidate passages 𝑃 = {𝑝 1 , . . . , 𝑝 𝑛 }, the task is to generate a freetext response 𝑦 that satisfies two main requirements. first, similar to the traditional maqa task, 𝑦 should incorporate all answers that appear in reference second, the response 𝑦 should indicate all conflicts, if any, between the answers within it. we assume that the response 𝑦 is in natural language, as typically generated by large language models (llms), and not necessarily in a structured format. for example, consider the question "what climate degree change is caused by greenhouse gases?". the different answers 0.45°c and 0.8°c cannot be simultaneously true, hence an ideal response should present both answers through contrastive language (e.g., by using the word "however", both answers can be communicated, and the conflict is established). in contrast, for the question "which domestic pets can potentially test positive for sars-cov-2?", the answers dogs, cats, and ferrets are non-conflicting and an ideal response should enumerate the different answers cohesively. since a question may elicit a mixture of conflicting and non-conflicting answers, we want to determine whether the model response 𝑦 accurately reflects the conflict or non-conflict relations between the different answer pairs. formally, for reference answers 𝐴 and a respective set 𝐶 = {𝑎 𝑖 , 𝑎 𝑗 } | 𝑎 𝑖 and 𝑎 𝑗 conflict , that lists the pairs of conflicting answers in 𝐴, the objective of response 𝑦 is to accurately incorporate 𝐴 and 𝐶. in accordance with the task definition, we define two evaluation criteria, adopted from related tasks @xcite : answer quality, measuring how well the model covers the set of correct answers; and conflict identification, assessing the model's ability to correctly identify conflicting answer pairs. we propose metrics for the evaluation criteria in §4.3. ## creating the natconfqa dataset obtaining annotations for a conflict-aware maqa dataset is challenging, because informational conflicts are infrequent in arbitrary sources from which answers can be collected. in this work, we approach this challenge by leveraging existing fact-checking datasets, a well-studied field with many datasets, which were annotated with large manual effort @xcite . some fact-checking datasets contain claims for which some pieces of evidence refute the claim, while others support it, as exemplified in figure 1 . this structure makes them particularly valuable for our task, as they naturally capture conflicting evidence. in this section, we describe our methodology for converting and annotating existing fact-checking datasets into a conflict-aware maqa dataset ( §3.1 and §3.2). we then describe the resulting new natconfqa dataset ( §3.3) finally, we examine its quality ( §3.4). ## experimental setup in this section, we outline our experimental setup for evaluating how well models detect and communicate conflicts. we describe two prompting modes for the conflict-aware maqa task ( §4.1) that will be applied on eight top-performing llms ( §4.2). their performance will be assessed using two evaluation criteria adapted from prior work ( §4.3). ## results and analysis in this section, we first present results on the two subsets of natconfqa under both prompting modes, and analyze the general trends ( §5.1, §5.2). we then conduct a manual error analysis on a sample of model responses ( §5.3), uncovering the techniques used by models to wrongly handle conflicts. ## related work in multi-answer qa, a question may have multiple valid answers, each supported by its own evidence @xcite . although most datasets for this task generally assume that the different answers are consistent and complementary @xcite @xcite @xcite , in real-world scenarios, a query can expose conflicts or discrepancies between the different textual sources. yet, there are only several qa datasets that address conflicting answers, each exhibiting its own set of limitations. other recent works focus on various types of conflicts in rag settings, where there is a single correct answer @xcite , multiple points of view @xcite , or a mix of different conflict types, such as temporal, misinformation, or opinion @xcite . most similar to our work, wikicontradict @xcite includes humanannotated qas that incorporate naturally-occurring (rather than synthetic) conflicting answers, found in wikipedia articles. yet, the instances in this dataset are limited to only two relatively short evidence passages, which always contradict each other. in contrast to existing resources, natconfqa is a human-annotated dataset composed of naturally occurring conflicts between the different answers, covering both yes/no and wh-questions. additionally, each instance includes on average 5.5 passages. importantly, our work is the first to collect finegrained annotations for each pair of answers, where some answer pairs are conflicting while others are not (the "mixed" category in ## conclusion in this work we enhance the conflict-aware multi-answer qa task by explicitly requiring conflict identification among answers. we create a dataset for the enhanced task, via a novel cost-effective methodology that leverages fact-checking datasets. our natconfqa dataset is a realistic, conflict-rich benchmark that challenges current strong models. we test several state-of-the-art llms on the dataset, and show that models still struggle with surfac-ing conflicting answers consistently, particularly in instances that contain both conflicting and nonconflicting answers, even when expressly prompted to be on the watch for potential conflicts. finally, an error analysis of model responses exposes manners in which llms mishandle conflicts.