Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation (2025)

Abstract: Faithfulness evaluators based on Large Language Models (LLMs) are often fooled by the fluency of the text and struggle with identifying errors in the summaries, usually leading to high false negative rate. We propose an approach to summary faithfulness evaluation in which multiple LLM-based agents are assigned initial stances (regardless of what their belief might be) and forced to come up with a reason to justify the imposed belief, thus engaging in a multi-round debate to reach an agreement. The uniformly distributed initial assignments here result in a greater diversity of stances leading to more meaningful debates and ultimately more errors identified. Furthermore, by analyzing the recent faithfulness evaluation datasets, we observe that naturally, it is not always the case for a summary to be either faithful to the source document or not. We therefore introduce a new dimension ambiguity and a detailed taxonomy to identify such special cases. Experiments demonstrate our approach can help identify ambiguities, and have even a stronger performance on non-ambiguous summaries.

## introduction summary evaluation has a long history, and over the years, different approaches have been applied to evaluate the quality of the generated summaries including n-gram based metrics @xcite , representation-based approaches @xcite , finetuned specialized evaluators @xcite @xcite @xcite and human evaluation. with recent advancements in llms and their superior ability to generate fluent text, automatic summary evaluation has gained even more attention. in particular, assessing aspects like faithfulness has become more challenging due to the high fluency of llmgenerated text. while overlap-based metrics usually show weak correlation with human judgments @xcite and finetuned approaches usually lack explainability, human evaluation is also costly with high turnaround time, low reproduciblity and low inter annotator agreement (iaa). with that said, efficient and accurate evaluation of summaries still remains a challenge. automatic evaluation using llms have shown promising results, overcoming some of the major bottlenecks of traditional approaches in efficient evaluation of the generated summaries. different single-llm and multi-llm settings have been applied on a wide range of tasks and are shown to be strong automatic evaluators @xcite @xcite @xcite . but even llms as evaluators fail to identify a large portion of the errors and are often fooled by the fluency of the llm-generated summaries. interestingly, when told that a given summary is unfaithful, llms can come up with correct reasoning and arguments that they couldn't otherwise, showing their inherent potential for error detection. to efficiently exploit the error detection capability of the llms to reason about the faithfulness of a given summary, we propose madisse, a multi-agent debate with initial stance for summary evaluation framework, in which llm-based agents will be assigned opposing initial stances (either faithful or unfaithful) as their beliefs on the faithfulness quality of the summary. forcing llms to come up with reasons to justify an initial stance might not always lead to correct prediction as the stances are random and might not be aligned with actual faithfulness labels. therefore, agents engage in multiple rounds of debate with each other, either support or refute others' arguments with the aim of resolving any inconsistencies and reaching an agreement on the final label. however, the main underlying assumption in faithfulness evaluation is that a summary always has a right answer and can either be classified as faithful or unfaithful which might not be the case. a summary can be interpreted in different correct and plausible ways and then depending on the interpretation can be seen as both faithful and unfaithful as shown in figure 2 . this would lead to low iaa regardless of the quality of the evaluators as they might only think of one interpretation and base their evaluation on that. the possibility of a summary having multiple interpretations leading to different faithfulness evaluations can impact the conclusions regarding system performance and ranking. we therefore introduce a new evaluation dimension, ambiguity, and we define it as when a summary can have multiple correct interpretations in context of the given document leading to opposing beliefs about the faithfulness of the summary. an optimal faithfulness evaluator should address any ambiguities before evaluating faithfulness and the initial step in doing so is to identify such ambiguous summaries. to facilitate this, we also provide a detailed taxonomy of ambiguities and a human annotated dataset by extending the tofueval meetingbank dataset @xcite with ambiguity annotations. our main contributions can be summarized as follows: (1) we propose madisse, a multi-agent debate setup with initial stance for improved faithfulness evaluation leading to stronger performance compared to single-llm and multi-llm setups for non-ambiguous scenarios by identifying more errors; (2) we introduce a new evaluation dimension, ambiguity, a detailed taxonomy of ambiguity types and provide ambiguity annotation on tofueval meetingbank dataset; (3) we show how the debate approach can help with identifying ambiguous cases and furthermore can even have a stronger performance in terms of accuracy and increasing iaa, when evaluated on non-ambiguous summaries. ## madisse faithfulness as a key evaluation dimension of summarization systems, measures whether the facts specified in the summary can be attributed to the source document. we focus on faithfulness as described above and consider summaries to be faithful if only they can be entailed from the source document foot_0 . formally, we define an evaluation model m to predict whether the summary s can be entailed from the source document d. m (d, s) ∈ {faithful, unfaithful} the overview of madisse can be seen in figure 1 . each madisse session consists of three main stages: initialization, debate and adjudication. in the initialization stage, a pool of evaluator agents a are assigned a random stance on whether they believe the summary is faithful or not. in the second stage, the agents engage in a debate for n rounds and each agent a @xmath0: u j @xmath1, l j i ) where e j i is the explanation to justify the decision and l j i is the faithfulness label assigned to the summary at round j for the i-th agent a i . if at any round j, all agents agree on the final label, the debate will be stopped and the final label of the summary is determined. if agents do not reach an agreement after n rounds, the debate will stop and then the final label is determined by adjudication. adjudicators j 1 , ..., j @xmath2. in the following sections, we will detail each component of madisse, describing their responsibilities and goals and how they achieve them. ## defining and annotating ambiguity faithfulness evaluation is usually done with a major underlying assumption: the summaries can al-ways be definitely classified as either faithful or with some faithfulness errors. however this might not always be true. a summary can be interpreted in different ways, all plausible but where one interpretation can make the summary faithful whereas with a different interpretation, one might consider the summary as unfaithful. given the example in figure 2 , depending on how one would parse the summary, two interpretations can emerge; the first one would make the summary faithful but the second one would give the unfaithfulness perception. we therefore define the notion of ambiguity as follows: an ambiguous summary can be correctly mrs merkel said she wanted friendly relations with both countries but europe now had to "fight for its own destiny". her comments come after mr trump refused to re-commit to the 2015 paris climate deal at the g7 summit. mrs merkel is on the campaign trail ahead of elections in september. "the times in which we could completely depend on others are on the way out. …. by all means keep friendly relations with trump's america and brexit britain, was the message-but we can't rely on them. …. german chancellor angela merkel has said the times when europe could "completely depend on others" are "on the way out" after us president donald trump's comments on climate change and brexit. parse 1: ... [after [ [us president donald trump's comments on climatechange and [brexit] ]] parse 2: ... [after [us president donald trump's comments on [ [climate change]and [brexit] ]] label: structural ambiguity document summary interpretations figure 2: a summary with structural ambiguity which can be interpreted using two different parses where interpretation with parse 1 makes the summary faithful whereas parse 2 makes the summary unfaithful. to better help with identifying ambiguous cases as defined above, we first introduce a detailed taxonomy of such cases along with the definitions and examples of each category in section 4.1. then, in a first attempt to identify ambiguous cases in summaries, we extend tofueval meetingbank @xcite with ambiguity human annotations and present the details in section 4.2. ## conclusion we have proposed madisse, a new automatic llm-based multi-agent summary faithfulness evaluation with stance initialization and multi-round debate shown to be capable of identifying more errors compared to other llm-based baselines. we have also identified a new evaluation dimension called ambiguity and a detailed taxonomy to identify ambiguous summaries that can be evaluated as both faithful and unfaithful depending on the how one would interpret them. we extend the meetingbank dataset by providing annotations for ambiguity dimension and show how filtering the ambiguous cases can help further improve the results and lead to higher iaa.