Enhancing Multi-Agent Debate System Performance via Confidence Expression (2025)

Abstract: Generative Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks. Recent research has introduced Multi-Agent Debate (MAD) systems, which leverage multiple LLMs to simulate human debate and thereby improve task performance. However, while some LLMs may possess superior knowledge or reasoning capabilities for specific tasks, they often struggle to clearly communicate this advantage during debates, in part due to a lack of confidence expression. Moreover, inappropriate confidence expression can cause agents in MAD systems to either stubbornly maintain incorrect beliefs or converge prematurely on suboptimal answers, ultimately reducing debate effectiveness and overall system performance. To address these challenges, we propose incorporating confidence expression into MAD systems to allow LLMs to explicitly communicate their confidence levels. To validate this approach, we develop ConfMAD, a MAD framework that integrates confidence expression throughout the debate process. Experimental results demonstrate the effectiveness of our method, and we further analyze how confidence influences debate dynamics, offering insights into the design of confidence-aware MAD systems.

## introduction multiple studies have demonstrated that llms possess emergent reasoning and reflection capabilities @xcite @xcite . by effectively harnessing these abilities, researchers can enhance the accuracy of llm responses, minimize hallucinations, and strengthen reasoning capabilities. building upon this foundation, @xcite , drawing inspiration from the society of mind @xcite and multiagent frameworks, proposed utilizing multi-agent systems composed of multiple llms to achieve superior performance on various tasks. specifically, they developed a multi-agent debate (mad) system where, when presented with a query, multiple instances of llm agents first generate independent candidate responses. subsequently, these agents engage in a structured debate about these responses, iteratively refining and updating their own contributions throughout the process. however, we've identified a key issue with current mad systems. llm agents with different knowledge and capabilities don't explicitly express their confidence level regarding their arguments and knowledge during communication, preventing full utilization of each agent's relative strengths. this issue may limit the performance of current mad systems, potentially leading to failure to converge or even convergence towards incorrect answers. as shown in figure 1 , in a basic debate setting @xcite , when only one llm provided the correct answer in the initial round on the bbh and mmlu datasets, fewer than 50% of such cases ultimately converged to the correct answer. to address the above issue, we propose an intuitive solution by incorporating confidence expression for each llm instance in mad systems. during the debate process, for a given query, each llm agent's debate content includes not only arguments (reasons) and an answer, but also a confidence score. additionally, considering that modern deep neural networks often exhibit overconfidence @xcite @xcite @xcite , we introduce calibration methods to further investigate the impact of confidence on the mad system. based on these considerations, we developed a mad framework with confidence expression called confmad foot_0 . we evaluated confmad on various benchmarks. as shown in figure 2 , our results demonstrate that introducing confidence expression can lead to notable accuracy improvements across multiple benchmarks. building on this foundation, we provide detailed discussions on how confidence scores influence the mad debate process and offer some insights for developing better confidence-aware mad systems. in summary, the contributions of our work are summarized as follows: in this section, we are going to introduce the design of confmad. the overall design of confmad is presented in figure 3 . the core design elements of confmad encompass confidence expression, calibration scheme, and debate workflow. ## conclusion in this work, we propose incorporating confidence expression into mad systems to enhance their performance. we explore two confidence expression methods and several calibration schemes to adjust confidence scores. to validate our approach, we develop confmad, a confidence-aware mad framework. experiments across multiple benchmarks show that confidence expression not only improves the accuracy of individual llm agents but also helps the system reach correct consensus more effectively. we further analyze the impact of different calibration methods and highlight the importance of aligning confidence scores across agents. our findings underscore the critical role of wellcalibrated confidence in enabling more reliable and effective multi-agent debate and offer insights for designing better confidence mechanisms in future mad systems.