R e C oncile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLM s (2024)

Abstract: Large Language Models (LLMs) still struggle with natural language reasoning tasks. Motivated by the society of minds (Minsky, 1988), we propose ReConcile, a multi-model multi-agent framework designed as a round table conference among diverse LLM agents. ReConcile enhances collaborative reasoning between LLM agents via multiple rounds of discussion, learning to convince other agents to improve their answers, and employing a confidence-weighted voting mechanism that leads to a better consensus. In each round, ReConcile initiates discussion between agents via a ‘discussion prompt’ that consists of (a) grouped answers and explanations generated by each agent in the previous round, (b) their confidence scores, and (c) demonstrations of answer-rectifying human explanations, used for convincing other agents. Experiments on seven benchmarks demonstrate that ReConcile significantly improves LLMs’ reasoning – both individually and as a team – surpassing prior single-agent and multi-agent baselines by up to 11.4% and even outperforming GPT-4 on three datasets. ReConcile also flexibly incorporates different combinations of agents, including API-based, open-source, and domain-specific models, leading to an 8% improvement on MATH. Finally, we analyze the individual components of ReConcile, demonstrating that the diversity originating from different models is critical to its superior performance.

## introduction a large body of recent work has focused on improving the reasoning capabilities of large language models (llms) by imitating various human cognitive processes @xcite @xcite . these include phenomena like reflecting on and critiquing one's own predictions, being receptive to feedback, and learning from feedback. of note, 1 code: https://github.com/dinobby/reconcile self-reflection is an introspective process that allows the model to improve its outputs by generating feedback from the model itself @xcite . however, self-reflection suffers from degeneration-of-thought -when the model is overly confident in its answer, it is unable to generate novel thoughts even after multiple rounds of feedback @xcite . to promote more diverse thoughts, past work has drawn inspiration from the concept of society of minds in multi-agent systems @xcite . it highlights the importance of communication and collaboration between multiple agents for complex decision-making tasks. while such collaborative frameworks like multi-agent debate @xcite increase the reasoning diversity through the process of a debate, multiple agents have typically been limited to different instances of the same underlying model like @xcite . 2 this results in an inherent model bias, a restricted knowledge scope, and a lack of external feedback from other models due to identical pre-training data and model architectures across all agents. in general, when multiple agents propose solutions to a problem, the success of such a multi-agent system is fundamentally reliant on (a) the diversity of the solutions, (b) the ability to estimate each agent's confidence, and (c) accordingly, convince other agents (with explanations) to reach a better consensus. this puts forward the question: if multiple diverse llms collaboratively solve a task, are they capable of discussing their solutions with each other to reach a better consensus? we aim to solve reasoning problems by learning from diverse insights and external feedback, originating from agents that belong to different model figure 1 : an illustration of the main differences between reconcile and prior works. while most current selfrefine and debating techniques rely on multiple instances of a single model (e.g., chatgpt), our method incorporates models from different families (e.g., chatgpt, bard, and claude2). our approach also emphasizes critical elements of effective discussion, including convincing another agent to improve their answers and incorporating the estimated confidence of all agents. for illustrative simplicity, we depict only one agent contemplating how to convince the other two agents. given a reasoning problem, reconcile begins with each agent first generating an answer, its uncertainty, and a corresponding explanation (as a chainof-thought @xcite ) for the answer. then all agents enter a multi-round discussion phase. each discussion round consists of all agents generating a revised explanation and answer based on all other agents' explanations and answers from the previous round. in particular, reconcile initiates a discussion by designing a discussion prompt for each agent, that lets it condition on (1) grouped answers from all agents, (2) corresponding explanations generated in the previous round, and (3) demonstrations of answer-rectifying human expla-nations for convincing other agents. we leverage them in an in-context learning framework to teach models to generate their own convincing explanations (see fig. 3 ). even in cases where an agent initially offers an incorrect answer and explanation, it can consider another agent's convincing explanation and amend its response accordingly. in each discussion round, we estimate an agent's uncertainty via a confidence-estimation prompt @xcite . once all agents converge to the same answer (i.e., a consensus has been reached), we employ these confidences to compute a weighted vote as the team answer. we primarily develop reconcile with three state-of-the-art llms: chatgpt (openai, 2022), @xcite . we also demonstrate the flexibility of reconcile with variants that employ a much stronger @xcite , an open-source llama-2-70b @xcite , or a domainspecific deepseekmath @xcite model as an agent. across seven benchmarks spanning commonsense reasoning, mathematical reasoning, logical reasoning, and natural language inference (nli), reconcile outperforms prior singleagent (e.g., self-refine @xcite and self-consistency @xcite ) and multiagent baselines (debate @xcite and judge @xcite ) that are built on top of the same underlying models. for example, rec-oncile, (1) on a date understanding task, outperforms the leading multi-agent debate baseline by 2023 )); self-consistency (sc, @xcite ); debate @xcite and judge @xcite . 11.4%, (2) on strategyqa, also outperforms gpt-4 by 3.4%, and (3) on math, outperforms both gpt-4 and a specialized deepseekmath model by 8%. moreover, detailed analyses of the individual components of reconcile demonstrate that leveraging diverse llm agents leads to maximum gains, and we further validate their higher response diversity via a bertscore-based diversity metric @xcite . finally, we show that reconcile not only leads to better team performance but also enables each agent to improve individually via the discussion process. in summary, our primary contributions are: • we propose reconcile, a reasoning framework involving diverse large language models in a round table conference. ## related work reasoning with llms. progress in llms has led to the development of advanced prompting and fine-tuning techniques for solving reasoning problems. representative methods include chain-of-thought (cot) @xcite @xcite and treeof-thought prompting @xcite , selfconsistency @xcite , meta-reasoning over multiple paths @xcite , use of scratchpads @xcite , training veri-fiers @xcite , self-collaboration @xcite @xcite , self-reflection @xcite @xcite , improved math reasoning @xcite and fine-tuning via bootstrapping models @xcite @xcite . eliciting reasoning from a single agent, while promising, is fundamentally limited by a lack of diverse insights. ## problem setup we assume that we are given a test problem q and there are n agents @xmath0. each agent is a distinct llm, potentially trained with different pre-training data and model architectures. all agents are capable of generating an answer and a corresponding chain-of-thought explanation @xcite for the test problem. for each agent a i , we utilize a small number of k demonstrations of convincing samples ## reconcile: a collaborative discussion framework prompt that instructs it to reason about the problem 'step-by-step'. see 'phase 1' in fig. 2 and the prompt is shown in fig. 5 in appendix a.2. phase 2: multi-round discussion. reconcile then enters a discussion phase, consisting of r rounds (see 'phase 2' in fig. 2). in discussion round r, for each agent a i , reconcile develops a discussion prompt d (r) i (as shown in fig. 5), consisting of the following three components. (a) grouped responses of all agents from the previous round. d (r) i consists of the answers {a (r-1) j } n @xmath1. to foster better discussions, reconcile summarizes this information by grouping the answers into distinct categories and appends all plausible explanations for each answer, as shown in our discussion prompt (appendix fig. 5) and on the left side of fig. 2. (b) confidence associated with the answers. all agents are not equally confident in their answers. hence, an effective discussion should also consider each agent's uncertainty. for all black-box models, we estimate its confidence p (r) i in round r by directly prompting the agent to verbally quantify its uncertainty, which in past work has been shown to be effective @xcite . see appendix fig. 5 for the usage of confidence in discussion. (c) convincing samples from all other agents. finally, the prompt contains convincing samples c j for all other agents a j̸ =i . 4 when an agent tries to reassess its reasoning in light of the reasoning provided by other agents, we hypothesize that it should benefit from conditioning on demonstrations that can convince other agents. in order to obtain such convincing samples for an agent a j , we select a small number of samples (4 in our experiments) for which the agent's initial answer is wrong but conditioning on the corresponding human explanation, rectifies the answer (see fig. 3 ). for datasets that do not come with human explanations (e.g., the date understanding task in our experiments), we develop reconcile without using any convincing sample in the discussion prompt and still obtain large improvements (see §6.2 for details). phase 3: team answer generation. recon-cile continues the discussion for a maximum of r rounds or terminates it as soon as a consensus is reached (i.e., all agents agree on the same answer). at the end of any round r, reconcile generates the team answer â(r) for that round using a weighted voting scheme (see the right side of fig. 2 ). in particular, we recalibrate each agent's confidence using a function f (•) and then use these as weights to compute the team answer, as follows: ## experimental setup agents in reconcile. we primarily implement reconcile with chatgpt, bard, and claude2 as the three agents, engaging them in up to three rounds of discussion. later in §6.1, we also show the generalizability of our reconcile framework with different choices of agents, including apibased (gpt-4), open-source (llama-2-70b), and domain-specific (deepseekmath) agents. datasets. we evaluate reconcile on seven benchmarks, including two commonsense, three math, one logical reasoning, and one nli task. these are: (1) strategyqa @xcite , (2) commonsenseqa (csqa; @xcite ), (3) gsm8k @xcite , (4) aqua @xcite , ( 5 ) math @xcite , (6) date understanding (big-bench collaboration, 2023), and (7) anli @xcite . baselines. we compare reconcile to prior works in three categories: • vanilla single-agent methods. in this category, we experiment with (1) zero-shot cot prompting @xcite with one of the interacting llms, and (2) eight-shot cot with claude2 where the number eight matches the number of convincing samples used in reconcile. ## conclusion we presented reconcile, a multi-agent framework for reasoning with diverse llm agents, engaged in multiple rounds of discussion via confidence estimation and generating explanations that can correctively convince other agents. recon-cile demonstrated strong results on multiple reasoning benchmarks, consistently outperforming prior single-agent and multi-agent baselines and even improving upon gpt-4 on some benchmarks.