Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2024)

Abstract: Modern large language models (LLMs) like ChatGPT have shown remarkable performance on general language tasks but still struggle on complex reasoning tasks, which drives the research on cognitive behaviors of LLMs to explore human-like problem-solving strategies. Along this direction, one representative strategy is self-reflection, which asks an LLM to refine the solution with the feedback generated by itself iteratively. However, our study shows that such reflection-style methods suffer from the Degeneration-of-Thought (DoT) problem: once the LLM has established confidence in its solutions, it is unable to generate novel thoughts later through reflection even if its initial stance is incorrect. To address the DoT problem, we propose a Multi-Agent Debate (MAD) framework, in which multiple agents express their arguments in the state of “tit for tat” and a judge manages the debate process to obtain a final solution. Clearly, our MAD framework encourages divergent thinking in LLMs which would be helpful for tasks that require deep levels of contemplation. Experiment results on two challenging datasets, commonsense machine translation and counter-intuitive arithmetic reasoning, demonstrate the effectiveness of our MAD framework. Extensive analyses suggest that the adaptive break of debate and the modest level of “tit for tat” state are required for MAD to obtain good performance. Moreover, we find that LLMs might not be a fair judge if different LLMs are used for agents.

## introduction large language models (llms) have shown remarkable performance on general language tasks @xcite ; bang 0.00 0.25 0.50 0.75 1.00 1 2 3 4 5 multi-agent debate self-reflection iteration avg. disagreement et al., 2023) but still struggle on complex reasoning tasks @xcite , which drives the research on cognitive behaviors of llms to explore human-like problem-solving strategies. in particular, self-reflection @xcite , a concept that usually refers to the process of introspection and examination of a person's own thoughts, has been explored to solve intricate tasks that could be challenging for a zeroshot generation or even chain-of-thought (cot) prompting @xcite . specifically, selfreflection involves an iterative refinement process such that the llm generates a new answer based on the answers and feedback in previous iterations and then provides feedback for the new answer. while self-reflection can be effective in creating better answers, it is highly dependent on the self-evaluation capabilities of llms, which are not formally guaranteed @xcite . in this work, we focus on the degeneration-of-thought (dot) problem in self-reflection, which is proposed and defined by us for the first time. formally, dot describes the following scenario: once the llm-based agent has established confidence in its answers, it is unable to generate novel thoughts later through self-reflection even if the initial stance is incorrect. to demonstrate this problem, we force the agents to engage in a debate or self-reflection for 5 rounds before reaching an answer. next, we manually determine the disagreement as 1 and agreement as 0 between two adjacent iterations. we define the average disagreement in iteration i as the percentage of opposition occurring between two debaters across multiple debates (or self-confliction in selfreflection). we show the trends in figure 1 . the low disagreement of self-reflection suggests that the llm sticks to the incorrect answers predicted by cot and is unable to engage in meaningful selfreflection. there are various factors @xcite that could result in dot, and we outline three here: (1) bias and distorted perception. self-perception can be influenced by biases, preconceived notions, and distorted thinking patterns, which can be learned from the massive amount of data during pretraining. if an llm's self-reflection is clouded by such biases or distorted thinking, it can lead to inaccurate conclusions instinctively. (2) rigidity and resistance to change. self-reflection often involves challenging one's beliefs, assumptions, and behaviors. if an llm is resistant to change or holds rigid beliefs, it may struggle to engage in meaningful self-reflection that leads to better answers. (3) limited external feedback. selfreflection is primarily an internal process, but external feedback can provide valuable perspectives and insights. without considering external feedback, an llm may miss important blind spots or alternative viewpoints that can enrich its self-reflection. to address the dot issue, we leverage another fundamental characteristic of human problemsolving, i.e., debate, to encourage divergent thinking in llms. specifically, we propose the mad framework, short for multi-agent debate, where two agents express their own arguments in the state of "tit for tat" and a judge monitors and manages the debate process to obtain a final solution. the nature of mad determines that (1) the distorted thinking of one agent can be corrected by the others; (2) the resistance to change of one agent will be complemented by the others; and (3) each agent can obtain external feedback from the others. therefore, mad is less susceptible to the factors of dot, and can explore divergent chain-of-thoughts to achieve accurate answers. we conducted experiments on both natural language generation and understanding through two challenging tasks, namely, commonsense machine translation (common mt) and counter-intuitive arithmetic reasoning (counter-intuitive ar). the common characteristic of the two tasks is that our instincts are mostly incorrect based on only the superficial expressions of the questions, and deeper levels of contemplation are required for better answers. experimental results demonstrate that our mad framework outperforms the baseline methods, especially, gpt-3.5-turbo with mad can surpass the performance of gpt-4 on common mt. the contributions of this work are summarized as follows: ## multi-agent debate framework figure 2 illustrates the general framework of mad, where two debaters and a judge are involved in a debate to resolve a math problem while self-reflection descends into the trap of dot. generally, our mad framework is composed of three components which are elaborated as follows: meta prompts. we use meta prompts to introduce the topic to be solved, the number of debaters, the iteration limit, and other requirements. an exampe of meta prompts for the arithmetic reasoning task in figure 2 is: you are a debater. hello and welcome to the debate competition. it's not necessary to fully agree with each other's perspectives, as our objective is to find the correct answer. the debate topic is stated as follows: <debate topic>. as seen, we require the agents to "tit for tat" (e.g. contents underlined in meta prompts above) so as to create an atmosphere of debate. reflection: based on the results of my selfreflection, i am confident that my answer is correct. therefore, circle a will revolve 3 times in total as it rolls around circle b once. let the radius of circle a be r and the radius of circle b be 3r. the distance circle a travels along the circumference of circle b is equal to the circumference of circle b, which is 6πr. the number of @xmath0. so, circle a will revolve 3 times in total as it rolls around circle b once. that's a valid point, however... i disagree with you. to find the total number of revolutions, we need to consider both the rotation around circle b and the rotation of circle a itself. therefore, circle a will revolve 3 times around its own center and 1 time around circle b, making a total of 4 revolutions. the two circles are externally tangent and there is no relative sliding. the radius of circle a is 1/3 the radius of circle b. circle a rolls around circle b one trip back to its starting point. how many times will circle a revolve in total? ## analysis in this section, we present a qualitative analysis to provide some insights how mad works. unless otherwise stated, we report the overall results on the common mt dataset. ## conclusion we propose and define the degeneration-of-thought (dot) problem in self-reflection, and address it by proposing the multi-agent debate (mad) framework to explore divergent chainof-thoughts. we demonstrate the effectiveness of mad on two challenging tasks and find that gpt-3.5-turbo with mad can even surpass gpt-4 on the common mt dataset. extensive analyses suggest that the adaptive break strategy of debate and the modest level of "tit for tat" state are required for mad to obtain good performance. complex samples require more rounds of debate. more interestingly, we find that llms might not be a fair judge if different llms are used for agents. future work includes scheduling more agents in the debate in an appropriate manner, multi-agent intelligence for board games, and ai feedback for model alignment.