Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key? (2024)

Abstract: Recent progress in LLMs discussion suggests that multi-agent discussion improves the reasoning abilities of LLMs. In this work, we reevaluate this claim through systematic experiments, where we propose a novel group discussion framework to enrich the set of discussion mechanisms. Interestingly, our results show that a single-agent LLM with strong prompts can achieve almost the same best performance as the best existing discussion approach on a wide range of reasoning tasks and backbone LLMs. We observed that the multi-agent discussion performs better than a single agent only when there is no demonstration in the prompt. Further study reveals the common interaction mechanisms of LLMs during the discussion. Our code can be found in https://github.com/HKUST-KnowComp/LLM-discussion .

## introduction large language models (llms) demonstrate strong abilities in language understanding and generation @xcite @xcite @xcite . however, llms still fall short for reasoning tasks due to model deficiencies like hallucination and reasoning perspective mistakes @xcite . to overcome these issues, numerous works have been proposed by simulating human reasoning. inspired by society of mind @xcite , multi-agent discussion frameworks such as debate @xcite , mad @xcite , and reconcile @xcite present a novel approach by involving multiple ai agents. powered by llms, these agents autonomously engage in discussions on given topics, improving the reasoning abilities of llms by emulating the human discussion process. to further improve the performance, most of multi-agent discussion frameworks leverage task-specific examples, which are often termed as demonstrations @xcite . this is based on the insights that llms can learn from the context demonstrations @xcite . we note that all these discussion frameworks claim that they outperform the conventional methods with a single agent, such as chain-of-thought @xcite . figure 1 presents a comparison between singleagent settings and multi-agent discussion frameworks on folio-wiki dataset @xcite . in this figure, 'demo' means that the tested single agent is provided with a demonstration case, 'q-desc.' indicates that the single agent is provided with detailed question description, while 'direct' refers to the single agent without demonstrations. we observe that the single agent 'demo' tends to reach a performance upper bound similar to that of all discussion frameworks, which will be further elaborated on later. this result suggests that the previous claim is not fully established. based on the observation from figure 1 , in this paper, we conduct systematic experiments to provide in-depth analysis. as a newly emergent topic, the number of available discussion frameworks still remains limited. to provide a more com-1 6106 after careful consideration, i think that participant a makes more sense. therefore, the proposition is ## cmd: conquer-and-merge discussion to provide a thorough comparison in following experiments, we identify and rectify the gap in pre-vious research, which has not suggested a groupdiscussion-based framework. to address this, we introduce a novel framework for multi-agent group discussion, referred to as cmd. in cmd, suppose there are n agents @xmath0, and agent a i is powered by an llm l i . we use an array h to store the history responses. the agents will discuss for r rounds. during each turn r, every agent a i generates a response (v i , e i ) = cmd(h|q, r), where v i is the viewpoint and e i denotes the explanation. detailed description can be seen in appendix c. ## experimental setups in the experiments, we contrast single-agent setup with four established multi-agent discussion frameworks: debate, mad, reconcile, and cmd under various prompt conditions. these methods are introduced in previous sections 2.2 and 3. ## experiments on single llm in this section, we conduct our experiments using a single llm, chatgpt-3.5. to gain the initial insights, we provide an in-depth analysis of the folio-wiki dataset, examining both single-agent settings and multi-agent discussions as detailed in section 5.1. afterward, we extend our experiments to two other datasets gsm8k and ecqa in section 5.2. we also investigate common mistakes made by discussions through a case study in section 5.3. finally, we summarize our findings in section 5.4. ## experiments on multiple llms in this section, we expand our experimental scope from a single llm to multiple llms for both single-agent settings and multi-agent discussion frameworks, which allows us to test the validity of our previous findings in multi-llm scenarios. specifically, we assess the performance of agents powered by three advanced llms: chatgpt-3.5, gemini pro, and bard. in section 6.1, we compare the performance of three single-agent configurations, each using a different llm, against multiagent discussions leveraging all three llms. our round-level analysis in section 6.2 yields another insight: an agent powered by stronger llm can enhance the performance of an agent powered by a less capable llm. more experimental results can be found in appendix a, where we study the open-source llms and how they interact with other llms in cmd. ## conclusion in this paper, we re-examine the claim that multiagent discussions are superior to a single agent in reasoning tasks by conducting systematic experiments. we introduce a novel framework cmd for a comprehensive and fair assessment. by conducting experiments over standard benchmarks, we find that (1) a single agent with a strong prompt and powered by a strong llm achieves comparable per-8 formance with multi-llm multi-agent discussions;