Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate (2023)

Abstract: Large Language Models (LLMs) have shown impressive capabilities in various applications, but they still face various inconsistency issues. Existing works primarily focus on the inconsistency issues within a single LLM, while we complementarily explore the inter-consistency among multiple LLMs for collaboration. To examine whether LLMs can collaborate effectively to achieve a consensus for a shared goal, we focus on commonsense reasoning, and introduce a formal debate framework (FORD) to conduct a three-stage debate among LLMs with real-world scenarios alignment: fair debate, mismatched debate, and roundtable debate. Through extensive experiments on various datasets, LLMs can effectively collaborate to reach a consensus despite noticeable inter-inconsistencies, but imbalances in their abilities can lead to domination by superior LLMs. Leveraging a more advanced LLM like GPT-4 as an authoritative judge can boost collaboration performance. Our work contributes to understanding the inter-consistency among LLMs and lays the foundation for developing future collaboration methods. Codes and data are available at https://github.com/Waste-Wood/FORD.

## introduction large language models (llms) like chatgpt recently demonstrate general intelligence @xcite and have been widely used as a foundation model in various applications @xcite . to solve complex tasks, multiple llms are further introduced to collaborate, with each targeting a different subtask or aspect @xcite . interestingly, do these llms possess a spirit of collaboration? are they capable of cooperating effectively and performantly towards a shared goal? ## preliminaries we first describe the datasets and llms used in this paper. then we define a metric incon to quantify the inter-inconsistency among multiple llms. finally, we define the baselines to verify our proposed formal debate framework ford. ## ford: formal debate framework to further explore the inter-inconsistency, inspired by the structure of debate @xcite , we design a formal debate framework (ford) for llms collaboration to solve shared tasks. in figure 2 , we use two llms for illustration, ford consists of 3 steps: (1) stance selection and argument generation expects llms to choose a stance and give an argument for each sample, aiming to select the inconsistent samples for future collaboration in step 2. ## mismatched debate in human discourse, when interacting with a dominant individual, we often tend to be influenced by their cognition, leading us to adopt their ideas and thought processes @xcite . we investigate this scenario in mismatched debate. ## roundtable debate in many scenarios, the debates or discussions often involve more than two participants, such as law @xcite and healthcare @xcite . since llama and vicuna are not good at debate, we design two roundtable debates among three llms: one mismatched debate chat-gpt & davinci-003 & gpt-4 (denoted as r1), and one fair debate chatgpt & davinci-003 & chatgpt-0301 (denoted as r2). following sec 5, we select e-care and piqa for experiments. in step 1 of ford, the samples that not all llms reach agreements would be sent to step 2. for r1 and r2 in step 2, chatgpt debates first, davinci-003 second to align with the fair debates between two llms. gpt-4 and chatgpt-0301 speak last in r1 and r2, respectively. the llms debate up to 9 rounds. refer to appendix d for more details and the prompts. the overall results of the roundtable debates are shown in ## conclusion in this paper, we explore inter-inconsistency issues among llms. then we propose ford to examine whether llms can effectively collaborate to ultimately achieve a consensus through debate. with ford, we explore three real-world debate scenar-ios: fair debate, mismatched debate, and roundtable debate. we find llms possess the spirit of collaboration to achieve a consensus for solving a shared goal. debates can improve the performance and inter-consistency of llms. stronger llms can be distracted by weaker llms when the debates are mismatched. these findings contribute to the future development of collaboration methods.