M ulti A gent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debate (2024)

Abstract: Large Language Models (LLMs) have shown exceptional results on current benchmarks when working individually. The advancement in their capabilities, along with a reduction in parameter size and inference times, has facilitated the use of these models as agents, enabling interactions among multiple models to execute complex tasks. Such collaborations offer several advantages, including the use of specialized models (e.g. coding), improved confidence through multiple computations, and enhanced divergent thinking, leading to more diverse outputs. Thus, the collaborative use of language models is expected to grow significantly in the coming years. In this work, we evaluate the behavior of a network of models collaborating through debate under the influence of an adversary. We introduce pertinent metrics to assess the adversary’s effectiveness, focusing on system accuracy and model agreement. Our findings highlight the importance of a model’s persuasive ability in influencing others. Additionally, we explore inference-time methods to generate more compelling arguments and evaluate the potential of prompt-based mitigation as a defensive strategy.

## introduction large language models (llms) have exhibited exceptional capabilities across various domains, such as reasoning @xcite , code generation @xcite , and mathematics @xcite . the expansion of their capabilities and their increasing commoditization are establishing llms as building blocks in the development of agents capable of performing more real-world tasks. this is achieved through their integration with tools, apis, and collaboration with other llms @xcite . similar to human interactions, collaboration between agents stands as a mechanism that can help solve more complex and real-world problems. code: https://github.com/amayuelas/multi-agent-attack figure 1 : agent collaboration can be vulnerable to adversarial attacks. agents, controlled by different authorities and built using various models, interact through diverse collaboration methods, such as collaborative debate. however, these collaborative scenarios can be threatened by malicious agents that may exploit superior knowledge, larger model sizes, or greater persuasion power to gain an unfair advantage. previous works suggest that llm collaboration and debate lead to more divergent thinking @xcite , better reasoning and factuality @xcite , and more faithful evaluations @xcite . these results demonstrate the importance of collaboration in achieving more capable agents, at the expense of greater computational resources and more complexity to build them. in the near future, agents are destined to collaborate with other agents controlled by different authorities and with varying capabilities. this raises critical questions: what if the agents do not share the same goal? what will the outcome be if one or more agents want to take advantage of or break the collaboration mechanism? how robust is the collaboration mechanism to an adversarial attack? in this work, we focus on answering these questions, where models must interact through debate to answer questions or complete tasks. some agents may act against the general goal and attempt to gain an unfair advantage through greater access to knowledge, larger model size, or superior persua-sive power. we believe it is crucial to address these questions to develop more robust communication and collaboration methods between llms. to evaluate this scenario, we selected four representative tasks: reasoning @xcite ), trustworthiness @xcite ), @xcite ), and legal @xcite ). the first two tasks address llm-specific challenges, while the latter two focus on high-risk applications. the evaluation involves a debate between llms. initially, the llms receive a question and independently provide answers. each response is then shared with the other models for reconsideration and revision over several rounds. an example of this debate is illustrated in figure 2 . to simulate an adversarial attack, the adversary selects an incorrect answer and tries to persuade the other agents to accept it as correct. this highlights the models' persuasive abilities and their susceptibility to persuasion. we evaluate this threat by measuring the drop in accuracy and the change in agreement with the adversary from the initial to the final rounds. additionally, we explore methods for generating more convincing arguments. from the experiments and analysis described, we can highlight the following insights: 1. collaboration via debate is usually vulnerable to an adversary. in general, the adversary is able to undermine the common objective with system accuracy decreases ranging from 10% to almost 40%, and individual accuracy decreases from the group models of up to 30%. 2. model's persuasiveness is an important ability to attack the collaborative setting. persuasion is a skill that has traditionally not gained a lot of attention in language models. we show how to evaluate it based on accuracy and agreement. and we highlight its relevance in language models due to its effect on collaboration. ## related work cooperation and collaboration between agents have been extensively studied @xcite . the emerging capabilities of language models have prompted research into the collaborative abilities of deep learning models @xcite . multi-agent networks can be very useful in applications such as software development and court simulations @xcite . multiagent collaboration. among agent collaboration techniques, debate emerges as the most effective method of communication. given that llms have become proficient in generating and understanding human language, they can leverage it to communicate with each other. inspired by the concept of the society of mind @xcite , debate among agents aims to harness collective knowledge, achieving superior results compared to individual efforts. this has been demonstrated in several studies: @xcite shows that multiagent debate can enhance factuality and reasoning, foster more divergent thinking @xcite , and even achieve state-of-the-art performance in mathematical reasoning @xcite . other research works have focused on studying collaboration mechanisms @xcite @xcite (e.g., hierarchical vs. same-level or role-playing), attack @xcite or their implications in competitive settings @xcite . recently, several frameworks have been developed to facilitate the implementation and integration of different collaboration mechanisms and role-playing scenarios, including autogen @xcite , camel @xcite , and metagpt @xcite . persuasiveness. as debate becomes the primary tool for llm coordination, persuasion emerges as a critical trait, enabling agents to influence others to deviate from their prompted tasks. this raises several research questions that have only recently been explored. one focus is the overall persuasiveness of llms, as studied by @xcite . another area of interest is the type of arguments that llms find convincing, discussed in @xcite . the ability of llms to recognize convincing arguments is explored in @xcite . furthermore, the potential persuasiveness of llms is investigated in @xcite . additionally, @xcite examines how a weaker model can assess a stronger one and how persuasiveness enables a model to identify truth in debates. figure 2 : sample debate (from mmlu). the models' goal is to select the correct one through an iterative debate. debate: initially, each model independently answers the question. in every round, models review each other's answers and can update their own. adversary: the adversary is given a wrong answer and attempts to convince the other models it is correct, succeeding in this example. a detailed version of this example is provided in appendix a. ## methods debate -debate serves as the primary mode of communication among llms, using human language for interaction. in this protocol, models engage in argumentation to justify their responses to a given question. we select a predetermined number of group models that will engage in the collaboration: m @xmath0, where the total number of models in the debate is m . the collaborative goal is to solve the task as accurately as possible. following the setup introduced by @xcite , all models are initially presented with the same question (q i ), to which each model provides an initial response. the debate proceeds for a predetermined number of rounds r t , from a total of t rounds. during the debate rounds, each model receives the responses from the other models and generates updated answers. this process is depicted in figure a . after t rounds, a final answer is selected through a majority vote. adversary -the goal of the adversary (a m ) is to convince other models in the debate to not provide the correct answer. the adversary is given an incorrect answer and prompted to convince the other agents that it is correct. after each round, the adversary is reminded to maintain its answer to avoid being influenced by the group. the attack is successful if the adversary manages to convince other models to change their answers. changing other models' answers or opinions has not been thoroughly studied, and we believe persuasiveness can become a key aspect in agent or human collaboration. optimizing for more persuasive arguments -in our threat scenario, it is crucial for the models to generate convincing arguments capable of persuading other agents involved in the conversation. we explore inference-time strategies to achieve this. drawing inspiration from the work by @xcite , we implement an argument selection mechanism (best-of-n). this mechanism generates multiple requests to the adversary llm (a m ) to produce several completions supporting the adversarial answer. a preference model (p m ) then ranks these responses. specifically, it compares each generated response to a dummy argument and computes the log probability for each response. the response with the highest rank is selected as the most convincing argument. further details are included in appendix c. ## results and analysis in this section, we evaluate the effectiveness of the adversary in the described setting based on multiagent collaboration via debate. we present general results, improved attack, and fine-grained analysis to identify the model's persuasive power. we also introduce ablation studies and possible mitigation. ## conclusion this work studies the vulnerabilities of language model collaborations via debate. we foresee the collaboration of llms becoming more relevant in the coming years and the interaction between distinct models controlled by different entities. therefore, we consider this topic of high importance. in particular, we analyze the behavior of the collaboration under the presence of an adversary. in our experiments, we show that an adversary can undermine the common objective of other models, highlighting the crucial skill of persuasion. the adversary's ability to persuade other models is key to the success of the attack. we measure this using a combination of system accuracy and adversary as the number of agents in the debate increases, the overall system accuracy is slightly higher. however, the accuracy drops after every round, indicating the network is not robust against the adversary. agreement, observing that the adversary generally convinces the other models. the attack remains effective with increasing debate rounds and agents, suggesting that model persuasion ability is the main driver. additionally, we propose methods to generate more convincing arguments based on added knowledge or improved argument generation at inference time. we believe this work is a first step toward developing robust communication and collaboration systems with llms, as well as raising awareness of the importance of persuasiveness in such environments. future work should focus on refining defensive strategies, exploring alternative collaboration protocols, and improving the inherent persuasive abilities of llms to resist adversarial attacks.