Preventing Rogue Agents Improves Multi-Agent Collaboration (2025)

Abstract: Multi-agent systems, where specialized agents collaborate to solve a shared task hold great potential, from increased modularity to simulating complex environments. However, they also have a major caveat – a single agent can cause the entire system to fail. Consider a simple game where the knowledge to solve the task is distributed between agents, which share information in a communication channel. At each round, any of the agents can terminate the game and make the final prediction, even if they are uncertain about the outcome of their action. Detection of such rogue agents __before they act__ may prevent the system’s failure. In this work, we propose to __monitor__ agents during action prediction and __intervene__ when a future error is likely to occur. To test our approach, we introduce WhoDunitEnv, a multi-agent collaboration environment that allows modular control over task complexity and communication structure. Experiments on WhoDunitEnv, code generation tasks and the GovSim environment for resource sustainability show that our approach leads to substantial performance gains up to 17.4%, 2.5% and 20%, respectively. Thorough analysis shows that our monitors successfully identify critical points of agent confusion and our interventions effectively stop agent errors from propagating.

## introduction recently there has been a growing interest in leveraging large language models (llms) to solve complex multi-agent tasks @xcite , such as improving factuality and reasoning, @xcite simulating software development teams @xcite @xcite , and human interactions @xcite . a key requirement for succeeding at such cooperative tasks is the agent's ability to communicate effectively. however, establishing effective communication is a major challenge for current llms, which often ignore critical information in the communication @xcite , become distracted by irrelevant messages @xcite , and introduce hallucinations @xcite @xcite @xcite . such failures introduce noise to the communication, which can be amplified as llm responses are generated for many steps @xcite @xcite . a single valid agent can be consequently affected by these issues and go rogue. this rogue agent could then cause the entire system to fail. to improve multi-agent collaboration, recent works have proposed introducing changes to the communication protocol @xcite , augmenting the communication with agent beliefs @xcite , and adding specific modules for reasoning and grounding @xcite . while these approaches often yield improvements in the overall task performance, they do not prevent cases where one rogue agent introduces noise to the communication and drags the system into failure (d' @xcite . in this work, we propose to improve multi-agent collaboration through live monitoring and interventions in agent communication. specifically, we hypothesize that many failures can be detected and prevented early on -before the agent becomes non-functional and the task fails -based on the state of the agent and how it observes the environment (see fig. 1 for an illustration). this approach is widely used in industrial systems, such as intrusion detection software and critical manufacturing processes @xcite @xcite @xcite , and is also observed in natural systems such as the human immune system, which monitors faulty behavior and prevents infections from spreading throughout the body. to test this hypothesis, we present a monitor and intervention framework ( §2). to monitor an agent's state, we employ simple classifiers that detect rogue agents based on intrinsic signals of uncertainty, such as the prediction's entropy during action selection. this monitor then triggers an intervention mechanism, which prevents the communication from being distorted and the agents from failing at the task. specifically, we experiment with simple interventions that roll back certain parts of the communication or the entire environment. to evaluate our approach, we create a modular collaborative environment, called whodunitenv, which allows analyzing different agent communication protocols at varying task-complexity levels ( §3). whodunitenv is inspired by the well-known game of guess who, foot_0 where players (here agents) collaborate to identify a culprit among a group of suspects based on their appearance properties. we conduct experiments ( §4) on two variants of whodunitenv, a code generation environment codegen, and the govsim resource sharing environment @xcite . these environments cover symmetric and asymmetric agents, structured and free-form communication protocols, and varying levels of task complexities. our results ( §5) show that, across all settings, monitoring rogue agents leads to substantial improvements in multi-agent collaboration. specifically, we observe gains of up to 20%. finally, we conduct ablations and qualitative analysis ( §6), showing that strong monitors and interventions are needed to improve performance, and that our monitors effectively identify a range of rogue agent behaviors, including hallucination of information in the communication channel. to summarize, our work makes the following contributions: (a) we propose the notion of live monitoring and interventions to prevent failures due to rogue agents in multi-agent systems, (b) we introduce whodunitenv, a modular environment for studying collaborative multi-agent systems where agents are either symmetrical/asymmetrical, (c) we show that our approach leads to substantial performance gains on whodunitenv, codegen, and govsim across models and complexity levels, and (d) we show that improvements are due to strong monitors and interventions. ## monitoring and intervening in multi-agent systems in multi-agent systems, agents collaborate in order to solve a task or use shared resources. the system comprises of agents @xmath0, a communication channel c that stores messages from the agents, and a shared task t . agents possess knowledge k i and perform actions a i . for example, agents that simulate human behavior have different memories based on their experiences @xcite , and different personas in a software development team have different actions, such as designing the code, programming, or writing tests @xcite . let c j be the communication channel at the j-th turn, an agent g i chooses their next action using a probability distribution over actions conditioned on their knowledge and the shared information: similarly, agents share information in the channel with a distribution over knowledge pieces: we propose to perform live, mid-run interventions to prevent single agents from causing a system-wide failure (fig. 1 ). our approach consists of monitoring agent action predictions to detect rogue agents, and intervening in the environment when a rogue agent is detected. we view monitoring as a function that estimates the probability of succeeding at the task at every turn, based on the agent's probability distribution over actions. namely, given p a i for agent g i at turn j, we wish to estimate p (success if a future failure is likely, namely, p (success | p a i , j, g i , t ) < τ for some threshold τ , we intervene to provide agents with an opportunity to reach a better state. an intervention is a causal operation that modifies the current state of the environment based on its current state and the monitoring output. for example, the intervention could revert the communication or augment it with additional content. ## whodunitenv: an environment for multi-agent collaboration whodunitenv is a modular multi-agent environment, where agents act as detectives working together to point out a culprit out of a suspect lineup. a game is comprised of n suspects, each with a unique set of attribute-value pairs that are randomly assigned from a predefined set. attributes include clothing (e.g., a green shirt), accessories (e.g., a silver watch), and personality traits (e.g., mood). one suspect is randomly chosen as the culprit. each agent receives partial information k i about the suspects or the culprit, and must collaborate to find the culprit and accuse them. turns move in a round robin fashion, and the game ends once either an agent accuses a suspect or a turn limit is reached. actions are tuples (a, t), consisting of a prime action @xmath1. we provide two variants of the environment, asymmetric and symmetric, that differ by the action set a and information available to each agent k i . see §f for prompts and §g for an example. whodunitenv-asym (fig. 2 ) this variant consists of exactly two agents -accuser and intel. k accuser contains the exact description of the culprit, but does not contain any information about the suspects. k intel is the complete description of all suspects, without any indication of the culprit. the set of actions available to accuser is a @xmath2, request-broad, accuse}, which allows it to request information about a specific attribute of a suspect, request broad information from intel for no specific suspect or attribute, and accuse a suspect, respectively. the set of actions available to intel is a @xmath3, respond-broad} which allows it to respond to accuser's request with a yes/no answer or return a broad message. when returning a broad message, the agent decides on an specific attribute value, such as "green hat", then lists all the suspects that have this property. thus, ## experiments we conduct experiments on whodunitenv and two other environments. for each environment, we use train and validation sets to train our monitors and a separate test set to evaluate them, as detailed below. ## results we find that monitoring and interventions consistently improve system performance across all environments and models, leading to substantial gains of up to 17.4% in whodunitenv, up to 20.0% in govsim and up to 2.5% in codegen. whodunitenv fig. 4 and 5 present the results for whodunitenv-asym and whodunitenv-sym, respectively. for whodunitenv-asym our method outperforms the base model by 6.1%, 10.6%, 10.3% for gpt-4o, llama-3.1-70b, and qwen-2.5-72b respectively, and by 4.1%, 7.0%, 4.5% for whodunitenv-sym. when one reset is allowed, our method outperforms the best baseline on average across models and the variants by 6.3%. for whodunitenv-asym, more resets yield additional gains (11.8% vs 9.0% with one reset across models on average). interestingly, the open-weight qwen-2.5-72b and llama-3.1-70b perform similarly to gpt-4o with two resets. however, a second reset did not further boost performance on whodunitenv-sym. we hypothesize that this is due to the structured form of communication in whodunitenv-sym, which would result in less communication-based mistakes. our interventions come at a cost of additional turns. the average number of turns increases by a factor of 1.9 and 1.6 with double and single reset for whodunitenv-asym and 1.6 and 1.4 for whodunitenv-sym. detailed results per variant and model can be found at §d.4. codegen tab. 1 presents the results for llama-3.1-70b on humaneval and livecodebench, obtained with the monitors trained on humaneval. our method reaches a pass@1 score of 83.5%, outperforming zero-shot prompting at 80.5% and a multi-agent network without monitoring which reaches 81.6%. applying the same monitors to livecodebenchreaches a pass@1 score of 21.6%, compared to 18.2% by zero-shot prompting and 20.4% in the multi agent system without monitoring. this shows that not only is our method effective for realistic tasks, but also that our monitors generalize well between similar tasks. govsim tab. 2 presents the results for govsim. with qwen-1.5-110b, our method leads to an increase of 20.0% in survival-rate and 1.2 steps in survival-time. efficiency remains similar, with 49.4% vs 48.8%. for gpt-4o, we increase efficiency by 6.9% on average, while maintaining a maximal survival-rate. overall, our method leads to better collaboration and performance on the task. ## ablations and analysis we conduct additional analysis on whodunitenv-asym, demonstrating that our monitors capture meaningful signals of agent uncertainty and the effectiveness of our interventions. ## related work multi-agent communication communication between agents have long been suggested in order to develop strong modular systems @xcite @xcite @xcite @xcite . one of the most exciting applications of language agents is environments where agents autonomously communicate, with examples of improving factuality and reasoning via agent debate @xcite , cooperation between embodied robots @xcite , simulating software development teams @xcite @xcite , and gaming environments @xcite . thus, several frameworks have been proposed to enable simple development of multi-agent environments @xcite @xcite . we contribute to this line of work by introducing whodunitenv, a modular environment for multi-agent communication, which allows easy configuration of difficulty levels and communication structures. ## conclusion we present live monitoring and mid-run interventions for multi-agent systems. we demonstrate that monitors based on simple statistical measures can effectively predict future agent failures, and these failures can be prevented by restarting the communication channel. experiments across multiple environments and models show consistent gains of up to 17.4%-20% in system performance, with an addition in inference-time compute. our work also introduces whodunitenv, a new environment for studying multi-agent cooperation.