R e D el: A Toolkit for LLM -Powered Recursive Multi-Agent Systems (2024)

Abstract: Recently, there has been increasing interest in using Large Language Models (LLMs) to construct complex multi-agent systems to perform tasks such as compiling literature reviews, drafting consumer reports, and planning vacations. Many tools and libraries exist for helping create such systems, however none support *recursive* multi-agent systems—where the models themselves flexibly decide when to delegate tasks and how to organize their delegation structure. In this work, we introduce ReDel: a toolkit for recursive multi-agent systems that supports custom tool-use, delegation schemes, event-based logging, and interactive replay in an easy-to-use web interface. We show that, using ReDel, we are able to achieve significant performance gains on agentic benchmarks and easily identify potential areas of improvements through the visualization and debugging tools. Our code, documentation, and PyPI package are open-source at https://github.com/zhudotexe/redel, and free to use under the MIT license.

## introduction a multi-agent system uses multiple large language models (llms) together to accomplish complex tasks or answer complex questions beyond the capabilities of a single llm. often, in such scenarios, each llm is provided with tools @xcite that it can use to give it additional capabilities, like searching the internet for real-time data or interacting with a web browser. in most cases, these systems are defined manually, with a human responsible for defining a static problem-decomposition graph and defining an agent to handle each subproblem in the graph @xcite @xcite . in a recursive multi-agent system, rather than a human defining the layout of multiple agents, a single root agent is given a tool to spawn additional agents. when faced with a complex task, the i'll need to find flights, hotels, trains, and food. delegation graph find hotels in tokyo, osaka, ... root agent can decompose the task into smaller subtasks, then delegate those tasks to newly-created sub-agents. each sub-agent can then either complete the task if it is small enough, or recursively decompose and delegate the task further foot_1 @xcite @xcite (figure 1 ). in the current landscape of multi-agent systems, the majority of tooling focuses on human-defined static systems, and poorly handles dynamic systems where agents are added to a computation graph at runtime. furthermore, much of this tooling is unsuitable for academic purposes @xcite or hidden behind paywalls and proprietary licenses. in this paper, we present redel, a fully-featured open-source toolkit for recursive multi-agent systems. redel makes it easy to experiment by providing a modular interface for creating tools, different delegation methods, and logs for later analysis. this granular logging and a central event-driven system makes it easy to listen for signals from anywhere in a system, and every event is automatically logged for post-hoc data analysis. redel also features a web interface that allows users to interact with a configured system directly and view replays of saved runs, making it easy for researchers and developers to build, iterate on, and analyze recursive multi-agent systems. in section 4 we use re-del to run recursive multi-agent systems on three diverse agentic benchmarks, and in section 5 we demonstrate how the toolkit can be used to explore complex behaviours of these systems. ## related work recursive multi-agent systems. recent work on recursive multi-agent systems has been done by @xcite , @xcite , @xcite , and @xcite . these works introduce the method of fine-tuning or few-shot prompting llms to decompose complex tasks and using sub-agents to solve each part (often called recursive or hierarchical decomposition). redel builds upon the methods introduced in these works by taking advantage of modern models' native tool use capability @xcite to decompose and delegate tasks zero-shot (i.e., without humanwritten examples in prompt) instead of using fewshot prompting or fine-tuning. as a framework, we provide an extensible interface to apply these approaches to additional tasks and domains. other multi-agent system methods such as agent evolution @xcite @xcite perturb human-written prompts and tools to create new variations of sub-agents on the fly. in this paper, we choose to explore delegation using zero-shot prompting and function calling without on-the-fly adaptation, but our framework is flexible enough to implement these alternate methods of agent delegation as well. ## system design redel consists of two main parts: a python package to define recursive delegation systems, log events, and run experiments, and a web interface to quickly and interactively iterate on defined systems or analyze experiment logs. in the following sections, we discuss these components in more detail. ## evaluation & case study to evaluate redel, we compare its performance to a baseline single-agent system and to the published state-of-the-art system on three different benchmarks. we include the logs and source code for all experiments in our code release. ## using redel for error analysis for our error analysis, we took the saved log files for each benchmark and manually investigated the logs of both the successful runs as well as the failed runs through the replay view of the redel web interface. through this investigation we observed two common failure cases in recursive multi-agent systems. these cases are as follows: we find that overcommitment commonly occurs when an agent performs multiple tool calls and fills its context window with retrieved information. in the redel web interface, this manifests as an abnormally small delegation graph, often consisting of only two nodes: the root node, and a single child which the root delegates to and which subsequently overcommits. in practice, this often, but not always, results in the overcommitting model "forgetting" the task it was meant to accomplish due to the original task being truncated its limited context window. an overcommitting model might fail a task because it outputs a summary of whatever remains in its context window instead of the answer to the original task, whereas a task failure due to causes other than overcommittment might look like a hallucinated result or a simple apology for being unable to complete the task. in contrast, we find that undercommitment commonly happens when the model incorrectly decides that it does not have the necessary tools to solve the problem and instead assumes that its future child will possess the required tools to solve the problem. in all three benchmarks, this led to failure as agents entered an infinite loop of delegation until they reached a configured depth limit or timed out. in the web interface, this manifests as a line of nodes in the delegation graph (figure 6 ). in given the prevalence of these two issues, we hypothesize that recursive multi-agent systems may still see further improvements to performance from interventions that target these behaviors. for example, one could fine-tune or prompt agents with domain-specific instructions that detail when the models should delegate and when they should perform tasks on their own. while implementing such improvements is beyond the scope of this paper, we believe that this case study helps to demonstrate the strengths of the redel system. using the delegation graph view, it is easy to identify and characterize errors in recursive multi-agent systems and we hope that through redel more research can be done to further refine such systems for maximum utility. ## conclusion we present redel, a novel toolkit for working with recursive multi-agent systems. redel allows academic developers to quickly build, iterate on, and run experiments involving dynamic multi-agent systems. it offers a modular interface to create tools for agents to use, an event framework to instrument experiments for later analysis, and a free and open-source web interface to interact with and explore developer-defined systems. we use re-del to demonstrate recursive multi-agent systems' performance on three diverse benchmarks, and we include the full logs of these runs in our demo release for reproducibility and further exploration foot_3 . redel opens the door for a new paradigm of recursive multi-agent systems, and we are excited to see how developers can utilize our system in the future.