V illager A gent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in M inecraft (2024)

Abstract: In this paper, we aim to evaluate multi-agent systems against complex dependencies, including spatial, causal, and temporal constraints. First, we construct a new benchmark, named VillagerBench, within the Minecraft environment. VillagerBench comprises diverse tasks crafted to test various aspects of multi-agent collaboration, from workload distribution to dynamic adaptation and synchronized task execution. Second, we introduce a Directed Acyclic Graph Multi-Agent Framework (VillagerAgent) to resolve complex inter-agent dependencies and enhance collaborative efficiency. This solution incorporates a task decomposer that creates a directed acyclic graph (DAG) for structured task management, an agent controller for task distribution, and a state manager for tracking environmental and agent data.Our empirical evaluation on VillagerBench demonstrates that VillagerAgentoutperforms the existing AgentVerse model, reducing hallucinations and improving task decomposition efficacy. The results underscore VillagerAgent’s potential in advancing multi-agent collaboration, offering a scalable and generalizable solution in dynamic environments. Source code is open-source on GitHub.

## introduction multi-agent collaboration using llm is a challenging research topic that aims to enable multiple autonomous agents to coordinate their actions and achieve a common goal @xcite @xcite @xcite . the collaboration process requires communication, planning, and reasoning among multiple intelligent agents. it has many applications in domains such as robotics, gaming @xcite , and social simulation @xcite . there are increasing interests in developing multiagent systems using llms. mindagent introduces 1 https://github.com/cnsdqd-dyb/villageragent the cuisineworld gaming scenario as a benchmark, utilizing the collaboration score (cos) to measure the efficiency of collaboration @xcite . agentverse organizes its framework into four essential stages: expert recruitment, collaborative decision-making, action execution, and evaluation, thereby effectively deploying multi-agent groups that outperform a single agent @xcite . metagpt, on the other hand, employs an assembly line approach, designating specific roles to agents and efficiently breaking down complex tasks into subtasks involving many agents working together @xcite . however, these multiagent collaboration models either tend to restrict agents to parallel-executable subtasks each round, even when unnecessary or bind them to a fixed pipeline and task stage, overlooking complex task dependencies. this may cause issues for tasks that need both sequential and parallel execution, thus limiting model generality and scalability @xcite @xcite . in this paper, we focus on multi-agent collaboration for problem-solving with complex dependencies. these dependencies can be of different types, such as spatial dependencies that constrain the locations of the sub-tasks, causal dependencies that affect the availability and effects of the sub-tasks, and temporal dependencies that impose constraints on the timing of the sub-tasks. it is crucial to understand and manage these dependencies for effective multi-agent collaboration, enabling the agents to reason about the long-term consequences of their actions and avoid potential conflicts. first, we introduce villagerbench, a new multiagent benchmark in the minecraft environment designed for the evaluation of complex dependencies (figure 7 ). some of the multi-agent research is being tested within the overcooked-ai @xcite . nevertheless, due to limitations in the number of agents, scenario flexibility, and task diversity, there is a desire for more comprehensive frameworks to test multi-agent cooperation. inspired by voyager @xcite , gitm @xcite , and mindagent @xcite , we construct a multi-agent and multi-task evaluation framework with greater degrees of freedom using minecraft. minecraft offers a rich and diverse set of tasks that can be used to benchmark and evaluate multi-agent systems, such as building and farming. it allows players to explore dynamic environments that pose various challenges for multi-agent collaboration, such as resource allocation, task decomposition, and coordination. specifically, we introduce three tasks, i.e., construction cooperation, farm-to-table cooking and escape room challenge. the construction cooperation task tests agents' aptitude for understanding task requirements and orchestrating team workload, focusing on the evaluation of spatial dependencies in multiagent collaboration. the farm-to-table cooking task assesses their agility in adapting to fluctuating environmental conditions, aiming to solve complex causal dependencies. the escape room challenge task tests agents on their ability to execute tasks both sequentially and in parallel, requiring the reasoning of temporal dependencies and the ability to synchronize actions. second, we introduce a directed acyclic graph multi-agent framework (villageragent) to tackle complex dependencies in multi-agent collaborations. each subtask is represented as a graph node in the dag. we dynamically adjust the graph structure and the agent roles according to the environment and the agent states. villageragent consists of task decomposer, agent controller, state manager and base agents. the task decomposer generate a directed acyclic graph (dag) of subtask nodes each round, while the agent controller oversees the assignment of these subtasks to the base agents for execution and self-reflection. meanwhile, the state manager is responsible for maintaining the status information of both the environment and the agents. we quantitatively evaluate our method on villager-bench. we demonstrate the superior performance of villageragent over agentverse @xcite by fewer hallucinations and enhancing the effectiveness of task decomposition. ## villagerbench design our villagerbench uses mineflayer (prismarinejs, 2013) to establish agent apis, offering a platform to examine cooperative behaviors in multi-agent systems via tasks such as construction, cooking, and escape room challenges (figure 1 ). we evaluate multi-agent systems powered by llms using three key metrics: completion (c) that measures the average task completion rate; efficiency (e) that assesses the speed of task execution and the utilization of resources; and balance (b) that examines the distribution of workload among agents, with higher values indicating a more equitable assignment of tasks. further details can be found in appendix a. ## conclusion in this study, we introduce villagerbench, a minecraft multi-agent benchmark platform. we design three distinct scenarios within villagerbench to evaluate collaborative tasks, aiming to assess the performance of our villageragent framework. our framework employs directed acyclic graphs (dag) to decompose tasks, enabling efficient and coordinated execution by agents. we benchmark the coordination skills of three llms using these metrics and demonstrate the effectiveness of our villageragent framework. we also explore how agent count and capability diversity impact framework performance.