M ind A gent: Emergent Gaming Interaction (2024)

Abstract: Large Foundation Models (LFMs) can perform complex scheduling in a multi-agent system and can coordinate agents to complete sophisticated tasks that require extensive collaboration.However, despite the introduction of numerous gaming frameworks, the community lacks adequate benchmarks that support the implementation of a general multi-agent infrastructure encompassing collaboration between LFMs and human-NPCs. We propose a novel infrastructure—Mindagent—for evaluating planning and coordination capabilities in the context of gaming interaction. In particular, our infrastructure leverages an existing gaming framework to (i) act as the coordinator for a multi-agent system, (ii) collaborate with human players via instructions, and (iii) enable in-context learning based on few-shot prompting with feedback.Furthermore, we introduce “Cuisineworld”, a new gaming scenario and its related benchmark that supervises multiple agents playing the game simultaneously and measures multi-agent collaboration efficiency. We have conducted comprehensive evaluations with a new auto-metric Collaboration Score: CoS for assessing the collaboration efficiency. Finally, Mindagent can be deployed in real-world gaming scenarios in a customized VR version of Cuisineworld and adapted in the “Minecraft” domain. Our work involving LFMs within our new infrastructure for general-purpose scheduling and coordination can elucidate how such skills may be obtained by learning from large language corpora.

## introduction large foundation models (lfms) have been driving the effort to develop general intelligent machines @xcite . although they are trained using large text corpora, their superior problem-solving capacity is not limited to canonical language processing domains. lfms can potentially tackle complex tasks that were previously presumed exclusive to human experts or domain-specific algorithms. recent research has shown the possibility of using lfms to generate complex plans for robots and game ai @xcite @xcite , marking an important milestone for lfms as general-purpose intelligent agents @xcite . in this paper, we investigate the planning capacity of lfms in the context of multi-agent systems @xcite . compared to planning for a single agent, which has been studied extensively @xcite , multi-agent planning imposes much higher problem-solving complexity due to an action space that grows exponentially with respect to the number of agents. the planner must simultaneously control multiple agents, avoid possible conflicts, and coordinate agents into achieving a shared goal that requires potentially sophisticated collaboration. to understand to what extent lfms can acquire multi-agent planning skills, we first develop a new benchmark, cuisineworld, which is illustrated in figure 1 . to incorporate agent ais into video games, we design mindagent, an infrastructure inspired by multi-agent task allocation optimization theories, to facilitate the multi-agent planning capabilities of lfms. our infrastructure enables lfms to perform complex coordination and scheduling of multiple agents in order to achieve task completion. we conduct comprehensive evaluations with recently introduced lfms, including gpt-4, claude, and llama, playing our cuisineworld game within our mindagent interactive multi-agent planning framework, leading to the following key observations: our experimental results validate its potential in helping develop generalist multi-agent planners. ## the cuisineworld game we introduce cuisineworld as a novel and flexible game for multi-agent scheduling and coordination in a virtual kitchen environment. in this game, a multi-agent system must supervise multiple agents and coordinate them, with the goal of completing as many dish orders as possible. the game is equipped with a textual interface since our focus is on evaluating lfm-based planning agents. our modularized design separates tasks and game engines, allowing inclusion of more tasks (dish types) and domains ("kitchen" implementation via textbased engine, unity, minecraft, etc.). ## mindagent gaming ai infrastructure our first foray into the challenging cuisineworld benchmark is an interactive multi-agent planning framework with lfms. it facilitates in-context learning and adopts a minimalist design for the purposes of demonstrating the scheduling and coordination capacity of lfms, while also bringing in exploratory prompting techniques that facilitate better planning and inform future approaches in this domain. our mindagent infrastructure comprises prompt, current state, and memory, as shown in figure 2 with details illustrated as follows: prompt incorporates four distinct sub-components: recipes, general instructions, inference knowledge, and a one-shot demo. recipes outline hierarchical procedures for preparing various dishes at a given level. they specify the ingredients necessary for each intermediate or final product, the appropriate tools, and the outcome. instructions detail the foundational rules of cuisineworld, delineating the array of actions agents can undertake within the game and enumerating the characteristics of every tool available. moreover, they inform agents about the base ingredients retrievable from storage, as well as all potential intermediate products they can procure. agents are also explicitly advised to remain cautious about feedback from the environment. inference knowledge encapsulates insights and helpful hints for the agent, which when utilized appropriately can guide agents to sidestep potential errors and improve their collaborative efficiency. one-shot demo presents a step-by-step demonstration of the preparation of a distinct dish, different from other dishes at the current level, spanning several time steps, each of which is incorporated as part of the prompt. the demonstration illustrates the major procedures for cooking a dish in cuisineworld, including obtaining ingredients, putting ingredients into different tools, transporting intermediate ingredients, and delivering the final dish to the serving table. more details of prompt are in appendix a. current state provides a snapshot of the prevailing observations from the environment. it encompasses information such as the locations of agents, the objects currently in the possession of agents, the tools that are accessible within the environment, the ingredients present within each tool, and the tools that are actively in use. moreover, it includes optional feedback from the environment, triggered when agent actions violate the rules of the environment; for instance, when assigning two distinct actions to the same agent. memory archives the history of interaction with the environment. specifically, it chronicles the state of the environment and the state of the agents at every time step. in addition to the prompt modules, other modules are implemented to help interface between lfms and cuisineworld: action extraction employs a regular expression matching procedure to distill agent actions from the textual output of the lfms. this module is indispensable because lfm output is not always clean, but may include information reflecting its internal thought processes or even issue apologies for prior missteps in reaction to environmental feedback. action validation utilizes a look-ahead checking mechanism. this module parses the proposed actions, assessing their feasibility. if an action is deemed unexecutable, an error message is returned. ## experiments and results we have conducted extensive experiments in cuisineworld. we first introduce the experiment settings and then present an analysis of our empirical results. we report lfm settings in appendix c. our experiments focused on addressing the following research questions: q1: how efficiently can the model dispatch multiple agents? q2: can the model dispatch agents for dynamic, on-the-fly goals across different tasks? q3: to what extent can the existing methods collaborate with human users? q4 what is the human perception of collaborating with numerous intelligent agents? q5: how do various components of the input prompt influence the model's performance? q6: how do other lfms perform compared to gpt-4? ## emergent abilities across our experiments, our mindagent framework exhibits the following emergent properties: ## novel game adaptation in line with our ongoing efforts to create collaborative, in-game, multi-agent systems, we integrated our infrastructure into minecraft (figure 5 ). in this adaptation, we designed several unique cooking tasks where two in-game agents, alex and steve, must cook various types of meat as shown in appendix f.2. after cooking, they must deposit the meats into a chest. see ## conclusion and prospects we introduced mindagent, an infrastructure for multi-agent collaboration through lfms across multiple gaming domains. we investigated its multi-agent planning capabilities, and we deployed our infrastructure into real-world video games that demonstrate its multi-agent and human-ai collaboration effectiveness. additionally, we presented cuisineworld, a text-based multi-agent collaboration benchmark that provides a new auto-metric collaboration score (cos) to quantify collaboration efficiency. we anticipate that our work will guide the development of future gaming systems in which human-ai collaboration is seamless and intuitive. we are optimistic that our insights and findings will catalyze the design of games that are both technologically advanced and significantly more engaging and enjoyable for players.