CB 2: Collaborative Natural Language Interaction Research Platform (2023)

Abstract: CB2 is a multi-agent platform to study collaborative natural language interaction in a grounded task-oriented scenario. It includes a 3D game environment, a backend server designed to serve trained models to human agents, and various tools and processes to enable scalable studies. We deploy CB2 at https://cb2.ai as a system demonstration with a learned instruction following model.

## introduction collaborative grounded natural language interactions involve multiple agents, either human or machine, working together to complete tasks while coordinating using natural language. a key obstacle in studying such scenarios is building the research interaction platform, a significant design and engineering undertaking. this requires building and designing the interaction environment, the task the agents collaborate on, an interface for both machine learning models and human agents, and a process to onboard human agents. each aspect dramatically influences the interaction and language elicited, and is critical to get right. we introduce cb2, a platform for the study of collaborative grounded natural language interaction, and demonstrate its use through the deployment of a learned collaborative natural language agent. cb2 largely instantiates the cerealbar scenario @xcite ), foot_0 but is implemented from scratch to emphasize research accessibility. cb2 is a customizable, scalable, and complete research platform, including server and clients for multi-agent human-machine interactions, tools for real-time data management, and processes to onboard crowdsourcing workers. the cb2 scenario poses learning and reasoning challenges, as well as opportunities. comprehending and producing instructions in cb2 requires addressing the symbol grounding problem @xcite , which is studied extensively in the instruction following (e.g., @xcite @xcite and generation (e.g., @xcite literature. however, the collaborative scenario remains relatively understudied. collaboration is not simply an added complication, but dramatically alters both interaction and learning through joint presence and action. it allows the instructor to ad-hoc modify the tasks they delegate based on the follower behavior, potentially recovering from system failures. at the same time, this adaptation creates constant distribution shift, a significant generalization challenge. learning is also drastically transformed through collaboration. the constant engagement of other agents (including humans), the ability to modify delegation strategies, and the shared task-based incentives bring about within-interaction signals that can be used for continual learning, reducing the dependency on annotated data and enabling model adaptation. we deploy a demonstration of cb2 with a learned baseline instruction following agent (section 7). players can connect to cb2 and collaborate with our agent or other human agents at https://cb2.ai/ . 2 the cb2 platform is available at https://github.com/lil-lab/cb2 . a video demonstration of cb2 is available at https: //youtu.be/talpx_kkmiw . ## related work cb2 is a re-implementation and extension of ce-realbar, a scalable platform to study natural language instruction collaboration @xcite . cerealbar was used to study instruction following @xcite , instruction generation @xcite , and linguistic change @xcite . cb2 is related to instruction following environments, such as sail @xcite , r2r @xcite , rxr @xcite , and alfred @xcite . in contrast, cb2 is focused on embodied multi-agent collaborations, including with human agents. symbol grounding @xcite , a core challenge in cb2, was studied extensively in the singleagent context of instruction following (e.g., @xcite @xcite and generation (e.g., @xcite @xcite . the cb2 scenario emphasizes multi-agent collaboration, an aspect that is significantly less studied with natural language instruction. the cards corpus @xcite presents a related scenario, which has been used for linguistic analysis. a related problem is studied by the emergent communication literature @xcite @xcite , but with less focus on collaboration with human agents. natural language collaboration between agents with asymmetric capabilities has also been studied with minecraft-based scenarios @xcite @xcite . cb2 differs from these in allowing both agents to effect changes on the environment, enabling ad-hoc modification and delegation of tasks. ## interaction scenario cb2 largely implements the interaction scenario introduced by @xcite in the cereal-bar environment with several modifications. the interaction takes place in a procedurally generated spatial environment and includes two agents that collaborate together to complete card collection tasks and coordinate using natural language. figure 1a shows an instance of the environment. the environment is a procedurally generated 3d map made of a grid of hexagons (figure 1a ). it includes lakes, mountains (figure 1c ), paths, open spaces, and landmarks. a new environment is generated for each game. cb2 includes improved visuals and generation compared to cerealbar. for example, cb2 map generation includes semantic biases: houses are generated to cluster together and form towns (figure 1b ) and paths are generated to connect between meaningful areas in the map, such as towns and mountains. landmark instances vary visually to elicit richer language. for example, houses are generated with different roof colors and number of floors (figure 1b ). the environment also includes randomly placed cards (figure 1d ). each card shows 1-3 copies of one of a few possible shapes in one of a few possible colors. the interaction involves two agents, a leader (figure 1f ) and a follower (figure 1g ), that collaborate together to complete tasks, but differ in their observations of the environments and abilities. both the leader and the follower move in the environment, by moving between neighboring hexagons or by turning in place to change orientation. the agents select and deselect cards by moving over them (figure 1e ). the goal of the agents is to select valid sets of cards. a valid set includes three cards, where each color, shape, and count are unique (figure 1i ). the agents select sets together. when the currently selected cards form a valid set, they disappear, the agents together receive one point, three new randomly selected cards appear in random positions, and the agents receive additional turns. the number of turns added diminishes with each set completion. asymmetries between the two agents make collaboration critical for success. the leader sees a complete overhead view of the environment (figure 1a ), while the follower only sees what is ahead from a first-person view (figure 1h ). cb2 introduces two optional observability features not present in cerealbar. first, the patterns on unselected cards may be hidden from the follower, instead displaying a quesiton mark on all cards. second, cb2 allows to control how far the follower sees ahead of them with a fog that is present only in the follower view. the observability gap means the leader is in charge of planning how the agents operate. if the follower acts independently of the leader plans, the interaction will be suboptimal, because follower actions are likely to conflict with leader actions and the partial view of the environment does not allow for optimal planning of goals and movement. the agents move in turns, with a limited number of steps per turn. each movement (forward, left, right, or backward) consumes a single step. turns are time limited to keep the interaction moving and avoid long wait periods for the inactive agent. the exact time budget is customizable, but we generally provide significantly more time for the leader turns, so they can plan as needed. turns alternate between the follower and leader. the follower has significantly more steps than the leader per turn. this means the follower is able to move further in each turn, and potentially accomplish much more in each turn. this ability gap makes it critical for the leader to collaborate with the follower, rather than ignore the follower and attempt to accomplish tasks on their own, a suboptimal strategy. the agents coordinate via uni-directional natural language instruction, the only form of coordination available. during a leader turn, in addition to moving in the environment, the leader can send text instructions to the follower. the follower executes the leader instructions and indicates when an instruction is complete. the leader can queue multiple instructions, but the follower only sees past instructions and the one they are currently executing. because the follower does not see future instructions, alignment between the actions recorded and the instruction displayed is guaranteed. the leader can also cancel the instruction the follower is executing alongside all future instructions in the queue during the follower turn. this is intended to halt very bad executions, and reduce their overall cost, for example by having to correct drastic departures from the leader plan. instruction writing and sending by the leader, and marking them as complete by the follower do not consume steps. leaders may write as many instructions as they wish during a single turn, and followers are not taxed if the tasks are given in multiple instructions that they need to mark as complete. exempting the language channel from the budget of actions per turn aims to reduce the influence of the turn systems on the language produced. the combination of collaboration incentives (i.e., because of the capability differences between the agents) and the exclusivity of the language channel for communication makes effective natural language instruction essential for successful interactions. 3 ## framework implementation the cb2 framework has three main components: a python server, a unity client, and a python headless client. the game logic is orchestrated from the server, allowing to customize the interaction without modifying unity code. the python client simplifies the interaction between learning processes and the system, for example during reinforcement learning. figure 2 visualizes the architecture. cb2's design emphasizes customizability, as much as possible, without modifying unity code, a skill that is less common among researchers. this motivates placing the game logic on the python 3 depending on the environment configuration, it is possible for one of the agents to operate alone if the cards forming a set are really close and the other agents does not move. this can allow 1-2 set completions. a higher score without collaboration via language coordination is extremely unlikely. server, a decision that dictates the client-server communication design. however, modifications that require updating the client user interface, such as adding bi-directional communication or translating the ui to other languages, do require modifying unity coding. ## example task formulations cb2 is well suited to study a variety of tasks, with emphasis on learning and evaluation in collaborative interactions with human agents, such as: instruction following the task of instruction following is to map a start state observation from the follower perspective and a leader instruction to a sequence of actions. after each action, the agent receives a new observation. @xcite studied this problem with cerealbar by learning from recorded human-human interactions, and @xcite studied it within a continual learning from human feedback scenario. both approaches were evaluated by deploying follower agents to interact with human leaders. ## crowdsourcing process cb2 poses several relatively demanding crowdsourcing tasks. human-human interactions require pairing two workers for real-time play over extended time. we design a process to collect cb2 interactions via crowdsourcing, either for games where both roles are controlled by human players, or where one of the sides is controlled by a learned model. the task-focused design of cb2 naturally allows an effective incentive structure by tying game performance with compensation. the key to our process is gradual training of workers. a new worker first starts with a tutorial and a qualifier quiz that covers the relatively simple role of the follower. the follower role requires following the leader instructions by controlling the character in the game. the worker is then qualified to the follower role only, and is paired by joining a dedicated follower-only queue in the lobby. focusing on the follower role only simplifies the learning curve, and much of the learning required for the leader role takes place on the job, as the worker collaborates with more experienced leaders. once the worker displays sufficient level of performance for several games, they are invited to qualify as a leader by taking a leader tutorial and a quiz. the second tutorial is both longer and more complex than the follower tutorial, and includes both planning and instruction writing. once the worker completes the tutorial and passes the quiz, they are qualified to the leader role, and can then participate in tasks as both leader or follower. we design the lobby to pair workers based on experience. because the leader role is significantly more critical to the effectiveness of the interaction and the quality of language data, we prioritize workers with better performance for it. we measure worker performance, keeping track of the mean game score in the most recent games. if two leader-qualified players are waiting in the lobby for matching, we will assign the leader role to the higher performing of the two. the pay structure includes a base pay for connecting to the game, and an increasing bonus for each point. both workers, the leader and follower, get the base pay and the additional bonus per point, tightly connecting compensation to their collaborative performance. because the leader role is more complex, we provide an additional relative bonus to the worker in the leader role. ## cb2 demonstration deployment we demonstrate the functionality and potential of cb2 via deployment, including collecting a corpus of human-human interaction that we release, training a follower baseline model, and evaluating it in interaction with human leaders. human games data we follow the crowdsourcing process outlined in section 6 to collect games between human leaders and followers. we collect 185 games containing 3,439 instructions. ## conclusion cb2 is a multi-agent research platform to study natural language instruction in collaborative, embodied environments. a core objective of cb2 is to enable scaleable studies where human agents interact with learned models, potentially over long periods of time. cb2 is designed to be easy to use and customize, with emphasis on accessibility for researchers with limited game development experience. it is designed from the ground up for machine learning, and includes a headless fast python client api to support learning processes and to deploy learned models to interact with human users.