MC -Saar-Instruct: a Platform for M inecraft Instruction Giving Agents (2020)

Abstract: We present a comprehensive platform to run human-computer experiments where an agent instructs a human in Minecraft, a 3D blocksworld environment. This platform enables comparisons between different agents by matching users to agents. It performs extensive logging and takes care of all boilerplate, allowing to easily incorporate new agents to evaluate them. Our environment is prepared to evaluate any kind of instruction giving system, recording the interaction and all actions of the user. We provide example architects, a Wizard-of-Oz architect and set-up scripts to automatically download, build and start the platform.

## introduction collaborative human-computer interaction can occur in different environments. while interaction in the physical world is often a desirable goal, it places a huge burden on automatic agents as perception is a hard problem, raising the barrier of setting up such experiments significantly. on the other end, interactions on a custom-built platform may be a good fit to explore specific phenomena, but they do not scale easily to different or complex problems. a good example for a custom-built virtual 3d world is the give challenge, where an instruction system must guide a player to press a specific sequence of buttons in a 3d environment while avoiding to step into traps @xcite . we instead use a generalpurpose 3d environment. we release an experimentation platform based on minecraft (see figure 1 ). minecraft is a game in which the players are situated in a 3d world, which mainly consists of blocks. the game can either be played locally as a single-player game or one can join an online server and play with others. the players can move around, place and remove blocks, and even craft new blocks or items. as such, minecraft can be seen as a classic blocksworld that can be scaled up a lot in complexity: blocks can have different types (wood, earth, stone, glass, lamps, . . . ), they can be combined into high-level objects, and special blocks even enable building circuits, resulting in turing-complete machinery. minecraft contains different game modes: a survival mode, which focuses on exploration and survival in the game world, and the creative mode, focusing on building. we make use of the creative mode. this feature-richness makes minecraft a perfect environment for the evaluation of all kinds of intelligent agents @xcite , from reinforcement learning agents @xcite , to instruction receiving @xcite and instruction giving assistants (narayan-chen et al., 2019). its popularity (minecraft is the most sold game of all time), together with the client-server architecture make minecraft a tool well-suited for crowd-sourcing with volunteers from all over the world. moreover, there are tons of instruction videos for minecraft on the internet which could be used as auxiliary datasets for offline instruction giving. this addresses several of the limitations minecraft client (player 1) minecraft client (player n) minecraft server broker architect server architect server architect 1 architect k architect n database . . . . . . . . . world status instructions logging figure 2: overview of the services in mc-saar-instruct. updates on the world state are passed along the full lines, instructions are forwarded along the dashed lines. the platform presented here makes it easy to set up and run instruction giving experiments in minecraft. in our research, we focus on instructing the user to build complex objects @xcite , but our platform can easily be used for other generation tasks. while there is only one minecraft server and one broker, there can be several different kinds of architects, each hosted by its own architect server. all interactions between these components are handled using the grpc library, foot_0 abstracting away the low-level networking and providing a succinct and type-save remote procedure call (rpc) interface. we provide example architect servers in java, but they can be written in any language with grpc bindings, such as python, go, and many more. ## defining and running experiments an experiment is defined by two components: the scenarios that the players are supposed to work on and the architects that should be evaluated. a scenario consists of a definition of an initial state of the world and architect-specific information instructing the architect of the goal. the initial world state is given by a list of blocks with their location and type (see figure 4 ). each scenario is identified by a unique name. we use a shared dependency for all components that contains the necessary descriptions of the world state when starting a scenario as well as the scenario-specific data for the architects, ensuring that the architects and the minecraft server use the same initial setup. ## wizard-of-oz architect we also ship a wizard-of-oz architect (woz) to perform human-human interaction experiments. this architect runs in a second minecraft server where only one player may log in. that player can neither move nor place or destroy blocks. once this architect is paired with a player by the broker, the viewpoint of the woz player is synchronized with the player, i. e. the woz player always sees exactly what the player sees. the woz player may send text messages and these are forwarded as instructions in the same manner as those generated by an automatic agent. we conducted initial experiments with spoken interaction and noticed that the instruction givers used patterns only possible with spoken interaction such as exactly timing single words to the instruction follower's actions and self-correction. the text-based wizard-of-oz setup on the other hand mirrors the setup with an automatic architect as closely as possible. ## post-experiment questionnaires after finishing an experiment, the participants fill out a questionnaire using the in-game chat. once the architect determines that a game is over (hopefully in a successful way), the broker takes over the communication channel and asks the user a series of configurable questions. the questions and answers to this post-experiment questionnaire are logged to the database. the in-game questionnaire allows to keep all interaction with the experiment platform inside a single medium by removing the need to e. g. open a website. it also ensures that the questionnaires and experiment data are always correctly matched. finally, the questionnaire mechanism can be used for fraud prevention @xcite . ## conclusions we introduced a system for researching situated human-computer dialogue in the minecraft domain. while primarily focused on instruction giving, it can potentially also be used for two-way text interaction. the framework abstracts away from most of the low-level system, providing a clean and easy to use interface for implementing instruction givers. the system also takes care of matching study participants with different architects and logging of all interactions. we ship several example architects, including a wizard of oz architect. we plan to implement a replay viewer which streams the previously recorded actions by a participant to a minecraft server. all necessary data is already being stored in the database. mc-saar-instruct as well as scripts to automatically download, build and run specific versions of it for reproducible experiments are available from https://minecraft-saar.github.io .