Learning to execute instructions in a M inecraft dialogue (2020)

Abstract: The Minecraft Collaborative Building Task is a two-player game in which an Architect (A) instructs a Builder (B) to construct a target structure in a simulated Blocks World Environment. We define the subtask of predicting correct action sequences (block placements and removals) in a given game context, and show that capturing B’s past actions as well as B’s perspective leads to a significant improvement in performance on this challenging language understanding problem.

## introduction there is a long-standing interest in building interactive agents that can communicate with humans about and operate within the physical world (e.g. @xcite ). the goal for agents in this scenario is to not only be able to engage in rich natural language discourse with their human conversation partners, but also to ground that discourse to physical objects, and execute instructions in the real world. traditional dialogue scenarios are either completely ungrounded @xcite , focus on slot-value filling tasks @xcite which instead require grounding to entities in a knowledge base, or operate within static environments, such as images @xcite or videos @xcite . relevant efforts in robotics have largely focused on single-shot instruction following, and are mostly constrained to simple language @xcite with limited resources @xcite @xcite . the recently introduced minecraft collaborative building task and the corresponding minecraft dialogue corpus @xcite is one attempt to bridge this gap within the simulated game world of minecraft. in this task, two players, an architect (a) instructs a builder (b) to con-struct a target structure out of multi-colored building blocks. the corpus consists of 509 game logs between humans that perform this task. narayan- @xcite focus on generating architect utterances. in this paper, we explore models for building an automated builder agent. 1 we focus on the subtask of predicting the builder's block placements, and leave the back-and-forth dialogue aspect of the overall task required of a fully interactive builder agent to future work. we define the builder action prediction (bap) task in section 2, describe our models in section 3, an approach to augment the training data in section 4, and our experiments in section 5. we analyze results and highlight challenges of the bap task in section 6. ## data augmentation the small size of the training set (3,709 examples) is a major limiting factor for training complex models. here, we explore ways of generating synthetic data to augment the size and variety of our data. for each game log in the original training data, we generate twenty new game logs by combining the following data augmentation techniques: utterance paraphrases: we generate paraphrases of the utterances in the dialogue by randomly substituting tokens with any of their synonyms in the hand-engineered synonym lexicon of narayan- @xcite . color substitutions: we permute block colors by applying one of the 6! possible permutations, chosen at random, to the entire game log. these substitutions also change the language in the synthetic dialogues to reflect the updated colors. spatial transformations: since the world contains no landmarks besides the built region, absolute coordinates are somewhat arbitrary. we sample one (0, 90, -90, 180) rotation in the ground plane (affecting all x, z coordinates, plus b's yaw and position) per synthetic log (subject to the constraint that the target still fit in the built region). ## experiments we evaluate our world state encoders, game history and data augmentation schemes. experimental setup our training, test and development splits contain 3709, 1616, and 1331 builder action sequences respectively. we increase the training data to 7418 (2x), 14836 (4x) and 22254 (6x) items by sampling items from the synthetic data of section 4. the average sequence length (in the development set) is 4.3 (with a std. deviation of 4.5). target structures in the test data do not appear in the training or development data. we train models with adamw @xcite and weight decay regularization with a weight decay factor of 0.1. we use a learning rate of 0.001 for the original data and a slightly lower learning rate of 0.0001 in the case of augmented data. we use a batch size of 1. during training, we use teacher forcing and minimize the sum of the cross entropy losses between each predicted and ground truth action sequence (the action sequence performed by the human). we stop training early when loss on the held-out development set has increased monotonically for ten epochs. we use greedy decoding (max. sequence length of 10) to generate action sequences, which seems to work better than beam search decoding (for fixed beam sizes between 5 and 20). we report net action f1 (section 2.3) on the test set. ## conclusion and future work in the minecraft collaborative building task, builders must be able to comprehend complex instructions in order to achieve their primary goal of building 3d structures. to this end, we define the challenging subtask of builder action prediction, tasking models with generating appropriate action sequences learned from the actions of human builders. our models process the game history along with a 3d representation of the evolving world to predict actions in a sequence-to-sequence fashion. we show that these models, especially when conditioned on a suitable amount of game history and trained on larger amounts of synthetically generated data, improve over naive baselines. in the future, richer representations of the dialogue history (e.g. by using bert @xcite or of past builder actions) combined with de-noising of the human data and perhaps more exhaustive data augmentation should produce better output sequences. for true interactivity, the builder must be augmented with the capability to determine when and how to respond when it is too uncertain to act. and, finally, an approach like the speaker-follower models of