C raft A ssist Instruction Parsing: Semantic Parsing for a Voxel-World Assistant (2020)

Abstract: We propose a semantic parsing dataset focused on instruction-driven communication with an agent in the game Minecraft. The dataset consists of 7K human utterances and their corresponding parses. Given proper world state, the parses can be interpreted and executed in game. We report the performance of baseline models, and analyze their successes and failures.

## introduction semantic parsing is used as a component for natural language understanding in human-robot interaction systems @xcite @xcite @xcite , and for virtual assistants @xcite @xcite . we would like to be able to apply deep learning methods in this space, as recently researchers have shown success with these methods for semantic parsing more generally, e.g. @xcite @xcite . however, to fully utilize powerful neural network approaches, it is necessary to have large numbers of training examples. in the space of human-robot (or human-assistant) interaction, the publicly available semantic parsing datasets are small. furthermore, it can be difficult to reproduce the end-to-end results (from utterance to action in the environment) because of the wide variety of robot setups and proprietary nature of personal assistants. in this work, we introduce a new semantic parsing dataset for human-bot interactions. our "robot" or "assistant" is embodied in the sandbox construc-tion game minecraft foot_0 , a popular multiplayer openworld voxel-based crafting game. we also provide the associated platform for executing the logical forms in game. situating the assistant in minecraft has several benefits for studying task oriented natural language understanding (nlu). compared to physical robots, minecraft allows less technical overhead irrelevant to nlu, such as difficulties with hardware and large scale data collection. on the other hand, our bot has all the basic in-game capabilities of a player, including movement and placing or removing voxels. thus minecraft preserves many of the nlu elements of physical robots, such as discussions of navigation and spatial object reference. working in minecraft may enable large scale human interaction because of its large player base, in the tens of millions. furthermore, although minecraft's simulation of physics is simplified, the task space is complex. while there are many atomic objects in the game, such as animals and block-types, that require no perceptual modeling, the player also interacts with complex structures made up of collections of voxels such as a "house" or a "hill". the assistant cannot apprehend them without a perceptual system, creating an ideal test bed for researchers interested in the interactions between perception and language. our contributions in the paper are as follows: grammar: we develop a grammar over a set of primitives that comprise a mid-level interface to minecraft for machine learning agents. data: we collect 7k crowd-sourced annotations of commands generated independent of our grammar. in addition to the natural language commands and the associated logical forms, we release the tools used to collect these, which allow crowd-workers to efficiently and accurately annotate parses. models: we show the results of several neural semantic parsing models trained on our data. execution: finally, we also make available the code to execute logical forms in the game, allowing the reproduction of end-to-end results. this also opens the door to using the data for reinforcement and imitation learning with language. we also provide access to an interactive bot using these models for parsing foot_1 . ## the assistant grammar in this section we summarize a grammar for generating logical forms that can be interpreted into programs for the agent architecture described in @xcite . ## the caip dataset this paper introduces the craftassist instruction parsing (caip) dataset of english-language commands and their associated logical forms (see appendix d for examples and appendix c for a full grammar specification). ## baseline models in order to assess the challenges of the dataset, we implement two models which learn to read a sentence and output a logical form by formulating the problem as a sequence-to-tree and a sequenceto-sequence prediction task respectively. ## experiments in this section, we evaluate the performance of our baseline models on the proposed dataset. training data: the caip datasets consists in a total of 6693 annotated instruction-parse pairs. in order for our models to make the most of this data while keeping the evaluation statistically significant, we create 5 different train/test splits of the data and report the average performance of models trained and evaluated on each of them. in each case, we hold out 650 examples from prompts and 350 from interactive for testing, and use the remaining 5693 as the training set. modeling choices: for the end-to-end trained sentencerec model, we use a 2-layer gru sentence encoder and all hidden layers have dimension @xmath0. we use pre-trained word embeddings computed with fasttext with subword information @xcite . the decoder uses a gru recurrent cell and 4-headed attention. the seq2seq model uses a variant of the bert-base-uncased provided in the transformer library foot_2 with 6 encoding and decoding layers. for the seq2seq model and the sentencerec with pre-trained encoder, we use the distilbert-base-uncased encoder from the same library. the seq2seq model uses beam search decoding with 15 beams. all models are trained with the adam optimizer with quadratic learning rate decay. we provide our model and training code along with the dataset for reproducibility purposes. overview of results: error analysis: we further investigate the errors of the seq2seq models on one of the data splits. we find that the model still struggles with span predictions: out of 363 errors, 125 only make mistakes on spans (and 199 get the tree structure right but make mistakes on leaves). figure 6 shows the nodes which are most commonly mistaken, with the number of false positive and false negatives out of these 363 mistakes. unsurprisingly, the most commonly confused span leaf is "has tag", which we use as a miscellaneous marker. aside from that "has tag" however, the span mistakes are evenly spread over all other leaves. the next most common source of mistakes comes from the model struggling between identifying whether a provided location corresponds to the target of the action or to the reference object, and to identify instructions which imply a repetition. the former indicates a lack of compositionality in the input representation: the model correctly identifies that a location is mentioned, but fails to identify its context. repeat conditions on the other hand challenge the model due to the wide variety of possible stop condition, a problem we suggest future work pay special attention to. ## conclusion in this work, we have described a grammar over a mid-level interface for a minecraft assistant. we then discussed the creation of a dataset of natural language utterances with associated logical forms over this grammar that can be executed in-game. finally, we showed the results of using this new dataset to train several neural models for parsing natural language instructions. consistent with recent works, we find that bert pre-trained models do better than models trained from scratch, but there is much space for improvement. we believe this data will be useful to researchers studying semantic parsing, especially interactive semantic parsing, human-robot interaction, and even imitation and reinforcement learning. the code, dataset and annotation tools described in the paper have been open-sourced foot_3 .