Learning to Execute Actions or Ask Clarification Questions (2022)

Abstract: Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also define two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly.

## introduction following instructions in natural language by intelligent agents to achieve a shared goal with the instructors in a pre-defined environment is a ubiquitous task in many scenarios, e.g., finding a target object in an environment @xcite , drawing a picture @xcite , or building a target structure @xcite . a number of machine learning (ml) research projects about following instructions tasks have been initiated by making use of the video game minecraft @xcite @xcite @xcite . building such agents requires to make progress in grounded natural language understanding -understanding complex instructions, for example, with spatial relations in natural languageself-improvement -studying how to flexibly learn from human interactions -synergies of ml components -exploring the integration of several ml and non-ml components to make them work together @xcite . the recently introduced minecraft corpus dataset @xcite proposes a collaborative building task, in which an architect and a builder can communicate via a textual chat. architects are provided with a target structure they want to have built, and the builders are the only ones who can control the minecraft avatar in the virtual environment. the task consists in building 3d structures in a block world-like scenario collaboratively, as shown in the figure 1 . earlier works in minecraft collaborative building tasks @xcite attempted to build an automated builder agent with a large action space but failed to allow the builder to take the initiative in the conversation. however, an intelligent agent should not only understand and execute the instructor's requests but also be able to take initiatives, e.g., asking clarification questions, in case the instructions are ambiguous. in the task defined by this dataset, builders may encounter ambiguous situations that are hard to interpret by just relying on the world state information and instructions. for example, in figure 1 , we provide a simple case where the architect fails to provide sufficient information to the builder, such as the color of the blocks. in this situation, it is clearly difficult for the builder to know exactly which action should be taken. if, however, the builder is able to clarify the situation with the architect, this ambiguity can be resolved. therefore, builders, besides following architects' instructions, should take the initiative in the conversation and ask questions when necessary. to this end, in this paper we annotate all builder utterances in the minecraft corpus dataset by categorizing them in the dataset into eight dialogue utterance types as shown in also, we propose a new automated builder agent that learns to map instructions to actions and decide when to ask questions. our model utilizes three dialogue slots, the action type slot, the location slot, and the color slot. this solution has the benefit of making the learning easier with respect to those models that work using a large action space @xcite . to solve the collaborative building task, both the dialogue context and the world state need to be considered. therefore, to endow our model with the ability to better learn the representations between the world state and language, our model implements a cross-modality module, which is based on the cross attention mechanism. experimental results on our extended minecraft corpus dataset show that our model achieves stateof-the-art performance with a substantial improve-ment for the collaborative building task. we also provide new baselines for learning to ask task and the joint learning of these two tasks. ## method in this section we introduce the proposed builder model, as shown in figure 2 . the model comprises four major components: the utterance encoder, the world state encoder, the fusion module, and the slot decoder. the utterance encoder (in sec. 4.1) and world state encoder (in sec. 4.2) learn to represent the dialogue context and the world state. these encoded representations are then fed into the fusion module (in sec. 4.3) that learns contextualized embeddings for the grid world and textual tokens through the single and cross modality modules. finally, the learned world and text representations are mapped into the pre-defined slot-values in the slot decoder (in sec. 4.4). dialogue history u grid world state encoder grid world state grid single modality module grid single modality module grid cross modality module text cross modality module grid cross modality module grid single modality module color module action type module location module ĉ t l last action + (n g -n t )× n t × text single modality module text single modality module dialogue context encoder w ## experiments, results and discussion in this section, we first compare our model against the baseline for the collaborative building task where models only need to learn the instruction following task (in sec. 5.1). then, we train our model to learn when to ask and evaluate on our extended minecraft dialogue corpus (in sec. 5.2). finally, we evaluate our model's ability on the combination of the two above-mentioned tasks (in the sec. 5.3). all training details are reported in the appendix. the software and data are available at: https://github.com/ zhengxiangshi/learntoask . we first compare our model against the only baseline 1 named bap @xcite by only using the "execution (original)" dataset as in for our models, we present the performance of two different dialogue context encoders: we use the pre-trained glove word embeddings with 300 dimensions @xcite as the initial word embeddings followed by a gru @xcite and contextual word embeddings using the pre-trained bert base model @xcite . for the action type slot, we pre-define three potential values: placement, removal, and stop. the value of the location slot can be one of 1,089 candidate voxels and the value of the color slot can 1 https://github.com/prashant-jayan21/ minecraft-bap-models be one of six candidate colors. during training we minimize the sum of the cross entropy losses of the location slot, the color slot, and the action type slot. the f1-score metric on the test set is used to evaluate model performance by comparing the model predictions against the action sequence performed by the human builder. results. in ## conclusion in this paper, we extend the minecraft corpus dataset by labelling builder utterances into eight types, in which two of them are relevant to clarification questions. this allows builder models to learn to take initiatives in the instruction following tasks. also, we have proposed a new model that achieves state-of-the-art performance on the minecraft collaborative building task with a large improvement. besides these contributions, we introduce a new learning to ask task for clarification questions and a jointly learning task with the collaborative building task. we leave the generation of clarification questions to future work.