tag E : Enabling an Embodied Agent to Understand Human Instructions (2023)

Abstract: Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as task and argument grounding for Embodied agents (tagE). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot’s established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.

## introduction robots in our daily surroundings are often engage with human beings for various purposes. as natural language interaction capability increases the acceptability and usability of a robot, many studies have focused on natural language interaction with a robot @xcite . this can be particularly useful if we can provide task instruction in natural language @xcite . however, large vocabulary and variation of word/phrases/sentences of any natural language (e.g., english) makes it very difficult for a robot to understand human intention and perform the task @xcite . recent advances in natural language processing (nlp), in particular, the rise of large-scale neural language models have simplified nlp tasks with high accuracy @xcite . however, in order for a robot to perform a task, the task intention has to be mapped to a known set of skills of the robot (task grounding) so that some action can be taken in the physical world. additionally, the objects associated with a task (task arguments) should be mapped to the objects within the environment (argument grounding). most robots use an object detector, which uses a fixed vocabulary. a human may not be aware of that vocabulary or may not remember it. as a result, a different word/phrase can be used to refer the same object. therefore, argument grounding becomes equally important in order to perform physical action in the environment. existing works on intent prediction map the intended task to the robot's capability @xcite . however, they can neither extract the arguments associated with a task nor handle complex instructions with multiple tasks. works on relation extraction generally find the relevant triplets in a natural language text, where the triplets have the form of head, relation, and @xmath0, r, @xmath1. one can use such a method to find a task-argument triplet (figure 1a ). but, multiple arguments may be associated with the same tasks. extracting such triplets where the head/tail is common in many such triplets is not straightforward (figure 1b , 1c ). moreover, the existing methods neither ground the task nor ground the arguments. for example, in figure 1b , the word 'keep' signifies the 'placing' task according to the robot's capability, and the word 'fridge' needs we have developed a novel deep neural network architecture, called tage (task and argument grounding for embodied agents) that extract the set of tasks and their relevant arguments from a natural language complex instruction. the contributions of this article are following. ## related work there are three different areas that are relevant to this work -sequence-to-sequence learning, structured information extraction, and natural language understanding of instructions for robots. sequence-to-sequence learning: encoderdecoder models are popular for the sequenceto-sequence (s2s) learning. there are different types of s2s tasks for which encoder-decoder architecture is used, e.g., neural machine translation @xcite @xcite , joint entity and relation extraction @xcite , crosslingual open information extraction @xcite , joint extraction of aspect-opinion-sentiment triplets @xcite , etc. ## proposed framework in this section, we formally define the task before describing the neural network architecture. given a natural language instruction @xmath2, x 2 , ..., x n } with n tokens, the goal is to extract a set of tasks, in figure 3 , we give an overview of our proposed model. we use an encoder-decoder model inspired from nayak and ng (2020); @xcite for this task. the encoder is used to encode the given natural language instruction and decoder is used to generate the task and argument sequence. however, nayak and ng (2020) proposed a linear decoding process for joint entity and relation extraction task and @xcite proposed an aspect-sentiment triplets extraction task. but such linear decoding scheme is not suitable for this task. we need to extract a list of tasks and for each task, we need to extract a list of arguments. thus, we propose a novel nested decoding approach for task and argument extraction. the outer decoder is responsible to generate the task sequence and inner decoder is responsible to generate the argument sequence for each task. for argument grounding, we use a 'bio' sequence labeling based approach. we jointly train our model for these three objectives of task extraction, argument extraction, and argument grounding together. ## experiments in this section, we describe the dataset, parameter settings, and evaluation metrics that are used to train and evaluate our model. ## results to evaluate our model, we have defined a number of baseline systems. @xcite proposed the first baseline where they have used a crf-based model for task and argument extraction (feature crf in as mentioned earlier, tage follows a generative approach to tackle the limitations of the existing the experiment shows that joint training performs much better than separate training. as ablation, we have experimented with different types of bert encoders. though all types of bert models use a number of stacked transformer layers to get a vector representation of natural language input, the variation stems from the number of transformer layers and dimension of the output vectors. we have experimented with 5 different variations -(i) mini with 4 layers and 256-dimensional vector, (ii) small with 4 layers and 512-dimensional vector, (iii) medium with 8 layers and 512-dimensional vector, (iv) base uncased with 12 layers and 786-dimensional vector, and (v) large uncased with 24 layers and 1024-dimensional vector. the pre-trained models for the variations of bert is provided by @xcite . a small number of layers and vector dimensions leads to less number of parameters, a smaller model size, and a smaller training & inference time. however, this impacts the accuracy of the system as evident in ## conclusions instructing a robot in natural language certainly improves the usability and acceptability of the robot. however, understanding the instructions given in unstructured text can be challenging for a robot. a robot needs to perform multiple jobs to completely understand a natural language instruction such as extracting the (sub)tasks, extracting their associated arguments, and grounding of the arguments. previously, researchers have tried to solve these jobs independently, thus misses the interaction among them in the given instruction. to overcomes this limitation, we propose a neural network model tage, which can solve these three tasks together in an end-to-end fashion. we also annotate a suitable dataset for experiments and our proposed approach outperforms some strong baselines on this dataset. in future, we would integrate a planner module that would generate an executable plan from the output of tage.