Compositional Data and Task Augmentation for Instruction Following (2021)

Abstract: Executing natural language instructions in a physically grounded domain requires a model that understands both spatial concepts such as “left of” and “above”, and the compositional language used to identify landmarks and articulate instructions relative to them. In this paper, we study instruction understanding in the blocks world domain. Given an initial arrangement of blocks and a natural language instruction, the system executes the instruction by manipulating selected blocks. The highly compositional instructions are composed of atomic components and understanding these components is a necessary step to executing the instruction. We show that while end-to-end training (supervised only by the correct block location) fails to address the challenges of this task and performs poorly on instructions involving a single atomic component, knowledge-free auxiliary signals can be used to significantly improve performance by providing supervision for the instruction’s components. Specifically, we generate signals that aim at helping the model gradually understand components of the compositional instructions, as well as those that help it better understand spatial concepts, and show their benefit to the overall task for two datasets and two state-of-the-art (SOTA) models, especially when the training data is limited—which is usual in such tasks.

## introduction one of the hallmarks of artificial intelligence is designing robots that can understand and execute natural language instructions in a grounded domain @xcite . there is a strong need for this technology in several applications @xcite @xcite , where the robot needs to ground relevant parts of the instruction to the environment. blocks world is a popular platform to study instruction understanding in physically grounded environments and presents several key reasoning challenges @xcite @xcite @xcite @xcite . in @xcite , the environment consists of a number of blocks placed on a board. the robot receives an instruction and the current block configuration as input and has to execute the instruction by manipulating appropriate blocks. there are two scenarios: the easier one where blocks have labels and the more challenging one in which the blocks are unlabeled, thus necessitating the use of involved referential expressions. in the labeled dataset, the blocks have names: move the nvidia block to the right of the hp block, making grounding easier. however, as shown in fig. 1 , instructions in the unlabeled dataset are highly complex, involving multiple spatial concepts and a high degree of compositionality. to identify the block to be moved (source block) one needs to ground the block nearest (to) the left hand corner and understand the above and right spatial concepts. similarly, identifying where the block is to be moved to (target location) requires grounding the block below the centermost block and understanding the spatial concepts below and right. in this paper we ask a fundamental question: do models trained on small data-sets for a grounded task, really learn compositional reasoning or do they merely over-fit to a particular data-set ? we show that existing models trained in an end to end manner to predict source block and target coordinates, given the instruction and current configuration of blocks, fail to generalize to simple instructions. existing approaches do not address the compositionality of these instructions nor deal directly with the complex spatial concepts. in this paper, we attempt to bridge this gap by augmenting the end-to-end training in a knowledge-free way, with (i) data augmentation and (ii) task augmentation, to improve performance on the standard test set. further, we show in sec 3.1, while existing models perform poorly on the generated atomic instructions, our approach removes this vulnerability. in (i) we use a few simple templates to automatically generate examples that focus on a single spatial concept. then, we pre-train the model on this synthetic data before training on the more complex, original training data. this approach is an instance of curriculum learning @xcite where the difficulty of an instance is related to the number of spatial concepts it contains. in (ii), we create auxiliary tasks which are coarser than the location prediction task (e.g. quadrant prediction) and train the model on these tasks jointly with the main task @xcite . the auxiliary tasks help teach the model spatial concepts by providing explicit supervision for these components in the instruction. we supervise the auxiliary tasks in an alternate fashion with the main task to train the model. we emphasize that both our proposed solutions require no additional supervision. we observe that compositional data augmentation and the auxiliary tasks improve generalization on the synthetic test data and on the standard test data. our method is evaluated on different datasets (labeled blocks and unlabeled blocks) and on different models @xcite . ## augmenting end-to-end training given the current configuration ((x, y, z) locations of a maximum of 20 blocks) and an instruction, the model has to move the corresponding block. this has two sub-tasks-(1) source prediction: predict the block to be moved and (2) target prediction: predict the location the source block is to be moved offset, reference are fully connected (fc) layers. details of the shared backbone is in @xcite . to. figure 2 shows the model architecture we build using a baseline model for eg: @xcite as the backbone. we also experiment with the stateof-the-art model from @xcite . importantly, @xcite treats both source and target sub-tasks as regression problems while, @xcite treats the source sub-task as a classification problem and the target as regression. ## experiments here we present empirical evidence that shows the role of (1) pre-training with simpler instructions. ## conclusion we showed the benefits of data and task augmentation for instruction understanding on two datasets and two existing models for this task, improving their shortcomings on simple examples. • for the quantification of spaces we allow 1, 2, 3, 4. a.2 quadrant subset filters the following keywords are used for filtering the test-set: top left / upper left / northwest / back left top right / upper right / northeast bottom left / lower left / southwest / front left bottom right / lower right / southeast