ALFRED - L : Investigating the Role of Language for Action Learning in Interactive Visual Environments (2022)

Abstract: Embodied Vision and Language Task Completion requires an embodied agent to interpret natural language instructions and egocentric visual observations to navigate through and interact with environments. In this work, we examine ALFRED, a challenging benchmark for embodied task completion, with the goal of gaining insight into how effectively models utilize language. We find evidence that sequence-to-sequence and transformer-based models trained on this benchmark are not sufficiently sensitive to changes in input language instructions. Next, we construct a new test split – ALFRED-L to test whether ALFRED models can generalize to task structures not seen during training that intuitively require the same types of language understanding required in ALFRED. Evaluation of existing models on ALFRED-L suggests that (a) models are overly reliant on the sequence in which objects are visited in typical ALFRED trajectories and fail to adapt to modifications of this sequence and (b) models trained with additional augmented trajectories are able to adapt relatively better to such changes in input language instructions.

## introduction recently a number of benchmark datasets have been proposed to study the ability of embodied agents to understand natural language in the context of egocentric visual observations and predict sequences of executable actions to answer questions @xcite , navigate to desired destinations @xcite , or additionally manipulate objects to complete tasks @xcite . although multi-modal transformer-based models have achieved tremendous progress on many of these datasets @xcite @@xmath0: walk to the coffee maker on the end of the counter. <i3>: go to the stove and focus on the top left burner. <i5>: turn around and go to the kitchen sink. in this work, we analyze models trained on the alfred dataset @xcite to better understand how they utilize language for embodied task completion. we chose alfred for our analysis as it requires object manipulation (object pick and place, opening and closing doors and more) in addition to navigation making it more challenging than most related datasets. in alfred, each example trajectory consists of a high-level natural language task description, followed by stepby-step (low-level) natural language instructions corresponding to logical subgoals, that when completed in sequence accomplish the task (see fig 1 ). in this work, we refer to this sequence of logical subgoals as a task structure -alfred trajectories consists of only 7 possible task structures. in each trajectory, an embodied agent is placed in the initial state of the environment, provided the language instructions, and expected to predict and execute a sequence of low level actions that accomplish the task described by the instructions using visual feedback from the execution of each action. we evaluate the sensitivity of alfred models to changes in language instructions in two ways. we first examine whether model predictions are affected by the removal of words indicative of spatial relationships, or by the removal of step-by-step instructions entirely ( §2). our experiments demonstrate that model performance is less affected by these changes than expected. in contrast, the task completion rate of models drops to 0% when deprived of visual inputs. in addition, we construct a new test set alfred-l, to test whether models can generalize to variations of alfred instructions that remove object manipulation steps or add navigation steps ( §3). a sample from this dataset is shown in fig 1 . intuitively, these changes do not require learning of new language understanding capabilities since following navigation instructions is already a prerequisite for successfully completing alfred tasks. consequently, existing alfred models should be able to generalize well to such instructions. however, experimental results on alfred-l demonstrate that models are incapable of such generalization. we hypothesize that models overfit to the task structure of alfred and ignore the addition of extra objects to be visited in alfred-l. further, we find that models trained using a larger number of visual scenes are able to adapt relatively better to such changes in input language instructions suggesting the importance of data augmentation techniques to make substantive progress on alfred. ## analysis by reducing instruction informativeness in this section, we examine the sensitivity of models to loss of information from language instructions when trying to complete alfred tasks. in our first experiment e1, we drop all words and phrases that indicate directional and spatial information from high and low level language instructions during inference. we note that 81% of the tokens in alfred instructions constitute directional and spatial information such as to the left, three steps forward, towards right, and over to the back. we hypothesize that the absence of such crucial information should cause models to be unable to correctly navigate or identify objects being referred to. in our second experiment e2, we discard all the step-by-step language instructions from the input foot_0 (see appendix a for more details). we analyze the following models trained on alfred: (a) episodic transformer (et): a model that uses a transformer to encode multimodal inputs @xcite ; et+synth: a version of et augmented with synthetic trajectories; et w/o pt: an ablated version that does not include language pretraining; (b) hitut: a hierarchical transformerbased model that explicitly predicts sub-goals in addition to low-level actions at every time step to enable backtracking to cope with execution failures @xcite ) (c) moca: a modular sequence-to-sequence model that separates action and object prediction (singh et al., 2020); (d) seq2seq: a simple sequence-to-sequence baseline @xcite . ## alfred-l for testing generalization alfred step-by-step language instructions involve a combination of navigation subgoals and a variety of object manipulation subgoals. intuitively, a model that can understand such language instructions should still be able to follow modified combinations of them that add or remove some steps. to evaluate the ability of alfred models to generalize in this manner, we create modified instructions using examples from alfred validation splits. we call this modified test split alfred-l. more concretely, training samples in alfred are typically a sequence of alternating navigation (gotolocation) and object manipulation (for example cleanobject, heatobject) subgoals, always ending with an object manipulation subgoal. models would thus expect to perform object manipulations at the end of each navigation subgoal and could memorize locations of objects commonly @xmath2, and exploit these instead of understanding instructions provided at inference time to determine objects to be navigated to. we create alfred-l to break some of these patterns -removing the need for object manipulation actions and adding instructions to navigate to additional objects. a model that sees a significant performance drop between alfred and alfred-l is likely overly reliant on the task structure of alfred and ignoring details present in language instructions. as shown in fig 1 and ## conclusion we evaluate embodied task completion models trained on alfred and find that they are not very sensitive to loss of spatial and directional information, or detailed task steps. we also present a new test split alfred-l to test generalization to novel task structures and find that models are unable to adapt to the addition of extra reverse navigation steps. we hope that our work guide the development of future embodied ai benchmarks (and models) to avoid the issues we identified with alfred. in addition, our analysis at the sub-goal level on unseen environments (unseen test) and on our proposed alfred-l test split help test different generalization aspects and therefore can potentially represent major failure modes in other embodied ai datasets.