Modular Networks for Compositional Instruction Following (2021)

Abstract: Standard architectures used in instruction following often struggle on novel compositions of subgoals (e.g. navigating to landmarks or picking up objects) observed during training. We propose a modular architecture for following natural language instructions that describe sequences of diverse subgoals. In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type. A sequence of modules to execute is chosen by learning to segment the instructions and predicting a subgoal type for each segment. When compared to standard, non-modular sequence-to-sequence approaches on ALFRED, a challenging instruction following benchmark, we find that modularization improves generalization to novel subgoal compositions, as well as to environments unseen in training.

## introduction work on grounded instruction following @xcite @xcite @xcite has recently been driven by sequence-to-sequence models @xcite , which allow end-to-end grounding of linguistically-rich instructions into equally-rich visual contexts @xcite @xcite . these sequence-to-sequence models are monolithic: they consist of a single network structure which is applied identically to every example in the dataset. monolithic instruction following models typically perform well when evaluated on test data from the same distribution seen during training. however, they often struggle in compositional generalization: composing atomic parts, such as actions or goals, where the parts are seen in training but their compositions are not (lake and @xcite @xcite . ## modular instruction following networks we focus on following instructions in embodied tasks involving navigation and complex object interactions, as shown in figure 2 . in training, each set of full instructions (e.g. "turn right and cross the room ... place the vase on the coffee table to the left of the computer.") is paired with a demonstration of image observations and actions. in training, we further assume that the full instruction is segmented into subgoal instructions, and each subgoal instruction is labeled with one of a small number (in our work, 8) of subgoal types , e.g. ["walk to the coffee maker.": goto ], ["pick up the dirty mug...": pickup], . . . , and paired with the corresponding segment of the demonstration. during evaluation, the agent is given only full instructions (which are unsegmented and unlabeled), and must predict a sequence of actions to carry out the instructions, conditioning on the image observations it receives. our modular architecture for compositional instruction following consists of a high-level controller (figure 2 , left), and modules for each subgoal type (figure 2 , right). the high-level controller chooses modules to execute in sequence based on the natural language instructions, and each chosen module executes until it outputs a stop action. the modules all share the same sequence-to-sequence architecture, which is the same as the monolithic architecture. we initialize each module's parameters with parameters from the monolithic model, and then fine-tune the parameters of each module to specialize for its subgoal. ## generalization evaluation we evaluate models on out-of-domain generalization in two conditions (see below) using the al-fred benchmark @xcite , comparing our modular approach to their non-modular sequence-to-sequence model. alfred is implemented in ai2-thor 2.0 @xcite , which contains a set of simulated environments with realistic indoor scene renderings and object interactions. the dataset contains approximately 25k expert instruction-trajectory pairs, comprised of 3 instructions for each of 8k unique trajectories. the instructions include both a high level instruction and a sequence of low level instructions. in our experiments, we do not use the high level instructions, which @xcite found to produce comparable results when evaluated on generalization to unseen environments with these architectures. figure 1 shows two example trajectories and their associated instructions. trajectories are composed (see sec. 2) of sequences of eight different types of subgoals: navigation (goto) and a variety of object interactions (e.g. pickup, clean, heat). each subgoal's subtrajectory is composed of a sequence of low-level discrete actions which specify commands for navigation or object interactions (which are accompanied by image segmentations to choose the object to interact with). ## conclusions we introduced a novel modular architecture for grounded instruction following where each module is a sequence-to-sequence model conditioned on natural language instructions. with the alfred dataset as a testbed, we showed that our modular model achieves better out-of-domain generalization, generalizing better at the subgoal level to novel task compositions and unseen environments than the monolithic model used in prior work. all of the module types in our model currently use separate parameterizations but identical architectures; future work might leverage the modularity of our approach by using specialized architectures, training procedures, or loss functions for each subgoal type. furthermore, unsupervised methods for jointly segmenting instructions and trajectories without requiring labeled subgoal labels and alignments would be a valuable addition to our framework.