Language-Aligned Waypoint ( LAW ) Supervision for Vision-and-Language Navigation in Continuous Environments (2021)

Abstract: In the Vision-and-Language Navigation (VLN) task an embodied agent navigates a 3D environment, following natural language instructions. A challenge in this task is how to handle ‘off the path’ scenarios where an agent veers from a reference path. Prior work supervises the agent with actions based on the shortest path from the agent’s location to the goal, but such goal-oriented supervision is often not in alignment with the instruction. Furthermore, the evaluation metrics employed by prior work do not measure how much of a language instruction the agent is able to follow. In this work, we propose a simple and effective language-aligned supervision scheme, and a new metric that measures the number of sub-instructions the agent has completed during navigation.

## introduction training agents to navigate in realistic environments based on natural language instructions is a step towards building robots that understand humans and can assist them in their daily chores. @xcite introduced the vision-and-language navigation (vln) task, where an agent navigates a 3d environment to follow natural language instructions. much of the prior work on vln assumes a discrete navigation graph (navgraph), where the agent teleports between graph nodes, both in indoor @xcite and outdoor @xcite ) settings. @xcite reformulated the vln task to a continuous environment (vln-ce) by lifting the discrete paths to continuous trajectories, bringing the task closer to real-world scenarios. @xcite supervised agent training with actions based on the shortest path from the agent's location to the goal, following prior work in vln @xcite @xcite . however, as jain et al. go toward the white table on the patio turn right and go inside through the sliding doors stop next to the stove and island in the kitchen go through the room with television and into the kitchen (2019) observed, such supervision is goal-oriented and does not always correspond to following the natural language instruction. our key idea is that language-aligned supervision is better than goal-oriented supervision, as the path matching the instructions (language-aligned) may differ from the shortest path to the goal @xcite . this is especially true in 'off the path' scenarios (where the agent veers off the reference path prescribed by the instructions). languagealigned supervision encourages the agent to move towards the nearest waypoint on the languagealigned path at every step and hence supervises the agent to better follow instructions (see figure 1 ). in the discrete nav-graph setting, @xcite interleave behavioral cloning and policy gradient training, with a sparse 'fidelity-oriented reward' based on how well each node is covered on the reference path. in contrast, we tackle the vln-ce setting and propose a simple and effective approach that provides a denser supervisory signal leading the agent to the reference path. a dense supervisory signal is especially important for vln-ce where the episodes have a longer average length of 55.88 steps vs. 4-6 nodes in (discrete) vln. to this end, we conduct experiments investigating the effect of density of waypoint supervision on task performance. to assess task performance, we complement the commonly employed normalized dynamic time warping (ndtw) metric @xcite with a intuitive waypoint accuracy metric. finally, to provide qualitative insights about degree of following instructions, we combine language-aligned waypoints with information about sub-instructions. our experiments show that our language-aligned supervision trains agents to more closely follow instructions compared to goal-oriented supervision. however, these methods fail to achieve similar performance in the more challenging vln-ce task, where the agent navigates in a continuous 3d simulation environment. @xcite propose a modular approach using topological environment maps for vln-ce and achieve better results. in concurrent work, @xcite propose a modular approach to predict waypoints on a panoramic observation space and use a lowlevel control module to navigate. however, both these works focus on improving the ability of the agent to reach the goal. in this work, we focus on the vln-ce task and on accurately following the path specified by the instruction. instruction following in vln. work in the discrete nav-graph vln setting has also focused on improving the agent's adherence to given instruc-tions. @xcite adopt bayesian state tracking to model what a hypothetical human demonstrator would do when given the instruction, whereas @xcite attends to specific objects and actions mentioned in the instruction. @xcite train the agent to follow shorter instructions and later generalize to longer instructions through a curriculum-based reinforcement learning approach. @xcite divide language instructions into shorter sub-instructions and enforce a sequential traversal through those subinstructions. they additionally enrich the roomto-room (r2r) dataset @xcite with the sub-instruction-to-sub-path mapping and introduce the fine-grained r2r (fg-r2r) dataset. more closely related to our work is @xcite , which introduced a new metric -coverage weighted by length score (cls), measuring the coverage of the reference path by the agent, and used it as a sparse fidelity-oriented reward for training. however, our work differs from theirs in a number of ways. first, in law we explicitly supervise the agent to navigate back to the reference path, by dynamically calculating the closest waypoint (on the reference path) for any agent state. in contrast to calculating waypoints, @xcite optimize accumulated rewards, based on the cls metric. moreover, we provide dense supervision (at every time step) for the agent to follow the reference path by providing a cross-entropy loss at all steps of the episode, in contrast to the single reward at the end of the episode during stage two of their training. finally, law is an online imitation learning approach, which is simpler to implement and easier to optimize compared to their policy gradient formulation, especially with sparse rewards. similar to @xcite al. (2019), ilharco et al. (2019) train one of their agents with a fidelity oriented reward based on ndtw. ## approach our approach is evaluated on the vln-ce dataset @xcite , which is generated by adapting r2r to the habitat simulator @xcite . it consists of navigational episodes with language instructions and reference paths. the reference paths are constructed by taking the discrete nav-graph nodes corresponding to positions of panoramic cameras (we call these pano waypoints, shown as gray circles in figure 2 2020 ) which predicts an action. we optimize the model using language-aligned supervision, which brings it back on the path toward the next waypoint. to create a ground truth reference path consisting of dense waypoints (step waypoints, see dashed path in figure 2 ) corresponding to an agent step size of 0.25m. we take waypoints from these paths as languagealigned waypoints (law) to supervise our agent, in contrast to the goal-oriented supervision in prior work. we interpret our model performance qualitatively, and examine episodes for which the groundtruth language-aligned path (law step) does not match goal-oriented shortest path (shortest) foot_0 . task. the agent is given a natural language instruction, and at each time step t, the agent observes the environment through rgbd image i t with a 90 training. we follow the training regime of vln-ce. it involves two stages: behavior cloning (with teacher-forcing) on the larger augmented dataset to train an initial policy, and then fine-tuning with dagger @xcite . dagger trains the model on an aggregated set of all past trajectories, sampling actions from the agent policy. rather than supervising with the conventional goal-oriented sensor, we supervise with a language-aligned sensor in both the teacher-forcing phase and the dagger phase. the language-aligned sensor helps bring the agent back on the path to the next waypoint if it wanders off the path (see figure 2 top). the training dataset @xmath0, w (i) } consists of instructions s (i) and reference path w (i) . for each episode (s, w ) ∼ d with the agent starting state as x 0 , we use cross entropy loss to maximize the log likelihood of the ground-truth action a * at each time step t: here, x t is the 3d position of the agent at time t, e a * is the one-hot vector for the ground-truth action a * , which is defined as a * = g(x t , φ(x t , w )). the set of languagealigned waypoints is @xmath1, ..., w m }. the waypoint in w that is nearest to a 3d position x t is obtained by φ(x t , w ). the best action based on the shortest path from a 3d position x t to w is denoted by g(x t , w). ## experiments dataset. we base our work on the vln-ce dataset @xcite . the dataset contains 4475 trajectories from matterport3d @xcite . each trajectory is described by multiple natural language instructions. the dataset also contains ∼150k augmented trajectories generated by @xcite adapted to vln-ce. to qualitatively analyze our model behavior, we use the fine-grained r2r (fg-r2r) dataset from @xcite . it segments instructions into subinstructions foot_1 and maps each sub-instruction to a corresponding sub-path. evaluation metrics. we adopt standard metrics used by prior work @xcite . in the main paper, we report success rate (sr), success weighted by inverse path length (spl), normalized dynamictime warping (ndtw), and success weighted by ndtw (sdtw). trajectory length (tl), navigation error (ne) and oracle success rate (os) are reported in the supplement. since none of the existing metrics directly measure how effectively waypoints are visited by the agent, we introduce waypoint accuracy (wa) metric. it measures the fraction of waypoints the agent is able to visit correctly (specifically, within 0.5m of the waypoint). the agent trajectory as we illustrate in figure 4 . implementation details. we implement our agents using pytorch @xcite and the habitat simulator @xcite . we build our code on top of the vln-ce codebase 3 and use the same set of hyper-parameters as used in the vln-ce paper. the first phase of training with teacher forcing on the 150k augmented trajectories took ∼60 hours to train, while the second phase of training with dagger on the original 4475 trajectories took ∼36 hours over two nvidia v100 gpus. ablations. we study the effect of varying the density of language-aligned waypoints on model performance. for all the ablations we use the cma model described in section 3. we use law # to distinguish among the ablations. on one end of the density spectrum, we have the base model which is supervised with only the goal (law#1 or goal). on the other end is law step which refers to the pre-computed dense path from the vln-ce dataset and can be thought of as the densest supervision available to the agent. in the middle of the spectrum, we have law pano, which uses the navigational nodes (an average of 6 nodes) from the r2r dataset. we also sample equidistant points on the language-aligned path to come up with law#2, law#4 and law#15 containing two, four and fifteen waypoints, respectively. the intuition is that law pano takes the agent back to the languagealigned path some distance ahead of its position, while law step brings it directly to the path. quantitative results. in ## conclusion we show that instruction following during the vln task can be improved using language-aligned supervision instead of goal-oriented supervision as commonly employed in prior work. our quantitative and qualitative results demonstrate the benefit of the law supervision. the waypoint accuracy metric we introduce also makes it easier to interpret how agent navigation corresponds to following sub-instructions in the input natural language. we believe that our results show that law is a simple but useful strategy to improving vln-ce.