B aby W alk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps (2020)

Abstract: Learning to follow instructions is of fundamental importance to autonomous agents for vision-and-language navigation (VLN). In this paper, we study how an agent can navigate long paths when learning from a corpus that consists of shorter ones. We show that existing state-of-the-art agents do not generalize well. To this end, we propose BabyWalk, a new VLN agent that is learned to navigate by decomposing long instructions into shorter ones (BabySteps) and completing them sequentially. A special design memory buffer is used by the agent to turn its past experiences into contexts for future steps. The learning process is composed of two phases. In the first phase, the agent uses imitation learning from demonstration to accomplish BabySteps. In the second phase, the agent uses curriculum-based reinforcement learning to maximize rewards on navigation tasks with increasingly longer instructions. We create two new benchmark datasets (of long navigation tasks) and use them in conjunction with existing ones to examine BabyWalk’s generalization ability. Empirical results show that BabyWalk achieves state-of-the-art results on several metrics, in particular, is able to follow long instructions better. The codes and the datasets are released on our project page: https://github.com/Sha-Lab/babywalk .

## introduction autonomous agents such as household robots need to interact with the physical world in multiple modalities. as an example, in vision-and-language navigation (vln) @xcite , the agent moves around in a photo-realistic simulated environment @xcite by following a sequence of natural language instructions. to infer its whereabouts so as to decide its moves, the agent infuses its visual perception, its trajectory and the instructions @xcite @xcite . arguably, the ability to understand and follow the instructions is one of the most crucial skills to acquire by vln agents. @xcite shows that the vln agents trained on the originally proposed dataset room2room (i.e. r2r thereafter) do not follow the instructions, despite having achieved high success rates of reaching the navigation goals. they proposed two remedies: a new dataset room4room (or r4r) that doubles the path lengths in the r2r, and a new evaluation metric coverage weighted by length score (cls) that measures more closely whether the groundtruth paths are followed. they showed optimizing the fidelity of following instructions leads to agents with desirable behavior. moreover, the long lengths in r4r are informative in identifying agents who score higher in such fidelity measure. in this paper, we investigate another crucial aspect of following the instructions: can a vln agent generalize to following longer instructions by learning from shorter ones? this aspect has important implication to real-world applications as collecting annotated long sequences of instructions and training on them can be costly. thus, it is highly desirable to have this generalization ability. after all, it seems that humans can achieve this effortlessly foot_0 . to this end, we have created several datasets of longer navigation tasks, inspired by r4r @xcite . we trained vln agents on r4r and use the agents to navigate in room6room (i.e., r6r) and room8room (i.e., r8r). we contrast to the performance of the agents which are trained on those datasets directly ("in-domain"). the results figure 1 : performance of various vln agents on generalizing from shorter navigation tasks to longer ones. the vertical axis is the newly proposed path-following metric sdtw @xcite , the higher the better. babywalk generalizes better than other approaches across different lengths of navigation tasks. meanwhile, it get very close to the performances of the in-domain agents (the dashed line). please refer to the texts for details. our findings are that the agents trained on r4r (denoted by the purple and the pink solid lines) perform significantly worse than the in-domain agents (denoted the light blue dashed line). also interestingly, when such out-of-domain agents are applied to the dataset r2r with shorter navigation tasks, they also perform significantly worse than the corresponding in-domain agent despite r4r containing many navigation paths from r2r. note that the agent trained to optimize the aforementioned fidelity measure (rcm(fidelity)) performs better than the agent trained to reach the goal only (rcm(goal)), supporting the claim by @xcite that following instructions is a more meaningful objective than merely goal-reaching. yet, the fidelity measure itself is not enough to enable the agent to transfer well to longer navigation tasks. to address these deficiencies, we propose a new approach for vln. the agent follows a long navigation instruction by decomposing the instruction into shorter ones ("micro-instructions", i.e., baby-steps), each of which corresponds to an intermediate goal/task to be executed sequentially. to this end, the agent has three components: (a) a memory buffer that summarizes the agent's experiences so that the agent can use them to provide the context for executing the next baby-step. (b) the agent first learns from human experts in "bitesize". instead of trying to imitate to achieve the ground-truth paths as a whole, the agent is given the pairs of a baby-step and the corresponding human expert path so that it can learn policies of actions from shorter instructions. (c) in the second stage of learning, the agent refines the policies by curriculum-based reinforcement learning, where the agent is given increasingly longer navigation tasks to achieve. in particular, this curriculum design reflects our desiderata that the agent optimized on shorter tasks should generalize well to slightly longer tasks and then much longer ones. while we do not claim that our approach faithfully simulates human learning of navigation, the design is loosely inspired by it. we name our approach babywalk and refer to the intermediate navigation goals in (b) as baby-steps. fig. 1 shows that babywalk (the red solid line) significantly outperforms other approaches and despite being out-of-domain, it even reach the performance of in-domain agents on r6r and r8r. the effectiveness of babywalk also leads to an interesting twist. as mentioned before, one of the most important observations by @xcite is that the original vln dataset r2r fails to reveal the difference between optimizing goalreaching (thus ignoring the instructions) and optimizing the fidelity (thus adhering to the instructions). yet, leaving details to section 5, we have also shown that applying babywalk to r2r can lead to equally strong performance on generalizing from shorter instructions (i.e., r2r) to longer ones. in summary, in this paper, we have demonstrated empirically that the current vln agents are ineffective in generalizing from learning on shorter navigation tasks to longer ones. we propose a new approach in addressing this important problem. we validate the approach with extensive benchmarks, including ablation studies to identify the effectiveness of various components in our approach. different from them, we focus on transferring an agent's performances on shorter tasks to longer ones. this leads to designs and learning schemes that improve generalization across datasets. we use a memory buffer to prevent mistakes in the distant past from exerting strong influence on the present. in imitation learning stage, we solve fine-grained subtasks (baby-steps) instead of asking the agent to learn the navigation trajectory as a whole. we then use curriculum-based reinforcement learning by asking the agent to follow increasingly longer instructions. ## notation and the setup of vln in the vln task, the agent receives a natural language instruction x composed of a sequence of sentences. we model the agent with an markov decision process (mdp) which is defined as a tuple of a state space s, an action space a, an initial state s 1 , a stationary transition dynamics ρ : s×@xmath0, a reward function r : s × @xmath1, and the discount factor γ for weighting future rewards. the agent acts according to a policy π : s × @xmath2. the state and action spaces are defined the same as in @xcite (cf. § 4.4 for details). for each x, the sequence of the pairs (s, a) is called a trajectory @xmath3, a 1 , . . . , s |y| , a |y| where |•| denotes the length of the sequence or the size of a set. we use â to denote an action taken by the agent according to its policy. hence, ŷ denotes the agent's trajectory, while y (or a) denotes the human expert's trajectory (or action). the agent is given training examples of (x, y) to optimize its policy to maximize its expected rewards. in our work, we introduce additional notations in the following. we will segment a (long) instruction x into multiple shorter sequences of sentences {x m , @xmath4, 2, • • • , m}, to which we refer as baby-steps. each x m is interpreted as a microinstruction that corresponds to a trajectory by the agent ŷm and is aligned with a part of the human expert's trajectory, denoted as y m . while the alignment is not available in existing datasets for vln, we will describe how to obtain them in a later section ( § 4.3). throughout the paper, we also freely interexchange the term "following the mth microinstruction", "executing the baby-step x m ", or "complete the mth subtask". we use @xmath5, |y|] to denote the (discrete) time steps the agent takes actions. additionally, when the agent follows x m , for convenience, we sometimes use t @xmath6, | ŷm |] to index the time steps, instead of the "global time" ## approach we describe in detail the 3 key elements in the design of our navigation agent: (i) a memory buffer for storing and recalling past experiences to provide contexts for the current navigation instruction ( § 4. ## experiments we describe the experimental setup ( § 5.1),followed by the main results in § 5.2 where we show the proposed babywalk agent attains competitive results on both the in-domain dataset but also generalizing to out-of-the-domain datasets with varying lengths of navigation tasks. we report results from various ablation studies in § 5.3. while we primarily focus on the room4room dataset, we re-analyze the original room2room dataset in § 5.4 and were surprised to find out the agents trained on it can generalize. ## discussion there are a few future directions to pursue. first, despite the significant improvement, the gap between short and long tasks is still large and needs to be further reduced. secondly, richer and more complicated variations between the learning setting and the real physical world need to be tackled. for instance, developing agents that are robust to variations in both visual appearance and instruction descriptions is an important next step. details on identifying baby-step instructions and aligning baby-steps with expert trajectories. ( § 4.3 and § 4.4 of the main text) • section b. implementation details of the navigation agent, reward function used in rl and optimization hyper-parameters. ( § 4.4 of the main text) • section c. additional experimental results, including in-domain & transfer results of different dataset trained models, sanity check of our reimplementation, and extra analysis of babywalk. ( § 5.1 and § 5.2 of the main text)