Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration (2022)

Abstract: Vision-language navigation (VLN) is a challenging task due to its large searching space in the environment. To address this problem, previous works have proposed some methods of fine-tuning a large model that pretrained on large-scale datasets. However, the conventional fine-tuning methods require extra human-labeled navigation data and lack self-exploration capabilities in environments, which hinders their generalization of unseen scenes. To improve the ability of fast cross-domain adaptation, we propose Prompt-based Environmental Self-exploration (ProbES), which can self-explore the environments by sampling trajectories and automatically generates structured instructions via a large-scale cross-modal pretrained model (CLIP). Our method fully utilizes the knowledge learned from CLIP to build an in-domain dataset by self-exploration without human labeling. Unlike the conventional approach of fine-tuning, we introduce prompt tuning to achieve fast adaptation for language embeddings, which substantially improves the learning efficiency by leveraging prior knowledge. By automatically synthesizing trajectory-instruction pairs in any environment without human supervision and instruction prompt tuning, our model can adapt to diverse vision-language navigation tasks, including VLN and REVERIE. Both qualitative and quantitative results show that our ProbES significantly improves the generalization ability of the navigation model.

## introduction teaching a robot to navigate following a natural language instruction has a broad impact in the field of human-robotic interaction. many related tasks have been proposed to delve into this problem. the vision-language navigation (vln) task @xcite is proposed where an agent is required to navigate in a photo-realistic environment stepby-step following a natural language instruction. recent tasks @xcite focus on target objects localization that asks an agent to identify an object in an unseen room. solving these tasks requires an agent to obtain a vision-text alignment ability that locates related objects and executes corrective actions according to the instruction. however, collecting a large-scale vln dataset is difficult and laborious since annotating the semantic of a trajectory within a sentence costs times of labor than annotating an image. existing navigation datasets are relatively small-scale, and learning on such datasets hinders the agent to obtain a good generalization ability. to solve this problem, envdrop @xcite uses a speaker model to generate instructions for sampled trajectories in unseen environments, but the generalization ability is not strong with limited vision-language understanding ability. recently, vln-bert @xcite introduces a visio-linguistic model pretrained on conceptual captions @xcite dataset to learn from image-caption pairs, which are quite different from trajectoryinstruction pairs from vln. to address this, airbert @xcite constructs a large-scale in-domain pretraining dataset with image-caption pairs collected from online marketplaces such as airbnb to finetune vilbert. however, airbert collects image captioning data on websites, which are still far from the scenario of vision-language navigation. different from previous methods that collect human-labeled data to train a navigation model, we suggest that automatically generating instruction-trajectory pairs by self-exploration for pretraining not only helps the model obtain better generalization ability but also achieves fast adaptation to downstream tasks. in this paper, we propose a method named prompt-based environmental self-exploration (probes) that generates navigation data with prior knowledge automatically and adapts pretrained model quickly to vln tasks. an overview of our proposed framework is shown in figure 1 . by using this method, a pretrained visio-linguistic model is able to adapt to the vln task automatically and efficiently. specifically, we build an in-domain dataset by self-exploration without labeling or crawler. to build such a dataset. we first generate templates by masking visual and action words in labeled instructions. then, we sample trajectories in the training environment. a pretrained clip @xcite model is used to recognize rooms and objects in the sampled trajectories and match described phrases with them. we construct instructions by filling the matched phrases into sampled templates. by leveraging the prior knowledge learned by clip, we are able to build a dataset automatically with rich semantic information. meanwhile, finetuning the whole pretrained model is time-consuming, we adopt prompt tuning @xcite , a lightweight alternative to finetuning. our prompt-based method can distill task-relevant knowledge from pretrained model and achieve fast adaption to downstream tasks. we evaluate probes on r2r @xcite and reverie @xcite datasets by discriminative and generative settings. results show that probes can match or surpass the performance of finetuning with substantially less training time. to sum up, our main contributions are as follows: ## related work vision-and-language navigation. anderson et al. @xcite proposed the first vision-language navigation (vln) benchmark combining real imagery @xcite and natural language navigation instructions. to solve this task, wang et al. @xcite proposed a novel serl model to learn reward functions from the expert distribution. and combining imitation learning and reinforcement learning @xcite has been proved to be beneficial for vln. since the vln dataset is relatively small-scale, some works propose augmentation approaches @xcite @xcite to improve robustness. auxiliary losses @xcite @xcite is used to take advantage of the additional training signals derived from the semantic information. some pretraining methods @xcite have been proposed to learn generic crossmodal representations. this is further extended to a recurrent model that significantly improves sequential action prediction @xcite . however, the limited number of environments in pretraining constrain the generalization ability to unseen scenarios. most related to this work, vln-bert @xcite transfers knowledge from abundant, but out-of-domain image-text data to improve path-instruction matching. in contrast, we not only propose an effective method to build an in-domain dataset by sampling trajectory and generating instructions with templates, but also present a prompt-based pretraining strategy to improve vln. ## conclusion in this work, we first introduce an effective way to generate in-domain data for pretraining the vln model: leveraging a large pretrained clip model to generate captions for each viewpoint and sampling actions in the environment. experiments show that the domain gap between pretraining data and vln tasks can be mitigated. we also propose a promptbased architecture, which introduces prompt tuning to adapt the pretrained model fastly. our proposed probes achieves better results compared to baseline on both r2r and reverie datasets, and ablations show the contribution of each module and the effectiveness of the generated data. walk past family room with mirror on your left, walk to dining room with mirror, wait at dining room.