Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation (2021)

Abstract: One of the most challenging topics in Natural Language Processing (NLP) is visually-grounded language understanding and reasoning. Outdoor vision-and-language navigation (VLN) is such a task where an agent follows natural language instructions and navigates in real-life urban environments. With the lack of human-annotated instructions that illustrate the intricate urban scenes, outdoor VLN remains a challenging task to solve. In this paper, we introduce a Multimodal Text Style Transfer (MTST) learning approach and leverage external multimodal resources to mitigate data scarcity in outdoor navigation tasks. We first enrich the navigation data by transferring the style of the instructions generated by Google Maps API, then pre-train the navigator with the augmented external outdoor navigation dataset. Experimental results show that our MTST learning approach is model-agnostic, and our MTST approach significantly outperforms the baseline models on the outdoor VLN task, improving task completion rate by 8.7% relatively on the test set.

## introduction a key challenge for artificial intelligence research is to go beyond static observational data and consider more challenging settings that involve dynamic actions and incremental decision-making processes @xcite . outdoor visionand-language navigation (vln) is such a task, where an agent navigates in an urban environment by grounding natural language instructions in visual scenes, as illustrated in fig. 1 . to generate a series of correct actions, the navigation agent must comprehend the instructions and reason through the visual environment. ## conclusion in this paper, we proposed the multimodal text style transfer learning approach for outdoor vln. this learning framework allows us to utilize outof-domain navigation samples in outdoor environments and enrich the original navigation reasoning training process. experimental results show that our mtst approach is model-agnostic, and our mtst learning approach outperforms the baseline models on the outdoor vln task. we believe our study provides a possible solution to mitigate the data scarcity issue in the outdoor vln task. in future studies, we would love to explore the pos-