N av H int: Vision and Language Navigation Agent with a Hint Generator (2024)

Abstract: The existing work on vision and language navigation mainly relies on navigation-related losses to establish the connection between vision and language modalities, neglecting aspects of helping the navigation agent build a deep understanding of the visual environment.In our work, we provide indirect supervision to the navigation agent through a hint generator that provides detailed visual descriptions.The hint generator assists the navigation agent in developing a global understanding of the visual environment. It directs the agent’s attention toward related navigation details, including the relevant sub-instruction, potential challenges in recognition and ambiguities in grounding, and the targeted viewpoint description. To train the hint generator, we construct a synthetic dataset based on landmarks in the instructions and visible and distinctive objects in the visual environment.We evaluate our method on the R2R and R4R datasets and achieve state-of-the-art on several metrics. The experimental results demonstrate that generating hints not only enhances the navigation performance but also helps improve the agent’s interpretability.

## introduction in many real-world applications, it is a crucial skill for an intelligent agent to perceive the visual environment and interact with humans using natural language. the vision and language navigation (vln) task @xcite is one of the popular problems in this direction that has attracted significant attention from computer vision, natural language processing, and robotic communities @xcite @xcite . with the increasing popularity of the vln task, many neural navigation models @xcite @xcite have been proposed. one line of research is to strengthen the connection of the vision and language modalities (ma turn around and go straight. walk towards the wall and stop. instruction candidate viewpoints view1 view2 view3 but i can see "wall" in all candidate viewpoints. however, there are "large window with wooden blinds, glass table with white chairs, and a ceiling lamp" that are specific to view3. ## method in the vln problem setting, the agent is given a natural language instruction, denoted as @xmath0, w 2 , • • • , w l }, l is the length of the sentence. at each navigation step, the agent perceives panoramic views with 36 1 discrete images. there are n candidate viewpoints that can be navigated to, denoted as @xmath1, i 2 , • • • , i n }. this task aims to generate a trajectory following the given instruction. in the following section, we first present our constructed navigation hint dataset. then, we introduce the hint generator. the navigation hint dataset is used to train the navigation agent and the hint generator jointly. ## conclusion in this paper, we equip the navigation agent with a hint generator to generate visual descriptions during navigation, which helps the agent's understanding of the visual environment. to train the hint generator, we create a navigation hint dataset that provides comprehensive supervision for training the agent. during navigation, the agent generates natural language descriptions about its visual environment at each step, including comparing various views and explaining ambiguities in recognizing the target destination. empirical results show that detailed visual description generation improves both navigation performance and the interpretability of actions taken by the navigation agent.