N av RAG : Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM (2025)

Abstract: Vision-and-Language Navigation (VLN) is an essential skill for embodied agents, allowing them to navigate in 3D environments following natural language instructions. High-performance navigation models require a large amount of training data, the high cost of manually annotating data has seriously hindered this field. Therefore, some previous methods translate trajectory videos into step-by-step instructions for expanding data, but such instructions do not match well with users’ communication styles that briefly describe destinations or state specific needs. Moreover, local navigation trajectories overlook global context and high-level task planning. To address these issues, we propose NavRAG, a retrieval-augmented generation (RAG) framework that generates user demand instructions for VLN. NavRAG leverages LLM to build a hierarchical scene description tree for 3D scene understanding from global layout to local details, then simulates various user roles with specific demands to retrieve from the scene tree, generating diverse instructions with LLM. We annotate over 2 million navigation instructions across 861 scenes and evaluate the data quality and navigation performance of trained models. The model trained on our NavRAG dataset achieves SOTA performance on the REVERIE benchmark.

## introduction vision-and-language navigation (vln) @xcite @xcite requires the agent to understand natural language instructions and navigate to the described destination in 3d environments. the immense semantic space and diverse forms of language instructions require massive data to train a vln agent capable of generalizing across different scenarios. however, the high cost of manual * corresponding author turn left, pass the bathtub, and exit the bathroom. then turn right, go down the stairs, and turn right to enter the living room. stop next to the sofa. please go to the living room at level 1 and find the light switch next to the sofa, turn it off for me. ## conclusion in this work, we propose navrag, a user demandoriented navigation data generation method through retrieval-augmented llm. unlike previous works that use trajectory-based instruction generators to translate navigation videos into step-bystep instructions, our navrag utilizes the environmental representations from a hierarchical scene description tree. by retrieving descriptions of different levels in a top-down manner and introducing the user demands, navrag effectively enhances the quality of instructions generated by the llm.