D iv S cene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes (2025)

Abstract: Large Vision-Language Models (LVLMs) have achieved significant progress in tasks like visual question answering and document understanding. However, their potential to comprehend embodied environments and navigate within them remains underexplored. In this work, we first study the challenge of open-vocabulary object navigation by introducing DivScene, a large-scale dataset with 4,614 houses across 81 scene types and 5,707 kinds of target objects. Our dataset provides a much greater diversity of target objects and scene types than existing datasets, enabling a comprehensive task evaluation. We evaluated various methods with LVLMs and LLMs on our dataset and found that current models still fall short of open-vocab object navigation ability. Then, we fine-tuned LVLMs to predict the next action with CoT explanations. We observe that LVLM’s navigation ability can be improved substantially with only BFS-generated shortest paths without any human supervision, surpassing GPT-4o by over 20% in success rates.

## introduction large vision-language models (lvlms), such as qwen2.5-vl @xcite , @xcite , and others @xcite , have demonstrated state-of-the-art performance on a variety of vision-and-language tasks. however, their ability to comprehend embodied environments and navigate within them has been less explored, primarily due to the limited diversity of scenes and objects in current navigation benchmarks. for example, matterport-3d @xcite only considers 21 types of target objects in 90 private homes, and procthor @xcite ) contains 16 object types in four kinds of rooms (i.e., bedroom, living room, kitchen, and bathroom). ## related work object navigation: the reinforcement learning has long been used for tackling object navigation tasks @xcite @xcite . however, recent studies @xcite show that rl requires extensive reward shaping, limiting its practicality. alternatively, imitation learning @xcite has also been applied to these tasks, inspiring many subsequent works @xcite @xcite . however, existing works only focus on closed-vocab object navigation tasks, mainly due to the limited diversity of current benchmarks. for example, matterport-3d @xcite only considers 21 kinds of target objects in private homes. procthor @xcite solely contains 16 target objects in four kinds of rooms. although hm3d-ovon @xcite attempts to broaden the diversity, the number of object categories is still limited to 379, and there are only 181 scenes, making it difficult to fully reflect the open-vocab nature of real-world scenarios. in constrast, we introduce divscene, a dataset comprising 5,707 categories of target objects among 22,696 object types, spanning 4,614 distinct scenes. therefore, our dataset reflects a more diverse nature of real-world settings with an open vocabulary. vlms and llms for embodiment: llms and vlms @xcite @xcite have emerged as the foundation for solving embodied tasks @xcite . a few methods have employed contrastive vlms @xcite as the visual encoders @xcite or object-grounding tools @xcite for navigation. @xcite utilizes an llm as the planning backbone with a captioning model for perception. following them, many improvements have been proposed @xcite @xcite . while these works still focus on a small set of target objects, we explore the potential of recent lvlms for open-vocabulary object navigation. images: store home leisure work public 81 types from 5 categories 1. scene type collection 2b. house description 2a. assign attributes value 1. lighting 2. objects 3. other… bright small sink … gpt-4 1. dental office 2. small sink,… a dental office with small sink and bright lighting… 4. holodeck environment building holodeck dental office add attribute gpt-4 house description episode sampling grid map floor lamp initial actions: moveahead, …, done … positions: (1.00, 2.25), … figure 2 : data collection process. on the left, we show the process of collecting scenes. we prompt gpt-4o to collect textual house descriptions and use holodeck to build houses. on the right, we show an episode built in the house, where we use bfs to find the shortest path. then, actions and observations are collected. ## task definition we study the open-vocab object navigation task involving an agent in an environment to find the target object belonging to a given category. the object categories in the open-vocabulary setting are denoted by @xmath0, c 1 , c 2 , . . .}, which is not restricted to a predefined set. the houses can be described by @xmath1, h 1 , . . . , h n }, and n is the total number of scenes. in each episode, the embodied agent is initialized at a random position p i with rotation r i in a house h i . then, the embodied agent is required to perform instance-level object navigation, where it gets a target object within the category c i at the position o i . thus, an episode can be represented as at each time step t, the embodied agent observes the environment and predicts the next action a t . following previous work @xcite , the observation comprises an rgb image of the egocentric view and the agent status (i.e., its position and rotation). our new houses are all collected on the ai2thor platform @xcite , and thus the action space, signified as a, covers: movea-head, rotateright, rotateleft, and done. by default, the moveahead action moves the agent 25 centimeters, and rotateleft (or rota-teright) rotates the agent 90 degrees. the done action is used by the agent to indicate that the navigation task is complete. when the agent takes the done action or reaches the max action limit, the episode is considered successful if the distance to the target object is below 1.5 meters. based on the actions, an environment can be discretized as a 0.25 × 0.25-meter grid map of all reachable positions. ## data collection for divscene and divscene ep existing object navigation studies only focus on limited types of scenes and objects. to fill this gap, we first curate a large-scale scene dataset divscene, featuring 4,614 scenes. then, 23k shortest-path episodes with 5,707 kinds of target objects were sampled using breadth-first search (bfs), forming the divscene ep dataset. we illustrate the details in figure 2 . ## natvlm in this section, we describe our natvlm model. it is fine-tuned from idefics 2 @xcite through imitation learning on shortest paths from divscene ep to enhance the navigation ability. at each time step t, the model processes environment observations s t and generates the next action a t in a manner consistent with instruction tuning. figure 1 provides an overview of natvlm. ## experiment we conduct extensive experiments to evaluate both off-the-shelf lvlms and llms, as well as models that have been fine-tuned using divscene ep . ## conclusion in this work, we first study the open-vocab object navigation and evaluate extensive lvlms and llms. meanwhile, we collect a large-scale scene dataset, divscene, featuring 4,614 scenes. over 22k episodes are sampled with the shortest paths by bfs. our evaluation shows that current lvlms and llms still fall short of the ability of openvocab object navigation. then, we also fine-tuned lvlms with the shortest paths through imitation learning, where we also introduced cot explanations. experiments show that the navigation ability of lvlms can be improved significantly. for future work, a promising direction is to enable lvlms to navigate over longer horizons.