O pen FMN av: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models (2024)

Abstract: Object navigation (ObjectNav) requires an agent to navigate through unseen environments to find queried objects. Many previous methods attempted to solve this task by relying on supervised or reinforcement learning, where they are trained on limited household datasets with close-set objects. However, two key challenges are unsolved: understanding free-form natural language instructions that demand open-set objects, and generalizing to new environments in a zero-shot manner. Aiming to solve the two challenges, in this paper, we propose **OpenFMNav**, an **Open**-set **F**oundation **M**odel based framework for zero-shot object **Nav**igation. We first unleash the reasoning abilities of large language models (LLMs) to extract proposed objects from natural language instructions that meet the user’s demand. We then leverage the generalizability of large vision language models (VLMs) to actively discover and detect candidate objects from the scene, building a *Versatile Semantic Score Map (VSSM)*. Then, by conducting common sense reasoning on *VSSM*, our method can perform effective language-guided exploration and exploitation of the scene and finally reach the goal. By leveraging the reasoning and generalizing abilities of foundation models, our method can understand free-form human instructions and perform effective open-set zero-shot navigation in diverse environments. Extensive experiments on the HM3D ObjectNav benchmark show that our method surpasses all the strong baselines on all metrics, proving our method’s effectiveness. Furthermore, we perform real robot demonstrations to validate our method’s open-set-ness and generalizability to real-world environments.

## introduction as a fundamental task in robotics and embodied ai, object navigation requires an agent to navi1 we show further information and demo videos on https://yxkryptonite.github.io/openfmnav/ . find the bed. find the bed with the blue mattress next to the window. i'm exhausted. i need to lie down and rest. ## navigation in the real world we further conduct real robot demonstrations to show our method's ability to understand free-form natural language instructions and perform open-set zero-shot navigation in the real world. ## conclusions in this paper, we presented a novel framework, openfmnav, for open-set zero-shot object navigation. by leveraging foundation models, our method could understand free-form natural language instructions, conduct reasoning, and perform effective zero-shot object navigation. extensive experiments showed the superiority of our framework. finally, we conducted real robot demonstrations to validate our method's open-set-ness and generalizability to real-world environments.