GOAT: GO to Any Thing

• Proposes GOAT (Go to anything): a universal navigation system with multimodal goal query (label, image, language description), lifelong (past experiences, episodic memory from previous goals), platform/robot agnostic; has instance-aware semantic memory. CLIP for image-language matching, SuperGlue for image-image matching. Store nodes of object instances with images and location in a top-down map of the environment; if query found in this: point-nav setting, else: exploration goal.
• Better than CLIP on Wheels (COW) for success rate and SPL (Success weighed by Path Length - agent’s path length over optimal path length) metrics on visually diverse homes (custom dataset) using Spot quadruped and Stretch; GOAT without memory (reset semantic map every query episode) is still better than COW (with memory, it improves further).
  • Benefits from: Modular system, SuperGlue keypoint correspondences giving geometric verification for image-to-image matching, filtering goal categories (match only within goal category).
• Agent takes RGB-D and pose, perception stack contains instance segmentation, depth estimation, geometric projection, and dynamic instance mapping; semantic map and object instance memory are agent state; global policy takes this, goal, and generates long-term goal (in the map); local policy takes the map and gives actions.
  • MaskRCNN (ResNet backbone, MS-COCO) for object detection and instance segmentation on RGB; fill holes by predicting depth (MiDaS), solving least squares for scale and position offset (with using actual depth readings), voxelize semantic map and project to BEV (top-view); got a KxMxM semantic map representation (K is object classes/categories + obstacles, explored area, current, and past locations) for each timestamp.
  • Object instance memory is through object (instance) locations in (semantic) map and storing images (with local matching done).
• Global policy for getting to goal: Category goals are checked in semantic map; language goals go through Mistral-7B and then CLIP features (language-image) are matched in Object Instance Memory; image goals are matched by category and then SuperGlue matching; if found in memory, then use the exact location as point navigation goal, else use frontier-based exploration (explore closest unexplored region).
  • Local policy is Fast Marching Method (waypoint controller in Spot, custom controller for Stretch). Supplementary material has instance matching strategy selection (CLIP vs SuperGlue, thresholds for matching, category filtering, etc.), quantitative and qualitative results for the same.

From UIUC (Saurabh Gupta), CMU, Georgia Tech, UC Berkeley (Jitendra Malik), Meta (Dhruv Batra) and Mistral (DS Chaplot).

Links: arxiv, website, GitHub