Where Are You? Localization from Embodied Dialog (2020)

Abstract: We present WHERE ARE YOU? (WAY), a dataset of ~6k dialogs in which two humans – an Observer and a Locator – complete a cooperative localization task. The Observer is spawned at random in a 3D environment and can navigate from first-person views while answering questions from the Locator. The Locator must localize the Observer in a detailed top-down map by asking questions and giving instructions. Based on this dataset, we define three challenging tasks: Localization from Embodied Dialog or LED (localizing the Observer from dialog history), Embodied Visual Dialog (modeling the Observer), and Cooperative Localization (modeling both agents). In this paper, we focus on the LED task – providing a strong baseline model with detailed ablations characterizing both dataset biases and the importance of various modeling choices. Our best model achieves 32.7% success at identifying the Observer’s location within 3m in unseen buildings, vs. 70.4% for human Locators.

## introduction imagine getting lost in a new building while trying to visit a friend who lives or works there. unsure of exactly where you are, you call your friend and start describing your surroundings ('i'm standing near a big blue couch in what looks like a lounge. there are a set of wooden double doors opposite the entrance.') and navigating in response to their questions ('if you go through those doors, are you in a hallway with a workout room to the right?'). after a few rounds of dialog, your friend who is familiar with the building will hopefully know your location. success at this cooperative task requires goal-driven questioning based on your friend's understanding of the environment, unambiguous answers communicating observations via language, ## related work image-based dialog several datasets grounding goal-oriented dialog in natural images have been proposed. the most similar settings to ours are cooperative visual dialog @xcite , in which a question agent (q-bot) attempts to guess which image from a provided set the answer agent (a-bot) is looking at, and guesswhat?! @xcite , in which the state estimation problem is to locate an unknown object in the image. our dataset extends these settings to a situated 3d environment allowing for active perception and navigation on behalf of the a-bot (observer), and offering a whole-building state space for the q-bot (locator) to reason about. ## where are you? dataset we present the where are you? (way) dataset consisting of 6,154 human embodied localization dialogs across 87 unique indoor environments. environments. we build way on matterport3d @xcite , which contains 90 buildings captured in 10,800 panoramic images. each building is also provided as a reconstructed 3d textured mesh. this dataset provides high-fidelity visual environments in diverse settings including offices, homes, and museums -offering numerous objects to reference in localization dialogs. we use the matterport3d simulator @xcite to enable first-person navigation between panoramas. task. a way episode is defined by a starting location (i.e. a panorama p) in an environment e. the observer is spawned at p 0 in e and the locator is provided a top-down map of e (see fig. 1 ). starting with the locator, the two engage in a turn-based dialog (l 0 , o 0 , . . . l t -1 , o t -1 ) where each can pass one message per turn. the observer may move around in the environment during their turn, resulting in a trajectory (p 0 , p 1 , . . . , p t ) over the dialog. the locator is not embodied and does not move but can look at the different floors of the house at multiple angles. the dialog continues until the locator uses their turn to make a prediction (p t ) of the observer's current location (p t ). the episode is successful if the prediction is within k meters of the true final position -i.e. ||p t -pt || 2 < k m. this does not depend on the initial position, encouraging movement to easily-discriminable locations. map representation. the locator is shown topdown views of matterport textured meshes as environment maps. in order to increase the visibility of walls in the map (which may be mentioned by the observer), we render views using perspective rather than orthographic projections (see left in fig. 1 ). we set the camera near and far clipping planes to render single floors such that multi-story buildings contain an image for each floor. ## modeling localization from embodied dialog while the way dataset supports multiple tasks, we focus on localization from embodied dialog as a first step. in led, the goal is to predict the location of the observer given a dialog exchange. ## conclusion and future work in summary, we propose a new set of embodied localization tasks: localization from embodied dialog -led (localizing the observer from dialog history), embodied visual dialog -evd (modeling the observer), and cooperative localization -cl (modeling both agents). to support these tasks we introduce where are you? a dataset containing ∼6k human dialogs from a cooperative localization scenario in a 3d environment. way is the first dataset to present extensive human dialog for an embodied localization task. on the led task we show that a lingunet-skip model improves over simple baselines and model ablations but without taking full advantage of the second half of the dialog. since way encapsulates multiple embodied localization tasks, there remains much to be explored.