Refer360 ^{\circ} : A Referring Expression Recognition Dataset in 360 ^{\circ} Images (2020)

Abstract: We propose a novel large-scale referring expression recognition dataset, Refer360°, consisting of 17,137 instruction sequences and ground-truth actions for completing these instructions in 360° scenes. Refer360° differs from existing related datasets in three ways. First, we propose a more realistic scenario where instructors and the followers have partial, yet dynamic, views of the scene – followers continuously modify their field-of-view (FoV) while interpreting instructions that specify a final target location. Second, instructions to find the target location consist of multiple steps for followers who will start at random FoVs. As a result, intermediate instructions are strongly grounded in object references, and followers must identify intermediate FoVs to find the final target location correctly. Third, the target locations are neither restricted to predefined objects nor chosen by annotators; instead, they are distributed randomly across scenes. This “point anywhere” approach leads to more linguistically complex instructions, as shown in our analyses. Our examination of the dataset shows that Refer360° manifests linguistically rich phenomena in a language grounding task that poses novel challenges for computational modeling of language, vision, and navigation.

## introduction imagine a scenario in which you are asked to retrieve medication from a bathroom. 'first, face the sink, then find the second drawer in the cabinet to your left. the pills should be inside that drawer behind the toothbrush." interpreting instruction sequences in order to locate targets in novel environments is challenging for ai systems (e.g. personal robots and self-driving cars). first, the system needs to ground the instructions into visual perception @xcite look towards the door leading outside the cafe. to study these challenges, we introduce a novel dataset, named refer360°1, for the task of localizing a target in 360°scenes given a sequence of instructions. figure 1 presents an example scenario partial fov (a) an example scene from the refer360°dataset. note that both annotators and systems cannot observe the shaded area. they only observe a partial field of view which can be updated dynamically. in refer360°, the target location could be any random location on the image. in (b), annotators chose an existing object as the target location. in (c), boxes for objects were used as targets. refer360°also seeks to increase the complexity of instruction following, making it more realistic by introducing a partial and dynamic fov rather than providing a holistic oracle-like view of the image. from refer360°. for this scenario, finding the target location requires first finding the door leading outside, then looking at the coffee pot, and finally finding the trash can, which is the nearest object to the target. here, instructions are given from the perspective of a partial field of view (fov) of the scene, and these fovs can dynamically be changed. thus, the correct interpretation of the sequence of instructions will require reasoning about what is currently visible in the fov (e.g., grounding of objects) but also what is not visible yet. these scenarios will often require adjusting the fov based on intermediate instructions. an important feature of the refer360°dataset is that the target location is not an object; instead, it can be any point in the scene, which makes the grounding task more challenging since it is harder to describe a location when we cannot readily refer to it with the name of an object. refer360°consists of 17,137 instruction sequences with ground-truth actions to complete these instructions in 360°scenes. refer360°has some unique characteristics which differentiate it from prior work. first, refer360°allows the scene to be viewed through a partial fov that can be dynamically changed as instructions are followed. this is in contrast with existing 360°scene-based datasets such as touchdown-sdr @xcite and 2d image-based referring expression datasets @xcite @xcite , where the visual input is either fixed, corresponding to a holistic, oracle-like view, or consists of fixed, cardinal fovs. the partial and dynamic fov in refer360°poses new challenges for language grounding (see figure 2a , 2b , and 2c for an illustrative comparison). for instance, the mentioned objects may not be visible in the current fov, and language may refer to the fov itself. further, since our annotators generate instructions while observing a partial and dynamic fov, and do so for a follower whose first fov will be initially located at random, the instruction following task is strongly sequential. to interpret the sequence of instructions to find the target correctly, a follower must reason about the sequence of fovs referenced by the instructor. ## motivation the vision behind refer360°is to build systems that perform localization of any point in 3d space, bringing us closer to human-like reasoning. this is an important milestone towards better collaboration between ai systems (e.g. personal robots) and humans, allowing them to act within the same space. it might also pave the way for ai-agents interacting with virtual worlds. the refer360°dataset was designed to address three technical challenges towards this vision. first, learning environments we create need to reflect the characteristics of human's perception of 3d space. in such an environment, the agent only observes a partial fov of the scene. this requires adjusting the fov in accordance with instructions so that current view and instructions are aligned. the agent's fov can be changed in a continuous manner, moving smoothly left, right, up, and down. this is analogous to a real-world robot performing motor actions to change its camera position, or a human changing their head's pitch and yaw. further, real scenes are 3d, but the fov is represented in 2d in our task. thus, interpreting some instructions will require inferences about depth. second, the paradigm of 360°scenes with par-tial fov will almost always necessitate instructions that consist of multiple intermediate steps. as the first intermediate step, the follower and instructor need to find a common referential fov. then, the instructor can continue giving guidance towards the target location, often by identifying objects that are physically related to the target location. this multistep process can serve as a natural benchmark for measuring whether systems achieve localization through a human-like process of progressively getting closer to the target location by interpreting intermediate steps. in other words, this setup may helps researchers make sure that our systems are arriving at the referred location for the right reasons. third, since any point in the scene could be of interest, instructions will be more complex: many points in the scene will not correspond to easily named objects, and thus, when such points are allowed as targets, more sophisticated instructions will be required to unambiguously refer to them. the instructor may rely on description of physical relationships with the closest easily named locations in the scene @xcite @xcite . for instance, in figure 2a , the target location is on the side of a trash bin, which is difficult to unique describe with a single word or a short phrase. in this case, the instructor may use the distance to the floor or to another object in the scene in order to describe the exact location of the target. this will additionally introduce description of degree (e.g. 'slightly above', 'a few inches away from') rather than more discrete spatial relationships (e.g. 'on top of the desk'). ## related work referring expression recognition. grounding a short phrase or a sentence into a visual modality such as video @xcite or imagery @xcite @xcite ) is a well studied problem in intelligent user interfaces @xcite , human-robot interaction @xcite @xcite , and situated dialogue @xcite . @xcite , @xcite , and @xcite introduce two benchmark datasets for the real-world 2d images. @xcite propose a model where the target and supporting objects (i.e. objects that are mentioned in order to disambiguate the target object) are identified and scored jointly. @xcite introduce a compositional approach where they assume that the referring expression can be decomposed into a triplet consisting of the target object, the supporting object, and their spatial relationship. similarly, @xcite propose a type of neural modular network @xcite where the grounding of referring expression depends on the parse tree of the input referring expression to learn to ground an unconstrained number of supporting objects. ## refer360°dataset in this section, we describe the details of the re-fer360°dataset, a vision-and-language benchmark for localizing a target point in a panoramic image. refer360°consists of 17,137 instruction sequences that describe randomly distributed target locations in 2,000 panoramic scenes from the sun360 @xcite dataset. we first explain the annotation procedure for collecting and validating the instruction sequences. later, we discuss the statistics of the refer360°dataset. ## analyses we conduct four analyses of the refer360°dataset. first, we investigate if the random selection process of target locations can mitigate possible bias issues. recent studies @xcite @xcite show that design decisions for collecting annotations may introduce bias into datasets. high-capacity machine learning models can exploit these issues which hinders the meaningful progress towards real language understanding @xcite . second, we study whether each instruction in an instruction sequence is critical in finding the target location, or whether some instruction sequences are overcomplete. it may be very well the case that, by just understanding the last instruction, one can easily locate the target location. third, we perform a qualitative analysis of refer360°to provide the types of linguistic reasoning required to find the target location accurately. finally, we analyze the performance of the state-ofthe-art on refer360°. ## conclusion we designed refer360°to study 3d spatial language understanding for real scenes. we collected a fine-grained set of annotations that support study at many levels of language grounding. refer360°is a versatile dataset and enables investigation along three axes: in our experiments, we presented one of these scenarios (single instruction, static, and pixellevel) since it was the closest to the pre-existing touchdown-sdr system. however, one can also study a much larger number of scenarios and modeling tasks using refer360°.