HOLM : Hallucinating Objects with Language Models for Referring Expression Recognition in Partially-Observed Scenes (2022)

Abstract: AI systems embodied in the physical world face a fundamental challenge of partial observability; operating with only a limited view and knowledge of the environment. This creates challenges when AI systems try to reason about language and its relationship with the environment: objects referred to through language (e.g. giving many instructions) are not immediately visible. Actions by the AI system may be required to bring these objects in view. A good benchmark to study this challenge is Dynamic Referring Expression Recognition (dRER) task, where the goal is to find a target location by dynamically adjusting the field of view (FoV) in a partially observed 360 scenes. In this paper, we introduce HOLM, Hallucinating Objects with Language Models, to address the challenge of partial observability. HOLM uses large pre-trained language models (LMs) to infer object hallucinations for the unobserved part of the environment. Our core intuition is that if a pair of objects co-appear in an environment frequently, our usage of language should reflect this fact about the world. Based on this intuition, we prompt language models to extract knowledge about object affinities which gives us a proxy for spatial relationships of objects. Our experiments show that HOLM performs better than the state-of-the-art approaches on two datasets for dRER; allowing to study generalization for both indoor and outdoor settings.

## introduction one of the fundamental challenges in building ai systems physically present in the world is addressing the issue of partial observability, the phenomenon where the entire state of the environment is not known or available to the system. people cope with partial observability by reasoning about what is not immediately visible (see example in figure 1 ). people combine their general knowledge about the world and adapt their knowledge ↗ ↗ ↗ ↗ figure 1 : illustration of our main contribution: hallucinating objects. knowledge about object relationships is helpful when navigating in an unknown and partially observed environment. in the example above, the tv is not visible, but the couch hints that a tv might be in front of it because usually couches face tvs. to specific contexts @xcite . general knowledge about kitchens can help to know approximately where to look for pans or utensils in a kitchen that has never been seen before. how can an ai system build general knowledge about objects and their environment to help with a similar task? even more interestingly, can we gather this information from language, using readily available resources such as language models trained on a large collection of unlabeled text? in this paper, we introduce a method called holm, hallucinating objects with language models, for reasoning about the unobserved parts of the environment. inspired by the recent successes of large pre-trained language models (lm) extracting knowledge about the real world, we propose a methodology based on spatial prompts to extract knowledge from language models about object. holm extracts spatial knowledge about objects in the form of affinity scores, i.e., how often a pair of objects are observed together. this knowledge of objects are combined with observed spatial figure 2 : illustration of the drer task with an example of language instruction and its recognition in four steps. the agent adjusts its fov by looking at different directions and navigate on the graph in the spherical view. note that objects mentioned in bold in the instruction are not visible at all until timestep 4. thus, the agent needs to reason about possible locations of the mentioned object using its partial view of the scene. the drer task can be formulated as a markov decision process (mdp) (howard, 1960) @xmath0: holm for the drer task. (top) we use language models trained on a large amount of text by prompting with the spatial relationship of objects to calculate co-occurrence statistics of objects. (bottom) the flow of our hallucination method. we determine objects of interest for each action. then, we combine objects of interest and co-occurrence table to hallucinate objects, i.e. what might appear after performing an action. s, a, p s , r where s is the visual state space, a is the discrete action space 1 , p s is the unknown environment probability distribution from which the next state is drawn, and @xmath1. for a time step t, the agent observes an image s @xmath2, and performs and action a @xmath3. as a result of this action, the environment generates a new observation s t+1 ∼ p s (• | s t , a t ) as the next state. this interaction continues sequentially and ends when the agent performs a special stop action or a pre-defined maximum episode length is reached. the resolution process is successful if the agent ends the episode at the target location. the training dataset d @xmath4, t } consists of m pairs of the instruction sequence @xmath5. the agent learns to navigate by learning a policy π via maximum like-1 for computational efficiency, we picked discrete action space. it could be continuous as well. ## experiments we designed our experiments to study and evaluate our proposed holm approach under five different research questions. rq1: what is the performance of holm when compared to other state-of-the-art approaches? rq2: what is the impact of lm as a source of knowledge for holm when compared to other more conventional sources (e.g., images)? rq3: how essential are external sources of data for learning knowledge about objects compared to in domain data? rq4: how accurate is holm for predicting objects in future observations? rq5: how do annotation-free language-based knowledge sources i.e., lms and word embeddings compare for holm? the following section explains the details of experimental setup. our results are presented and discussed in section 4.2. ## related work our work on drer is closely related to previous studies focusing on referring expression recognition (rer), vision-and-language navigation (vln), and methods we propose are related to pretraining language models for vision-and-language tasks, model-based reinforcement learning, and coocccurrence modeling for computer vision. we review these studies in this section. rer is the task of localizing a target object or a point in an image described by a natural language expression. the most of existing datasets poses the task in 2d images with objects as being the target @xcite @xcite @xcite @xcite . several lines of work are proposed to address rer @xcite @xcite @xcite @xcite @xcite @xcite @xcite . in touchdown @xcite and refer360 • @xcite the target is a point not an object in a 360 • image. in the drer setup, we also use 360 • images of touchdown and refer360 • , but we do not provide the full panoramic view of the scene. instead, in a more realistic scenario, the agent observes a partial and dynamic view of the scene, i.e. the agent needs to adjust its fov to find the target location. closer to our work, in reverie @xcite an embodied setup is proposed where the agent needs to first navigate to a location where the target object is visible. similar to touchdown and refer360 • , at the final position, the full 360 • view is visible to the agent. unlike ours and similar to 2d image-based rer, the target is an object rather than a point in the scene. vln is a vision-and-language task where an agent in a simulated environment observes a visual input and is given a natural language instruction to navigate to a target location. the earlier work @xcite @xcite studies the task with synthetic images or in a very small scale @xcite . @xcite proposes room-to-room (r2r) benchmark and revisit vln task with a modern look. in r2r, the agent observes panoramic scans of a house @xcite and needs to carry out the natural language instruction. envdrop @xcite ) model shows generalization to unseen environments by dropping visual features. prevalent @xcite tackles the data sparsity problem with a pretraining scheme. @xcite show that a pre-trained multi-modal can be enhanced with a memory state for the vln task by recurrently feeding a contextualized state feature after each time step. drer also poses a navigation task where locations in physical space in vln correspond to fovs in a fixed location. in drer, a trajectory of the agent corresponds to its resolution process for finding the goal location. pre-trained models for vision-and-language has been recently studied after the huge success of transformer-based models @xcite in nlp @xcite @xcite @xcite @xcite . numerous studies extend these approaches to the multimodal domain @xcite @xcite @xcite @xcite . they achieve the-state-of-the-art results in several tasks such as image captioning, text-to-image retrieval, or referring expression recognition. our work differs from these studies in the sense that the previous approaches use large scaled paired image-text data @xcite @xcite @xcite to learn efficient representations @xcite for visual and textual modalities whereas we are interested in spatial information learned in unimodal text representations. language priors for vision were explored in recent studies. @xcite use word embeddings in a language module to learn a representation for a object-predicate-object triplet for visual relationship detection task. @xcite propose an approach to extend pre-trained transformerbased lms for multimodal tasks. similarly, @xcite ; @xcite show that pre-trained lms can be finetuned to perform well in few-shot settings for image classification and opendomain visual question answering @xcite . @xcite also show that multimodal transformer architectures capture implicit knowledge for a pair of objects. our work differs from these studies (1) we use only unimodal models, (2) we do not finetune models -we do not update models during training. the most similar work to ours, @xcite show that pre-trained lms can perform reasonably well on visual question generating @xcite out of the box. one difference is that we use object labels rather than object features or the appearance of objects to query the language model; however, they use object features as a visual token to the language model. prompts we use in our work shares similarities with prompts designed in piqa @xcite , but our work is evaluated in a multimodal setup. in con-trast, piqa is evaluated for textual commonsense reasoning tasks. hallucination idea is also related the work on predicting future observations in long horizons @xcite which has been studied in the context of learning planning @xcite and acquiring skills for control problems @xcite , and efficient policy learning @xcite , and vision-and-language navigation @xcite . all these approaches are interested in longer horizons; however, in our work, we study predicting single-step future observation. more recent work @xcite @xcite study view synthesis from a single visual observation. unlike these approaches, holm does not generate pixel-level views rather abstractions of views with object labels. affinity scores are mainly studied in computer vision tasks in the form of object cooccurrences. previous studies have shown that object co-occurrences are efficient representations of visual prior for object categorization for object segmentation @xcite @xcite and zero shot object-recognition @xcite , and scene understanding @xcite . our work differs from these studies: we do not calculate co-occurrence statistics, i.e. we do not count the frequency of times they appear together; instead, we calculate a probability measure using language models. ## conclusion in this paper, we introduced holm -a model that can extract prior knowledge about objects from lms and hallucinate objects in future observations. our experiments showed that holm approach improves over various baselines from the literature. surprisingly, our model which used background knowledge from lms outperformed models with knowledge from human-annotated data showing that lms learn useful knowledge about the world without requiring any visual observations. we also showed that out approach generalizes to both indoor and outdoor scenarios. our work has limitations in the following ways. first, the hallucination process solely conditions on the current field of view. however, the instruction and the previous observations are available to the system. conditioning on these sources of infor-mation could improve the hallucination accuracy by getting more targeted information from the language model. second, we assume a fixed lexicon of object labels for hallucination. for both the visual side i.e., the object detector, and the language side i.e., the language model, when an unknown object appears the system cannot use this object for hallucination. another issue is the scalability, i.e, the affinity scores scale with o(n 2 ) where n is the number of objects, which might be challenging when n is large. we hope the follow-up work could address these limitations. future work will explore the use of background knowledge in other domains such as vision-andlanguage navigation @xcite and dialog @xcite . we also believe background knowledge of objects would be handy in complex scenarios such as manipulating objects in a simulated environment @xcite . our method examines extracting background knowledge in a zero-shot manner. however, the literature shows that learning how to prompt could be helpful in finding better @xcite . we strictly compared unimodal approaches for hallucination. future work extend our work by comparing multimodal models @xcite @xcite @xcite @xcite . another interesting direction would be to study the capability of transferring knowledge from indoor to outdoor settings and vise versa. finally, the success of prevalent @xcite and other pre-training approaches for vln could stem from their ability to implicitly encode prior knowledge about objects. hopefully, future studies examines this phenomenon.