Learning to Generate Unambiguous Spatial Referring Expressions for RealWorld Environments Referring to objects in a natural and unambiguous manner is crucial for effective humanrobot interaction. Previous research on learningbased referring expressions has focused primarily on comprehension tasks, while generating referring expressions is still mostly limited to rulebased methods. In this work, we propose a twostage approach that relies on deep learning for estimating spatial relations to describe an object naturally and unambiguously with a referring expression. We compare our method to the state of the art algorithm in ambiguous environments (e.g., environments that include very similar objects with similar relationships). We show that our method generates referring expressions that people find to be more accurate (30% better) and would prefer to use (32% more often).

Grounding Language Attributes to Objects using Bayesian Eigenobjects We develop a system to disambiguate object instances within the same class based on simple physical descriptions. The system takes as input a natural language phrase and a depth image containing a segmented object and predicts how similar the observed object is to the object described by the phrase. Our system is designed to learn from only a small amount of humanlabeled language data and generalize to viewpoints not represented in the languageannotated depth image training set. By decoupling 3D shape representation from language representation, this method is able to ground language to novel objects using a small amount of languageannotated depthdata and a larger corpus of unlabeled 3D object meshes, even when these objects are partially observed from unusual viewpoints. Our system is able to disambiguate between novel objects, observed via depth images, based on natural language descriptions. Our method also enables viewpoint transfer; trained on humanannotated data on a small set of depth images captured from frontal viewpoints, our system successfully predicted object attributes from rear views despite having no such depth images in its training set. Finally, we demonstrate our approach on a Baxter robot, enabling it to pick specific objects based on humanprovided natural language descriptions.