Robust and Interpretable Grounding of Spatial References with Relation Networks (2020)

Abstract: Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations. In this paper, we develop effective models for understanding spatial references in text that are robust and interpretable, without sacrificing performance. We design a text-conditioned relation network whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. This design choice provides interpretability of learned intermediate outputs. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17% improvement in predicting goal locations and a 15% improvement in robustness compared to state-of-the-art systems.

## introduction grounding spatial references in text is essential for effective human-machine communication through natural language. spatial reasoning is ubiquitous in many scenarios such as autonomous navigation @xcite , situated dialog @xcite and robotic manipulation @xcite . despite tremendous applicability, understanding spatial references is a highly challenging task for current natural language processing (nlp) systems, requiring a solid contextual understanding of language dependent on other observations from the environment. figure 1 demonstrates two tasks where the interpretation of the instruction or statement changes completely with the observation provided. in the first example, the westernmost circle may lie to the left or right of the navigating agent's starting location (red dots). in the second, the validity of the statement depends on the relative orientation of the relevant objects -the triangle and the pentagon. some of the earliest work in this field @xcite investigated the grounding of spatial prepositions (e.g., "above", "below") to perceptual processes like visual signals. such early grounding efforts were limited by computational bottlenecks but several deep neural architectures have been recently proposed that jointly process text and visual input @xcite @xcite @xcite @xcite . while these approaches have made significant advances in improving the ability of agents at following spatial instructions, they are either not easily interpretable or require pre-specified parameterization to induce interpretable modules @xcite . moreover, their end-to-end formulations and lack of explicit entity reasoning make them susceptible to noise and perturbations in the input data, as we demonstrate in our experiments (section 5). in this paper, we develop a model to perform robust and interpretable grounding of spatial references in text. our objective is to understand and analyze how to ground spatial concepts in a robust and interpretable way. in particular, we focus on the class of deictic spatial references @xcite , which specify a location or object in terms of one or more reference objects. our key idea is to decompose the spatial reasoning process into two important steps: (1) identifying the reference object(s) (e.g., rock, circle) from the instructions, and (2) accurately inferring the spatial direction (e.g., left, top-right) of the goal with reference to that object. we use a relation network @xcite to implicitly enable this factorization by computing representations for each location in the environment based on its interactions with neighboring entities. the parameters of the relation network are dynamically derived from a vector representation of the input text followed by an attention module conditioned on the observations of the environment. this architecture provides three key benefits: (1) the dynamically computed parameters of the relation network enable fine-grained modeling of spatial references, (2) since the model considers relations between pairs of entities, it is more robust to noisy inputs, and (3) the explicit multi-modal representations learned for each entity pair are highly interpretable. we empirically test our model on three different task settings -classification, value map regression and reinforcement learning (rl) for navigation, and compare its performance to existing state-ofthe-art methods. we find that our approach is competitive with or outperforms the baselines under several different evaluation metrics. for example, in the navigation task with rl, our model obtains up to 13.5% relative improvement in policy quality over the best performing baseline. our approach is also more robust to noisy inputs -for instance, after adding unseen objects as noise to a value map regression task, our model's performance degrades by only around 10% compared to over 20% for the best baseline. finally, we also present several visualizations of relation and value maps produced by the model, which demonstrate appropriate grounding of reference objects as well as spatial words. ## related work the role of language in spatial reasoning has been explored since the 1980s @xcite @xcite . most early papers dealt with the question of representing spatial prepositions @xcite @xcite and grounding them to spatial templates @xcite . @xcite introduced the influential attention vector-sum model which accurately predicted human spatial judgements for words like "above" and "below". the use of neural networks to computationally ground spatial terms to geometric orientations was first explored by @xcite and later by @xcite . while spatial reasoning in general is a wide-ranging problem, in this paper, we focus on grounding deictic spatial references in third person, which involve referring to a goal location using one or more referent objects @xcite . ## experimental setup tasks. we perform several empirical studies and compare our model with prior work in terms of accuracy and robustness. as previously mentioned, we focus on deictic spatial references, which involve talking about a location or object in terms of other referent objects. we consider three different prediction tasks with accompanying text. these tasks all involve joint reasoning over both observations and the text in order for a system to perform well. the tasks are as follows: the aim is to produce the optimal value function map with a value v (s) for each location in the map with respect to the goal. for this task, we use two recently proposed instruction following datasets -isi @xcite and puddleworld (janner et al., 2018). isi contains a set of blocks, each with a different company logo or number, with the instruction being to move a particular block to a desired location. puddleworld (pw) is a navigation environment with a positive reward for reaching the goal and negative rewards for stepping in puddles. pw consists of local instructions with local neighborhoods in references e.g., "two cells to the left of the triangle" and global instructions with description about the entire map e.g., "the westernmost rock." the ground truth value maps for each instance are obtained using the value iteration algorithm @xcite . these three tasks cover different application settings for spatial reasoning. while the prediction objective is different in each one, the input observations are also varied -shapeworld uses pixels, while isi and puddleworld are grid worlds. all three are fully observable 2-d environments, and do not contain first-person points of view or certain kinds of spatial references such as intrinsic relations @xcite . however, these environments are sufficient to demonstrate the accuracy, robustness and interpretability of our approach and there is nothing inherently preventing the model from generalizing to more scenarios (e.g., 3-d, different points of view, etc.). more statistics on the datasets including train-test splits are provided in evaluation metrics. we use several different quantitative metrics to evaluate the models: 1. accuracy (acc) of predictions for the binary classification task. 2. mean square error (mse) between the predicted and ground truth value maps for the regression task. 3. policy quality (pq), which is a normalized reward score obtained by the agent's policy compared to the optimal policy @xcite . 4. manhattan distance (md) which measures the distance between the agent's final position and the ground truth goal location. the last two measures (pq and md) are naturally applicable to the navigation task with rl, but we also apply them to the regression task by inducing a policy from the predicted value map as π(s) = arg max a r(s, a) + γt (s |s, a) ṽ (s ). baselines. for binary classification on shape-world, we use the text-vgg net (t-vgg) from @xcite , which contains a convolution network and two dense layers of size (512, 512) with tanh activation functions, followed by a softmax layer. for the other two tasks, we compare with a text-conditioned universal value function approximator (uvfa) @xcite , and the text-conditioned cnn (t-cnn) architecture of @xcite which has been shown to obtain state-of-the-art performance on the pw and isi datasets. both models learn multi-modal representations by either concatenating text and observation (a) observational noise (b) textual noise vectors or using text vectors as a kernel in a convolution operation over the observation. further details on the models are in appendix a. ## conclusion we have presented an approach to learn robust and interpretable models for handling spatial references in text. we use a text-conditioned relation network to capture fine-grained spatial concepts between entity pairs, with dynamically computed weights using a cross-modal attention layer. our empirical experiments over various domains demonstrate that our model matches or outperforms existing state-ofthe-art systems on several metrics, e.g. achieving up to 16.7% improvement in goal localization error. further, we show that our approach is more robust to noise compared to the baselines, in terms of both unseen objects (observational noise) and randomly injected words (textual noise). finally, we demonstrate that our model's intermediate representations provide a way to interpret its predictions. future research can explore other types of spatial relations as well as techniques to scale relation networks to larger observations spaces (e.g., egocentric vision tasks) in a computationally efficient manner. in addition, our approach can be readily extended to more scenarios by considering longerrange cells and incorporating the object detector to extract the object in the scene. (a) (b) figure 8: examples in isi. the goal is to place the block in the right position specified by the instructions. put the code on the designated folder. finally, go to the example folder and execute the code using python command.