Modeling Semantics and Pragmatics of Spatial Prepositions via Hierarchical Common-Sense Primitives (2021)

Abstract: Understanding spatial expressions and using them appropriately is necessary for seamless and natural human-machine interaction. However, capturing the semantics and appropriate usage of spatial prepositions is notoriously difficult, because of their vagueness and polysemy. Although modern data-driven approaches are good at capturing statistical regularities in the usage, they usually require substantial sample sizes, often do not generalize well to unseen instances and, most importantly, their structure is essentially opaque to analysis, which makes diagnosing problems and understanding their reasoning process difficult. In this work, we discuss our attempt at modeling spatial senses of prepositions in English using a combination of rule-based and statistical learning approaches. Each preposition model is implemented as a tree where each node computes certain intuitive relations associated with the preposition, with the root computing the final value of the prepositional relation itself. The models operate on a set of artificial 3D “room world” environments, designed in Blender, taking the scene itself as an input. We also discuss our annotation framework used to collect human judgments employed in the model training. Both our factored models and black-box baseline models perform quite well, but the factored models will enable reasoned explanations of spatial relation judgements.

## introduction prepositions in general and spatial prepositions in particular form a notoriously difficult lexical class because of their inherent vagueness and polysemy. pragmatics plays crucial role in determining both which prepositions are licensed for usage in a given situation and the range of configurations (i.e., locations of the arguments) of which the licensed preposition holds true. spatial senses of prepositions are sensitive to miscellaneous factors such as shapes and salience of the argument objects, presence of meronymy (part-of) relations, typicality, etc. on provides a good example of such a semantically rich preposition. when we say that one object is on another one, we strongly imply the relation of physical support between them. but support relation comes in many forms and occurs in diverse physical configurations: a) an apple on the table b) a book on the shelf c) a picture on the wall d) a fly on the ceiling e) a shirt on the person f) a lamp on the post g) a fish on a hook h) a sail on a ship such variety makes capturing the meaning in a computational model difficult. yet, locative expressions involving prepositions are pervasive in natural languages and, therefore, interpretation and understanding of their meaning is important for ai, especially in use cases involving grounded human-machine interactions. another important requirement for modern ai systems is interpretability and explainability. while neural networks can efficiently learn complex statistical distributions from large datasets, they are predominantly opaque from the common-sense analysis perspective. our approach to computational models for spatial prepositions is based on the following considerations. to begin with, even though the range of senses of spatial relations together with the heavy dependence on pragmatic considerations make capturing their meaning with simple mathematical criteria difficult, it is still possible to account for many of the above aspects in a principled way. people's judgments about whether a particular relation holds in a given case can be quite variable; therefore it should suffice to provide models that estimate the probability that arbitrary judges would consider the relation to hold. this approach is aligned with a view of predicate vagueness as variability in applicability judgments @xcite , enabling bayesian interpretation. next, since the usage of locative expressions is pragmatic, the ultimate success criterion in assessing models of prepositional predicates should also be pragmatic; i.e, in physical settings we often use such predicates to identify a referent (the blue book in front of the laptop) or to specify a goal (put the laptop on the table), so our models should allow a natural language system to interpret such usages as a human would. last, but not least, our approach facilitates explainability. each relation is built from a combination of simpler relations, whose value can be retrieved and used to provide a justification for a particular judgement. for example, in order for one object to be next to another, they need to be close to each other and at about the same elevation. thus, the latter criteria are included as factors in determining the value of the next-to relation, and their values could be used to generate meaningful explanations for any particular judgement made by the model. in the following sections, we discuss related work, and then outline our modeling framework by examining the primitive concepts that are used as building blocks, and showing how these concepts come together in modeling a specific preposition. we then evaluate our approach in a "room world" domain, making use of blender graphics software. we discuss two different sets of models, one purely neural network-based, implemented as a collection of multi-layer perceptrons, and another where models are implemented as trees, where each node computes a probabilistic rule. we describe our annotation framework for collecting human spatial judgments and evaluate our models. we summarize our contributions, and directions for future work, in the concluding section. ## related work in what follows, the first and second arguments of a preposition are referred to as figure and ground, respectively, when used in locative settings @xcite . the 3d approach to modeling spatial relations, as opposed to modeling based on 2d images, is informed by the cognitive science perspective. it is likely that people conceptualize their immediate surroundings as a 3d space defined by the three principal orientation axes of the body @xcite . moreover, 2d map-like space representations employed in navigation can be easily computed from a 3d "mental image" of the environment. it seems reasonable to assume that a potential embodied agent, such as robot, would also benefit from constructing such 3d "mental images" of its surroundings. indoor scenarios for spatial modeling are particularly conducive to such approach @xcite . developing computational models for spatial prepositions is a long-standing problem in the field of computational linguistics and nlp, and the attempts date back to the late 1960s. early work followed mainly geometric intuitions, relying on the concepts of contiguity, surface, etc. @xcite . a very good review of the semantic and pragmatic issues involved in spatial expressions is contained in @xcite . herskovits' analysis identified a variety of important factors that influence correctness judgments in the application of spatial prepositions, illustrating these factors with many striking examples (e.g., the role of object types and typicality in contrasts such as the house on the lake vs. *the truck on the lake, or the role of the figure/ground distinction and object size and type in contrasts like the bicycle is near mary's house vs. ?mary's house is near the bicycle). herskovits also proposed various abstract principles constraining the meaning and use of spatial prepositions. our work borrows many of the elements of herskovits' analysis, but is more narrowly focused on application to a particular setting (the room world), and is distinguished by our emphasis on developing computational models capable of actually evaluating the truth of prepositional relations in the chosen domain. a number of methodologies rooted in application of topological notions to defining semantics of spatial prepositions arose aiming at spatial reasoning using abstract qualitative primitives to encode relations between objects @xcite . one example of such an approach is the region connection calculus (rcc) and its modifications @xcite . at the heart of rcc lies the notion of connectedness. two nonempty regions are con-nected if and only if their topological closures have a nonempty intersection. starting with this primitive, one may proceed to define more useful spatial relations such as part-of (x is a part of y if every object that is connected to x is also connected to y) and overlapping (x and y overlap if there is a z that is a part of both x and y). continuing in the same fashion one can define several other topological notions and then use them to describe spatial configurations of objects. while mathematically appealing and facilitating rigorous inference, these qualitative methods are too strict and unable to capture the semantic richness of natural language descriptions of spatial configurations of objects, since they neglect aspects such as orientation, size, shape, and argument types. conceptually, the way we define the spatial relations in our model is similar to the spatial template approach, discussed in @xcite . this approach is based on the idea of defining a region of acceptability around the reference object that captures the typical locations of the relatum for this relation and determining how well the actual relatum fits this region. our work is also similar in spirit and goals to the work by @xcite , which combined the imagistic space representations with spatial templates and applied it to a story understanding task. in their approach, the authors used explicit blender graphics modeling of a scene to represent the objects in question and their relative configurations. in their model, each region of acceptability is a three dimensional rectangular region (more precisely, a prism with a rectangular base) representing the set of points for which the given spatial relation holds. for example if one has a pair of two objects, a and b, and wants to determine whether a is on top of b, a is checked to determine whether it is in the region of acceptability located directly above b. probabilistic reasoning is supported by using values from 0 to 1 to represent the portion of the relatum that falls into a particular region of acceptability. in recent years, attempts have been made to use statistical learning models, especially deep neural networks, to learn spatial relations. the work by @xcite is concerned with learning to transduce verbal instructions, e.g., "move the mcdonald's block so it's just to the right (not touching) the twitter block" into block displacements in a simulated environment. this system, unlike ours, relies on deep learning and does not use high-level cognitively-motivated spatial relation models. the clevr dataset @xcite and its modified versions, such as @xcite , lays out an explicit spatial question answering challenge that has inspired a flurry of visual reasoning works, e.g., @xcite and @xcite , which achieves near-perfect scores on the clevr questions. common shortcomings of these approaches are reliance on synthetic data of limited variety (only a few simple geometric shapes are present), two-dimensional image-based model of the world, very limited ground-truth models of spatial relations (e.g., left means any amount laterally to the left, regardless of depth or intervening objects, etc.), and use of domain-specific procedural formalisms for linguistic semantics. other noteworthy recent examples of datasetdriven work are @xcite and @xcite . the former inverts the learning problem, in a sense; the task was not to learn how to describe object relationships, but rather to automatically generate a scene based on a textual description. the latter employed models of spatial relations to locate and identify similar objects in several video streams. we should separately mention the spatial modelling studies by @xcite and, especially, @xcite , which apply deep neural networks to learning spatial templates for triplets of form @xcite . the latter work does this in an implicit setting, that is, it uses relations that indirectly suggest certain spatial configurations, e.g., (person, rides, horse). their model is capable not only of learning a spatial template for specific arguments but also of generalizing that template to previously unseen objects; e.g., it can infer the template for (person, rides, elephant). these approaches, however, rely on the analysis of 2d images rather than attempting to model relations in an explicitly represented 3d world. our approach can be seen as an attempt at quantitative implementations inspired by the criteria that have been discussed in psychologically and linguistically oriented studies @xcite @xcite . studies of human judgements of spatial relations show that overly formal qualitative models with sharp boundaries generally cannot do justice to the usage of locative expressions in natural settings. we previously mentioned a study @xcite that applied 3d graphics scene modeling to a story understanding task, allowing reasoning about the relative configuration and visibility of objects in the scene. another example of an imagistic reasoning system was implemented as part of the planning system for the robot ripley @xcite . ripley used three-dimensional representation of its body, operator and workspace, reconstructed from two-dimensional view coming from ripley's cameras. our work is very similar in spirit and execution to @xcite and @xcite . all these studies model prepositions using specially designed 3d environments in blender or unity and employ similar sets of metrics to define the meaning of the prepositions. the studies by platonov & schubert differ from the present work in that the rules were less flexible (fewer parameters) and parameter values were hand-adjusted, while in our work we use gradient descent-based optimization to learn optimal values. the studies by richard-bollans ete al. relied on the prototype and exemplar approaches, using learning from data to estimate the prototype parameters or the exemplar configuration. our work is, by contrast, rule-based (although one might argue that the parameters in our rules implicitly encode prototype properties). none of the prior studies explore generation of justifications for the spatial judgements. ## task description we explore spatial prepositions as applied to the socalled "room worlds" -3d scenes depicting room interiors filled with common everyday items such as furniture, appliances, food items, etc. the objects in the scene are designed in a particular way, so that their meronomy corresponds to that of the real objects. that is, the mesh consists of parts that are usually distinguished by people (e.g., for a chair, its seat, legs, back, ets., are separate objects that can be accessed by our system). this is useful for part-based inferences, e.g., a book is on a bookshelf when it is on one of the shelves. the objects are also annotated with other additional tags such as frontal vectors that indicate where the "front" of an object is, object type, etc. we have designed 52 scenes containing about 10-30 objects admissible for annotation as figure objects. since our annotation task involves describing the location of a figure object in relation to other objects (grounds), objects that form the environment (walls, ceiling, floor) are not admissible as figures (however, they can be used as grounds as in the poster is on the north wall). this serves as a realistic domain for evaluating spatial relations. we designed the annotation task so as to achieve a balance between obtaining a significant number of annotations and collecting some information about human preferences. in each annotation instance the annotator is presented with a screenshot of a room world scene and is asked to describe the location of a highlighted figure object. first, the annotator is to pick a single best-fitting preposition and a corresponding ground object. after that, they are to indicate all other relations that they believe to hold between the figure and the ground (if the most appropriate relation chosen was between, they are to indicate which relations hold between the figure and the first ground object). they are then asked to repeat the same procedure for up to two more times. the reason for such an approach is that while the second part of the annotation (choosing all the relations holding between the given and the selected objects) produces coverage of pairwise relations, many such judgements feel forced and unnatural to human annotators (during earlier explorations it was noted that vagueness of locative expressions leads to annotators overthinking when making judgements). the laws of conversational implicature predict that, in everyday usage, various locatives will not occur with uniform frequencies. when several possible prepositions are applicable, people tend to choose those prepositions that disambiguate better or imply stronger relations, e.g., on is preferable to touching or near even though these relations often co-occur. hence, the first part of the annotation process allows annotators to freely choose the most natural or "obvious" options. at the moment, because of the scarcity of data (see ## model details we have developed two kinds of models. the first one is a series of simple multi-layer perceptrons (one per each relation), and the second is our main rule-based model, which is implemented as a network (more precisely, an arborescence) of nodes that compute meaningful hand-crafted relations used for determining the values of the prepositions. each node realizes one or more differentiable operations which allows us to train the model using standard gradient descent-based optimization. the main reasons for developing the pure nn-based solution are to provide the baseline performance metric against which we compare our main models. each model is essentially a binary classifier used to predict the likelihood that a particular relation holds between given objects. ## evaluation and discussion evaluation data for both types of models are presented in table . 2. overall, both models performed reasonably well, apart from the cases such as in front of, behind and touching where the rule-based model performed better thanks to additional available information. the results clearly show that it is possible to produce reasonable judgments for most spatial relations even with purely geometric information. however, our main goal was to demonstrate that even when they fall short, our rule-based models still compare reasonably well with pure neural network-based approaches, with the added benefit of being interpretable thanks to their formulation in terms of meaningful decision criteria that correspond to human intuitions about spatial relations. another important aim of our exploration was to evaluate whether the factors we selected are appropriate and sufficient for modeling the semantics of the locative senses of prepositions. while it is difficult to extrapolate our performance results to novel settings, we believe that our room worlds are representative of a significant subset of everyday settings where locative expressions are apt to be used. the annotation process is still ongoing and we are working on an additional set of scenes depicting outdoor environments. as such, the numbers in the table are subject to change, as the breadth of configurations covered and annotation data is increased. scale differences between the two domains might affect the boundaries of applicability of the prepositions as well as their relative psychological preference ordering. whenever possible we rely on approximations to the real 3d meshes of objects, using centroids and bounding boxes. this allows us to focus on the most salient features of objects' shapes and maintain relatively high performance. ness during a dialogue with the user (see the next subsection). ## conclusion we considered the problem of designing intuitive computational models of spatial prepositions that combine geometrical information as well as some pieces of commonsense knowledge and contextual information about the arguments. our main aim was to develop spatial semantic models that rely on psychologically plausible criteria and facilitate justification of spatial judgements produced by the models, and to compare such an approach against a more mainstream black box statistical learning architecture acting as a baseline. we believe that combining the power of data-driven methods and interpretable, algorithmic models is the way forward in ai in general and, in particular, is necessary in order to incorporate context and background knowledge information needed to model spatial expressions properly. this work is one step in that direction.