A Cognitively Motivated Approach to Spatial Information Extraction (2020)

Abstract: Automatic extraction of spatial information from natural language can boost human-centered applications that rely on spatial dynamics. The field of cognitive linguistics has provided theories and cognitive models to address this task. Yet, existing solutions tend to focus on specific word classes, subject areas, or machine learning techniques that cannot provide cognitively plausible explanations for their decisions. We propose an automated spatial semantic analysis (ASSA) framework building on grammar and cognitive linguistic theories to identify spatial entities and relations, bringing together methods of spatial information extraction and cognitive frameworks on spatial language. The proposed rule-based and explainable approach contributes constructions and preposition schemas and outperforms previous solutions on the CLEF-2017 standard dataset.

## introduction human-centered technologies, such as in selfdriving cars or cognitive assistance systems in aviation, face the challenge of having to provide cognitively plausible decisions, i.e., decisions that are understandable and accepted by humans. this is needed for an effortless interaction between artificial and human agents. yet, even in spatial language, with its advantageous characteristic of following recurring, uniform patterns @xcite , drawing automated inferences in a dynamic spatial environment remains to be an open research question. as illustration consider example ( ) foot_0 : "i poured water from the bottle into the cup until it was full" foot_1 . consider the question: "what was full, the cup or the bottle?". a cognitively plausible explanation for the answer the cup, could be as follows: "as water was poured from the bottle into the cup, it is likely that the water is in the cup. normally, if water is poured into the cup then the cup is full. thus, there is some evidence that the cup is full." in order to produce and accept "the cup" as the plausible answer, humans do not require particular reasoning effort or expert knowledge. yet building systems that provide such simple explanations remains to be a challenge. in the past, a considerable amount of literature has been published on automated spatial semantic analysis. however, most of the solutions focus on specific spatial phenomena suffering from a lack of generalizability (e.g. @xcite , or are generalizable machine learning approaches (e.g. @xcite , but lack explainability of provided decisions. also cognitive linguistics contributed to the understanding of spatial language by proposing various cognitive models to this end @xcite . taking spatial cognitive models as a starting point, we propose a novel spatial representation method and investigate which type of spatial information, such as entities of spatial scenes and relations holding between them, can be extracted from natural language (nl) expressions. to this end, we rely on construction grammar and cognitive linguistic theories. for example ( ), the following spatial roles apply: agent ("i"), figure ("water"), ground ("bottle" and "cup"), and simultaneously source ("bottle") and goal ("cup"). figure 1 illustrates the automated spatial semantic analysis (assa) framework proposed in this paper. assa provides an automated extraction method based on construction grammar and image schemas @xcite , which are spatio-temporal relationships built from sensorisection 4.1 -4.2 semantic analysis of spatial preposition source or path, via (if applicable), goal or orientation section 4.3 dynamic static figure 1: external sources are in white, input/ output are in light gray and own contributions are in black boxes. to the best of our knowledge, this is the first generalizable and explainable framework grounded in construction grammars and cognitive linguistic theories to be empirically validated. we firmly believe and show that exploiting insights gained from cognitive linguistics can help building such a framework. this framework contributes to the automated analysis of spatial semantics in nl and thereby to the analysis of the human experience of space and human-centered applications. our contributions are: (1) a semantic analysis of spatial prepositions based on image schemas, (2) an automated extraction method that unifies constructionist theory and cognitive linguistic insights on spatial language, and (3) a prototype system of assa fully evaluated on a standardized dataset with competitive performance. ## related work our work combines two strands of research, namely spatial semantic analysis and (automated) spatial role labeling. spatial semantic analysis -the process of detecting spatial entities, their relations, and motion -has been a long-standing research endeavor in areas such as cognitive linguistics (e.g. @xcite , geographical information systems (gis) (e.g. @xcite , spatial language under-standing in robotics (e.g. @xcite as well as spatial role labeling @xcite . at times, works focus on one aspect of spatial semantic analysis only, such as prepositions @xcite and spatial frames and their relations @xcite . we consider three works as particularly closely related to the proposed framework. first, @xcite similarly to our work distinguish dynamic and static spatial relations, however, focus on robot-robot interactions. second, the embodied construction grammar @xcite brings together construction and cognitive grammar, which is, however, exemplified rather than fully evaluated. third, @xcite propose a knowledge-based approach on fictive motion, who rely on cognitive linguistic theories, rules and verbs to differentiate actual motion from fictive motion. however, they focus on fictive motion in the geographic domain, such as "the valley ran towards the sea". though such methods are good starting points for extracting variations of spatiotemporal information, they are less suited to reconstruct static and dynamic spatial scences from linguistic descriptions, which we propose to do with a generalizable and evaluated analysis framework. methods specific to spatial role labeling have mostly utilized machine learning. @xcite apply a step-wise approach with two probabilistic classifiers, where the first step consists in uncovering the spatial indicator (a preposition) and the second in extracting the related trajector and landmark. while this machine learning method has a lot to offer, it disregards the influence of verbs as indicators of movement. however, verbs play a central role in identifying the relation between trajector and landmark and represent a vital element of our proposed method. one approach testing on the same evaluation dataset, the spatial role labeling shared task at clef-2017 @xcite , is the lip6 system @xcite . they re-implement saul @xcite for the two subtasks of identifying spatially-related entities (trajector, spatial indicator, landmark) and organizing them into spatial triplets. to this end, a sparse perceptron classifier is trained for each of these entities, utilizing lexical, syntactical, and contextual features. while the results are partially competitive to ours, this machine learning method requires complex feature engineering and allows for little insight into decisions taken for each entity. in contrast, assa is highly transparent and each decision taken can be fully explained based on the linguistic properties of the spatial expression and the assigned constructional schema and cognitive semantic roles. ## theoretical foundations our approach is based on cognitive linguistic theories, specifically talmy's cognitive semantic theory, goldberg's construction grammar, and lakoff's and johnson's image schemas. ## spatial semantic analysis framework the spatial semantic analysis framework covers three components depicted on the right of figure 1 : section 4.1 specifies the semantic roles, section 4.2 develops the spatial constructional schemas for the identification of these roles and their relations, and section 4.3 provides the semantic analysis framework for the proposed preposition schemas. ## procedure of assa we describe the procedure of assa for our empirical validation of the proposed framework in section 4. our prototype system is written in python and utilizes the stanford dependency parser @xcite . figure 1 depicts the process, where the following steps will be detailed here: ## evaluation we are not aware of any specific shared task as a suitable evaluation for the novel approach proposed here. spatial role labeling (sprl), a subtask of semeval @xcite seems to be the closest related task for our purpose. we use the test data from the sprl task of clef-2017 to test the coverage of our system. it has 3 subtasks: (1) label spatial indicators and their landmark(s)/trajector(s), (2) extract triplets of spatial indicator-trajector-landmark, (3) label relations with region, direction and distance. only the first two are relevant for the evaluation of our framework foot_5 . consider the following example: about 20 kids in traditional clothing and hats waiting on stairs. for subtask 1 this means identifying the trajector kids, spatial indicator on and landmark as stairs. for subtask 2 the system should extract spatial relation(kids, on, stairs). ## conclusion although the extraction of spatial information has been widely studied, no general and human understandable approach has been proposed up to this point. based on its interdisciplinary nature by exploiting the developments made in various areas, our contributions are two-fold: assa uni-fies methods from automated spatial information extraction with cognitive models on spatial language. the constructional schemas developed here can extract relevant spatial information, and further (in case of a dynamic construction) apply them to image schemas in order to identify corresponding semantic roles. second, an evaluation of a prototype system outperforms machine learning systems. in terms of future work, it would be interesting to apply the prototype to a larger dataset with more dynamic spatial expressions as well as test the procedure on different languages. as discussed above, it would also be interesting to utilize the insights gained from our constructions and preposition schemas to provide natural language explanations for system decisions. finally, it would be interesting to investigate the interaction of assa with existing resources, such as framenet @xcite to further improve on its precision and provide background knowledge to our framework. it would also be interesting to formally represent our constructional schemas, e.g. utilizing qualitative trajectory calculus (van de @xcite and similar formalisms, in order to benefit from reasoning algorithms for spatial information extraction.