Systematic Analysis of Image Schemas in Natural Language through Explainable Multilingual Neural Language Processing (2022)

Abstract: In embodied cognition, physical experiences are believed to shape abstract cognition, such as natural language and reasoning. Image schemas were introduced as spatio-temporal cognitive building blocks that capture these recurring sensorimotor experiences. The few existing approaches for automatic detection of image schemas in natural language rely on specific assumptions about word classes as indicators of spatio-temporal events. Furthermore, the lack of sufficiently large, annotated datasets makes evaluation and supervised learning difficult. We propose to build on the recent success of large multilingual pretrained language models and a small dataset of examples from image schema literature to train a supervised classifier that classifies natural language expressions of varying lengths into image schemas. Despite most of the training data being in English with few examples for German, the model performs best in German. Additionally, we analyse the model’s zero-shot performance in Russian, French, and Mandarin. To further investigate the model’s behaviour, we utilize local linear approximations for prediction probabilities that indicate which words in a sentence the model relies on for its final classification decision. Code and dataset are publicly available.

## introduction in the tradition of embodied cognition, image schemas have been proposed by @xcite and @xcite as spatio-temporal cognitive building blocks that capture recurring sensorimotor experiences. for instance, in early infancy we experience many objects with the properties of a con-tainer, i.e., having an inside and an outside separated by a boundary. the image schema contain-ment captures this experience and is subsequently used to make sense of new experiences while at the same time also influencing how we think and talk 1 https://tinyurl.com/24haedv5 about abstract concepts, such as thinking, emotions, or life (see figure 1 ). in order to systematically analyse the occurrence of image schemas in natural language, we propose to build on the recent success of multilingual pretrained language models and a small set of examples from image schema literature @xcite to train a supervised classifier based on xlm roberta (xlm-r) @xcite to classify natural language expressions into image schemas. an image schema detection model as ours could help linguists to explore the use of image schemas efficiently and effectively in large text corpora. it can guide researchers who, for instance, investigate how the use of image schemas differs across languages and cultures (e.g., @xcite , how the language of children with spatial impairments differs (e.g., @xcite or which image schemas occur in various literary works (e.g., @xcite . moreover, we hope that analysing image schemas in large text corpora allows us to contribute to image schema theory directly and to investigate how we think and talk about abstract concepts. our proposed method has significant advantages over previously proposed methods. several corpus linguistic studies (e.g., @xcite and unsupervised machine learning approaches (e.g., @xcite for image schema extraction rely on specific partsof-speech (pos) as indicators of spatio-temporal events. these approaches using pos-tags conventionally portray prepositions as excellent spatial indicators and verbs as movement indicators (e.g., @xcite . however, spatial language might be expressed with prepositions (he walked across the room) or without (he crossed the room) @xcite . in both examples, the underlying image-schematic structure is that of source-path-goal, i.e., the way through the room. since not all spatial expressions in language rely on prepositions, a more general, word classindependent method is needed, which we propose in form of a supervised training procedure based on a multilingual pretrained language model. in contrast to these previous methods, we make use of a small annotated image schema corpus that not only allows us to extract image schemas in different languages without relying on manually created patterns, but also provides a gold standard to evaluate our model. natural language examples of image schemas in literature have been collected in a repository @xcite . however, this database is rather inconsistent in its formatting and image schema annotation. thus, we cleaned and complemented it with other examples from metanet @xcite . our classification method is trained and primarily evaluated in english and german. we also analyse the model's zero-shot performance on a small set of sentences in french, russian, and mandarin, representing different language families. to further investigate the model's behaviour, we utilize the explainable artificial intelligence model lime @xcite that provides local linear approximations for prediction probabilities for each word in the input expression in relation to each available target class, i.e., image schema. thereby, we can provide an analysis of which words in the input sequence the model primarily relies on to make its predictions. ## related work most previous automated approaches for image schema extraction rely on handwritten rules and pattern matching to annotate natural text with image schemas (e.g., @xcite . however, such rules and patterns have to be spec-ified for each image schema as well as for each language resulting in a substantial manual effort. moreover, such patterns lead to low recall and have no mechanisms to handle polysemous words. the only existing machine learning approach clusters triples of syntactically dependent nouns, verbs, and prepositions in order to group them by image schema in an unsupervised manner @xcite . since this approach relies on assumptions about word classes, especially preposition, the range of expressions that can be considered is limited. fields with thematically related objectives are metaphor extraction and spatial role labeling, where recent state-of-theart approaches rely on pretrained neural language models (e.g., @xcite and on contextualized embeddings created for trajector, landmark, and preposition candidates @xcite . ## foundation embodied cognition, a field that builds on the hypothesis that cognitive processes are grounded in perception and sensorimotor interactions with the world, has experienced significant traction in cognitive linguistics. in this tradition, @xcite and @xcite introduce image schemas as cognitive concepts that are firmly rooted in sensorimotor experiences that eventually shape higher-level cognition, including natural language. ## data the data combined from the image schema repository @xcite and metanet @xcite consist of a total of 1,949 samples: 1,535 in english and 414 in german. the exact distribution per image schema can be seen in the cleaning of the image schema repository consisted in deduplicating and ensuring a consistent processable format and annotation. additionally, the authors of this paper and chao xu for mandarin manually curated small test datasets of image schematic language in russian, french, and mandarin, consisting of 35, 40, and 55 samples respectively, for evaluating the zero-shot performance of the classifier. sources for the additional language samples consisted of image schema literature, novels, and online news articles. ## conclusion we introduce a novel approach to perform image schema extraction from natural languages based on multilingual, pretrained neural language models. thereby, a supervised training procedure can be implemented by finetuning the pretrained model with only a few training samples without making any prior assumptions about word classes. the model shows a strong cross-validated performance in english and german, and even shows the ability to generalize to languages unseen during finetuning. explanations generated by the explainable ai approach show insights and shortcomings regarding the model behavior as well as the dataset annotation. to further improve the differentiation between image-schematic classes, a more equal distribution of training data would be beneficial. in terms of future work, we intend to add non-imageschematic samples to further enable the trained classifier to distinguish image-schematic from nonimage-schematic expressions. in addition, the task should be devised as a multi-label classification task to account for the frequent phenomenon of image schema collocations. lastly, we would like to improve the aggregation of local explanations and utilize it in order to systematically analyse image schematic language in a text corpus.