Figurative Language Processing: A Linguistically Informed Feature Analysis of the Behavior of Language Models and Humans (2023)

Abstract: Recent years have witnessed a growing interest in investigating what Transformer-based language models (TLMs) actually learn from the training data. This is especially relevant for complex tasks such as the understanding of non-literal meaning. In this work, we probe the performance of three black-box TLMs and two intrinsically transparent white-box models on figurative language classification of sarcasm, similes, idioms, and metaphors. We conduct two studies on the classification results to provide insights into the inner workings of such models. With our first analysis on feature importance, we identify crucial differences in model behavior. With our second analysis using an online experiment with human participants, we inspect different linguistic characteristics of the four figurative language types.

## introduction in recent years, transformer-based language models (tlms) have achieved groundbreaking performance in various nlp tasks. along with such progress, there has been an increasing demand for understanding the reasons for the decisions made by the tlms, as this is often required for humans to trust the models @xcite . as of now, researchers working on interpretability have mostly neglected the precise investigation of how tlms models process non-literal language. non-literal language, or figurative language, is a type of language where the intended meaning of an expression is incongruent with its literal meaning @xcite . typical cases of figurative language include sarcasm, e.g., saying 'lovely weather' on a stormy day, or metaphors, e.g., describing a person that always goes to bed late as a 'night owl' even though the person is not an actual owl. this discrepancy between the surface form and the intended message ? authors with equal contribution, listed alphabetically. makes tasks involving figurative language complex both for humans and for models; therefore, it is harder for humans to trust the output of a model in such tasks without a precise understanding of the motivations behind specific models' decisions. as the figurative meaning is not literally articulated in words, humans grasp it via pragmatic enrichment processes, i.e., inferring the speaker's communicative intention that is not uttered @xcite @xcite . although such processes often rely on non-deterministic factors such as social and cultural background @xcite , studies have shown that humans also utilize more explicit contextual cues to achieve pragmatic enrichment and identify the figurative meanings (see, e.g., @xcite @xcite . these cues include contextual incongruity, semantic relations between words, or explicit syntactic forms. given such multi-step nature of figurative language processing, we are interested in investigating the inner workings of tlms in processing different types of figurative language. specifically, we focus on two research questions (rqs): rq 1 -when the explicit cues that help the identification of the figurative meaning exist, do tlms attend to them as humans do, or do tlms adopt totally dissimilar strategies from humans? rq 2 -how do the performance and the feature attention behavior of tlms compare to those of intrinsically interpretable white-box models such as regression models or decision-tree-based models? would the attention mechanism enable them to grasp those cues better? to explore these two questions, we probe three black-box tlms along with two white-box models as baseline on the task of figurative language classification, using a dataset that provides a rich range of figurative language classes with different opacity degrees, i.e., some classes have obvious cue words, whereas others do not. based on the classification results, we conduct two analyses that compare 1) the behavior of different models and 2) the behavior of models vs. humans. our main contributions are two-fold: first, we show that even though different tlms achieve the same level of performance in the figurative language classification task, they show a striking discrepancy in the features they attend to, suggesting different levels of interpretability of different models. second, we bridge existing work in psycholinguistics and theoretical linguistics with our data analysis results to gain a better understanding of figurative language processing in both machines and humans. foot_0 2 related work nlp researchers have attempted to build models that can comprehend figurative meaning. the public availability of large-scale annotated corpora, e.g., the sarcasm corpus v2 @xcite , the vu amsterdam dataset of metaphor @xcite , and the magpie dataset for potentially idiomatic expressions @xcite , has encouraged the task of figurative language detection. before the extensive use of neural networks, most studies have treated figurative language processing as a classification task and utilized theoreticallyderived features. for example, incongruent sentiment expressions have often been used for sarcasm detection @xcite ; while abstractness of words @xcite and topic transition information @xcite have been used for metaphor detection. recent studies have been using neural models (e.g., @xcite @xcite , and tlms especially have shown good performance (see, e.g., @xcite @xcite @xcite . some recent work using tlms treats figurative language processing as a natural language inference (nli) task instead of a classification task @xcite , which is a step closer to comprehending figurative language. despite the successful model performance of tlms, little research has attempted to delve into their inner workings. several studies have probed into whether knowledge of figurative meaning is encoded in tlms @xcite @xcite . though this strand of work has confirmed that this knowledge is encoded in tlms to some extent, it does not provide information about what motivates the output of the models in the task of figurative language processing. our work attempts to fill this gap by inspecting the behavior of different models in processing different types of figurative language. we zoom into the most salient lexical properties used by different tlms in distinguishing four types of figurative language and compare such properties with those used by humans. ## figurative language classification as tlms are not intrinsically explainable, one way of inspecting the reasons behind their decisions is by a post-hoc feature analysis. we treat figurative language processing as a classification task where the models classify 4 different types of figurative language. we then conduct analyses on the classification results to compare the behavior of 1) different models (section 4) and 2) models vs. humans (section 5). in this section, we report the results from our classification experiments using a variety of black-box and white-box models. all supplementary details of this experiment are provided in appendix a. ## feature analysis 1: models vs. models in our first analysis, we investigate the impact of each feature on the model predictions. we compare the features that different models deem important for each figurative language class (cross-model comparison). we also identify the common behavior of the five models for each figurative language class (cross-class comparison). ## feature analysis 2: models vs. humans the classification results in section 3 suggest that models especially struggle with identifying idioms and metaphors. these results are in line with various studies in cognitive science and psycholinguistics showing that the processing of idioms and metaphors is also complex for humans. idioms are defined as "constructions whose meanings cannot be derived from the meanings of its con-stituents" @xcite . therefore, identifying idioms often relies on memory retrieval rather than syntactic and semantic analyses @xcite and the speaker's familiarity to them @xcite . similarly, the difficulty of and the speaker's familiarity to metaphors are factors that influence metaphor processing @xcite . drawing in on these intricacies, we build a classification task for human participants, aiming to investigate how human behavior differs from model behavior in figurative language classification, and whether humans also struggle more with identifying idioms and metaphors. ## conclusion with our two experiments, we provide insights into both the behavioral differences between different language models and the varying linguistic properties of several figurative language types. our first analysis reveals contrasting behavior among the black-box models. this highlights different degrees of interpretability of the tlms in the task of figurative language processing despite their similar performance. it also provides evidence-based indicators for choosing the best model that deals with rich linguistic information in an extended range of nlp applications. in the second analysis with human participants, we show that the general tendency found in the performance of all models is aligned with that of human participants; this manifests the varying complexity levels of different figurative language types.