Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages (2022)

Abstract: Human languages are full of metaphorical expressions. Metaphors help people understand the world by connecting new concepts and domains to more familiar ones. Large pre-trained language models (PLMs) are therefore assumed to encode metaphorical knowledge useful for NLP systems. In this paper, we investigate this hypothesis for PLMs, by probing metaphoricity information in their encodings, and by measuring the cross-lingual and cross-dataset generalization of this information. We present studies in multiple metaphor detection datasets and in four languages (i.e., English, Spanish, Russian, and Farsi). Our extensive experiments suggest that contextual representations in PLMs do encode metaphorical knowledge, and mostly in their middle layers. The knowledge is transferable between languages and datasets, especially when the annotation is consistent across training and testing sets. Our findings give helpful insights for both cognitive and NLP scientists.

## introduction pre-trained language models (plms) @xcite , are now used in almost all nlp applications, e.g., machine translation @xcite , question answering @xcite , dialogue systems @xcite , and sentiment analysis @xcite . they have sometimes been referred to as "foundation models" @xcite due to their significant impact on research and industry. metaphors are important aspects of human languages. in conceptual metaphor theory (cmt) @xcite , metaphor is defined as a cognitive phenomenon associating two different concepts or domains. this phenomenon is built in cognition and expressed in language. the creativity and problem solving (i.e., generalization to new equal contribution. problems) depend on the analogies and metaphors a cognitive system, like our brain, relies on. modeling metaphors is therefore essential in building human-like computational systems that can relate emerging concepts to the more familiar ones. so far, there has been no comprehensive analysis of whether and how plms represent metaphorical information. we intuitively assume that plms must encode some information about metaphors due to their great performance in metaphor detection and other language processing tasks. confirming that experimentally is a question that we address here. specifically, we aim to know whether generalizable metaphorical knowledge is encoded in plm representations or not. the outline of our work is presented in figure 1 . we first do probing experiments to answer questions such as: (i) with which accuracies and extractablities do different plms encode metaphorical knowledge? (ii) how deep is the metaphorical knowledge encoded in plm multi-layer representations? we take two probing methods, edge probing @xcite and minimum description length @xcite , and apply them to four metaphor detection datasets, namely lcc @xcite , trofi @xcite , vua pos, and vua verbs @xcite . to better estimate the generalization of metaphorical knowledge in plms, we design two setups in which testing comes from a different distribution than training data: cross-lingual and cross-dataset metaphor detection. each setup can reveal important information on whether or not the metaphorical knowledge is encoded consistently in plms. four languages (english, farsi, russian and spanish) and four datasets (lcc, trofi, vua pos, and vua verbs) are considered in these generalization experiments. in summary, this paper makes the following contributions: ## related work metaphor detection using plms. the metaphor detection task @xcite @xcite is a good fit for analyzing the metaphorical knowledge. using plms for metaphor detection has been common in recent years, setting new state-ofthe-art results, indicating implicitly that plms represent metaphorical information. @xcite introduce a new architecture that integrates metaphor detection theories with bert. they use the definitions as well as example usages of words jointly with plm representations. similarly, @xcite presents a new perspective on metaphor detection task by framing it as relation classification, focusing on the verbs. these approaches beat the earlier work of using plms @xcite @xcite , rnn-based @xcite and feature-based approaches @xcite . note that our goal is not to compete with these models, but to probe and analyze the relevant knowledge in plms. @xcite present cross-lingual metaphor detection models using linguistic features and word embeddings. bilingual dictionaries map different languages. their datasets are quite small ( 1000 training and 200 testing examples), making them unsuitable for a robust evaluation. however, this paper still remains as the only crosslingual evaluation of metaphor detection, to the best of our knowledge. here, using multilingual plms, we perform zero-shot cross-lingual transfer for metaphor detection. our goal is to test if plms represent metaphorical knowledge transferable across languages. ## inspecting metaphorical knowledge in plms metaphors are used frequently in our everyday language to convey our thoughts more clearly. there are related theories in linguistics and cognitive science. following linguistic theories, metaphoricity is mostly annotated using metaphor identification procedure (mip). mip identifies a word in a given context as a metaphor if it has a basic or literal meaning that contrasts with its context words. based on conceptual metaphor theory (cmt) @xcite , one target domain (e.g., argument) is explained using a source domain (e.g., war). the source domain is usually more concrete or physical, while the target is more abstract. relating these two theories, metaphors are expressed in language connecting two contrasting domains. for example, in "we won the argument", the domain of argument is linked to the domain of war by using the word "won". the word "won" is a "metaphor" here since its primary domain contrasts with its contextual domain. the same word "won" in a sentence like "the allies won the war" refers to its literal meaning and therefore is not a metaphor. the task of metaphor detection is defined to do this classification of "literal" and "metaphor". accordingly, when designing a metaphor detection system, to figure out if a token is a metaphor in a particular context, we assume following a process like: (i) finding if the token has multiple meanings in different domains, including a more basic, concrete, or body-related meaning. for example, "fight", "win" and "mother" satisfy this condition. (ii) finding if the source domain of the token contrasts with the target domain. here the contrast is important and finding the exact domains might not be necessary. the source domain, in which its literal / basic meaning resides, is a non-contextual attribute, while the target domain is mainly found using the contextual clues (war and argument for "won" in the above example). here, we use the metaphor detection datasets annotated based on these theories and analyze the plm representations to see if they encode metaphorical knowledge and if the encoding is generalizable. to do so, we first probe plms for their metaphorical information, generally and also across layers. this gives us intuition on how well metaphoricity is encoded and how local or contextual that is. then, we test if the knowledge of metaphor detection can be transferred across languages and if multilingual plms capture that. finally, the generalization of metaphorical knowledge across datasets is examined to see if the theories and annotations followed by different datasets are consistent, and if plms learn generalizable knowledge rather than dataset artifacts. ## discussion and conclusion metaphors are important in human cognition, and if we seek to build cognitively inspired or plausible language understanding systems, we need to work more on their best integration in the future. therefore, any work in this regard is impactful. our probing experiments showed that plms do in fact represent the information necessary to do the task of metaphor detection. we assume this information is related to metaphorical knowledge learned during pre-training. further, the layer-wise analysis confirmed our hypothesis that middle layers are more informative. even though our probing experiments did show that metaphorical knowledge is present in plms, it was still unclear if this knowledge is generalizable beyond the training data. so, to probe the probe and evaluate generalization, we ran crosslingual and cross-dataset experiments. our results showed that the transferability across languages works quite well for the four languages in lcc annotation. however, when the definitions and annotations were inconsistent across different datasets, the cross-dataset results were not satisfactory. overall, we conclude that metaphorical knowledge does exist in plm representations and in middle layers mainly, and it is transferable if the annotation is consistent across training and testing data. we will explore more the cross-lingual transfer of metaphors and the impact of cross-cultural similarities in the future. also, the application of metaphorical knowledge for text generation is something important that we will address.