M el BERT : Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories (2021)

Abstract: Automated metaphor detection is a challenging task to identify the metaphorical expression of words in a sentence. To tackle this problem, we adopt pre-trained contextualized models, e.g., BERT and RoBERTa. To this end, we propose a novel metaphor detection model, namely metaphor-aware late interaction over BERT (MelBERT) . Our model not only leverages contextualized word representation but also benefits from linguistic metaphor identification theories to detect whether the target word is metaphorical. Our empirical results demonstrate that MelBERT outperforms several strong baselines on four benchmark datasets, i.e., VUA-18, VUA-20, MOH-X, and TroFi.

## introduction as the conceptual and cognitive mapping of words, a metaphor is a common language expression representing other concepts rather than taking literal meanings of words in context @xcite . for instance, in the sentence "hope is on the horizon," the word "horizon" does not literally mean the line at the earth's surface. it is a metaphorical expression to describe a positive situation. therefore, the meaning of "horizon" is context-specific and different from its literal definition. as the metaphor plays a key role in cognitive and communicative functions, it is essential to understand contextualized and unusual meanings of words (e.g., metaphor, metonymy, and personification) in various natural language processing (nlp) tasks, e.g., machine translation @xcite , sentiment analysis @xcite , and dialogue systems @xcite . a lot of existing studies have developed various computational models to recognize metaphorical words in a sentence. automated metaphor detection aims at identifying metaphorical expressions using computational models. existing studies can be categorized into three pillars. first, feature-based models employ various hand-crafted features @xcite @xcite @xcite . although simple and intuitive, they are highly sensitive to the quality of a corpus. second, some studies @xcite @xcite utilize recurrent neural networks (rnns), which are suitable for analyzing the sequential structure of words. however, they are limited to understanding the diverse meanings of words in context. lastly, the pre-trained contextualized models, e.g., bert @xcite and roberta @xcite , have been used for detecting metaphors @xcite @xcite . owing to the powerful representation capacity, such models have been successful for addressing various nlp tasks @xcite and document ranking in ir @xcite . based on such an advancement, we utilize a contextualized model using two metaphor identification theories, i.e., metaphor identification procedure (mip) @xcite and selectional preference violation (spv) @xcite . for mip, a metaphorical word is recognized if the literal meaning of a word is different from its contextual meaning @xcite . for instance, in the sentence "don't twist my words", the contextual meaning of "twist" is "to distort the intended meaning", different from its literal meaning, "to form into a bent, curling, or distorted shape." for spv, a metaphorical word is identified if the target word is unusual in the context of its surrounding words. that is, "twist" is metaphorical because it is unusual in the context of "words." although the key ideas of the two strategies are similar, they have different procedures for detecting metaphorical words and their contexts in the sentence. to this end, we propose a novel metaphor detection model using metaphorical identification theories over the pre-trained contextualized model, namely metaphor-aware late interaction over bert (melbert). melbert deals with a classification task to identify whether a target word in a sentence is metaphorical or not. as depicted in figure 2 , melbert is based on a siamese architecture that takes two sentences as input. the first sentence is a sentence s with a target word w t and the second sentence is a target word w t itself. melbert independently encodes s and w t into each embedding vector, which avoids unnecessary interactions between s and w t . inspired by mip, melbert then employs the contextualized and isolated representations of w t to distinguish between the contextual and literal meaning of w t . to utilize spv, melbert employs the sentence embedding vector and the contextualized target word embedding vector. melbert identifies how much the surrounding words mismatch from the target word. lastly, melbert combines two metaphor identification strategies to predict if a target word is metaphorical or not. each metaphor identification theory is non-trivial for capturing complicated and vague metaphorical words. to overcome these limitations, we incorporate two linguistic theories into a pre-trained contextualized model and utilize several linguistic features such as pos features. to summarize, melbert has two key advantages. first, melbert effectively employs the contextualized representation to understand various aspects of words in context. because melbert is particularly based on a late interaction over contextualized models, it can prevent unnecessary interactions between two inputs and effectively distinguish the contextualized meaning and the isolated meaning of a word. second, melbert utilizes two metaphor identification theories to detect whether the target word is metaphorical. experimental results show that melbert consistently outperforms state-of-the-art metaphor detection models in terms of f1-score on several benchmark datasets, such as vua-18, vua-20, and vua-verb datasets. ## melbert in this section, we propose a novel metaphor detection model over a pre-trained contextualized model. to design our model, we consider two metaphor detection tasks. given a sentence @xmath0, . . . , w n } with n words and a target word w @xmath1, the classification task predicts the metaphoricity (i.e., mataphorical or literal) of w t . given a sentence s, the sequence labeling predicts the metaphoricity of each word w we aim at developing a metaphor detection model for the classification task. our model returns a binary output, i.e., 1 if the target word w t in s is metaphorical or 0 otherwise. by sequentially changing the target word w t , our model can be generalized to classify the metaphoricity of each word in a sentence, as in sequence labeling. ## evaluation in this section, we first present the experimental setup, then report empirical results by comparing our model against strong baselines. ## conclusion in this work, we proposed a novel metaphor detection model, namely, metaphor-aware late interaction over bert (melbert), marrying pre-trained contextualized models with metaphor identification theories. to our best knowledge, this is the first work that takes full advantage of both contextualized models and metaphor identification theories. comprehensive experimental results demonstrated that melbert achieves state-of-the-art performance on several datasets.