CATE : A Contrastive Pre-trained Model for Metaphor Detection with Semi-supervised Learning (2021)

Abstract: Metaphors are ubiquitous in natural language, and detecting them requires contextual reasoning about whether a semantic incongruence actually exists. Most existing work addresses this problem using pre-trained contextualized models. Despite their success, these models require a large amount of labeled data and are not linguistically-based. In this paper, we proposed a ContrAstive pre-Trained modEl (CATE) for metaphor detection with semi-supervised learning. Our model first uses a pre-trained model to obtain a contextual representation of target words and employs a contrastive objective to promote an increased distance between target words’ literal and metaphorical senses based on linguistic theories. Furthermore, we propose a simple strategy to collect large-scale candidate instances from the general corpus and generalize the model via self-training. Extensive experiments show that CATE achieves better performance against state-of-the-art baselines on several benchmark datasets.

## introduction conceptual metaphors are figurative languages widely used in our daily communication, implying a mapping between two conceptual domains @xcite . at a linguistic level, metaphor is defined as a linguistic expression representing other concepts rather than taking literal meanings of words in context @xcite . for instance, in the sentence "i have digested all this information," the word digested does not literally mean converting food into absorbable substances. instead, this word means "arrange and integrate in the mind" in the context. 1 this metaphor conceptualizes the concept of ideas in terms of the properties of food. metaphorical associations as such are broad generalizations that allow us to project knowledge and inferences across domains and are beneficial for various downstream nlp applications, such as machine translation @xcite , sentiment analysis @xcite , and dialogue systems @xcite . given the prevalence of metaphors in human communication, the effective detection of metaphors plays an essential role in natural language understanding. hence, many efforts have been devoted to metaphor detection (md), which aims to identify metaphorical expressions in a text automatically. most previous methods @xcite @xcite for md are based on various hand-crafted linguistic features and rely on manually annotated resources to extract them. recently, significant progress has been made in applying deep learning techniques for md @xcite @xcite @xcite . these methods directly embed textual semantic information into a low-dimensional space by deep neural networks. nevertheless, these methods are unable to model the multiple meanings of polysemous words in context @xcite . with the rapid development of contextualized representations, a number of methods @xcite @xcite adopt pre-trained language models to effectively capture context-dependent information with respect to the target words and fine-tune them to obtain state-of-the-art performances for md. although these pre-trained models have achieved promising results, several problems remain unsolved. first, the current models lack the discrimination between the literal meaning and non-literal meaning of the target words, which can be enhanced by analogical comparison in the specific context based on metaphor identification procedure (mip) @xcite . second, one challenge for fine-tuned language models is they still require large amounts of labeled data for obtaining state-of-the-art performances on downstream tasks @xcite @xcite . however, due to the expensive and labor-intensive labeling, existing public md datasets are relatively small. in addition, labeling metaphorical words can be influenced by subjective input and may need expert knowledge @xcite , which poses a significant challenge for metaphor detection. the above challenges motivate us to propose a contrastive pre-trained model (cate) for metaphor detection, using a contrastive objective to model the distance between the target word's literal and metaphorical senses, enhancing the model generalization performance via self-training with unlabeled data generated by a simple strategy. firstly, we utilize pre-trained models (i.e., bert and roberta) to capture contextual information about a target word in the sentence. if the target word is a metaphor, its semantic meaning is context-specific and different from its literal meaning. the word's literal meaning can be described through non-metaphorical instances. therefore, we incorporate a contrastive objective to enhance contextual representations between the literal and metaphorical meaning of a target word to make it more distinguishable, in which way the classifier can make a more informed decision. to address the label scarcity issue, we propose a simple target-based generating strategy to automatically generate training data inspired by a distantly supervised paradigm @xcite . concretely, if a given word serves as the detection target in a sentence, all sentences containing this word in a specific corpora are retrieved and regarded as candidate instances. to expand the training data, we use the pre-trained model to generate pseudo-labels for these candidate instances and incorporate them into training data, where the pre-trained model is first fine-tuned on the original training set, as shown in figure 1 . we update the pseudo-labels and the model iteratively by selftraining for improving the generalization power. in summary, the contributions of this paper are as follows: (1) we propose a novel pre-trained model with a contrastive objective for capturing the semantic incongruence in metaphors based on mip linguistic theories. (2) to our best knowledge, this is the first attempt to combine semi-supervised learning with self-training to alleviate the label scarcity issue for md. (3) empirically, we perform experiments on widely used datasets to verify the effectiveness of our approach. experimental results show that our approach obtains state-of-the-art performance over several benchmark datasets. ## related work early approaches mainly use a variety of linguistic features to detect metaphors, such as part of speech, unigrams @xcite , concreteness/abstractness @xcite , wordnet supersenses @xcite , and sensory features @xcite , etc. they rely heavily on numerous carefully designed feature engineering. in recent years, various models have been widely used in md based on end-to-end neural architectures. @xcite reformat the md task as a sequence labeling problem and combine cnn and lstm layers with ensemble learning to generate the best performance in the naacl-2018 metaphor shared task @xcite . subsequently, @xcite presented simple bil-stm augmented with contextualized word representation, which achieved better results. @xcite further adopted two linguistic theories on top of the structure of @xcite . in addition, some approaches employed multi-task learning to transfer knowledge from the related tasks and resources to improve the performance of md @xcite @xcite . these neural models are capable of properly capturing the relations between metaphors and their contexts without linguistic analyses. however, the superficial structures make them difficult to represent different aspects of words in context. recently proposed pre-trained language models @xcite @xcite have shown dramatic improvements on several nlp tasks with appropriate fine-tuning. therefore, some efforts @xcite @xcite are made to leverage the strong expressive power of pre-trained models, such as bert, roberta, to effectively capture general semantics and contextdependent information of target words for improving the performance of metaphor detection. despite their success, one bottleneck for fine-tuning pretrained models is the requirement of labeled data. when labeled data are scarce, the fine-tuned models often suffer from degraded performance, and the large number of parameters can lead to severe overfitting @xcite @xcite ). however, it is time-consuming and human-intensive to manually annotate large-scale training data for md. ## proposed method the md task is to predict whether a target word in a given sentence is metaphorical or literal. some previous work @xcite @xcite regards metaphor detection as a sequence labeling task that predicts the metaphoricity of each word in a given sentence. nevertheless, this format introduces the noise of treating all nontarget words as literal, which negatively impacts the model learning the difference between literal and metaphorical words @xcite . in this paper, we convert the md task as a classification task based on the target word, like @xcite . formally, given a sentence @xmath0, w 2 , ..., w n } with n words and a target word w @xmath1, the task involves predicting a binary label l @xmath2, 1} to indicate the metaphoricity (i.e., metaphorical or literal) of the target word w t . figure 2 gives an overview of cate. ## conclusion this paper takes advantage of self-training and designs a simple but effective metaphor detection model based on the pre-trained backbone to capture the contextualized features. to be specific, we incorporate a contrastive objective into the model to capture the semantic incongruence in metaphors and use a simple strategy to automatically construct substantial training data ready for self-training. the evaluation on multiple benchmarks has shown that our model can achieve state-of-the-art performance. in future work, we plan to explore how to use unlabeled data more effectively and discover potentially valuable metaphor examples to reduce efforts of manual annotation.