CLCL : Non-compositional Expression Detection with Contrastive Learning and Curriculum Learning (2023)

Abstract: Non-compositional expressions present a substantial challenge for natural language processing (NLP) systems, necessitating more intricate processing compared to general language tasks, even with large pre-trained language models. Their non-compositional nature and limited availability of data resources further compound the difficulties in accurately learning their representations. This paper addresses both of these challenges. By leveraging contrastive learning techniques to build improved representations it tackles the non-compositionality challenge. Additionally, we propose a dynamic curriculum learning framework specifically designed to take advantage of the scarce available data for modeling non-compositionality. Our framework employs an easy-to-hard learning strategy, progressively optimizing the model’s performance by effectively utilizing available training data. Moreover, we integrate contrastive learning into the curriculum learning approach to maximize its benefits. Experimental results demonstrate the gradual improvement in the model’s performance on idiom usage recognition and metaphor detection tasks. Our evaluation encompasses six datasets, consistently affirming the effectiveness of the proposed framework. Our models available at https://github.com/zhjjn/CLCL.git .

## introduction as a ubiquitous yet special class of expressions in natural languages, non-compositional expressions (e.g., the idiom under the weather) have specific communicative intents @xcite and are individually rare but collectively frequently appearing widely across genres @xcite . they are characterized by non-compositionality in their meaning because of which, their meaning cannot be inferred by composing the meaning of their constituent words @xcite . in addition, many non-compositional expressions can be used either figuratively or literally, in a context dependent manner. for example, the phrase "clean house" can be interpreted literally, as in we can not promise you good weather but we can promise you a clean house and a really good breakfast and can be understood figuratively, as in indeed , the kursk crisis may provide him with an opportunity to further clean house in the military. nlp systems intending to process these noncompositional expressions need to decide if these expressions are used in the figurative or literal sense before modeling their meaning. this is the traditional and popular non-compositional language processing task called usage disambiguation foot_0 which aims to differentiate the literal (i.e.,compositional) from the figurative (i.e., non-compositional) usage of these expressions in given contexts, dubbed as idiom usage recognition for idiomatic expressions and metaphor detection for metaphorical expressions @xcite @xcite @xcite . however, compared to the abundance of resources for tasks related to compositional expressions, the available resources for idiom usage recognition and metaphor detection are very limited. successful disambiguation of the usages of the non-compositional expressions involves overcoming two challenges: (1) the linguistic challenge of handling non-compositionality and (2) the resourcerelated challenge of learning from scarce training data. previous works @xcite @xcite primarily focus on designing complex architectures for modeling non-compositionality, while also ignoring the representational aspect to model non-compositionality under a limited-resource scenario to address the second challenge. the focus of this work is a method to solve the above two challenges jointly and find sense-specific representations of the idiomatic expressions. with the same idioms used in different ways as natural positive and negative examples whose representations could be better by using contrastive learning, we utilize contrastive learning to address the first challenge to produce a better representation of non-compositional expressions for recognizing their usage. successful idiom usage recognition and metaphor detection require different representations of the same expression when they are used in a literal and figurative way, respectively. therefore, we incorporate a contrastive objective to enhance the difference between the contextualized representations of the figurative sense and the literal sense for the same expression. in this way, we enable the classifier to make context-dependent decisions in the embedding space. secondly, to make better use of the scarce available data, we use curriculum learning @xcite , which enables the models to gradually proceed from easy training instances to harder ones, when the instances are themselves ordered according to a difficulty measure. therefore, curriculum learning naturally consists of (1) measuring the difficulty level for each training example, and (2) scheduling training examples based on their difficulty levels. furthermore, we combine contrastive learning and curriculum learning together by utilizing contrastive objectives to measure the difficulty level of the training examples. during model training, the contrastive objective is dynamically updated, and thus the difficulty levels of the training examples are also updated in accordance with the current ability of the model. our study is the first to jointly alleviate the problems caused by non-compositionality and limited data resources by strategically and dynamically combining contrastive learning and curriculum learning, and deploying it for idiom usage recognition and metaphor detection. our proposed framework enables the model to first learn from simple non-compositional expressions and then from harder ones by building better representations of non-compositional expressions via contrastive learning. the contributions of our work are as follows: prior studies mainly focus on disambiguating their figurative/literal use @xcite @xcite @xcite @xcite , i.e., performing the idiom usage recognition task. early works heavily rely on designing representative features, e.g., canonical form @xcite , to decide literal and figurative usages. with the emergence of word embeddings and neural networks, richer features are encoded into word embeddings and utilized for idiom usage recognition @xcite . recently proposed pre-trained language models have shown great improvement on various nlp tasks leading to efforts that leverage the power of large pre-trained language models for this task @xcite . however, due to non-compositionality and scarcity of available data resources, previous works mainly focused on designing complex architectures while ignoring the representational aspect to model noncompositionality under a limited-resource scenario. our study is the first to focus on solving both of these two challenges to fill this research gap. metaphor detection. like other figurative expressions, metaphors play a crucial role in cognitive and communicative functions @xcite , because of which computationally recognizing and understanding the metaphorical meanings of words becomes important. early approaches utilized various linguistic features to detect metaphors, such as word imageability @xcite , semantic supersenses @xcite , and unigrams @xcite . in recent years, different neural architectures have been widely used for metaphor detection, including cnn @xcite , lstm @xcite . beyond these, the prominence of large pre-trained language models on various nlp tasks has prompted their use for metaphor detection. @xcite uses roberta as the backbone model to get contextualized representations of words and @xcite combines other linguistic features in a roberta architecture for the purpose of metaphor detection. the subpar performance of large pretrained models when labeled data are scarce has led to studies exploring data augmentation @xcite . however, utilizing augmented data with pseudo labels could be even more detrimental to the performance due to the noise in the augmented data. our proposed curriculum learning framework can potentially alleviate data scarcity by using the limited data more effectively without introducing additional noise. this is the first work to show its positive impact on both tasks of idiom usage recognition and metaphor detection. contrastive learning. contrastive learning aims to learn meaningful representations by pulling semantically similar examples closer and pushing semantically dissimilar examples further apart in the embedding space. widely considered to be effective for building meaningful representations, contrastive learning has garnered increasing attention from researchers in different areas. for example, prior works in nlp have leveraged contrastive learning to produce better word embeddings @xcite and sentence embeddings @xcite . more recently, with the dominance of transformer-based models, contrastive learning is also being used to train transformer models @xcite @xcite . similarly, in this work, for a given non-compositional expression, we use contrastive learning to pull the expression embeddings that are used in the same figurative/literal sense closer while pushing the embeddings between figurative and literal senses apart. thereby we set a precedence of utilizing contrastive learning to enhance the representation quality of idiomatic expressions for modeling non-compositionality. besides, we also propose to utilize the contrastive objective to design curriculum learning, for reducing the training data quantity needed for transformers. curriculum learning first proposed by @xcite , curriculum learning aims to enable the models to gradually learn from easy to harder examples according to a difficulty measure for each example during training. therefore, curriculum learning enables the model to better utilize available data. with growing research interests, curriculum learning has been applied in different fields. in computer vision, curriculum learning has been applied to a range of tasks, such as image classification @xcite , human attribute analysis @xcite , and visual question answering @xcite , however, its nlp application is mainly limited to neural machine translation @xcite @xcite . so, prior works on curriculum learning on nlp, including their difficulty measurement and scheduling strategy, are mainly designed for compositional language processes, which are largely different from non-compositional expressions, i.e., idioms and metaphors. in this study, we propose a new curriculum learning method specifically designed for non-compositional expression recognition. moreover, for the first time we show how curriculum learning based on contrastive learning, results in performance gains in the idiomaticity-related tasks. ## framework in this section, we introduce our proposed framework as a combination of contrastive learning and curriculum learning. overall, we first utilize contrastive learning to obtain the contrastive objective, which is then used as a measurement of the difficulty level for each sentence containing idioms or metaphors. then, our proposed dynamic scheduling strategy is used to re-arrange the training examples. finally, the model is trained via the classification objective and the contrastive objective. ## analysis ablation study to investigate the effects of the different components in our method, i.e., contrastive learning and curriculum learning, we compare variants of our method without curriculum learning (w/o cl) and without contrastive learning (w/o cts). as shown in analysis on data splits. our method's effectiveness is most prominent on unseen idiomatic expressions, as shown in analysis on the datasets. results shown in analysis on the cross-task transfer. results shown in for the transfer study, we use the random split of magpie dataset and vua18. when trained on the dataset for one task and tested on the dataset for another task, our method always outperforms the baseline method, melbert. besides, we observe that the models achieve good results in idiom usage recognition when trained in metaphor detection. however, when trained on idiom usage recognition, the models' performance on metaphor detection is much worse. therefore, the symbolic knowledge learned during the task of metaphor detection could be transferred to perform the idiom usage recognition while the idiomatic knowledge cannot help with the metaphor detection. we leave the deeper study of this phenomenon to future research. ## conclusion and future work in this paper, we propose a novel method specifically for non-compositional expression detection, including idiom usage recognition and metaphor detection. our proposed method combines contrastive learning and curriculum learning. contrastive learning is used to build better representations to model non-compositionality. besides, the difficulty levels obtained from the contrastive learning objective are dynamically updated during the training, based on which the training examples are dynamically scheduled. as a result, the model could be trained in an easy-to-hard manner. we evaluate our proposed method on both idiom usage recognition and metaphor detection. experiment results affirm the effectiveness of our method on both tasks. detailed ablation studies and analyses are provided to support our claims. as a result, our work is the first to propose a framework for idiom usage recognition and metaphor detection. our proposed framework also shows better cross-task transfer ability based on idiom usage recognition and metaphor detection.