Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss (2024)

Abstract: Accurately modeling idiomatic or non-compositional language has been a longstanding challenge in Natural Language Processing (NLP). This is partly because these expressions do not derive their meanings solely from their constituent words, but also due to the scarcity of relevant data resources, and their impact on the performance of downstream tasks such as machine translation and simplification. In this paper we propose an approach to model idiomaticity effectively using a triplet loss that incorporates the asymmetric contribution of components words to an idiomatic meaning for training language models by using adaptive contrastive learning and resampling miners to build an idiomatic-aware learning objective. Our proposed method is evaluated on a SemEval challenge and outperforms previous alternatives significantly in many metrics.

## introduction among multiword expressions (mwes), idiomatic expressions (ies) are difficult to model as their meaning is often not straightforwardly related to the meaning of the component words @xcite . these expressions, which are also commonly referred to as non-compositional expressions, often take on figurative meanings. for example, eager beaver has a figurative meaning of an enthusiastic person who works very hard different from the literal meanings of its component words like impatient rodent @xcite . they are a common occurrence across various genres @xcite . accurately understanding idiomatic expressions has posed a significant challenge, as word and phrase representations may favor inherently compositional usages at the levels of both words and subwords to minimize their vocabulary @xcite . indeed recent models are mainly driven by compositionality, which is at the core of tokenization @xcite and selfattention mechanism @xcite . pretrained language models including static and contextualised embeddings do not seem to be wellequipped to capture the meanings of ies, as ies with similar meanings are not close in the embedding space @xcite . this reveals a need for models that can accurately capture idiomatic language. ensuring precise representation of ies is crucial for their precise handling in various downstream applications, such as sentiment analysis @xcite , dialog models @xcite ) and text simplification @xcite . to address this issue, previous methods often rely on new datasets with human annotations or on data augmentation @xcite . however, the use of alternative training processes has also been effective, including regression objective functions with a siamese network @xcite or substitute objectives @xcite to break the compositionality of idiomatic phrases, as finding an objective to stand for idiomatic representation is difficult. our work focuses on the development of idiomatic-aware language models, which are designed to better represent mwes of various degrees of idiomaticity in natural language text. to achieve this, we adopt the definition of idiomaticaware models from semeval 2022 task 2 (tayyar @xcite that when using the model, the semantic similarity between an ie and its incorrect paraphrase equals the semantic similarity between a correct and an incorrect paraphrase. our approach involves fine-tuning a pre-trained model using a bespoke triplet loss function that is specifically designed for capturing the asymmetry between the surface forms of the component words and their semantic contribution to the meaning of the expression. to build this idiomatic-aware language model, we use in-batch positive-anchornegative triplets @xcite . our model is trained on extracted triplets, where sentences with the idiomatic expressions and their synonyms correspond to positive and anchor respectively, and vice-versa, figure 1 . the aim of this training is to enable the model to learn the difference between the literal meanings of the component words of an mwe when used in isolation and their idiomatic meanings as part of the mwe. we use the "learnto-compare" paradigm of contrastive learning (cl), which has been successfully adopted for obtaining better text embeddings @xcite including for polysemous words @xcite . this framework fits well with our objective of distinguishing between the figurative and literal meanings of mwes. to evaluate the approach, a set of models with varying sizes and pre-training strategies is trained using this novel training method that we proposed. the best models achieved new state-of-the-art results in the dataset containing expressions of varying levels of idiomaticity, and our best model demonstrated a substantial improvements in both idiom-only performance and overall performance compared to the previous best results. our contributions are: an efficient approach for creating language models that can represent mwes of varying levels of idiomaticity. this is achieved through a specialized training process using a triplet loss function and in-batch positive-anchornegative triplets. an idiomatic-aware loss function tailored to directly optimize the representation of idiomatic language and the potentially asymmetric and non-compositional contributions of the component words. this function plays a crucial role in training to discern the nuanced differences between idiomatic and literal meanings of mwes. new state-of-the-art performance models that enable the understanding of idiomatic language. this advancement represents a major leap forward opening up new possibilities for more nuanced and accurate language understanding. the paper starts with an overview of previous work on idiomaticity representation in section 2. it also introduces contrastive learning in nlp and ie evaluation methods. section 3 presents our method using a triplet loss and data mining to do efficient training. section 4 describes our experiments, and section 5 analyzes the results. ## related work idiomaticity representation can be challenging even for large language models @xcite @xcite @xcite . for instance, gpt-3 (brown et al., 2020) reaches only 50.7% accuracy in idiom comprehension @xcite . this may be possibly due to idiomatic expressions being non-compositional and having figurative meanings that go beyond its individual words @xcite . methods that have been used for representing idiomaticity include combining compositional components with adaptive weights @xcite , representing mwes with single tokens @xcite @xcite and creating phrase embeddings that effectively capture both compositional and idiomatic expressions @xcite . the latter involves an adaptive learning process that adjusts to the nature of the phrases to generate accurate representation. an adapter-based approach is proposed that augmenting the bart model with an "idiomatic adapter" trained on dedicated idiom datasets @xcite . this adapter acts as a lightweight expert, enhancing bart's ability to capture figurative meanings alongside literal interpretations. pier @xcite , a language model based on bart, specifically addresses the challenge of representing non-compositional expressions, such as idioms, in natural language. traditional compositionality-based models often struggle with these expressions, as their meaning cannot be simply derived from the sum of their parts. pier overcomes this by incorporating an "idiomatic adapter" which learns to represent figurative meanings alongside literal ones. additionally, @xcite proposed a novel approach to idiomatic machine translation through retrieval augmentation and loss weighting, which significantly improves the translation quality of idiomatic expressions by leveraging context retrieval mechanisms and adjusting loss functions to better handle idiomaticity. contrastive learning contrastive learning is a method in machine learning that trains a model to distinguish between similar and dissimilar pairs of data. in recent times, significant progress has been made in sentence embeddings through contrastive learning @xcite @xcite @xcite . it also has been widely applied in other nlp research fields, such as text classification @xcite , machine translation @xcite , information extraction @xcite , question answering @xcite ) and text retrieval @xcite . despite their shared goal of acquiring high-quality text representations @xcite @xcite , the exploration of idiomatic representation and related research through contrastive learning is still yet to be fully explored. contrastive learning with triplet loss involves training the model on triplets: an anchor sample, a positive sample (similar to the anchor), and a negative sample (dissimilar to the anchor). the goal is to minimize the distance between anchors and positive samples while maximizing the distance between anchors and negative samples. this approach has recently been applied to tasks such as idiom usage recognition and metaphor detection @xcite . ## idiomaticity-aware objective our strategy for improving ie representation in language models utilizes a contrastive triplet loss adapted to prioritize idiomaticity and employs a miner to generate positive-anchor-negative triplets for training the model. ## experiment this section presents the comprehensive methodology employed to our model training. we begin by detailing the experiment implementation, including the hyperparameter setting, models used, evaluation method, and the overall setup. ## results and analysis in this section, we report results and analyze them in different settings. ## discussion the proposed model for training requires the identification of idiomatic expressions (ies) in each sentence beforehand. this step is crucial for reducing the difficulty of the training process. without identifying the ies beforehand, the model may not perform optimally, and its accuracy may be compromised. therefore, it is essential to ensure that the text has ies identified to achieve the best results. ## conclusion idiom representations have always been a challenge due to the non-compositional nature of idiomatic expressions. the performance of downstream tasks, such as translation and simplification, is dependent on the quality of the representations. this paper proposes a new method to train language models using adaptive contrastive learning with triplets and resampling miners. in this way, our method can build a better optimization objective, which makes the training very efficient. the proposed method, evaluated on the idiomatic semantic text similarity tasks, significantly outperforms previous methods. with limited idiomatic text data, the sole training signal of the contrastive objective is not sufficient to learn general semantic representations. therefore, the model is initialized with other pre-trained semantic-aware models. a series of base models in different sizes and pre-training strategies are trained in the proposed training loss. the best models achieve new state-of-the-art results with a significant improvement in overall over the previous best in the evaluation task.