Leveraging Contextual Embeddings and Idiom Principle for Detecting Idiomaticity in Potentially Idiomatic Expressions (2020)

Abstract: The majority of studies on detecting idiomatic expressions have focused on discovering potentially idiomatic expressions overlooking the context. However, many idioms like blow the whistle could be interpreted idiomatically or literally depending on the context. In this work, we leverage the Idiom Principle (Sinclair et al., 1991) and contextualized word embeddings (CWEs), focusing on Context2Vec (Melamud et al., 2016) and BERT (Devlin et al., 2019) to distinguish between literal and idiomatic senses of such expressions in context. We also experiment with a non-contextualized word embedding baseline, in this case word2Vec (Mikolov et al., 2013) and compare its performance with that of CWEs. The results show that CWEs outperform the non-CWEs, especially when the Idiom Principle is applied, as it improves the results by 6%. We further show that the Context2Vec model, trained based on Idiom Principle, can place potentially idiomatic expressions into distinct ‘sense’ (idiomatic/literal) regions of the embedding space, whereas Word2Vec and BERT seem to lack this capacity. The model is also capable of producing suitable substitutes for ambiguous expressions in context which is promising for downstream tasks like text simplification.

## introduction the task of determining whether a sequence of words (a multiword expression -mwe) is idiomatic has received lots of attention @xcite . especially for mwe type idiomaticity identification @xcite , where the goal is to decide if an mwe can be idiomatic regardless of context, high agreement with human judgments has been achieved, for instance, for compound nouns @xcite . however, as this task does not take context into account, these techniques have limited success in the case of ambiguous mwes where the same expression can be literal or idiomatic depending on a particular context. for example, such models would always classify hit the road as idiomatic (or conversely always as literal) while the expression could be idiomatic in one context and literal in another. as a consequence, for practical nlp tasks, especially machine translation and information retrieval, token idiomaticity identification is needed, with the classification of a potential idioms as literal (or idiomatic) in context. for example, hit the road must be translated differently in "the bullets were hitting the road and i could see them coming towards me a lot faster than i was able to reverse", and "the ulster society are about to hit the road on one of their magical history tours" @xcite . we argue that successful classification of potentially idiomatic expressions as idiomatic/literal is not possible without taking the context into account. recently introduced contextualized word embeddings (cwes) are ideal for this task as they can provide different embeddings for each instance of the same word type. cwes such as context2vec @xcite and bert @xcite proved successful in the task of word sense disambiguation (wsd) @xcite . we also argue that disambiguation of potentially idiomatic expressions is analogous to wsd in a sense that it also tries to assign the most appropriate sense to an idiom, i.e. literal, or idiomatic depending on its respective context. moreover, we hypothesize that in order to fully exploit the capacity of cwes, an idiom should be treated as a single token both in training and testing. this hypothesis is inspired by the evidence from psycholinguistic studies which support the idea that the idiomatic expressions are stored and retrieved as a whole from memory at the time of use @xcite . it is also rooted in the idea that different types of information are captured in vectors depending on the type of input, i.e. word, character, phrase to the model @xcite . moreover, this method proved successful in other tasks. for instance, carpuat and diab (2010) conducted a study for integrating mwes in statistical machine translation (smt) and improved the bleu score by treating mwes as single tokens in training and testing. contribution: we show that cwes can be utilized directly to detect idiomaticity in potentially idiomatic expressions due to their nature of providing distinct vectors for the same expression depending on its context. we also apply the idiom principle @xcite when training the models which improves the results as expected and supports our hypothesis that an mwe should be treated as a single token both in training and testing the models. we further show that context2vec trained based on idiom principle is able to provide suitable replacement for mwes in both the idiomatic and literal senses. to the best of our knowledge, this is the first attempt to integrate idiom principle into cwes and directly use them for the task of identification idiomaticity in mwes at the token level. ## related work distributional semantic models (dsm) are computational models based on the distributional hypothesis @xcite and the idea that words occurring in similar contexts tend to have a similar meaning. recently, two flavours of distributional models have been introduced and utilized which are known as contextualized and non-contextualized embedding models. the former produces different embeddings for a word depending on the context and the latter offers only one embedding for a word regardless of the context. researchers have leveraged dsms along with linguistic knowledge to deal with identifying mwes at type @xcite @xcite and token level @xcite . for instance, the degree of linguistic fixedness was used as the basis for @xcite to apply an unsupervised method to distinguish between idiomatic and literal tokens of verb-noun combinations (vncs). they argue that idiomatic vncs come in fixed syntactic forms in terms of passivation, determiner, and noun pluralization. they extracted these forms using known idiomatic/literal vnc patterns and among all variations they determined which were the canonical form(s). then they classified new tokens as idiomatic if they appeared in their canonical forms. @xcite leveraged the idea of canonical forms and the distributional hypothesis and built co-occurrence vectors representing the idiomatic and literal meaning of each expression based on their context and (canonical) forms. the problem with this model is relying solely on the canonical form to label an expression as idiomatic/literal which is not enough as there are many mwes, e.g. kick the bucket that can be in their canonical form and yet have a literal meaning depending on the context they appear in. hence, each mwe should be disambiguated in its own individual context. @xcite also built their work based on distributional hypothesis and the principle of compositionality to classify mwes as idiomatic/literal. their idiomaticity identification model at the type level works well for mwes that are either idiomatic or literal but falls short for idiomaticity identification at the token level when the mwe is ambiguous. @xcite used different types of contextualized and non-contextualized word embeddings from character-level to word-level models to investigate the capability of such models in detecting nuances of non-compositionality in mwes. when evaluating the models, they considered the mwes out of their context which is problematic especially in case of utilizing cwes as the reason behind the success @xcite @xcite of these models is in their ability to produce context-specific embeddings for each token. the main drawback of above-mentioned works is that they do not take the context of each individual expression into account when classifying them. however, there have been some attempts to detect idiomaticity in mwes in context (at token level) using distributional models. @xcite exploited contextual information captured in word embeddings to automatically recognize idiomatic tokens. they calculate the inner product of the embeddings of the context words with the embedding of target expression. they argue that since the literal forms can predict the local context better, their inner product with context words is larger than that of idiomatic ones, hence they tell apart literals from idiomatic forms. @xcite exploited skip-thought vectors @xcite to represent the sentential context of an mwe and used svm and k-nearest neighbours to classify mwes as idiomatic or literal in their context. they compared their work against a topic model representation that include the full paragraph as the context and showed competitive results. @xcite proposed a model based on distributed representations, non-cwe to classify vnc usages as idiomatic/literal. first, they represented the context as the average embeddings of context words and trained a support vector machines (svm) classifier on top of that. they further showed that incorporating the information about the expressions canonical forms boosted the performance of their model. a related task of metaphor token detection has seen successful results with the combination of cwes and non-cwes, along with linguistic features @xcite . for instance, @xcite used word2vec and elmo @xcite as embeddings, with a bidirectional lstm to encode sentences, and a feed-forward neural network for classifying them as literal or metaphoric. rohanian et al. ( 2020 ) presented a neural model and bert, to classify metaphorical verbs in their sentential context using information from the dependency parse tree and annotations for verbal mwes. they showed that incorporating the knowledge of mwes can enhance the performance of a metaphor classification model. we follow the intuition that cwes can be directly used for the task of token level identification of idiomaticity in mwes due to their ability to produce different embeddings for the different tokens of the same mwe. our work is also inspired by the idiom principle which explains how human distinguish idiomatic expressions. ## distributional models and idiom principle in this work, we use word2vec as non-cwes and leverage the context2vec and bert as cwes in combination with the idiom principle to detect idiomaticity in potentially idiomatic expressions. the embedding models and idiom principle are briefly described here. ## experimental setup to test the hypothesis, we build 6 different context representations using three embedding models: word2vec, context2vec and bert in two different settings: 1-their original models where each expression is not treated as a single token 2-our own models, which we call idiom-principle-inspired models, where each expression is treated as a single token. for the first setting, we use the original pre-trained models, word2vec, context2vec and bert-baseuncased. for the second setting, we use bnc corpus @xcite and first lemmatize it using spacy @xcite . then, we tokenize it where each mwe is treated as a single token with an underline between the first and the second part (e.g. blow the whistle is mapped to blow whistle). finally, we build three semantic spaces, using word2vec, context2vec, and bert. our goal is to determine the correct sense of an mwe in context, based on a manually tagged dataset, vnc @xcite . following @xcite , we use the simple non-parametric version of the knn classification algorithm @xcite with @xmath0, we rely on cosine distance of the vectors. as we do not do any extra training on the dataset, we divide it into evaluation and test sets. to classify a test mwe instance in context, we consider all the instances of the same mwe in the evaluation set and find the instance whose context embedding is the most similar to the context embedding of the test instance using the context-to-context similarity measure. finally, we use the label of that instance as the correct label for the mwe in the test set. the rationale behind such a simple classification model is to make the comparison between the representations easy so that each model's success can be attributed directly to the input representations. model original idiom-principle-inspired word2vec context2vec bert ## dataset and evaluation we use vnc-tokens dataset @xcite to evaluate our models. the dataset includes sentences containing verb-noun combinations (vnc) tokens labelled as either idiomatic (i) / literal (l) (or "unknown"). for our experiments, we only use vncs that are annotated as i or l as in @xcite . we evaluate the models using five-fold cross-validation and calculate the precision, recall, and f-score per each expression and then report the average scores as in @xcite , the results are reported per sense. we also investigate to see how well different models encode information such as distinguishable senses in their vector space. ## experimental results in this section, we report the results of the first set of experiments where we create context representation using the original pre-trained models and then we present the results of the second set of experiments in which inspired by the idiom principle, we train our own models by treating each mwe as a single token. then context embeddings are inferred using these trained models. we did not expect the results to improve for word2vec as it always conflates the senses so it will not be able to learn different embeddings for different senses no matter how mwes are treated in pre-processing step. in regard to bert, we cannot see the improvement observed for context2vec. we speculate this might be due to the models inability to provide quality embeddings for rare words @xcite as treating each mwe as a single token turns it into a rare word for which the models need to learn an embedding. we will be investigating this in our future work. nevertheless, the improvement on the results are noticeable (even for bert) as we used a much smaller corpus to train our models compared to those used by the original models. we used bnc which contains 100 million words whereas the original models were trained on the corpora of much bigger size, namely google news dataset with 100 billion words for word2vec, ukwac @xcite with 2 billion words for context2vec, and the entire wikipedia with about 2,500 million words and a book corpus with 800 million words for bert. @xcite reported 0.69 f-score for the same dataset and then they used extra feature, the expression being in its canonical form or not, and increased the f-score to 0.77. however, this method is limiting as it requires feature engineering while our model of context2vec is capable of producing on par results, 0.75, without any external knowledge and by only relying on the features extracted by the model itself. we also used much smaller corpus to train our model in comparison with what they used, which was a snapshot of wikipedia from september 2015, consisting of approximately 2.6 billion tokens @xcite . moreover, they did extra training on the dataset after extracting the embeddings from their model whereas we did not do any training on the dataset. to see how robust the models are across different expressions, we created the box plot for the models using the average f-scores of the models per expression. this is illustrated in figure 1 and figure 2 of we are also curious to see whether the models are capable of placing different senses of an expression in different segments of their semantic space. for this, we use t-sne (van der @xcite to map these high-dimensional spaces into two dimensional spaces. as you can see, the word2vec and bert embeddings hardly allow to distinguish any clusters as the senses are scattered across the entire plot, both in the original and idiom-principle-inspired settings. however, in context2vec embedding space, senses are placed in clearly separable clusters especially in the idiom-principle-inspired setting. this made us dig deeper into the context2vec model and probe its level of understanding through a lexical substitution task. in the lexical substitution task, the goal is to find a substitute word for a given target word in sentential context. to do so, we remove the mwe from a sentence and then get the embeddings of the remaining sentence which is in fact the context of the mwe. then we find the embeddings of which words have the highest cosine similarity with the embeddings of the context. mwe sense sentence # proposed substitute kick heel i 1 wait, stay, stop l 2 clap, barefoot, kick hit road i 3 go, start, embark l 4 smash, drop, shoot ## conclusion and future work in this work, we used contextualized word embeddings (cwe), i.e. context2vec and bert to include contextual information for distinguishing between the idiomatic and literal senses of an idiom in context. moreover, inspired by the idiom principle, we hypothesized that to fully exploit the capacity of cwe, an idiom should be treated as a single token both in training and testing; the results showed that by applying idiom principle to cwe, especially context2vec, we can build a model to distinguish between literal and idiomatic senses of a potentially idiomatic expression in context. through dimensionality reduction and lexical substitution, we further showed that context2vec is capable of placing literal and idiomatic senses in distinct regions of semantic space; besides, the model has a good level of understating of the meaning as it suggests suitable replacement for both literal and idiomatic senses of set of mwes. in our future work, we are interested in improving the results for bert. we also would like to train the idiom-principle-inspired models on a bigger corpus to investigate how the results compare to what were achieved here.