Disambiguation of Potentially Idiomatic Expressions with Contextual Embeddings (2020)

Abstract: The majority of multiword expressions can be interpreted as figuratively or literally in different contexts which pose challenges in a number of downstream tasks. Most previous work deals with this ambiguity following the observation that MWEs with different usages occur in distinctly different contexts. Following this insight, we explore the usefulness of contextual embeddings by means of both supervised and unsupervised classification. The results show that in the supervised setting, the state-of-the-art can be substantially improved for all expressions in the experiments. The unsupervised classification, similarly, yields very impressive results, comparing favorably to the supervised classifier for the majority of the expressions. We also show that multilingual contextual embeddings can also be employed for this task without leading to any significant loss in performance; hence, the proposed methodology has the potential to be extended to a number of languages.

## introduction by definition, a multiword expression (mwe) is idiomatic in the sense that its meaning cannot be derived from the meanings of its components. however, whereas sometimes a sequence of words corresponding to an mwe only has the idiomatic interpretation (e.g., by and large), there is often also a literal interpretation of the same sequence, resulting in an ambiguity: such multiword expressions are commonly referred as potentially idiomatic expressions (henceforth, pie) and determining the correct meaning of a pie in context is shown to be crucial for many downstream tasks including sentiment analysis @xcite , automatic spelling correction @xcite and machine translation @xcite . most of the previous work capitalizes on the differences between the contexts where pies are used idiomatically and literally. following that insight, we investigate the applicability of recent contextual embedding models to disambiguation of pies. contextual embeddings, e.g. elmo @xcite and bert @xcite , have emerged in the last few years and quickly become the standard in a variety of tasks. these are very deep neural language models which are pre-trained on large-scale corpora. unlike the conventional static word embeddings, such as word2vec @xcite where each word type is represented by a fixed vector, these models assign distinct representations for each input token dependent on their context. hence, they are called contextual word embeddings, highlighting their sensitivity to the context. for example, in the sentence "can you throw this can away?" the first and second occurrence of the token can are supposed to be assigned substantially different embeddings. the extent of the contextuality of these embeddings, on the other hand, is still an open research topic @xcite . in this work, we specifically investigate whether such contextual embeddings provide sufficient contextual information to distinguish literal usages of pies from idiomatic ones. to this end, we represent the pie tokens in a certain context by their corresponding bert embeddings @xcite and perform both supervised and unsupervised pie disambiguation. the results suggest that the plain bert model, without any fine-tuning or further training, is able to encode the different usages of pies to the extent that, even a with simple linear classifier, we can substantially improve the state-of-theart on common datasets in two different languages. the unsupervised classification, on the other hand, is performed via hierarchical clustering, accompanied with a simple heuristic, that pies with literal interpretations are semantically closer to their context than the idiomatic ones. for the most of the time, the unsupervised classification also achieves unprecedented performance although not as consistently as its supervised counterpart, failing completely with some expressions. finally, we compare the performance of the monolingual bert models with the multilingual-bert (mbert) to investigate the applicability of our approach to other low resource languages as well as to provide further insight regarding the cross-lingual capabilities of the multilingual contextual embeddings when they are employed directly; that is, without any fine-tuning in the target language. the results show that multilingual-bert achieves comparable results to monolingual models across all datasets, suggesting that the proposed methodology can straightforwardly be extended to other languages. ## related work a number of models have been proposed in the literature to disambiguate pies, with a trend shifting from employing linguistic features to more neural approaches, similar to the rest of the field. @xcite adopt an unsupervised approach relying on the hypothesis that multiword expressions are more likely to occur in different canonical forms when used literally. @xcite propose a generalized method (as opposed to "per-idiom classification") employing cohesion graphs which initially include all the words in the sentences. they hypothesize that a pie is used figuratively if the removal of the pie improves the cohesion. @xcite prepares a dataset consisting of high confidences instances found by @xcite and train a supervised classifier to classify the rest of the instances. @xcite use a variety of features including bag of all content words along with their concreteness measures and train a l2 regularized logistic regression (l2lr) classifier @xcite . @xcite also utilize the cues the context of the pie provides and adopt an ensemble learning approach based on three different classifiers trained on different representations of the context. @xcite propose a "literal usage metric" which quantifies the literalness of pie. this metric is computed as the average similarity between the words in the sentence and the "literal usage representation" which is the set of the words similar to the literal meanings of the pie's main constituent words found in large corpus. do @xcite use a multi-task learning approach covering four different non-literal language using tasks including classification of idiomatic use of infinitive-verb compounds in german using recurrent sluice networks @xcite . similar to @xcite , @xcite adopt a generalized approach and propose a novel "semantic compatibility model" which is a modified version of cbow, adapted specifically to the disambiguation of the pies task. in a related line of research, contextual embeddings are successfully applied to the general problem of word sense disambiguation (wsd). @xcite show that bert embeddings form distinct clusters for different senses of a given word in line with its promise to be contextual. @xcite approach wsd as a sentence pair classification task and fine-tune bert where the input consists of a sentence containing the target word and the one of its glosses and the objective is to classify if the gloss matches the sense of the target word in the sentence. ## method the task here is to distinguish the compositional (literal) and non-compositional (idiomatic) usages of a known pie in a certain context as opposed to mwe extraction which is the task of discovering mwes in a corpus. hence, the input to our method is a set of sentences containing a target pie. we regard disambiguation of pies as a word sense disambiguation problem. our basic assumption is that the context, in which pies occur literally and figuratively are distinct enough from each other to be assigned a fundamentally different contextual representations. below, we briefly introduce the contextual language model we use in the experiments, bert, followed by the descriptions of the supervised and the unsupervised classifiers. ## experiments we conduct our experiments on the widely used datasets in two languages: the vnc dataset @xcite and semeval5b @xcite for english and the horbach dataset for german @xcite . in order to have comparable results, we follow the the official train/test split of semeval5b dataset whereas for vnc dataset, we used multiword expressions which have at least 10 instances with both literal and idiomatic usage following @xcite . since there is not any official train/test split for both vnc and horbach datasets, we report the results of 5-fold cross-validation for the former foot_3 and 10-fold for the latter. we use scikit-learn library @xcite and agglomerative clustering. the learning rate of the perceptron is set to 1 × 10 -5 . the embeddings are normalized before they are fed into the classifiers. as the length of the available context differ for each dataset, we limit the context to the sentence containing the pie. we use the embeddings from the last layer of the bert models in the experiments; yet, we conduct a layer-wise analysis as well (see section 6). ## results our average results with a detailed comparison with the previous studies are provided in ## discussion in this section, we further discuss some implications of our results. overall, we comprehensively evaluated our approach in three datasets. the performance of the supervised classification is pretty consistent across all the pies in two languages, ranging between 0.77 to 1.00 f-score with a mean of 0.92 (±0.06). hence, the increase in the average results are not due to a significant increase in a subset of pies but constant improvement in all pies covered in the datasets. as for unsupervised classification, in line with our hypothesis, most of the time bert embeddings form distinct enough clusters corresponding the different usages of pie, allowing high performing unsupervised classification. yet, the unsupervised classifier is more prone to make errors as it completely fails with certain expressions which significantly lowers its overall performance (see figure 1 ). we group the errors of the unsupervised classification under two categories: • clustering errors occur due to the formation of poor clusters, consisting of pies with different usages. clustering errors happen relatively rarely in english, where there are only four expressions ( "blow whistle", "pull leg", "break a leg", "in the fast lane") with f-@xmath0.6; as opposed to german where the unsupervised classifier achieves only 0.59 f-score on average. we suspect that behind the high error rate in german lies the fact that german mwes exhibit a wider range of polysemy both in literal and figurative interpretation. @xcite also discusses this point as one of the challenges during annotation, stating that there are not very clearly separated uses of the respective verbs in the dataset, as opposed to, e.g., "bread and butter" in english which has a dominant figurative interpretation. for example, according to @xcite , stehen+bleiben (stand+still) has a large number of meanings, some of which are (i) a person's heart may stand still; (ii) people may stand still in their mental development; (iii) you can claim that a statement cannot "remain standing" (remain uncontradicted). this point is also visible in the dendrograms of german pies where there are more distinct clusters on the lower levels (figure 3 ). a preliminary analysis of these clusters show tendencies towards this direction, but a more systematic evaluation is left for future work. the most representative example of this error is the expression "break a leg" where the supervised classification achieves the f-score of 0.89 whereas the unsupervised classifier completely fails as our heuristics fail to label the clusters correctly. we ran a further experiment with an updated heuristics where we directly measure the cosine similarity between the sentence and the pie by (a) "bread and butter" (b) "through the roof" (c) "liegenbleiben" (d) "stehenbleiben" figure 3: hierarchical clustering of several cherry-picked english and german pie embeddings obtained from the respective monolingual bert model. the leaves corresponding to idiomatic examples are labeled whereas the rest are left empty in order to visualize how the idiomatic and literal instances are separated across clusters. furthermore, we ran all the experiments without performing lemmatization on the target expressions (see section 3.2) to see if lemmatization had any adverse effect on bert. overall, lemmatization turned out to lead to mixed results (1 to 2 point change in f-score) but surprisingly mostly positive; the surface (unlemmatized) forms achieve slightly better performance (+1 f-score) only on vnc dataset in the supervised setting and on semeval dataset in the unsupervised setting when multilingual embeddings are employed. however, as discussed in section 3.2, without lemmatization it is not possible to know if the classifiers exploit the possible correlation between the surface forms and associated usages. therefore, we believe lemmatization is a necessary pre-processing step as it allows us for that correlation, without harming the performance. we, additionally, conducted a layer-wise analysis as different layers of bert is shown to capture different properties of the language @xcite . in addition to each layer, we experiment with the concatenation of the last four layers following the original bert paper @xcite which claims that it yields the best contextualized embeddings. the results show that the sixth layer and upwards yield better performances where the concatenation of the last four layers leads mixed results, leading a slight drop on two datasets and increase in one (figure 2 ). finally, as can be seen in figure 1 , the performance of the supervised classifier with mbert embeddings are consistent across pies which suggests that disambiguation of pies can be performed with high accuracy in a large number of languages, requiring only a small set of annotated sentences, e.g. the portion of the vnc dataset used in the experiments contains only 61 sentences annotated per mwe on average (see ## conclusion in the current paper, we have proposed two methods, one supervised and one unsupervised, for disambiguation of potentially idiomatic expressions in running text. our models utilize contextual embeddings which are able to recognize the different usages of the same lexical units and assign representations accordingly. experimental results in two languages show both of our classifiers substantially outperform the previous state-of-the-art; yet, there is much room for improvement, especially with unsupervised classification which is less stable. the proposed methodology, furthermore, is shown to have a high potential to be extended into a large number of languages thanks to the multilingual contextual embeddings.