Contextualized Embeddings Encode Monolingual and Cross-lingual Knowledge of Idiomaticity (2021)

Abstract: Potentially idiomatic expressions (PIEs) are ambiguous between non-compositional idiomatic interpretations and transparent literal interpretations. For example, “hit the road” can have an idiomatic meaning corresponding to ‘start a journey’ or have a literal interpretation. In this paper we propose a supervised model based on contextualized embeddings for predicting whether usages of PIEs are idiomatic or literal. We consider monolingual experiments for English and Russian, and show that the proposed model outperforms previous approaches, including in the case that the model is tested on instances of PIE types that were not observed during training. We then consider cross-lingual experiments in which the model is trained on PIE instances in one language, English or Russian, and tested on the other language. We find that the model outperforms baselines in this setting. These findings suggest that contextualized embeddings are able to learn representations that encode knowledge of idiomaticity that is not restricted to specific expressions, nor to a specific language.

## introduction multiword expressions (mwes) are lexicalized combinations of multiple words, which display some form of idiomaticity @xcite . in this paper we focus on potentiallyidiomatic expressions (pies), i.e., expressions which are ambiguous between a semanticallyopaque idiomatic interpretation, and a compositional literal meaning. in the following example, the english pie hit the road has an idiomatic meaning corresponding roughly to 'start a journey': on the other hand, hit the road, can also be used literally, as in the example below: 2. two climbers dislodged another huge block which hit the road within 18 inches of one of the estate's senior guides. 1 pies occur across languages, with one particularly common class of pie cross-lingually being verb-noun combinations @xcite -i.e., pies consisting of a verb with a noun in its direct object position -such as hit the road in the example above. although vncs are common, pies also occur in other syntactic constructions, with english examples including combinations of a verb and prepositional phrase -e.g., skating on thin ice (which can be used idiomatically to mean roughly 'at risk') -and prepositional phrasese.g., off the hook (with a potential idiomatic meaning of roughly 'out of danger'). distinguishing between literal and idiomatic usages of pies could be particularly important for down-stream natural language processing applications such as machine translation @xcite . previous work has considered both unsupervised and supervised approaches to predicting the tokenlevel idiomaticity of pies. however, annotated data to train supervised approaches is not available for all pies in all languages. this makes unsupervised approaches (e.g., @xcite @xcite , which do not have this resource requirement, appealing. on the other hand, supervised approaches (e.g., @xcite tend to outperform unsupervised approaches, but are restricted to languages and pies for which annotated training data is available. in this paper we consider supervised approaches based on contextualized embeddings @xcite @xcite to predicting usages of pies as idiomatic or literal; however, we measure the ability of these approaches to generalize to expressions that were not observed during training, and also to generalize across languages. we begin by considering monolingual experiments for english and russian in which we train and test on instances of the same pies. for english, we focus on vncs @xcite . for russian, we consider a wider-range of types of pies @xcite . we then consider a second monolingual setting in which we evaluate on pies, again either english or russian, that were not observed during training. finally, we consider cross-lingual detection of idiomaticity. here we train on instances of pies in one language, english or russian, and evaluate on instances of pies in the other language. our findings evaluating on expressions that were observed during training are similar to those of @xcite ; we achieve strong improvements over baselines, and on english outperform previous approaches based on conventional word embeddings @xcite . in monolingual experiments evaluating on pies that were not observed during training, we again improve over baselines, and in the case of english, also over a strong linguistically-informed unsupervised baseline. in cross-lingual experiments, in which the model is evaluated on instances of pies in a language that was not observed during training, we again improve over baselines, and remarkably observe performance roughly on par with that of monolingual experiments evaluating on expressions not observed during training. these findings suggest that contextualized embeddings are able to learn representations that encode knowledge of idiomaticity that is not restricted to specific expressions, nor to a specific language. focusing on specific languages and types of expressions can improve unsupervised approaches. for example, focusing on vncs, the idiomatic interpretations of vncs are typically lexicosyntactically fixed. returning to the hit the road example from section 1, the idiomatic interpretation is typically not accessible if the determiner is indefinite (e.g., hit a road), the noun is plural (e.g., hit the roads), or the voice is passive (e.g., the road was hit); in such cases typically only the literal interpretation is available. @xcite propose an unsupervised statistical method based on the lexico-syntactic fixedness of vncs to determine the canonical forms -with respect to the determiner, number of the noun, and voice of the verb -of vncs. they observe that idiomatic usages of vncs tend to occur in canonical forms, and that literal usages tend to occur in non-canonical forms. a strong, linguistically-informed unsupervised baseline for distinguishing literal from idiomatic vnc usages is therefore to label canonical form usages as idiomatic, and non-canonical form usages as literal. @xcite propose a supervised approach to predicting the token-level idiomaticity of pies, focusing on english vncs, based on training an svm on skip-thoughts @xcite representations of sentences containing pies. @xcite achieve better results using a simpler sentence representation based on average of word embeddings. moreover, king and cook show that adding a single binary feature to the sentence representation indicating whether the vnc occurs in a canonical form -based on the method of @xcite -gives substantial improvements. @xcite propose a supervised approach in which pie instances are treated as single units by fusing their lexicalized component words, and learning representations of these units using word and contextualized @xcite embeddings. hashempour and villavicencio also focus on vncs. although they show improvements by treating vnc instances as fused units, they do not outperform king and cook; they do, however, train their models on smaller corpora. @xcite use representations of spans of tokens based on contextualized embedding for predicting a range of mwe properties. most closely related to our work, they consider light-verb construction and verb-particle construction classification, for both of which there is an ambiguity between mwe usages and similar-on-the-surface literal combina-tions. shwartz and dagan do not, however, consider english vncs or russian idioms as we do. @xcite propose a supervised approach to classifying instances of potentiallyidiomatic expressions, as idiomatic or literal, based on contextualized embeddings. they represent mwe instances as the average of the contextual embeddings for the tokenized pieces of their lexicalized component words, which are lemmatized in a preprocessing step, and use a single-layer perceptron for classification. their findings indicate that their approach improves over previous approaches on english and german pies. in this paper, similarly to kurfalı and östling, we consider an approach based on contextualized embeddings, but we consider experimental setups in which classifiers are evaluated on expressions, and also languages, that are unobserved during training. ## predicting pie idiomaticity with contextualized embeddings previous supervised approaches to identifying idiomatic instances of pies have represented pie instances with sentence embeddings @xcite . we consider a similar approach here using contextualized embeddings from bert @xcite , roberta @xcite , rubert @xcite , and mbert @xcite . specifically we represent a pie instance using the cls (classification) token for the context in which it occurs. 2 for representing english pies we use the sentence in which the target expression occurs as the context. for representing russian pies, the dataset we use (discussed in section 4.1) does not include sentence segmentation, and so we instead use a context of up to 300 characters to the left and right of the target expression. 3 because we focus on vncs for english experiments, following @xcite , for our monolingual experiments on english vncs, we also consider whether incorporating informa2 in preliminary experiments we also considered representations of english vnc instances formed by averaging and concatenating contextualized representations of the verb and noun components of a target vnc (where the verb and noun representations are themselves averages of the representations of the word pieces they are segmented into). we found these approaches to perform roughly on par with representing vnc instances using the cls token, and so only consider this approach here. 3 we did not attempt to tune this context window size, although there is scope to do so in future work. we fine tune pre-trained bert, roberta, ru-bert, and mbert models for binary classification of pie token instances as idiomatic or literal. we use two fully-connected layers on top of the contextualized embedding model. the first layer has the same dimensionality as the representation of the vnc (i.e., 768 dimensions, the hidden layer size of each of the contextualized embedding models considered, and an additional dimension when the cf feature is used) and uses the relu activation function. the second layer has 512 dimensions and uses the softmax activation function. ## materials and methods in this section we describe our datasets (section 4.1), experimental setups and evaluation metric (section 4.2), and then the implementation of our models and the parameter settings used (section 4.3). ## monolingual results in this section, we present results for the unseen and all expressions experimental setups, for monolingual experiments on english (section 5.1) and russian (section 5.2). in section 6 we present results for cross-lingual experiments. ## cross-lingual results in this section we consider cross-lingual experiments in which we train on instances of pies in a source language, and evaluate on instances of pies in a (different) target language. we consider the case of both english-to-russian and russianto-english. for english we consider both en-dev and en-test. in these experiments we train on the entire source language dataset (i.e., when russian is the source language we train on russian, and when english is the source language we train on either en-dev or en-test), and evaluate on the entire target language dataset. we use the best hyper-parameter settings for en-dev using bert for the unseen expressions experimental setup from section 5.1; i.e., we do not attempt any hyperparameter tuning for this cross-lingual experimental setup. we again compare results against a mostfrequent class baseline, and when english is the target language, also against the unsupervised cform baseline @xcite . results are shown in whereas the findings for the monolingual unseen expressions experimental setup indicate that the classifier is able to generalize to expressions that are unseen during training, these findings for cross-lingual experiments indicate that the classifier is able to generalize across languages. this suggests that the classifier has learned information about idiomaticity that is not restricted to specific expressions, nor to a specific language. the crosslingual findings furthermore seem to be inline with the findings of @xcite that cross-lingual transfer with mbert works reasonably well even when languages do not share the same script (as for english and russian), but works less well when the languages do not share the same word order (where english is an svo language, and russian has freer word-order, but svo is considered dominant @xcite ). ## conclusions in this paper we proposed a supervised model based on contextualized embeddings to predict the idiomaticity of pie instances. in contrast to most prior work on this topic, we considered the ability of the model to generalize to expressions that were not observed during training, and also to generalize across languages. code to reproduce these experiments is available. 8 we first considered monolingual experiments for english, focusing on verb-noun combinations, a common type of pie. in experiments in which we train and test on instances of the same pies, we demonstrated that an approach based on contextualized embeddings improves over previous approaches based on conventional word embeddings. we then considered experiments in which we evaluate on pies that were not observed during training, and showed that the proposed approach improves over a strong, linguistically-informed unsupervised baseline. we further found that, in con-trast to prior models based on conventional word embeddings, incorporating information about the lexico-syntactic fixedness of vncs does not lead to clear improvements, suggesting that contextualized embeddings capture this rich linguistic knowledge. in monolingual experiments on russian we considered a wider range of types of pies. here we showed that, as for english, the proposed approach improves over baselines when evaluating on expressions that were, and were not, observed during training. the experimental setup in which the model is tested on instances of pie types that were not observed during training is particularly interesting because we do not expect to have annotated instances of all pie types available for training supervised models. the findings in this experimental setup, for both english and russian, indicate that the model is capturing knowledge of pie idiomaticity that is not restricted to specific expressions. finally, we considered cross-lingual experiments in which we train on instances of either english or russian pies, and evaluate on pie instances in the other language. here the proposed model again improves over baselines, and achieves performance that is roughly on par with that of monolingual experiments in which we evaluate on pies that were not observed during training. this finding indicates that contextualized embeddings encode knowledge of pie idiomaticity that is not restricted to specific expressions, nor to a specific language. in future work, we plan to further explore crosslingual idiomaticity prediction. we would like to include more languages in the analysis to be able to measure the impact of training on multiple source languages. we further intend to consider including the target language amongst the source languages, to measure the impact of augmenting training data for the target language with data from other languages. finally, we intend to consider cross-lingual approaches for other mwe prediction tasks, such as predicting noun compound compositionality.