Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification (2022)

Abstract: Many linguistic expressions have idiomatic and literal interpretations, and the automatic distinction of these two interpretations has been studied for decades. Recent research has shown that contextualized word embeddings derived from masked language models (MLMs) can give promising results for idiom token classification. This indicates that contextualized word embedding alone contains information about whether the word is being used in a literal sense or not. However, we believe that more types of information can be derived from MLMs and that leveraging such information can improve idiom token classification. In this paper, we leverage three types of embeddings from MLMs; uncontextualized token embeddings and masked token embeddings in addition to the standard contextualized word embeddings and show that the newly added embeddings significantly improve idiom token classification for both English and Japanese datasets.

## introduction potentially idiomatic phrases are often used both in the idiomatic and literal sense. for example, "blew whistle" in (1) is used in the literal sense, whereas that in (2) is used in the idiomatic sense, that is, the meaning of the phrase has shifted and in this case it means accuse. deciding whether each occurrence of a potentially idiomatic phrase is a literal or idiomatic usage is an essential process for text understanding. we call this processing idiom token classification following @xcite . recently, contextualized word embeddings have been shown to be useful for word sense disambiguation @xcite . furthermore, * ryosuke takahashi is currently at sb technology corp. @xcite showed that the contextualized embeddings including bert @xcite are useful for recognizing meaning shift of words in idioms. however, they only used contextualized embeddings, even though comparing them with the standard embeddings of the target word can be beneficial for precise detection of meaning shifts. thus, in this paper, we propose a method to improved a bert-based idiom token classifier by leveraging uncontextualized word embeddings. specifically, we use the token embedding of bert, which is the uncontextualized embedding that is input to bert and the same vector as is used for the prediction in the task of masked language model. our assumption can be explained using ( 1 ) and (2) as follows: since "whistle" in (2) is used as a part of an idiomatic phrase, its contextualized embedding differs more from the uncontextualized embedding of "whistle" than in the case of (1). furthermore, we also leverage the masked token embedding of the target word in bert, which is generated when the target phrase constituents are masked. this embedding can be considered to represent the meaning inferred from its context, and we assume that if the target phrase is used in the literal sense, as in (1), the output embedding will not significantly differ from the original embedding and thus the differences between the bert embeddings without masking and those with masking are expected to be small. ## leveraging additional embeddings the relatively high performance of bert[v v ; v n ] in a zero-shot setting indicates that the standard bert embeddings contain information about how much the meaning differs from the standard meaning of the words that comprise the phrase. however, the performance of idiom token classification can be improved by explicitly incorporating the standard meaning of the constituent words and the meaning inferred from its context. ## related work several researchers have tackled the task of idiom token classification. @xcite is one of the earliest works. they created a japanese annotated data for idiom token classification and proposed an svm-based model with a set of features that commonly used for wsd. @xcite proposed statistical measures that quantify the degree of lexical, syntactic, and overall fixedness of a verb noun combination. sporleder and @xcite proposed a model for unsupervised idiom token classification based on the observation that literally used expressions typically exhibit cohesive ties with the surrounding discourse, while idiomatic expressions do not. @xcite explored various features, such as global lexical context, discourse cohesion, syntactic structure, and local lexical features. they reported that global lexical context and discourse cohesion were most effective for idiom token classification. @xcite treated id-iom token identification as a problem of outlier detection. they extracted topics from paragraphs containing idioms and from paragraphs containing literals by using latent dirichlet allocation (lda). a broad range of neural network-based models have been proposed in recent years. @xcite obtained phrase representations by averaging skip-gram @xcite vectors of words that appear around the target phrase and applied them to idiom token classification. @xcite constructed an svm-based classifier using the distributed representation of sentences generated by the skip-thought model @xcite . @xcite improved the performance of word embedding-based methods by incorporating syntactic and lexical patterns of idiomatic expressions. more recently, methods using contextualized word embeddings such as elmo @xcite and bert @xcite have been proposed. @xcite showed that the contextualized embeddings of constituent words were useful for recognizing meaning shifts of phrases. @xcite and @xcite worked on the idiom token classification task using bert embeddings and reported that the bert-based model achieved high accuracy in a phrase-specific setting. @xcite proposed probing measures to examine how accurately idiomaticity in noun compounds is captured in vector space models and concluded that idiomaticity is not yet accurately represented by contextualized word embeddings. studies that used multiple types of embeddings in bert, similar to our method, include the work by zhang et al. ( 2020) ## conclusion we demonstrate that leveraging uncontextualized token embeddings and masked token embeddings in addition to the standard contextualized word embeddings significantly improve idiom token classification in a zero-shot setting. we also show that the results of investigating the similarities of these embeddings for each of the literal and idiomatic cases support our assumption that the uncontextu-alized token embeddings and the masked token embeddings tend to be similar to the standard bert embeddings when the target phrase is used in the literal meaning. one of the advantages of the proposed method is that it does not require training a new model because it extracts and uses embeddings with different properties from the same language model. we believe that the three types of embedding introduced in this study can be applied to other natural language tasks.