Does BERT Understand Idioms? A Probing-Based Empirical Study of BERT Encodings of Idioms (2021)

Abstract: Understanding idioms is important in NLP. In this paper, we study to what extent pre-trained BERT model can encode the meaning of a potentially idiomatic expression (PIE) in a certain context. We make use of a few existing datasets and perform two probing tasks: PIE usage classification and idiom paraphrase identification. Our experiment results suggest that BERT indeed can separate the literal and idiomatic usages of a PIE with high accuracy. It is also able to encode the idiomatic meaning of a PIE to some extent.

## introduction understanding idiomatic expressions is important for nlp tasks such as sentiment analysis @xcite and machine translation @xcite . however, due to the non-compositionality of idioms, it remains a challenge to model the semantic meanings of idioms effectively @xcite . bert is a contextualized pre-trained language model that has been widely used and proven to be highly effective for many nlp tasks @xcite . to better understand how bert works, recently the community has adopted the approach of probing, where a probing task is designed to test whether bert encodings contain sufficient information to perform the task well. examples of probing tasks include pos tagging and parsing @xcite as well as semantic reasoning tasks such as understanding numbers @xcite . it is therefore also natural to ask whether bert encodes any knowledge about the usage and meanings of idioms, given that bert was trained on huge corpora, which must contain many idiomatic expressions. however, this problem has not been well explored. to the best of our knowledge, the closest existing work is by @xcite , who studied whether pre-trained (static and contextualized) word embeddings can detect meaning shift and implicit information of phrases, with the help of several probing tasks. however, we believe there is a need for further exploration. we note that @xcite did not specifically focus on idioms; only one of the six probing tasks was directly related to idioms, and only idiomatic noun compounds were studied. since english idioms have different syntactic structures, it would be useful to experiment with a higher coverage of different types of idioms. in this paper, we focus on probing bert to understand whether bert embeddings can encode the meanings of a diverse range of different types of idioms. we propose two probing tasks to test whether bert understands idioms. first, given a context containing a potentially idiomatic expression (pie), the task is to decide whether the meaning of the pie is literal or idiomatic, based on the bert-encoded contextualized embedding of the pie. we hypothesize that if pre-trained bert could perform the task well, it would indicate that bert knows the difference between literal and idiomatic usages of the same expression based on its context. for this task, we use a large dataset recently released by @xcite , which covers 1756 unique idioms and 50k contextual sentences, much larger and more diverse than the idiomatic noun compounds dataset used by @xcite . however, this task is not sufficient to show whether bert truly understands the idiomatic meaning of a pie. in order to test this, we design a second probing task based on existing idiom paraphrase datasets. the task is to select the correct paraphrase of an idiom among a set of candidate phrases based on the cosine similarity between the idiom's bert embedding and these candidate phrases' bert embeddings. we hypothesize that if the correct paraphrase could be ranked higher than other irrelevant phrases, it would indicate that bert indeed understands the idiomatic meaning of the idiom. it is important to note that our objective is not to improve the performance of the two tasks by designing effective learning methods; rather, the objective is to use these two tasks to probe pretrained bert in order to understand how much bert encodes the meanings of idioms. therefore, the models for the two probing tasks are simple models without many parameters to be learned. through our empirical study using both the original bert and ernie2 @xcite (an improved version of bert), we find that compared with non-contextualized embedding representations of pies, contextualized bert and ernie2 embeddings of pies can clearly achieve higher accuracy for pie usage classification, with an accuracy level around 90%, suggesting that bert can use the context to accurately guess whether an expression is used literally or idiomatically. for paraphrase identification, we find that bert and ernie2 perform significantly better than a random baseline, although the absolute performance is still considered low. since paraphrase identification is itself challenging, to put things in perspective, we also compare with paraphrase identification for general multi-word expressions (mwes). contrary to our expectation, we find that identifying paraphrases for general mwes does not necessarily fare better than for idioms. further analysis reveals that this is because bert contextualization actually hurts paraphrase identification for general mwes but not so for idioms. ## probing tasks we design two probing tasks to answer two research questions: (1) can bert distinguish the idiomatic usage of a pie from its literal usage? (2) can bert understand the idiomatic meaning of an idiom? both questions are related to the capabilities of bert to understand idioms, but the second task is more demanding than the first. the two tasks also share similar objectives as the probing tasks designed by @xcite , which aimed to test whether pre-trained word embeddings can detect the shift of meaning of a phrase from its component words, and whether pre-trained word embeddings understand the implicit meaning of a phrase. however, they are con-ducting probing at word level, which focuses on whether the meaning of a word in a noun compound (nc) is literal. the dataset only has 90 noun compounds @xcite . although they try to augment the dataset using @xcite , the dataset is still limited to 3k. the paraphrase identification task used by them also uses compounds and addresses whether the paraphrase describes the semantic relation between two words of a noun compound @xcite . in this paper, we use a much larger dataset called magpie @xcite ) that covers much more potentially idiomatic expressions for phraselevel literal-idiomatic classification. to make the task more challenging, we choose to split the data such that the idiomatic expressions in the training, development, and test sets do not overlap. we further adapt several paraphrase datasets @xcite @xcite to compare phrasal semantic relatedness for idioms. we compare the effect of bert encodings at different layers for the two probing tasks to better understand the effect of contextualization. ## experiments for each of the two probing tasks above, we use pretrained bert 2 and ernie2 3 to process each context (w 1 , w 2 , . . . , w n ). following standard practice, we prepend the [cls] token to the beginning of the sequence and append the [sep] token to the end. the sequence is then fed into an l-layer 2 huggingface.co/bert-base-uncased 3 huggingface.co/nghuyong/ernie-2.0-en bert. let h k @xmath0. when @xmath1, h 0 i denotes the combined representation of the word embedding, the position embedding and the token type embedding before it is fed into the transformer-based encoder. for each pie, we get a sequence of hidden vectors at the kth layer for the m tokens inside this pie as follows: for both probing tasks, we experiment with both the original bert @xcite and ernie2 @xcite , which supports phrase masking by using lexical analysis and chunking tools to get the boundary of phrases in the sentences. our code and data are released on github foot_1 . ## conclusion in this paper, we use two probing tasks to study whether bert understands english idioms. in conclusion, we find that bert is able to detect idiomatic usages of a pie with a high accuracy, and with more contextualization as the layer increases, bert helps the most in recognizing literal usages of pies. however, this only proves that bert is effective in detecting meaning shift for idiomatic expressions. to further probe if the shifted meanings are closer to their paraphrases, we adopt the paraphrase identification task by gathering three different types of paraphrase pairs, mwes-mwes, mwes-idioms and idioms-idioms. our experiments show that bert is able to encode the idiomatic meaning to some extent. however, contextualization may have different effects for mwes and idioms, which still requires further exploration to fully explain.