BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERT ology (2021)

Abstract: An important question concerning contextualized word embedding (CWE) models like BERT is how well they can represent different word senses, especially those in the long tail of uncommon senses. Rather than build a WSD system as in previous work, we investigate contextualized embedding neighborhoods directly, formulating a query-by-example nearest neighbor retrieval task and examining ranking performance for words and senses in different frequency bands. In an evaluation on two English sense-annotated corpora, we find that several popular CWE models all outperform a random baseline even for proportionally rare senses, without explicit sense supervision. However, performance varies considerably even among models with similar architectures and pretraining regimes, with especially large differences for rare word senses, revealing that CWE models are not all created equal when it comes to approximating word senses in their native representations.

## introduction contextualized word embedding (cwe) models such as bert @xcite , which were enabled by transformers @xcite , have yielded great improvements in a variety of nlp tasks. however, because bert and other cwe models are deep neural networks with complicated architectures and very high parameter counts, it is not easy to understand exactly which aspects of linguistic form and meaning contextualized word embeddings are able to capture. in response to these difficulties, much work has been done attempting to interpret cwe models, most notably in the field of bertology. over 100 bertological studies have been published since bert was introduced in 2018, covering a diverse range of linguistic phenomena @xcite such as pos tags, constituency, and event factuality @xcite . the methods developed in this area are usually applicable to other cwe models. an important question for cwe models is how well they can represent rare word senses. word sense disambiguation systems have been observed to be most lacking in the long tail of rare word senses @xcite , and sense-awareness is of fundamental importance for many nlp applications. thus a better understanding of how well cwe models understand senses, and especially rare senses, would aid the interpretation of nlp systems. to address this question, we perform an evaluation we call cwe similarity ranking on two senseannotated english corpora using several popular cwe models. we find that while all models outperform a random baseline on the evaluation, models differ substantially in their performance on rare word senses, with significant differentiation even between models which are closely related. foot_0 2 previous work ## cwe similarity ranking we formulate a query-by-example task in which a word in sentence context is used to query for similar usages of the same word in other sentences. in a process we call contextualized word embedding (cwe) similarity ranking, we will assume some embedding function f and two corpora of senselabeled text segmented into sentences: a larger "database" corpus d, and a smaller "query" corpus q. we will use sentences from q to rank sentences in d, as detailed below. 1. select a query sentence from q consisting of tokens w (q) 1 ,...,w (q) n , where a token w (q) i has been designated as the target token and has sense s i . 2. find embeddings for the query sentence, h ## experimental setup we use the cwe similarity ranking method to evaluate several popular pretrained cwe models retrieved from huggingface.co @xcite . corpora our two corpora are ontonotes 5.0 @xcite , which has sense annotations for nouns and verbs, foot_2 and the pattern dictionary of english prepositions (pdep) corpus @xcite , which has sense annotations for prepositions. 4 for ontonotes, we discard instances labeled with "none-of-the-above" senses, as we expect them to be heterogeneous. for pdep, we query: but with all the money and glamour of high finance come the relentless pressures to do well; pressure to pull 4 off another million before lunch [...] 1: "sometimes," he says, "we'll pull 3 someone off phones for more training." 2: hence, they have never lacked their own stately or amusing charms to pull 2 in wealth and keep it within a household. 3: i can't pull 4 it off. 4: bulatovic says kostunica was able to pull 4 off the balancing act because he is not really anti-american. query: these days i take five pills a day, but at one point i was on 20 about 20. 1: i'm happier than i was three years ago, when i was drinking and i was on 20 cocaine. 2: i was on 20 about sixty cigarettes a day. 3: i was on 20 heavy duty painkillers for 48 hours. 4: he had been put on 20 prescription drugs to help him cope with coming off crack. 5: recorded in 21 days in a mitcham garage, the fact that it is on 2 chrysalis is a mere coincidence. 6: you can phone now on 12 o-five-hundred , four-o-four , treble zero to put your views to tory mp phil gallie . ## results main results are given in two example queries and top results appear in figure 1 . consider the results for the "on" query at right: out of a large haystack (1728 available tokens of "on"), the system has correctly retrieved 6 of the 14 relevant needles, featuring 4 at the top of the list! interestingly, even though the complement of on in the query instance is just "20" (a fused-head construction; elazar and goldberg, 2019) instead all models perform well above the random baseline across all buckets, and show little differentiation for buckets with only non-proportionally-rare senses @xmath0.25. dramatic differences emerge, however, for buckets with proportionally rare senses @xmath1.25, with gpt-2 @xcite showing the poorest performance. in line with general trends, the distilled variants of models @xcite generally track only a few points behind their undistilled variants. overall, bert-base-cased performance is best. albert @xcite usually performs similarly, and roberta @xcite and @xcite show noticeably worse performance on rare senses. it is especially surprising that roberta performs so much worse than bert on rare senses (cf. ## conclusion we have presented an evaluation method for probing the word sense content of contextualized word embeddings and applied it to several popular cwe models. we find that their ability to capture rare word senses using their representations is variable and not easily explained by each model's training and architectural characteristics. moreover, we find that performance of models on this word sense evaluation is not directly proportional to their performance on downstream task benchmarks like glue. we view these differences between models as reason for further evaluation of cwe models to investigate their word sense representations, and inquiry into factors that affect word sense content of contextualized word embeddings. zhilin yang, zihang dai, yiming yang, jaime g. carbonell, ruslan salakhutdinov, and quoc v. le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in proc. of neurips, pages 5754-5764, vancouver, bc, canada.