Does She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models (2021)

Abstract: Recent progress in pretraining language models on large corpora has resulted in significant performance gains on many NLP tasks. These large models acquire linguistic knowledge during pretraining, which helps to improve performance on downstream tasks via fine-tuning. To assess what kind of knowledge is acquired, language models are commonly probed by querying them with ‘fill in the blank’ style cloze questions. Existing probing datasets mainly focus on knowledge about relations between words and entities. We introduce WDLMPro (Word Definitions Language Model Probing) to evaluate word understanding directly using dictionary definitions of words. In our experiments, three popular pretrained language models struggle to match words and their definitions. This indicates that they understand many words poorly and that our new probing task is a difficult challenge that could help guide research on LMs in the future.

## introduction natural language processing (nlp) has advanced drastically in the last decade with the design of larger and more sophisticated models, availability of larger corpora and increasing computational power. pretrained word embeddings @xcite popularized the use of distributed word representations, which became a fundamental building block for nlp systems. @xcite introduced lstmbased deep contextual representations and obtained large performance gains by fine-tuning on tasks after unsupervised pretraining @xcite . more recently, the attention based transformer architecture was shown to use context more effectively @xcite and several subsequent models achieved state of the art results in many nlp tasks by combining the transformer architecture with unsupervised pretraining and task specific fine-tuning @xcite . @xcite showed that language models can be applied to a variety of tasks without task specific fine tuning. this is demonstrated on a much larger scale by @xcite . deep models improve performance. however, what they actually learn about language and word meaning is still to a large extent unclear due to their uninterpretable nature. for static word embeddings, researchers used word similarity @xcite and word analogy @xcite tests to shed light on what information is captured in these dense vector spaces. for language models, a great amount of linguistic knowledge is stored in the model parameters @xcite . several studies proposed using 'fill in the blank' type cloze statements to test knowledge learned by these models during unsupervised pretraining. @xcite proposed the lama (language model analysis) probe to test the factual and common sense knowledge stored in language models. similarly, @xcite introduced wnlampro (wordnet language model probing) to assess the ability of language models to understand words based on their frequency. in wn-lampro, cloze style questions are generated based on antonym, hypernym and cohyponym relations among words extracted from wordnet. the existing probing datasets mainly focus on investigating the knowledge about relations between words or entities. however, a more direct way of testing whether a language model understands the meaning of a word is to use its dictionary definition. if a pretrained language model truly understands the meaning of a word, then it should be able to match it with its dictionary definition. based on this motivation, we introduce the word definition language model probing (wdlmpro) dataset; foot_0 it is a challenging benchmark for testing nlp models for their ability to understand words. wdlmpro is essentially a set of thousands of synset groups; each synset group consists of a target word (with its definition) and its taxonomic sisters (with their definitions). using taxonomic sisters, rather than random word groups, makes the task more challenging for statistical models that are based on the distributional hypothesis since these words have similar distributional characteristics @xcite . we evaluate two masked language models, bert and roberta, and the auto-regressive model gpt-2 on wdlmpro using two different probing tests: (i) match definition to word (d2w) (ii) match word to definition (w2d). we also provide a baseline using static fasttext embeddings @xcite . we find that all three language models perform clearly better than the baseline. nevertheless, they have great difficulty matching words and their definitions, implying a poor understanding of word meaning. this is an important result that could help guide research on lms in the future. ## wdlmpro in this section, we introduce wdlmpro (word definition language model probing), a dataset to test how well nlp models can match nouns and verbs with their definitions. we view this as a test of how well the models understand lexical meaning. ## results tables 4 and 5 present w2d and d2w results for bert, roberta and gpt-2 along with fasttext and random baselines. language models perform clearly better than both baselines. larger models perform generally better than smaller ones and roberta consistently outperforms bert. this might be an indication for the correlation between performance on wdlampro and downstream performance. however, further investigation is necessary to show the correlation more clearly. for w2d, best performance is achieved by gpt-2 xl for nouns (47.3 p@1, 0.81 rs) and by roberta large for verbs (50.8 p@1, 0.84 rs). performance on d2w is much lower than for w2d for all models. for nouns, roberta large and gpt-2 xl perform similarly (28.8 and 29.8 p@1, 0.70 and 0.73 rs) while roberta large achieves the best results for verbs (38.6 p@1, 0.80 rs). poor performance on d2w compared to w2d might be due to language models' ability to distinguish different definitions better than individual words since definitions are more informative than individual words. overall gpt-2 models perform better than masked language models (with the exception of roberta large for verbs), despite using a single pattern as opposed to the multiple patterns used by masked language models. this might indicate that the alm objective is better at learning word meaning than the mlm objective. to investigate the effect of frequency, we stratify words into rare (fewer than 10 occurrences), medium (10 to 99 occurrences) and frequent (100 or more occurrences), based on occurrences in wwc foot_4 @xcite ), analysis. the correct definition of the medium frequency verb 'beckon' is 'signal with the hands or nod'. gpt-2 xl predicts 'signal by winking'. the correct definition of the frequent noun 'roleplaying' is 'acting a particular role (as in psychotherapy)' gpt-2 xl predicts 'acting the part of a character on stage'. so gpt-2 xl understands that beckoning is signaling and that roleplaying is acting, but it has not learned to distinguish between different types of signaling and acting. this points to an important future goal for lms: they should be developed to gain an understanding of words that goes beyond the current superficial state of the art. human performance on wdlampro. it is beyond the scope of this paper to evaluate human performance on the entirety of wdlampro. however, we provide a comparison with human performance on a small subset to provide an intuition about the difficulty of the task. for each of the two tasks, 20 synset groups that have a maximum of 10 candidates are randomly sampled from wd-lampro. then two native english speakers are asked to rank the candidates. human performance is the upper bound for many nlp tasks. we believe that this is not the case for wdlampro: arguably, we should aim for models with an excellent understanding of the meanings of words even if it is better than average human understanding. knowledge based tasks are an analogous case: we should strive for models that know as many facts as possible even if that performance is above average human performance. ## conclusion we introduced wdlmpro, a probing test that helps analyze how well a model understands word meaning. wdlmpro is complementary to existing probing tests that are about relations between words or entities. we evaluated three popular pretrained language models on the w2d (word to definition) and d2w (definition to word) tasks. our findings show that, despite their remarkable performance on many downstream tasks, these models struggle to match a word and its true definition, suggesting an insufficient understanding of word meaning. relatively poor performance of these powerful models on wdlmpro can be seen as evidence for the limitations of purely distributional systems and the need for incorporating external knowledge. wdlmpro provides an important evaluation benchmark, encouraging design and training of models with precise word understanding.