Analyzing BERT ’s Knowledge of Hypernymy via Prompting (2021)

Abstract: The high performance of large pretrained language models (LLMs) such as BERT on NLP tasks has prompted questions about BERT’s linguistic capabilities, and how they differ from humans’. In this paper, we approach this question by examining BERT’s knowledge of lexical semantic relations. We focus on hypernymy, the “is-a” relation that relates a word to a superordinate category. We use a prompting methodology to simply ask BERT what the hypernym of a given word is. We find that, in a setting where all hypernyms are guessable via prompting, BERT knows hypernyms with up to 57% accuracy. Moreover, BERT with prompting outperforms other unsupervised models for hypernym discovery even in an unconstrained scenario. However, BERT’s predictions and performance on a dataset containing uncommon hyponyms and hypernyms indicate that its knowledge of hypernymy is still limited.

## introduction large pretrained language models @xcite have set new standards for performance on nlp tasks. many of these tasks, such as question answering and natural language inference, might seem to require human-like syntactic and semantic capabilities to perform well, leading to many studies on this topic @xcite . however, evidence that llms have human-like semantic capabilities is mixed. with respect to polysemy, @xcite find that the sense of a polysemous word can be disambiguated using a k-nearest-neighbor approach on bert's representation of the word. in contrast, @xcite note that while polysemous words' senses are linearly separable based on their bert embeddings, these embeddings do not form clusters based on their senses alone. in terms of bert's knowledge of lexical semantics, @xcite find that bert's embeddings capture useful lexical-type knowledge, performing better than non-contextual embeddings on a lexical relation prediction task. that said, bert's knowledge of lexical semantics is not equal for all words: it struggles with rare words @xcite . in this work, we further examine bert's knowledge of lexical semantics, more specifically that of hypernymy, using a cloze / prompting methodology. in the most basic form of this framework, we simply run bert on an input sentence such as "an apple is a @xcite .", and extract the probabilities assigned to each token that could fill in the blank; here, we would expect an answer such as "fruit". while @xcite also approach bert's lexical semantics using a prompting task, they focus on simple prompts and examine only if bert can predict a word's canonical hypernym. in contrast, we explore how more complex prompts affect bert's ability to predict hypernyms, propose new methods of evaluating hypernym discovery that that take into account the fact that a word may have multiple hypernyms, and find effective prompts for hypernym discovery. ## background: prompting although fine-tuning is the methodology with which llms shattered existing state-of-the-art on many downstream tasks, the use of prompting has also gained interest in recent years. unlike finetuning, prompting involves no further training to solve downstream tasks; instead, a prompt is fed to the language model, which either predicts the continuation of the prompt (autoregressive language modeling) or the identity of one or more masked words (masked language modeling). the output of the language model is then used as the response to the task at hand. prompting has seen use for diverse tasks such as knowledge base completion @xcite as well as summarization and translation @xcite . while performance on these tasks is not always up to par with supervised state of the art, prompting has the advantage of being unsupervised. however, various studies have found that task performance is highly dependent on the prompt used @xcite ; these same studies propose automatic or handmade improvements to prompts to improve them. supervised techniques have also been devised to automatically develop better prompts @xcite . beyond downstream tasks, prompting is also useful for investigating language models as a test subject. while probing @xcite , for example, is useful, it involves training an auxiliary model. this auxiliary model can be troublesome; @xcite show that the probes themselves can learn tasks independently of whether the model encodes linguistic structure. moreover, a model can contain information within its internal representations without relying on that information to make predictions. in contrast, prompting requires no auxiliary models that might complicate matters. ettinger (2020) uses prompting to compare bert's linguistic competencies and tendencies to that of humans; among those investigated is hypernymy. they find that bert easily identifies noun hypernyms, but fails to adapt to negated sentences: given the prompts "a hammer is a [mask]" and "a hammer is not a [mask]", bert predicts the words "hammer", "tool", and "weapon" for both. in a follow-up, @xcite , investigate whether bert has a systematic understanding of hypernymy. they find that bert's understanding of hypernymy is not systematic: bert fails when tasked with finding the hypernym of a hyponym in the plural; it also fails more often when the hyponym in question is uncommon, or has not been seen with its hypernym in wikipedia. in contrast to the practically-oriented works, these studies that analyze llms use primarily simple prompts to extract bert's predictions. these two uses of prompting, for downstream tasks and for analysis of llms, are somewhat in tension. on one hand, studies searching for knowledge in bert treat it as a test subject, and use prompting to search for knowledge of hypernymy using very simple prompts. on the other hand, from practically-oriented studies, we know that using simple prompts may lead to bert underperforming, if only because it is not responding to the task at hand. the use of simple prompts might thus obscure knowledge of hypernymy in bert. in this paper, we attempt to balance these two approaches. we explore more complex prompts, in an attempt to reveal knowledge of hypernymy in bert that might be hidden by challenges with prompting. however, we limit prompts to those which are human-understandable, and can be created without any supervision (excluding, thus, prompt-tuning approaches). moreover, we compare bert's performance on a hypernym discovery task to that of other systems, to give context to its hypernym discovery ability. ## bert diagnostics in this section, we use prompting to determine how well bert understands hypernymy. in order to predict the hypernym of a word (hyponym) using bert and prompting, we run bert on a prompt including a mask token, using it to predict the identity of the masked token. one such prompt might be "an apple tastes good. an apple is a @xcite .", where the hyponym is "apple", and its canonical hypernym is "fruit." ## comparison with other hypernym discovery models although the goal of this paper is to learn more about bert, we also compare bert to other models used for hypernym discovery, the task of predicting a word's hypernym. to do so, we evaluate on the test split of the semeval 2018 task 9 (camacho-collados et al., 2018) dataset, which consists of 1500 words, each with a list of valid hypernyms. this dataset is not very compatible with our approach. the hypernyms listed for each hyponym do not necessarily fit into one token. moreover, the hyponyms in the dataset are abstract nouns. many are also not necessarily common knowledge, being instead specific individuals, places, or technical terms. in contrast, the category norms dataset was designed to include only hyponyms that almost anyone would know. as in the prior experiments, we use the bert-base-uncased pretrained bert model from huggingface @xcite . we test the prompts "x is a type of [mask]" and "my favorite [mask]is x", to account for hypernyms starting with either consonants or vowels. we also test the high-performing "such-as" prompt "a [mask], such as x". we predict the top 15 hypernyms for each hypernym and evaluate using the following metrics: mean average precision (map), mean reciprocal rank (mrr), and precision at k (p@k). mean average precision can be defined as the average over k of p@k; we compute this average for @xmath0, . . . , 15. for all prompts, bert outperforms all of the unsupervised models on all metrics, as well as the worst-performing supervised model. the "such-as" prompt even manages to outperform the supervised ## qualitative analysis -does bert understand hypernymy? having quantitatively analyzed bert's outputs on various prompts, we now turn to qualitatively analyze them. the immediate results-that the quality of bert's responses depends heavily on the prompt, and none of the prompts are perfectare not terribly surprising, and follow from those of other works on prompting. however, a closer look reveals that while no prompt was successful at recovering the hypernyms we desired, certain prompts were able to most often produce reasonable hypernyms, even if they were not exactly the desired hypernyms. past works @xcite have suggested that bert's answers to hypernym queries show a lack of knowledge of hypernymy. some of our prompts display the same behavior. for example, the "type-of" prompt yields "wine", "coffee", and "grape" for the hypernyms of the color "mauve". bert guesses items that may be mauve in color, but are certainly not hypernyms of mauve; this prompt suggests that bert does not understand the hypernymy relation. however, the performance of the "such-as" prompt tells a different story. it is, in fact, more accurate than the numbers suggest-it predicts in the battig dataset, for example, "felony" to be the hypernym of "embezzlement", which is marked wrong, but is in fact correct. follow-up manual analysis of 100 (out of 419) incorrect predictions from the battig dataset using this prompt showed that 35% of answers marked as incorrect were actually correct, suggesting a true p@1 of 66.4%. this is also true of the "favorite" prompt, in which the same analysis suggested a true p@1 of 76%, higher than the "such-as" prompt's. despite this seemingly-good performance, a closer look at the incorrect answers provided by bert in response to these prompts complicates the notion that bert understands hypernymy. when using the "such-as" prompt, a pattern of errors appears with respect to city names: an oftguessed hypernym is "few". while this makes sense in the context of the prompt-consider a phrase such as "most cities, except for a [few, such as minneapolis]"-this is not a valid hypernym. however, it is a convenient failure case that bert can predict for almost any hyponym; it appears in the top 15 predictions for 438 of 1500 entries in the semeval dataset. a similar pattern appears with the "favorite prompt". on the battig dataset, bert sometimes yields hypernyms that are too generic-for example, as the hypernyms of medicine, it yields "thing", "subject", and "topic". worse, on the semeval dataset bert predicts "word" as the top hypernym for 305 of the 1500 words in the dataset; while "word" is a grammatically and logically correct answer to the prompt, it does not show any understanding of hypernymy. furthermore, the next 4 top predictions by bert cover another 427 of the 1500 entries, suggesting that bert is unable to provide a meaningful hypernym for each word. in contrast, the top hypernym on the semeval dataset for the "type-of" prompt is assigned to only 37 of 1500 words. so, the "type-of" prompt does not guess the same hypernym for many words. in light of these phenomena, it is difficult to say whether either of these two high-performing prompts has actually elicited knowledge of hypernymy from bert. the "such-as" prompt performs well, and if not for vowel / consonant issues, might perform even better. unfortunately, since the prompt does not specifically ask for the hypernym of x, its incorrect answers leave unclear whether bert is failing to systematically understand hypernymy, or is simply performing general language modeling when we would prefer that it extract a hypernym. the "favorite" prompt also performs well, and even its failures are often correct, albeit toogeneric, hypernyms. the prompt also implies that "[mask]" is the hypernym of x. moreover, because some of the hypernyms (e.g. "crime") are rather unlikely to have appeared in bert's training data in the context of the prompt (i.e. because "my favorite crime is. . . " is not a common phrase), it seems reasonable to attribute some of bert's performance to genuine learning of hypernymy, rather than simple memorization of training data. however, this is not true of all hypernyms: it is possible that bert's use of generic hypernyms like "thing" stems from a memorization of frequent constructions like "my favorite thing" in the training data. ## conclusion in this paper, we conduct a thorough investigation of bert's knowledge of hypernymy via prompting. inspired by recent work on prompting, we use a wide variety of prompts to search for knowledge of hypernymy. we find that bert does have some knowledge of hypernymy, and performs better than other unsupervised models for hypernym discovery. furthermore, two prompts, "[mask], such as x" and "my favorite [mask]is x.", often elicit correct hypernyms from bert. however, even with these prompts, bert occasionally fails, producing either non-hypernyms or extremely general hypernyms. moreover, this prompting methodology does not allow us to distinguish between understanding of hypernymy and memorization of hypernyms from training data. thus, we conclude that while some of the prompts elicited many correct hypernyms, we cannot claim that bert has fully understood hypernymy.