On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020)

Abstract: Contextualized word representations have become a driving force in NLP, motivating widespread interest in understanding their capabilities and the mechanisms by which they operate. Particularly intriguing is their ability to identify and encode conceptual abstractions. Past work has probed BERT representations for this competence, finding that BERT can correctly retrieve noun hypernyms in cloze tasks. In this work, we ask the question: do probing studies shed light on systematic knowledge in BERT representations? As a case study, we examine hypernymy knowledge encoded in BERT representations. In particular, we demonstrate through a simple consistency probe that the ability to correctly retrieve hypernyms in cloze tasks, as used in prior work, does not correspond to systematic knowledge in BERT. Our main conclusion is cautionary: even if BERT demonstrates high probing accuracy for a particular competence, it does not necessarily follow that BERT ‘understands’ a concept, and it cannot be expected to systematically generalize across applicable contexts.

## introduction hierarchical representations of concepts play a central role in reasoning and understanding natural language @xcite . they have long been studied as a core nlp objective in their own right through tasks requiring the identification of hypernyms @xcite @xcite , and as components for use in downstream nlp tasks, such as recognizing textual entailment (rte), metaphor detection, text generation and question answering (qa) @xcite @xcite @xcite @xcite . recently, pretrained language models (plms), such as bert @xcite , have emerged as a popular and successful approach to a variety of nlp tasks. thus, there has been community interest in evaluating their representations for the 'knowledge' they contain, including information about concept abstraction @xcite @xcite . we distinguish research that investigates knowledge encoded in bert through two broad perspectives: instrumentative and agentive. we view the instrumentative perspective as treating plms as a tool to mine or store knowledge, like hypernymhyponym and other relations, from text @xcite @xcite @xcite . the primary purpose of these investigations is to identify effective techniques to extract information from plms for use in downstream pipelines. in contrast, a growing body of work adopts an agentive perspective @xcite , treating plms as artificial intelligence (ai) agents and analyzing their linguistic competencies and world knowledge, sometimes through tasks such as natural language inference @xcite or story completion @xcite @xcite . in this work, we examine the agentive perspective, focusing specifically on the validity of conclusions drawn from probing studies. a popular approach to probing knowledge in pre-trained language models is the zero-shot masked-lm probing task. for example, given the statement 'a robin is a [mask]', a plm that produces the correct completion 'bird' is considered successful. 2 past work has studied this competency in bert @xcite , offering bert's ability to correctly retrieve noun hypernyms in cloze tasks as evidence that it successfully encodes hypernymy information. but to what extent does this knowledge of hypernymy generalize? among many systematic generalization abilities desirable in plms, we select the following two. (1) syntagmatic generalization: a model that has knowledge of a fact will be able to correctly answer queries about it and apply it across different contexts; (2) paradigmatic generalization: a model with a particular competency will be able to generalize to novel cues and items. we implement these generalization requirements through a set of diagnostic probing tasks, in which a model must demonstrate consistency in applying its knowledge across different selective contexts, and by generalizing in trained probe settings to novel, unseen items belonging to the same semantic category or relation. in particular, we focus on the setting of ettinger (2020), which demonstrates that bert is "very strong at associating nouns with hypernyms." we propose consistency tasks to illuminate the limits and generality of this ability, as illustrated in figure 1 . our consistency tasks combine related zero-shot probes in such a way that a model that succeeds on one probe, if it is drawing on a systematic, general ability, should also succeed on the paired probe. our evaluation with a grammatical number consistency task sheds light on the fragility of bert's ability to associate correct noun hypernyms and demonstrates that pre-trained lms have considerable room for improvement to reach a human-like level of understanding. 3 contributions: we demonstrate success on a hypernymy probing benchmark does not necessarily correspond to a systematic conceptual understanding of the phenomena in bert, as discovered by probes. we further formulate evaluation protocols for characterizing the generalizability of plm knowledge, in order to draw more reliable conclusions from probing studies. foot_2 ## experimental methodology saussure (1916) expounds on syntagmatic relations, studying how words acquire relations based on the ways in which they are chained together in language context. the syntagmatic relation is based on groups of terms, in this case the hyponym and hypernym that are communicated together. in this work, we study whether plm probes generalize syntagmatically, by evaluating the ability of models to produce correct predictions for hyponymhypernym items across both singular and plural contexts. we also examine the ability of probes to generalize paradigmatically, that is, do probing studies uncover paradigms embedded in text (in this case the relations between items and their abstractions)? foot_3 ## paradigmatic generalization representations: we use the following representations in the encode-embed-predict architecture described in §2.2.1: for static representation baselines, we use word2vec, glove and fasttext, and for our contextualized representation we study bert. detailed descriptions of the architectures can be found in §3. 2 baselines: (1) majority baseline: performance of classifier that always predicts the majority class in the test set. (2) partial: partial-input baselines have revealed biases in natural language inference @xcite @xcite and question-answering @xcite datasets. @xcite discuss the propensity of classifiers to rely on 'protoypical hypernyms' in hypernymy detection datasets, and not to solve the detection task. to control for potential dataset biases caused by the selection of items in the study, for each model we train a partial counterpart baseline, which is only provided the hypernym as input. if the dataset is unbiased in this aspect, partial baselines should achieve similar performance to a random classifier. results: ## related work there has been considerable interest in probing the capabilities of plms @xcite . much recent work focuses on the grammatical and syntactic capabilities of bert @xcite @xcite @xcite @xcite @xcite . in contrast, our focus is on probing studies that aim to uncover "knowledge" in bert. there have been several such studies @xcite study physical commonsense encoded in bert. @xcite probe bert for its understanding of object attributes, finding that it learns physical concrete norms (is made of wood) better than abstract ones (is strong). @xcite find a 'surprising degree of numeracy' is present in contextualized word representations. @xcite probe bert for capabilities at particular types of symbolic reasoning, such as comparison, conjunction and composition. our work focuses specifically on the validity of conclusions drawn from such probing studies that aim to discover knowledge in bert, using the setting of @xcite as a case-study. we further distinguish between the instrumentative and agentive perspectives on probing. for example, there has been considerable research attention focused on querying language models for their encoded information @xcite @xcite , which we consider as an instrumentative effort using plms as a tool. our focus in this work is instead on agentive studies, and our conclusion is that the probes we study should not be used to reveal evidence of some systematic knowledge or competence in plms-although plms can still be utilized as tools to extract such knowledge from text. closest to our work, kassner and schütze (2020) find that plms do not differentiate between negated and non-negated statements. negation is a notoriously hard phenomenon for neural nlp models @xcite @xcite ; our work demonstrates that even affirmative factual knowledge that can be extracted from bert does not systematically generalize. our work is also closely related to recent challenge set construction efforts, which aim to serve as sanity checks on the knowledge and commonsense capabilities of models @xcite @xcite . for example, @xcite show that bert finetuned for the natural language inference task, relies heavily on shallow heurestics instead of acquiring adequate commonsense knowledge. our work is complementary, demonstrating through a simple consistency task that bert's capabilities, as discovered through probes, may not correspond to some systematic general ability. our work examines, in particular, hypernymy knowledge encoded in bert representations. the identification of hypernyms is studied extensively in cognitive science and philosophy. some prominent theories include rosch's category theory @xcite ) and tversky's category resemblance approach @xcite . this work does not account for either of these interpretations of hypernymy, but instead relies on prior cognitive studies on category norms @xcite and relations defined with these super-ordinate categories in wordnet @xcite . additionally, our work ties into the rich history on modeling hypernymy in nlp systems @xcite @xcite @xcite @xcite and evaluating distributional semantic models on their ability to represent it @xcite @xcite @xcite . ## discussion and summary we briefly discuss our findings and offer some guiding principles for future work. frequency and memorization effects: we find that bert is particularly vulnerable to lowfrequency phenomena in the training data, and succeeds at examples in the probe which have explicitly occurred in the training data. we speculate based on this evidence that bert may just be memorizing the vast amount of training data it has been exposed to, rather than performing any kind of deeper reasoning. caution with cloze-style probes: bert's masked-lm format lends itself easily to clozestyle probes, which consider filling in a missing token correctly as evidence of plm knowledge. despite the accessibility of this format to investigate the behavior of plms, we speculate that, by design, the model is expected to fill in tokens whose context matches the provided template. the designer of the probing task may include templates to extract knowledge based on their intuitions, which (1) may or may not be the right template to extract the targeted kind of knowledge, (2) may provide enough inductive bias that it is unclear if the model understands the relation or understands how to match a particular template (which has been chosen so well based on the practitioners knowledge that it mimics the model actually understanding the deeper phenomena). we speculate that data-driven methods @xcite can be designed to mitigate (2), but will exacerbate (1).