What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020)

Abstract: Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event prediction— and, in particular, it shows clear insensitivity to the contextual impacts of negation.

## introduction pre-training of nlp models with a language modeling objective has recently gained popularity as a precursor to task-specific fine-tuning. pretrained models like bert @xcite and elmo @xcite have advanced the state of the art in a wide variety of tasks, suggesting that these models acquire valuable, generalizable linguistic competence during the pre-training process. however, though we have established the benefits of language model pretraining, we have yet to understand what exactly about language these models learn during that process. this paper aims to improve our understanding of what language models (lms) know about language, by introducing a set of diagnostics targeting a range of linguistic capacities drawn from human psycholinguistic experiments. because of their origin in psycholinguistics, these diagnostics have two distinct advantages: they are carefully controlled to ask targeted questions about linguistic capabilities, and they are designed to ask these questions by examining word predictions in context, which allows us to study lms without any need for task-specific fine-tuning. beyond these advantages, our diagnostics distinguish themselves from existing tests for lms in two primary ways. first, these tests have been chosen specifically for their capacity to reveal insensitivities in predictive models, as evidenced by patterns that they elicit in human brain responses. second, each of these tests targets a set of linguistic capacities that extend beyond the primarily syntactic focus seen in existing lm diagnostics-we have tests targeting commonsense/pragmatic inference, semantic roles and event knowledge, category membership, and negation. each of our diagnostics is set up to support tests of both word prediction accuracy and sensitivity to distinctions between good and bad context completions. although we focus on the bert model here as an illustrative case study, these diagnostics are applicable for testing of any language model. this paper makes two main contributions. first, we introduce a new set of targeted diagnostics for assessing linguistic capacities in language models. 1 second, we apply these tests to shed light on strengths and weaknesses of the popular bert model. we find that bert struggles with challenging commonsense/pragmatic inferences and role-based event prediction; that it is generally robust on within-category distinctions and role reversals, but with lower sensitivity than humans; and that it is very strong at associating nouns with hypernyms. most strikingly, however, we find that bert fails completely to show generalizable understanding of negation, raising questions about the aptitude of lms to learn this type of meaning. ## motivation for use of psycholinguistic tests on language models it is important to be clear that in using these diagnostics, we are not testing whether lms are psycholinguistically plausible. we are using these tests simply to examine lms' general linguistic knowledge, specifically by asking what information the models are able to use when assigning probabilities to words in context. these psycholinguistic tests are well-suited to asking this type of question because a) the tests are designed for drawing conclusions based on predictions in context, allowing us to test lms in their most natural setting, and b) the tests are designed in a controlled manner, such that accurate word predictions in context depend on particular types of information. in this way, these tests provide us with a natural means of diagnosing what kinds of information lms have picked up on during training. clarifying the linguistic knowledge acquired during lm-based training is increasingly relevant as state-of-the-art nlp models shift to be predominantly based on pre-training processes involving word prediction in context. in order to understand the fundamental strengths and limitations of these models-and in particular, to understand what allows them to generalize to many different taskswe need to understand what linguistic competence and general knowledge this lm-based pre-training makes available (and what it does not). the importance of understanding lm-based pre-training is also the motivation for examining pre-trained bert, as we do in the present paper, despite the fact that the pre-trained form is typically used only as a starting point for fine-tuning. because it is the pre-training that seemingly underlies the generalization power of the bert model, allowing for simple fine-tuning to perform so impressively, it is the pre-trained model that presents the most important questions about the nature of generalizable linguistic knowledge in bert. ## related work this paper contributes to a growing effort to better understand the specific linguistic capacities achieved by neural nlp models. some approaches use fine-grained classification tasks to probe information in sentence embeddings @xcite @xcite , or token-level and other sub-sentence level information in contextual embeddings @xcite . some of this work has targeted specific linguistic phenomena such as function words @xcite . much work has attempted to evaluate systems' overall level of ''understanding'', often with tasks such as semantic similarity and entailment @xcite @xcite @xcite , and additional work has been done to design curated versions of these tasks to test for specific linguistic capabilities @xcite @xcite . our diagnostics complement this previous work in allowing for direct testing of language models in their natural setting-via controlled tests of word prediction in context-without requiring probing of extracted representations or task-specific fine-tuning. more directly related is existing work on analyzing linguistic capacities of language models specifically. this work is particularly dominated by testing of syntactic awareness in lms, and often mirrors the present work in using targeted evaluations modeled after psycholinguistic tests @xcite @xcite @xcite . these analyses, like ours, typically draw conclusions based on lms' output probabilities. additional work has examined the internal dynamics underlying lms' capturing of syntactic information, including testing of syntactic sensitivity in different components of the lm and at different timesteps within the sentence @xcite , or in individual units @xcite . this previous work analyzing language models focuses heavily on syntactic competence-semantic phenomena like negative polarity items are tested in some studies @xcite , but the tested capabilities in these cases are still firmly rooted in the notion of detecting structural dependencies. in the present work we expand beyond the syntactic focus of the previous literature, testing for capacities including commonsense/pragmatic reasoning, semantic role and event knowledge, category membership, and negation-while continuing to use controlled, targeted diagnostics. our tests are also distinct in eliciting a very specific response profile in humans, creating unique predictive challenges for models, as described subsequently. we further deviate from previous work analyzing lms in that we not only compare word probabilities-we also examine word prediction accuracies directly, for a richer picture of models' specific strengths and weaknesses. some previous work has used word prediction accuracy as a test of lms' language understanding-the lambada dataset @xcite , in particular, tests models' ability to predict the final word of a passage, in cases where the final sentence alone is insufficient for prediction. however, although lambada presents a challenging prediction task, it is not well-suited to ask targeted questions about types of information used by lms for prediction-unlike our tests, lambada is not controlled to isolate and test the use of specific types of information in prediction. our tests are thus unique in taking advantage of the additional information provided by testing word prediction accuracy, while also leveraging the benefits of controlled sentences that allow for asking targeted questions. finally, our testing of bert relates to a growing literature examining linguistic characteristics of the bert model itself, to better understand what underlies the model's impressive performance. @xcite analyze the dynamics of bert's selfattention mechanism, probing attention heads for syntactic sensitivity and finding that individual heads specialize strongly for syntactic and coreference relations. @xcite also examine syntactic awareness in bert by syntactic probing at different layers, and by examination of syntactic sensitivity in the self-attention mechanism. @xcite test a variety of linguistic tasks at different layers of the bert model. most similarly to our work here, @xcite tests bert on several of the targeted syntactic evaluations described earlier for lms, finding bert to exhibit very strong performance on these measures. our work complements these approaches in testing bert's linguistic capacities directly via the word prediction mechanism, and in expanding beyond the syntactic tests used to examine bert's predictions in @xcite . ## leveraging human studies the power in our diagnostics stems from their origin in psycholinguistic studies-the items have been carefully designed for studying specific aspects of language processing, and each test has been shown to produce informative patterns of results when tested on humans. in this section we provide relevant background on human language processing, and explain how we use this information to choose the particular tests used here. ## experiments as a case study, we use these three diagnostics to examine the predictive capacities of the pretrained bert model @xcite , which has been the basis of impressive performance across a wide range of tasks. bert is a deep bidirectional transformer network @xcite pre-trained on tasks of masked language modeling (predicting masked words given bidirectional context) and next-sentence prediction (binary classification of whether two sentences are a sequence). we test two versions of the pretrained model: bert base and bert large (uncased). these versions have the same basic architecture, but bert large has more parameters-in total, bert base has 110m parameters, and bert large has 340m. we use the pytorch bert implementation with masked language modeling parameters for generating word predictions. 8 for testing, we process our sentence contexts to have a [mask] token-also used during bert's pre-training-in the target position of interest. we then measure bert's predictions for this [mask] token's position. following @xcite , we also add a [cls] token to the start of each sentence to mimic bert's training conditions. bert differs from traditional left-to-right language models, and from real-time human predictions, in being a bidirectional model able to use information from both left and right context. this difference should be neutralized by the fact that our items provide all information in the left context-however, in our experiments here, we do allow one advantage for bert's bidirectionality: we include a period and a [sep] token after each [mask] token, to indicate that the target position is followed by the end of the sentence. we do this in order to give bert the best possible chance of success, by maximizing the chance of predicting a single word rather than the start of a phrase. items for these experiments thus appear as follows: logits produced by the language model for the target position are softmax-transformed to obtain probabilities comparable to human cloze probability values for those target positions. foot_7 first we report bert's results on the cprag-102 test targeting common sense, pragmatic reasoning, and sensitivity within semantic category. ## results for role-88 next we turn to the role-88 test of semantic role sensitivity and event knowledge. ## results for neg-136 finally, we turn to the neg-136 test of negation and category membership. ## discussion our three diagnostics allow for a clarified picture of the types of information used for predictions by pre-trained bert models. on cprag-102, we see that both models can predict the best completion approximately half the time (at @xmath2, and that both models rely non-trivially on word order and full sentence context. however, successful predictions in the face of perturbations also suggest that some of bert's success on these items may exploit loopholes, and when we examine predictions on challenging items, we see clear weaknesses in the commonsense and pragmatic inferences targeted by this set. sensitivity tests show that bert can also prefer good completions to bad semantically related completions in a majority of items, but many of these probability differences are very small, suggesting that the model's sensitivity is much less than that of humans. on role-88, bert's accuracy in matching top human predictions is much lower, with bert large at only 37.5% accuracy. perturbations reveal interesting model differences, suggesting that bert large has more sensitivity than bert base to the interaction between subject and object nouns. sensitivity tests show that both models are typically able to use noun position to prefer good completions to role reversals, but the differences are on average even smaller than on cprag-102, indicating again that model sensitivity to the distinctions is less than that of humans. the models' general ability to distinguish role reversals suggests that the low word prediction accuracies are not due to insensitivity to word order per se, but rather to weaknesses in event knowledge or understanding of semantic role implications. finally, neg-136 allows us to zero in with particular clarity on a divergence between bert's predictive behavior and what we might expect from a model using all available information about word meaning and truth/falsity. when presented with simple sentences describing category membership, bert shows a complete inability to prefer true over false completions for negative sentences. the model shows an impressive ability to associate subject nouns with their hypernyms, but when negation reverses the truth of those hypernyms, bert continues to predict them nonetheless. by contrast, when presented with sentences that are more ''natural'', bert does reliably prefer true completions to false, with or without negation. although these latter sentences are designed to differ in naturalness, in all likelihood it is not naturalness per se that drives the model's relative success on them-but rather a higher frequency of these types of statements in the training data. the latter result in particular serves to highlight a stark, but ultimately unsurprising, observation about what these pre-trained language models bring to the table. whereas the function of language processing for humans is to compute meaning and make judgments of truth, language models are trained as predictive models-they will simply leverage the most reliable cues in order to optimize their predictive capacity. for a phenomenon like negation, which is often not conducive to clear predictions, such models may not be equipped to learn the implications of this word's meaning. ## conclusion in this paper we have introduced a suite of diagnostic tests for language models to better our understanding of the linguistic competencies acquired by pre-training via language modeling. we draw our tests from psycholinguistic studies, allowing us to target a range of linguistic capacities by testing word prediction accuracies and sensitivity of model probabilities to linguistic distinctions. as a case study, we apply these tests to analyze strengths and weaknesses of the popular bert model, finding that it shows sensitivity to role reversal and same-category distinctions, albeit less than humans, and it succeeds with noun hypernyms, but it struggles with challenging inferences and role-based event prediction-and it shows clear failures with the meaning of negation. we make all test sets and experiment code available (see footnote 1), for further experiments. the capacities targeted by these test sets are by no means comprehensive, and future work can build on the foundation of these datasets to expand to other aspects of language processing. because these sets are small, we must also be conservative in the strength of our conclusions-different formulations may yield different performance, and future work can expand to verify the generality of these results. in parallel, we hope that the weaknesses highlighted by these diagnostics can help to identify areas of need for establishing robust and generalizable models for language understanding.