Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (2020)

Abstract: Building on Petroni et al. 2019, we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated (‘‘Birds cannot [MASK]”) and non-negated (‘‘Birds can [MASK]”) cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add “misprimes” to cloze questions (‘‘Talk? Birds can [MASK]”). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.

## introduction plms like transformer-xl @xcite , elmo @xcite and bert @xcite have emerged as universal tools that capture a diverse range of linguistic and factual knowledge. recently, @xcite introduced lama (language model analysis) to investigate whether plms can recall factual knowledge that is part of their training corpus. since the plm training objective is to predict masked tokens, question answering (qa) tasks can be reformulated as cloze questions. for example, "who wrote 'dubliners'?" is reformulated as " @xcite wrote 'dubliners'." in this setup, @xcite show that plms outperform automatically extracted knowledge bases on qa. in this paper, we investigate this capability of plms in the context of (1) negation and what we call (2) mispriming. querying plms with these pairs and comparing the predictions, we find that the predicted fillers have high overlap. models are equally prone to generate facts ("birds can fly") and their incorrect negation ("birds cannot fly"). we find that bert handles negation best among plms, but it still fails badly on most negated probes. in a second experiment, we show that bert can in principle memorize both positive and negative facts correctly if they occur in training, but that it poorly generalizes to unseen sentences (positive and negative). however, after finetuning, bert does learn to correctly classify unseen facts as true/false. we could have manually generated more natural misprimes. for example, misprime "regent of antioch" in "tancred, regent of antioch, played a role in the conquest of [mask]" tricks bert into chosing the filler "antioch" (instead of "jerusalem"). our automatic misprimes are less natural, but automatic generation allows us to create a large misprime dataset for this initial study. contribution. we show that plms' ability to learn factual knowledge is -in contrast to human capabilities -extremely brittle for negated sentences and for sentences preceded by distracting material (i.e., misprimes). data and code will be published. 1 2 data and models lama's cloze questions are generated from subject-relation-object triples from knowledge bases (kbs) and question-answer pairs. for kb triples, cloze questions are generated, for each relation, by a templatic statement that contains variables x and y for subject and object (e.g, "x was born in y"). we then substitute the subject for x and mask for y. in a question-answer pair, we mask the answer. lama is based on several sources: (i) google-re. 3 relations: "place of birth", "date of birth", "place of death". (ii) t-rex @xcite . subset of wikidata triples. 41 relations. (iii) con-ceptnet @xcite . 16 commonsense relations. the underlying corpus provides matching statements to query plms. (iv) squad @xcite . subset of 305 context-insensitive questions, reworded as cloze questions. we use the source code provided by @xcite and @xcite to evaluate transformer-xl large (txl), elmo original (eb), elmo 5.5b (e5b), bert-base (bb) and bertlarge (bl). negated lama. we created negated lama by manually inserting a negation element in each template or question. for conceptnet we only consider an easy-to-negate subset (see appendix). misprimed lama. we misprime lama by inserting an incorrect word and a question mark at the beginning of a statement; e.g., "talk?" in "talk? birds can @xcite ." we only misprime questions that are answered correctly by bertlarge. to make sure the misprime is misleading, we manually remove correct primes for squad and conceptnet and automatically remove primes that are the correct filler for a different instance of the same relation for t-rex and conceptnet. we create four versions of misprimed lama (a, b, c, d) as described in the caption of our assumption is that the correct answers for a pair of positive question and negative question in summary, plms poorly distinguish positive and negative sentences. we give two examples of the few cases where plms make correct predictions, i.e., they solve the cloze task as human subjects would. for "the capital of x is not y" (trex, 1-1) top ranked predictions are "listed", "known", "mentioned" (vs. cities for "the capital of x is y"). this is appropriate since the predicted sentences are more common than sentences like "the capital of x is not paris". for "x was born in y", cities are predicted, but 2 a reviewer observes that spearman correlation is generally high and wonders whether high spearman correlation is really a reliable indicator of negation not changing the answer of the model. as a sanity check, we also randomly sampled, for each query correctly answered by bert-large (e.g., "einstein born in [mask]"), another query with a different answer, but the same template relation (e.g., "newton born in [mask]") and computed the spearman correlation between the predictions for the two queries. in general, these positive-positive spearman correlations were significantly lower than those between positive ("einstein born in [mask]") and negative ("einstein not born in [mask]") queries (t-test, @xmath0.01). there were two exceptions (not significantly lower): t-rex 1-1 and google-re birth-date. for "x was not born in y", sometimes countries are predicted. this also seems natural: for the positive sentence, cities are more informative, for the negative, countries. balanced corpus. investigating this further, we train bert-base from scratch on a synthetic corpus. hyperparameters are listed in the appendix. the corpus contains as many positive sentences of form "x j is a n " as negative sentences of form "x j is not a n " where x j is drawn from a set of 200 subjects s and a n from a set of 20 adjectives a. the 20 adjectives form 10 pairs of antonyms (e.g., "good"/"bad"). s is divided into 10 groups g m of 20. finally, there is an underlying kb that defines valid adjectives for groups. for example, assume that g 1 has property a @xmath1. then for each x @xmath2, the sentences "x i is good" and "x i is not bad" are true. the training set is generated to contain all positive and negative sentences for 70% of the subjects. it also contains either only the positive sentences for the other 30% of subjects (in that case the negative sentences are added to test) or vice versa. cloze questions are generated in the format "x j is [mask]"/"x j is not [mask]". we test whether (i) bert memorizes positive and negative sentences seen during training, (ii) it generalizes to the test set. as an example, a correct generalization would be "x i is not bad" if "x i is good" was part of the training set. the question is: does bert learn, based on the patterns of positive/negative sentences and within-group regularities, to distinguish facts from non-facts. ## discussion whereas @xcite 's results suggest that plms are able to memorize facts, our results indicate that plms largely do not learn the meaning of negation. they mostly seem to predict fillers based on co-occurrence of subject (e.g., "quran") and filler ("religious") and to ignore negation. a key problem is that in the lama setup, not answering (i.e., admitting ignorance) is not an option. while the prediction probability generally is somewhat lower in the negated compared to the positive answer, there is no threshold across cloze questions that could be used to distinguish valid positive from invalid negative answers (cf. we suspect that a possible explanation for plms' poor performance is that negated sentences occur much less frequently in training corpora. our synthetic corpus study ( the mispriming experiment shows that bert often handles random misprimes correctly ( our work is a new way of analyzing differences between plms and human-level natural language understanding. we should aspire to develop plms that -like humans -can handle negation and are not easily distracted by misprimes. ## related work plms are top performers for many tasks, including qa @xcite . plms are usually finetuned @xcite , but recent work has applied models without finetuning @xcite . @xcite investigate plms' common sense knowledge, but do not consider negation explicitly or priming. a wide range of literature analyzes linguistic knowledge stored in pretrained embeddings @xcite @xcite @xcite @xcite . our work analyzes factual knowledge. @xcite show that bert finetuned to perform natural language inference heavily relies on syntactic heuristics, also suggesting that it is not able to adequately acquire common sense. @xcite investigate bert's understanding of how negative polarity items are licensed. our work, focusing on factual knowledge stored in negated sentences, is complementary since grammaticality and factuality are mostly orthogonal properties. @xcite investigate understanding of negation particles when plms are finetuned. in contrast, our focus is on the interaction of negation and factual knowledge learned in pretraining. @xcite defines and applies psycho-linguistic diagnostics for plms. our use of priming is complementary. their data consists of two sets of 72 and 16 sentences whereas we create 42,867 negated sentences covering a wide range of topics and relations. @xcite test for comprehension of minimally modified sentences in an adversarial setup while trying to keep the overall semantics the same. in contrast, we investigate large changes of meaning (negation) and context (mispriming). in contrast to adversarial work (e.g., @xcite ), we do not focus on adversarial examples for a specific task, but on pretrained models' ability to robustly store factual knowledge. ## conclusion our results suggest that pretrained language models address open domain qa in datasets like lama by mechanisms that are more akin to relatively shallow pattern matching than the recall of learned factual knowledge and inference. implications for future work on pretrained language models. (i) both factual knowledge and logic are discrete phenomena in the sense that sentences with similar representations in current pretrained language models differ sharply in factuality and truth value (e.g., "newton was born in 1641" vs. "newton was born in 1642"). further architectural innovations in deep learning seem necessary to deal with such discrete phenomena. (ii) we found that plms have difficulty distinguishing "informed" best guesses (based on information extracted from training corpora) from "random" best guesses (made in the absence of any evidence in the training corpora). this implies that better confidence assessment of plm predictions is needed. (iii) our premise was that we should emulate human language processing and that therefore tasks that are easy for humans are good tests for nlp models. to the extent this is true, the two phenomena we have investigated in this paper -that plms seem to ignore negation in many cases and that they are easily confused by simple distractors -seem to be good vehicles for encouraging the development of plms whose performance on nlp tasks is closer to humans.