B irds have four legs?! N umer S ense: P robing N umerical C ommonsense K nowledge of P re- T rained L anguage M odels (2020)

Abstract: Recent works show that pre-trained language models (PTLMs), such as BERT, possess certain commonsense and factual knowledge. They suggest that it is promising to use PTLMs as “neural knowledge bases” via predicting masked words. Surprisingly, we find that this may not work for numerical commonsense knowledge (e.g., a bird usually has two legs). In this paper, we investigate whether and to what extent we can induce numerical commonsense knowledge from PTLMs as well as the robustness of this process. In this paper, we investigate whether and to what extent we can induce numerical commonsense knowledge from PTLMs as well as the robustness of this process. To study this, we introduce a novel probing task with a diagnostic dataset, NumerSense, containing 13.6k masked-word-prediction probes (10.5k for fine-tuning and 3.1k for testing). Our analysis reveals that: (1) BERT and its stronger variant RoBERTa perform poorly on the diagnostic dataset prior to any fine-tuning; (2) fine-tuning with distant supervision brings some improvement; (3) the best supervised model still performs poorly as compared to human performance (54.06% vs. 96.3% in accuracy).

## introduction pre-trained language models (ptlms), such as bert @xcite , have yielded stateof-the-art performance on many natural language processing tasks. given ptlms' cited ability to create general, yet useful text representations, an investigation of their ability to encode commonsense knowledge into representations is warranted--commonsense knowledge is often required to have a full understanding of language. recently there have been a few recent works that do investigate the inquiry of whether ptlms possess commonsense knowledge @xcite @xcite . overall, these prior studies suggest that ptlms are 1 https://inklab.usc.edu/numersense/ 1st:fly (79.5%) 2nd:sing (9.1%) 1st:four(44.8%) 2nd:two (18.7%) ## the numersense probing task we introduce our numerical commonsense reasoning probing task, as well as the creation process of the namesake dataset, numersense. then, we provide a breakdown of what types of knowledge are covered by the probes and finally include additional high-quality distant supervision to test if fine-tuning can improve performance. ## empirical analysis we introduce the set-up of the experiments and then present results from different ptlms in both a zero-shot setting and a distantly supervised finetuned one. we will also provide some analysis on the robustness and biases in the various models, and finally a study of the performance of a state-ofthe-art open-domain question-answering model. ## case studies object bias. recall the example "a bird usually has [mask] legs," which bert-large predicts to be "four". does bert-large always predict "four" as long as the adjacent word after the [mask] is 'legs'? to investigate if the bias exists, we show some case studies in ## open-domain 'how-many' questions the examples in the numersense can be also seen as open-domain questions targeting 'howmany' commonsense--"how many legs does a fly usually have?" answering these open-domain numerical commonsense questions is a practical downstream application of models that are successful in the numersense. thus, as a side note, we also report the performance of the state-of-the-art open-domain qa model @xcite . we use the model that is trained on the natural question (nq) dataset @xcite , where we replace the '[mask]'s in our examples with 'how many', so that our probes are in a similar format to nq examples. for example "a fly usually has [mask] legs" is converted to "how many legs a fly usually has?" 4 the accuracy of the state-of-the-art model is only 15.4%, which is even lower than using bert-base without fine-tuning. this indicates that improving performance on nu-mersense can help improve the performance on answering open-domain "how-many" questions. ## conclusion we present a probing task, numersense, to induce numerical commonsense knowledge from pretrained language models. we collect a new diagnostic dataset carefully verified by human annotators, which covers 8 different topics. powerful pre-trained models such as bert and roberta perform surprisingly poorly, even after fine-tuning with high-quality distant supervision. we hope our findings and probing dataset will provide a basis for improving pre-trained masked language models' numerical and other concrete types of commonsense knowledge.