Recent works show that pre-trained masked language models , such as BERT, possess certain linguistic and commonsense knowledge. However, it remains to be seen what types of commonsense knowledge these models have access to.In this vein, we propose to study whether numerical commonsense knowledge -- commonsense knowledge that provides an understanding of the numeric relation between entities -- can be induced from pre-trained masked language models and to what extent is this access to knowledge robust against adversarial examples? To study this, we introduce a  probing task with a diagnostic dataset, NumerSense, containing 3,145 masked-word-prediction probes . Surprisingly, our experiments and analysis reveal that: (1) BERT and its stronger variant RoBERTa perform poorly on our dataset prior to any fine-tuning; (2) fine-tuning with distant supervision does improve performance ; (3) the best distantly supervised model still performs poorly when compared to humans (47.8 \% vs 96.3\%  ).
Recent works show that pre-trained language models (PTLMs) , such as BERT, possess certain commonsense and factual knowledge. They suggest that it is promising to use PTLMs as "neural knowledge bases" via predicting masked words. Surprisingly, we find that this may not work for numerical commonsense knowledge (e.g., a bird usually has two legs). In this paper, we investigate whether and to what extent we can induce numerical commonsense knowledge from PTLMs as well as the robustness of this process. To study this, we introduce a novel probing task with a diagnostic dataset, NumerSense, containing 13.6k masked-word-prediction probes (10.5k for fine-tuning and 3.1k for testing). Our analysis reveals that: (1) BERT and its stronger variant RoBERTa perform poorly on the diagnostic dataset prior to any fine-tuning; (2) fine-tuning with distant supervision brings some improvement ; (3) the best supervised model still performs poorly as compared to human performance (54.06 \% vs 96.3\% in accuracy ).
R
masked language models
language models (PTLMs)
R
linguistic and commonsense knowledge. However, it remains to be seen what types of commonsense knowledge these models have access to.In this vein, we propose to study whether numerical commonsense knowledge -- commonsense knowledge that provides an understanding of the numeric relation between entities -- can be induced from pre-trained masked language models
commonsense and factual knowledge. They suggest that it is promising to use PTLMs as "neural knowledge bases" via predicting masked words. Surprisingly, we find that this may not work for numerical commonsense knowledge (e.g., a bird usually has two legs). In this paper, we investigate whether
R
is this access to knowledge robust against adversarial examples?
we can induce numerical commonsense knowledge from PTLMs as well as the robustness of this process.
A
novel
R
3,145
13.6k
R
. Surprisingly, our experiments and analysis reveal
(10.5k for fine-tuning and 3.1k for testing). Our analysis reveals
R
our
the diagnostic
R
does improve performance
brings some improvement
D
distantly
R
when compared to humans (47.8
as compared to human performance (54.06
A
in accuracy