N egat ER : U nsupervised D iscovery of N egatives in C ommonsense K nowledge B ases (2021)

Abstract: Codifying commonsense knowledge in machines is a longstanding goal of artificial intelligence. Recently, much progress toward this goal has been made with automatic knowledge base (KB) construction techniques. However, such techniques focus primarily on the acquisition of positive (true) KB statements, even though negative (false) statements are often also important for discriminative reasoning over commonsense KBs. As a first step toward the latter, this paper proposes NegatER, a framework that ranks potential negatives in commonsense KBs using a contextual language model (LM). Importantly, as most KBs do not contain negatives, NegatER relies only on the positive knowledge in the LM and does not require ground-truth negative examples. Experiments demonstrate that, compared to multiple contrastive data augmentation approaches, NegatER yields negatives that are more grammatical, coherent, and informative—leading to statistically significant accuracy improvements in a challenging KB completion task and confirming that the positive knowledge in LMs can be “re-purposed” to generate negative knowledge.

## introduction endowing machines with commonsense, which is knowledge that members of a culture usually agree upon but do not express explicitly, is a major but elusive goal of artificial intelligence @xcite @xcite . one way to capture such knowledge is with curated commonsense knowledge bases (kbs), which contain semi-structured statements of "everyday" human knowledge. as such kbs are increasingly being used to augment the capabilities of intelligent agents @xcite , automatically expanding their scope has become crucial @xcite @xcite . ## problem definition as the problem of negative knowledge has not yet been addressed in the commonsense kb completion literature, we begin by defining meaningful negatives in commonsense kbs. ## framework we propose the negater framework to match our definition of negative knowledge. as shown in figure 1 , negater consists of two steps: first, a pretrained lm is fine-tuned on a given commonsense kb using a contrastive approach to acquire strong positive beliefs. then, a set of grammatical (r1) and topically consistent (r2) out-of-kb candidate statements are fed to the lm and ranked by the degree to which they "contradict" the lm's finetuned positive beliefs (r3), such that the higherranking statements are more likely to be negative. we emphasize that ground-truth negative examples are not required at any point, which means that we trade off some accuracy (i.e., the true negative rate) for cost efficiency (i.e., the cost of gathering ground-truth negative examples for training via expert annotation). we describe each step in detail in the remainder of this section. ## fine-tuning evaluation in this section, we evaluate the efficacy of the finetuning step of negater ( § 3.1). in the following sections, we will evaluate the efficacy of the ranking step of negater ( § 3.2) from quantitative and qualitative perspectives. ## task-based evaluation we next evaluate the efficacy of the ranking step in negater. specifically, we next show how the topranking negative examples from negater can be informative (r3) for training kb completion models. similar to the previous section, we finetune pretrained bert and roberta models for a commonsense triple classification task. however, here we use a more challenging dataset split, and vary the ways that negatives are sampled at training time. ## human evaluation finally, we collect qualitative human judgments on the examples output by each negative sampler. ## conclusion in this paper we rigorously defined negative knowledge in commonsense kbs and proposed a framework, negater, to address this problem. importantly, our framework does not require groundtruth negatives at any point, making it an effective choice when gold training examples are not available. we empirically demonstrated the strength of negater over many competitive baselines in multiple evaluations, including the strength of our fine-tuning approach, the task-based utility of negater statements, and the intrinsic quality of these statements. a promising future direction is to explore new reasoning tasks that can be improved with negative knowledge from negater, for example multiple-choice commonsense qa @xcite , which often requires high-quality negative examples for training.