Get Your Vitamin C ! Robust Fact Verification with Contrastive Evidence (2021)

Abstract: Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness—improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation.

## introduction determining the truthfulness of factual claims by comparing them to textual sources of evidence has received intense research interest in recent years. an underlying, but often overlooked, challenge for this paradigm, however, is the dynamic nature of today's written resources. an extraordinary amount of new information becomes available daily; as a result, many consequential facts are established, changed, or added to over time. we argue that the quality of fact verification systems should be its population is estimated to be 86,205, almost 14% more than the 2000 census figure of 76,129. ## the vitaminc dataset vitaminc (abbreviated vitc) is based on revisions to english wikipedia. wikipedia has become a comprehensive online resource that is rigorously maintained by a large and active community @xcite . while adversaries do try to insert disinformation, popular pages are usually quickly corrected @xcite . furthermore, wikipedia's policies dictate that its content should be written from a neutral perspectiveor should otherwise objectively state all points of view. 3 these properties make wikipedia a suitable source of evidence for fact verification models. in the following section, we outline our process for mining factual revisions from wikipedia. ## vitaminc tasks the unique structure of vitaminc allows us to derive annotations that provide a novel source of supervision for several fact-verification-related tasks. we describe the four main tasks we consider in this work, along with baseline models: (1) factual revision flagging, (2) fact verification, (3) word-level rationales, and (4) factually consistent generation. figure 2 illustrates an example from vitaminc. we use the following notations: the outbreak was first identified in wuhan, hubei, china in december 2019 and recognized [...]. the outbreak was first identified in wuhan, hubei, china in 17 november 2019 and recognized [...]. s t-1 s t revision id: 945689803 the revision (s t-1 ,s t ) is factual because it modifies when the first covid-19 case was identified ## experiments we present and analyze results for the models described in section 4. our analysis attempts to evaluate several questions: (1) how well can the current state-of-the-art models perform on the vitaminc tasks? (2) does vitaminc increases the robustness of models against adversarial examples? (3) can vitaminc improve interpretability by providing supervision for anchoring words? ## conclusion we presented vitaminc, a large-scale dataset for training and evaluating fact verification models using contrastive contexts. our novel method of leveraging factual revisions to wikipedia enabled us to create challenging examples in which a claim is paired with contexts that are lexically similar, yet factually opposing. our results illustrated that training on vitaminc improves classifier sensitivity to subtle changes in evidence, and increases their robustness to adversarial examples. furthermore, we formulated several new, important tasks for fact verification that vitaminc allows us to test. we showed how the dataset's unique "before and after" structure lends itself to training classifiers to flag factual revisions. in addition, for factual revisions, the edits reveal which words in the evidence are the most critical-which helps supervise word-level rationale models for better interpretability. finally, we demonstrated that vitaminc can help with factually consistent text generation. we hope that this work and the range of tasks it presents will motivate and support the fact verification field in developing reliable models that can adapt to dynamically changing evidence.