Maintaining Quality in FEVER Annotation (2020)

Abstract: We propose two measures for measuring the quality of constructed claims in the FEVER task. Annotating data for this task involves the creation of supporting and refuting claims over a set of evidence. Automatic annotation processes often leave superficial patterns in data, which learning systems can detect instead of performing the underlying task. Humans also can leave these superficial patterns, either voluntarily or involuntarily (due to e.g. fatigue). The two measures introduced attempt to detect the impact of these superficial patterns. One is a new information-theoretic and distributionality based measure, DCI ; and the other an extension of neural probing work over the ARCT task, utility . We demonstrate these measures over a recent major dataset, that from the English FEVER task in 2019.

## introduction the fever task frames verification of claims given knowledge as a retrieval and three-class entailment problem. given a claim, supporting or refuting text must be found, and a judgment made as to whether or not the text supports the claim. one way in which annotation performance lapses present is with the use of shortcuts. an easy shortcut for this task would be to insert a few direct negation words into claim texts, thus making them clash with the associated evidence. a recent study of arct, the argument reasoning comprehension task, in which systems have to pick a warrant given a claim a premise, found that annotators were prone to inserting words such as 'not' when constructing negative examples, which later models (such as bert) could then pick up on @xcite . these superficial shortcuts were prevalent to the extent that removing this information led to a significant drop in bert argument reasoning performance, from 77% to 50%. mindful of the similar nature of the arct and fever tasks, we apply an extended version of niven & kao's metric to the fever dataset, and present an information theoretic measure over skipgrams in fever claims to detect candidate superficial features. ## annotation in fever the annotation process for fever is involved. the fever dataset @xcite comprises a total of 185,445 claims created from wikipedia articles and annotated as either supports, refutes or notenoughinfo. additionally, claims that are labelled supports and refutes also come with the evidence against which this judgement has been made. this fever data was created with the help of 50 annotators and in two stages: first creating claims from wikipedia articles, then labelling them against evidence from wikipedia. the claim generation stage entails providing annotators with a randomly sampled sentence from the introductory section of an english wikipedia article and asking them to create claims about the article's entity. in addition to basing their claims on the provided sentence alone, annotators were also given the choice to utilize information from hyperlinked articles to allow for more complex claims @xcite . annotators were also asked to create different variants of these claims by, for example, negating, generalizing or replacing part of the claim. this was done to introduce refutable and non-verifiable claims into the dataset. while trialing, the authors realized that "the majority of annotators had difficulty generating non-trivial negation mutations [...] beyond adding 'not' to the original" @xcite . we investigate the impact of these trivial negations on the quality of the dataset later on. in the second stage, annotators labeled the previously created claims as either supports, refutes or notenoughinfo. for the first two classes, annotators also marked the sentences they used as evidence for their decision. once again, the annotators had access to articles hyperlinked in the entity's article as well. the final dataset is segmented into multiple subsets, with the training set retaining a majority of the claims at a size of 145,449. the quality of their annotations is ensured by cross-checking labels through five-way agreement, super-annotators and even validation by the authors themselves. yet, despite spotting the issue with non-trivial negations early in the process, they do not report on any investigations into the quality of their claims. one might argue that annotation accuracy loses its importance if the task is performed on the basis of biased data. nevertheless, as with most complex annotation tasks over language, the complex nature of this annotation process is prone to annotation exhaustion and shortcuts @xcite . ## quality metrics we propose two quality metrics for fever. the goal of fever data is to help train inference/verification/entailment tools that are wellgeneralised. thus, a quality metric should help detect when annotated data risks being unsuitable for that purpose. the new metrics outlined here are generic and can be applied to data for other classification tasks. they are proposed with the goal of identifying surface-level linguistic patterns that 'leak' class information, helping dataset builders improve the quality of their data. ## discussion applying productivity, utility and coverage indicates a dearth of the sort of superficial features in fever that were present in previous tasks (namely the arct dataset). this is somewhat at odds with other work over fever. schuster et al. (2019) find that local mutual information (lmi) reveals some n-grams that are strongly-associated with negative examples, and are able to predict claim veracity based on claims alone. the phrases that schuster et al. find match those top-ranked by our dci metric. we can therefore see that mutual informationbased measures (lmi, dci) find different biases to frequency-associative measures, such as those use to find cues in the arct task. it may be worth applying e.g. lmi or dci to the arct data to see if complementary results emerge. note that we examine all n-and skip-grams in the dataset, without smoothing. @xcite experiment with removing named entities and rare noun-phrases from their dataset when training models. while this is likely to reduce variances in the data representation, enhancing the signal, the goal of this work is to find the strongest signals, and go down from there, rather than remove noise in a "bottom-up" fashion. this is not the first investigation into biases related to crowdsourcing and human annotation: @xcite find patterns in corpora for inference. @xcite and @xcite discuss best practices in crowdsourcing for corpus creation. notably, the number of annotations created by a single annotator should be capped strongly, to avoid nuances of a single worker's style disrupting the data significantlyrather, many annotators should contribute to the data. we propose further controlling quality by looking for superficial patterns during the annotation process, and asking annotators to consider re-formulating their input choices if such patterns are present. ## conclusion annotators are prone to introducing artefacts, certainly in the construction of datasets involving synthesis of claims and counterclaims. this paper presented metrics and an analysis of the english fever dataset with three previously-used measures: productivity, coverage and utility; and a new measure, dataset-weighted cue information. we find that the fever dataset is somewhat free of superficial artefacts, and present a truncated set of its most-informative (or most distracting) patterns.