Re-Examining F act B ank: Predicting the Author’s Presentation of Factuality (2022)

Abstract: We present a corrected version of a subset of the FactBank data set. Previously published results on FactBank are no longer valid. We perform experiments on FactBank using multiple training paradigms, data smoothing techniques, and polarity classifiers. We argue that f-measure is an important alternative evaluation metric for factuality. We provide new state-of-the-art results for four corpora including FactBank. We perform an error analysis on Factbank combined with two similar corpora.

## introduction the term factuality foot_0 refers to an author's presentation of a proposition @xcite as a fact, i.e., she is committed to the truth of the proposition. a lot of language use introduces propositions that are not presented as facts but as only possibly true, as a wish or as a hypothesis, or as someone else's belief. if we want to understand what an author is communicating, we need to distinguish these cases. over the last 15 years, this question has received a lot of attention. multiple corpora have been created, and these corpora have been used to explore machine learning architectures for factuality prediction. the machine learning studies often report results on all corpora, but these studies do not examine what the machine learning architecture can learn, nor how and why combining corpora can help. a notable exception is @xcite , who carefully analyze what exactly is learned from the cb corpus. in this paper, our goal is to determine how to combine corpora in order to maximize performance, and to understand why the specific combination of corpora works better than others. we choose a single resource to focus on so that we can gain insights by performing a careful study, and we choose factbank @xcite because it is one of the first carefully constructed datasets for factuality prediction. we show how insights gained from working with fact-bank can be used to improve performance on the cb corpus @xcite . there are three main contributions of this work: the paper is organized as follows. a survey of previous work is provided in section 2. we summarize the factbank representation of factuality in section 3 and present our correction to the projection from factbank in section 4. in section 5, we discuss metrics for evaluating factuality prediction. the redone experiments from @xcite with our corrected projection are described in section 6. we then report on machine learning experiments on three corpora: factbank in section 7, the cb corpus in section 8, and the ldc corpus in section 9. we conclude in section 10. ## related work corpora many corpora explore the notion of factuality including: factbank @xcite , lu @xcite , uw @xcite , ldccb (ldc) @xcite , meantime (mt) @xcite , megaveridicality (mv) @xcite , uds-ih2 (ud2) @xcite , commitment-bank (cb) @xcite , and rp @xcite . these corpora differ along several dimensions; we list dimensions which are salient for this paper. experiments early work on event factuality prediction used rule-based systems; for example, @xcite propose a recursive polarity propagation algorithm which uses implication signatures from clause-embedding verbs. @xcite predict factuality using implication signatures combined with lexical and dependency tree features. early machine learning work on event factuality prediction consists of svms or other supervised learning approaches. @xcite and @xcite use svms and crfs along with lexical and dependency tree features for predicting author belief, evaluating on f-measure. @xcite also use an svm along with lexical and dependency tree features on the uw corpus which they created, and evaluate on pearson correlation and mean average error (mae), as does all following work. @xcite al. (2017) we discuss jiang and de marneffe (2021) in more detail in section 6.1. our work differs from the related work by offering two salient contributions: first, we analyze specific corpus combinations to help improve on a specific corpus (factbank) instead of focusing on improving all corpora. second, we perform error analyses on corpus combinations to determine why and where factuality corpora can help each other or why factuality corpora can be incompatible. ## representation of factuality as discussed in section 2, some corpora represent the factuality judgments using numbers in an interval, while others use categorial labels with a defined meaning. the corpus creators determine that there are several distinct categories that annotators can identify. in the examples (all from factbank, some simplified), the head of the phrase which presents the evaluated situation is bolded. certain (factbank label ct): the author commits to the truth of the presented situation. note that the commitment is independent of tense. probable (factbank label pr): the author presents the situation as probable. possible (factbank label ps): the author presents the situation as possible. fully underspecified (factbank label uu): the source does not know what is the factual status of the event, or does not commit to it. in addition, factbank annotates polarity on types of author belief, i.e., whether the author presents the situation as factual, or the absence of the situation as factual. polarity is only added to the non-uu values, resulting in a label set containing seven labels, ct-, pr-ps-, uu, ps+, pr+, ct+. we discuss our mapping between the categorical labels and numerical labels in section 7 (see ## corrected factbank dataset label correction @xcite developed a projection from the complex factbank corpus to a conll-formatted file that includes only factuality judgments by the author, enabling an annotation with a single value at the word level. this data set has been extensively used in nlp experiments (see section 2). we have found that there is a systematic error in this factbank data set projection. consider (4) above. factbank annotates the denial event as seen as factual by the author (ct+). the notification event is annotated twice: according to the author, the minister sees it as certainly false (ct-). the author herself does not express her view of the factuality and her perspective is labeled uu, as explained in section 3. the old projection of the factbank data set incorrectly picks up the ct-label. we correct the projection by sticking to the author's perspective and supply the uu label. we will make the correct projection of the fact-bank data set available, see appendix a for details. ## evaluation of factuality prediction to date, the category labels in factbank have been translated to numbers for training and testing (from -3 for ct-to +3 for ct+). a regression head predicts a number, and the results are evaluated with the pearson correlation r between the predicted and gold numbers. the publications to date also provide mae (which need not correlate with correlation). we propose an additional evaluation, namely f-measure on the categories and macro averaged f-measure. usually in nlp, there is no single correct evaluation metric: the best evaluation metric to use depends on the downstream use we want to make of the module we are evaluating. in some applications, we need to know the specific level of commitment of the writer. for example, in argumentation analysis we need to detect claims to which the writer is fully committed, and when analyzing hedging as a marker of politeness or power structure, we need to identify the pr/ps family. but the numerical evaluation makes the same difference between pr+ and ps+ on the one hand, and pr+ and ct+ or ps+ and uu on the other hand. f-measure clearly separates these cases. in this paper, we show results using correlation, mae, and the f-measures (macro-averaged f1 and per-label f1). jiang and de marneffe (2021) provide state-of-theart results for many of the corpora discussed in section 2 using a simple architecture. in this section, we redo their experiments with the updated fact-bank data set. specifically, we redo the multi-task learning experiments using the same underlying architecture (the selfattentivespanextractor developed by @xcite ) and the same training parameters as @xcite . this training paradigm allows all corpora to share the same bert parameters, but with each corpus having a regression head with corpus-specific parameters. the authors find that fine-tuning on bert-large performs best. however, with the corrected factbank dataset, we find that robertalarge outperforms bert-large, and we therefore use it for our experiments. for each experiment, we do three runs with different seeds and report the average for pearson correlation and mae, and for most experiments we also provide the standard deviation. ## experiments on factbank in this section, we evaluate exclusively on fact-bank (using our new factbank dataset described in section 4). our goal is to provide the best results on fact-bank. for all experiments reported in this section, we follow the same setup as section 6, except that we train two models per setup, one for regression which we evaluate numerically (r and mae), and one for classification which we evaluate using f1. in classification, we collapse fact-bank labels pr and ps (probable and possible), as they are rare, and their distinction is less important. ## testing on cb we have seen that the corpora that most help factbank are cb and mv. can factbank and mv help cb? in section 6, we trained on all corpora (following the lead of jiang and de marneffe (2021)). here, we train on only cb, factbank, and mv, and evaluate on cb. one major difference, however, is genre. ldc consists of discussion forum posts with many typos and fragmentary language, while factbank consists entirely of newswire. train r mae ldc 0.822 0.361 fb 0.616 0.630 we carefully examine this combination and show why some factuality corpora, even if they have similar annotation goals and annotators, may be incompatible. using the same system and experimental setup as appendix b, we perform two experiments: first, we train on ldc and test on ldc, and second, we train on factbank and test on ldc. ## conclusion after correcting an error in a widely used data set derived from factbank, we report new best results on four corpora: factbank ( given the targeted help cb can provide on fact-bank predictions for embedded clauses, and given the current weakness on hypotheticals in factbank, we suggest a new targeted annotation of factivity on sentences in hypothetical contexts. yuzhe yang, kaiwen zha, ying-cong chen, hao wang, and dina katabi. 2021.