Document-level Event Factuality Identification via Machine Reading Comprehension Frameworks with Transfer Learning (2022)

Abstract: Document-level Event Factuality Identification (DEFI) predicts the factuality of a specific event based on a document from which the event can be derived, which is a fundamental and crucial task in Natural Language Processing (NLP). However, most previous studies only considered sentence-level task and did not adopt document-level knowledge. Moreover, they modelled DEFI as a typical text classification task depending on annotated information heavily, and limited to the task-specific corpus only, which resulted in data scarcity. To tackle these issues, we propose a new framework formulating DEFI as Machine Reading Comprehension (MRC) tasks considering both Span-Extraction (Ext) and Multiple-Choice (Mch). Our model does not employ any other explicit annotated information, and utilizes Transfer Learning (TL) to extract knowledge from universal large-scale MRC corpora for cross-domain data augmentation. The empirical results on DLEFM corpus demonstrate that the proposed model outperforms several state-of-the-arts.

## introduction this paper focuses on document-level event factuality identification (defi) task, which is defined as identifying the factuality of a specific event based on a document from which the event is derived. as a sub-task in event factuality identification (efi), different from sentence-level event factuality identification (sefi) focusing on just a single sentence, defi requires comprehensive understanding documents with regard to events. figure 1 illustrates an example of documentlevel event factuality. the current event is e1, i.e., "barack obama joins joe biden's cabinet", and the sentences s2-s6 contain the event mentions referring to e1. from figure 1 we can know that: 1) factuality of event mentions vary among sentences. s4, s5 and s6 negate e1 by the negative cues "not" and "denied", and commit to e1 as "certain negative/ct-". but some other sentences express different factuality with regard to e1. s2 evaluates e1 as "possible positive/ps+" according to the speculative cue "speculation", and s3 commits to e1 as "underspecified/uu" since the event mention is in the clause led by "if". however, the document-level factuality of e1 is unique, i.e., ct-. 2) in addition to e1, there are irrelevant mentions of other events in the document as well, e.g., the ps+ event "susan rice and michèle flournoy were considering for key posts under biden"in s6, and the ct+ event "barack obama was elected president in 2008" in s7 , which may cause e1 to be identified as ps+ or ct+ falsely. currently, most efi studies limited to sefi @xcite @xcite . while defi is still in its early stage, and previous work @xcite @xcite usually regarded defi as a typical text classification task. currently, defi is mainly faced ## approach this section introduces our defi-mrc-tl model, where overview ( §2.1) gives the definitions of defi task, even factuality values, and transfer learning used by our model, then §2.2 and §2.3 present the data formalization and detailed structure of ext-tl and mch-tl model, respectively. ## experimentation this section first introduces experiment settings ( §3.1), including target and source datasets, evaluation metrics, implementation details, and baselines. then, experimental analysis focuses on the following aspects, i.e., 1) results and analysis ( §3.2) discusses the comparisons of our model with baselines; 2) ablation ( §3.3) inspects the effectiveness of source datasets and networks in defi-mrc-tl model; 3) case study ( §3.4) illustrates what our ext-tl model can learn from several samples to reveal the internal mechanism of ext-tl. ## conclusion this paper designs a novel framework formalizing document-level event factuality identifi-cation as mrc tasks, and considers both spanextraction and multiple-choice mrc. furthermore, our model takes into account transfer learning as cross-domain data augmentation capturing extra knowledge from large-scale corpus in typical mrc. experiments on dlefm corpus demonstrate that our model can achieve state-of-the-art performance. in the future, we will explore cross-document event factuality identification and apply more effective data augmentation method.