Towards Generative Event Factuality Prediction (2023)

Abstract: We present a novel end-to-end generative task and system for predicting event factuality holders, targets, and their associated factuality values. We perform the first experiments using all sources and targets of factuality statements from the FactBank corpus. We perform multi-task learning with other tasks and event-factuality corpora to improve on the FactBank source and target task. We argue that careful domain specific target text output format in generative systems is important and verify this with multiple experiments on target text output structure. We redo previous state-of-the-art author-only event factuality experiments and also offer insights towards a generative paradigm for the author-only event factuality prediction task.

## introduction the term factuality refers to the author's or speaker's presentation of an event as factual, i.e. as an event that has happened, is happening, or will happen. often times, an author does not only talk about what they believe is factual, but also about what others believe is factual. thus, when a speaker presents an event, they communicate their view of the factuality of the event, and they can also at the same time attribute a factuality judgment about the same event to another source. over the past 15 years, the task of event factuality prediction has received a lot of attention, but only in predicting the factuality of an event according to the author's presentation. multiple corpora have been created alongside multiple machine learning architectures which solely focus on predicting the author's presentation of factuality. an exception is the factbank corpus @xcite , which not only annotates the author's presentation of factuality, but also annotates the nested sources assigning factuality values to events in text. in this paper, our goal is to predict the presentation of factuality of the nested sources mentioned in a text alongside their target events. we choose the factbank corpus @xcite as it is the only corpus annotating nested source factuality and it is carefully annotated and constructed. we attempt combinations with other corpora, namely author-only event factuality corpora and source and target cognitive state corpora, to improve on predicting nested source and target factuality. we perform all of these experiments with a novel generative approach and create a new version of the event factuality prediction task. there are four main contributions of this work: we first present the problem we are solving (section 2). we then present a survey of previous work (section 3). in section 4, we present the factbank database architecture. section 5 details our generative experimental details and modeling framework. finally, in sections 6 and 7 we report experiments on the factbank source-and-target and author-only tasks, respectively. ## background and motivation to understand the notion of factuality, consider the following sentence from the factbank corpus (we have replaced a pronoun for clarity in this exposition). this sentence reports on three events: a selling event, a saying event, and a doubling event. note that, in this paper, we are not interested in temporal relations, and the notion of factuality applies independently of whether an event is in the past, happening at utterance time, or in the future. we can identify four different factuality claims in this sentence: the first three are claims from the author's perspective, while the last one is from wu's perspective. we refer to the bearer of the perspective as the source, and the event (or state) that the factuality judgment is about as the target. factbank, following mpqa @xcite , represents the source of a factuality judgment as an ordered list of sources, since the sentence does not directly tell us about michael wu's factuality judgment, but rather the author's claim about michael wu's factuality judgment. in this paper, we do not address the explicit reconstruction of such attribution chains. in the above example, we have seen two factuality values: certain factual, and unknown. we can identify additional values by allowing for noncertain factuality (something may have happened), 1 1 factbank divided this category into the probable and the possible, but this leads to data fragmentation, and it can also be hard for humans to distinguish these two cases. and by incorporating polarity (something has happened or has not happened). this gives us a set of five possible values for factuality, as shown in in nlp, there is a distinct task of determining whether a statement is true or not (fact checking). unfortunately, this other task is sometimes also called "factuality prediction" (see, for example, @xcite ). the difference is that we are interested in how the author presents the event, not ground truth. so despite the same or similar name, there are two different tasks and we only deal with the presentation task, not the ground truth task. ## related work author-only factuality corpora all eventfactuality corpora focus on the presentation of factuality according to the author of the text, with the exception of factbank, which also annotates the factuality of the mentioned sources besides the author. these corpora include lu @xcite , uw @xcite , ldccb (ldc) @xcite , meantime (mt) @xcite , megaveridicality (mv) @xcite , uds-ih2 (ud2) @xcite , commitmentbank (cb) @xcite , and rp @xcite . these corpora mainly differ as to what is defined as an annotatable event, the genre of the text, the type of annotators, and the annotation scale. these corpora were unified under a continuous annotation scale in the range [-3, 3] by @xcite (though the author-only factuality value in factbank was misinterpreted, see @xcite for details). factbank the main focus of this paper is the factbank corpus, which annotates all events introduced in a corpus of exclusively newswire text. the factbank corpus not only annotates the factuality presented by the author of a text towards an event, but also the factuality of events according to their presentation by sources mentioned inside of the text. @xcite were the first to investigate and perform experiments on the source and target annotations in factbank. however, we cannot perform an apples-to-apples comparison, as their system neither recognizes events nor identifies sources mentioned in the text. rather, in their evaluation, this information was created from manual annotation, fed to the system, and then tested on the whole factbank corpus. we choose to focus on factbank because of its expert-level annotations and its detailed source and target annotations. because of the complexity of the factbank corpus, we build a robust and efficient database representation of factbank, which includes all sources including the author, the targets of the factuality attributions, and their respective relations. machine learning architectures all previous approaches on the event-factuality prediction task use author-only corpora and predict factuality according to the author of the text. early approaches to the event factuality prediction task used rulebased systems or lexical and dependency tree based features @xcite . expanding on these rule-based approaches, other work on the event factuality prediction task used svms alongside these dependency tree and lexical based features @xcite @xcite . early neural work includes lstms with multi-task or single-task approaches @xcite or using bert representations alongside a graph convolutional neural network @xcite . jiang and de marneffe (2021) expand on these previous works by using other event factuality corpora in multiple training paradigms while also introducing a simpler architecture. these previous neural approaches evaluate on pearson correlation and mean absolute error (mae). in previous work, we provide the first end-to-end evaluation using f-measure of the author-only event factuality prediction task @xcite . our work differs from the previous work in two major ways: first, we are the first to provide a novel and end-to-end generative approach for the event factuality prediction tasks (both author-only and source-and-target). furthermore, besides our own previous work @xcite , all previous works assumed gold event heads. our system is by default end-to-end, making it usable in real world applications. second, we perform experiments on the nested sources and target event's factuality, while other works only focused on the presentation of factuality according to the author. absa and orl two tasks close in formulation to our task and from which we adopt ideas and insights are the aspect-based sentiment analysis (absa) task and the opinion role labelling task (orl). @xcite create the aspect sentiment triplet extraction task to predict triplets consisting of aspects, opinions, and sentiment polarity. @xcite are the first to use a generative approach for absa fine-tuning on t5. expanding on this, @xcite achieve state-of-the-art results on all absa corpora using a multi-task learning approach through task-specific prompts. the orl task aims to discover opinions, the sources of opinions, and the associated targets of opinions using the mpqa 2.0 corpus @xcite . @xcite build an end-to-end system creating span representations and using a multi-task learning framework. they achieve state-of-the-art results in the end-to-end setting on the exact match f1 metric. ## factbank database we present a generalized database structure for capturing cognitive states expressed in language. the goal is to unify multiple annotated corpora in one format, and to make it simple for users to extract the information they need in various formats. in this paper, we describe only how we use it to hold the annotations of event factuality corpora, and of factbankin particular, whether in the authoronly perspective or source-and-target perspective. however, given the diversity of corpora, with each corpus having its own focus, annotation rules, and annotation styles, our database structure is sufficiently broad and abstract to accommodate various corpora equally well and yet to preserve the rich-ness of information that each corpus offers, so as to facilitate combining corpora in future experiments with as little data loss as possible. our goal of preserving the distinct details of individual corpora serves as a step in the direction of bringing human knowledge to bear upon otherwise black-box machine learning techniques. as an example, consider the factbank and lu @xcite corpora. the lu data was published as gate-formatted xml files with annotation targets and annotations given in xml elements, whereas factbank was published as a set of text files, each of which represents a relation in what amounts to a relational database. from both of these data sources, we may want to construct, for each training and testing example, a set of triples (sentence, target-marked-elements, label), where target-marked-elements are the tokens of the sentence that describe the target of the factuality judgment by the author, and to which the label refers. if we used the original factbank release and created a database from it, eliciting triples satisfactory for machine learning wouldl require a complex query with many joins and filters. this is because the structure of the factbank (implicit) database is oriented toward event-time relations rather than factuality labeling. accordingly, we designed a new database structure more amenable to queries to support machine learning and developed code to translate corpora including factbank into this database model. database structure to build the unified database, we needed a stable, fast, and lightweight tool. python's extensive library support for sqlite database interactions fit those requirements. the unified database's schema is composed of four tables: sentences, mentions, sources, and attitudes. we provide a graphic of the database schema in appendix c. the sentences table stores each sentence and any relevant identifying metadata. thus far, we have not encountered any corpora with suprasentential information encoded as labels. in principle, however, this table can be refactored to accommodate possible future suprasentential information. elements within each sentence marked for labeling are stored in the mentions table, with an entry being composed of the surface text of the element, which may be one or more tokens, and its character offset within the sentence. each sentence may contain more than one marked element. the sources table represents not only sources but their possible nested relations within sentences. these "according-to" relations form a list, as in mary said that john said that jane was coming to dinner. here, the embedded source for the coming event is (@xmath0. these "according-to" relations may form a tree, as in mary said that john said that jane was coming to dinner, but bob said that she was not. here, the embedded source for the coming event is (@xmath1. the author may have more than one child source, as in mary said that john was coming to dinner, but bob said that john was staying home. here, we have (@xmath2, and (@xmath3. each sentence may have more than one source, but each source has at most one mention. the implied author has no mention, and a named source mentioned repeatedly is listed once for each mention, since we do not apply anaphora resolution. finally, the attitudes table aggregates a sentence, its marked elements, and the factuality or sentiment label; the table accommodates both labels but could be refactored to support further label types. each source may have a distinct attitude toward each of several targets, and each target may have more than one source with its own attitude toward that target. thus, each source-target pair drawn from mentions has a single listing in attitudes. using event factuality corpora annotated on source-and-target factuality is inherently complex and requires structure induction, source linking, and complex database-like operations. our database structure is an initial step to address the complexity of corpora while also making easyto-use software for corpus projection and conversion. our database for factbank is available at https://github.com/t-oz/factbankunidb . ## experiments: source and target in this section, we perform experiments on the stf task. we evaluate exclusively on fbst. our goal is to achieve the best results on this projection of the corpus. ## experiments: author only in this section, we perform experiments on the aof task. we evaluate exclusively on fbao, performing our experiments with the same model and training paradigm. we use three styles of target representation mentioned in section 5: one style where we extract event words and their associated factuality values as tuples, an in-line annotation style used by @xcite , and finally a mtl triplet generation task with the source and target projection of factbank where we generate triplets of format (@xmath4, @xmath5, factuality label). furthermore, we also factor polarity in our experiments. @xcite found that separately predicting polarity and factuality for the event factuality task can lead to error reductions since polarity is often expressed independently of the degree of factuality. we treat the addition of polarity as a triplet generation task generating triplets of format (@xmath6, factuality label, polarity). we reduce the factuality label to the strength of factuality (true, possibly true, unknown), with the polarity being one of (negative, unknown, positive). ## conclusion we provide a new generative framework for the event factuality prediction task using flan-t5 and focusing on output format, individual task prefixes, and multi-task learning. to tackle the complexity of the factbank corpus, we create a database representation that simplifies extracting sources, targets, and factuality values for all projections of fact-bank, which we will publicly release. our sourceand-target experiments show that careful output formatting can yield improvements (