B e L eaf: Belief Prediction as Tree Generation (2024)

Abstract: We present a novel approach to predicting source-and-target factuality by transforming it into a linearized tree generation task. Unlike previous work, our model and representation format fully account for the factuality tree structure, generating the full chain of nested sources instead of the last source only. Furthermore, our linearized tree representation significantly compresses the amount of tokens needed compared to other representations, allowing for fully end-to-end systems. We achieve state-of-the-art results on FactBank and the Modal Dependency Corpus, which are both corpora annotating source-and-target event factuality. Our results on fine-tuning validate the strong generality of the proposed linearized tree generation task, which can be easily adapted to other corpora with a similar structure. We then present BeLeaf, a system which directly leverages the linearized tree representation to create both sentence level and document level visualizations. Our system adds several missing pieces to the source-and-target factuality task such as coreference resolution and event head word to syntactic span conversion. Our demo code is available on https://github.com/yurpl/beleaf and our video is available on https://youtu.be/SpbMNnin-Po.

## introduction the term "factuality" (or belief 1 ) refers to what extent an event mentioned by the author or by sources in a text is presented as being factual. in other words, the task aims to predict whether the author or the mentioned sources in the text believes the event happened. the event factuality prediction task (efp) has received a lot of attention over the past few years, but only in the perspective of the author of the text, disregarding the factuality of events according to all sources (lee et al., 1 we use the terms interchangeably since our system is called beleaf. factuality is closely related to the notion of "belief" as used in cognitive science and ai. 2015; @xcite @xcite . two notable exceptions are the factbank corpus @xcite and the modal dependency parsing corpus (mdp) @xcite . both corpora annotate event factuality according to the author of the text, and also according to the sources mentioned in the text, with some slight differences. factbank represents factuality on the sentence level, while the mdp corpus represents factuality as a document-level modal dependency structures (mds) proposed by @xcite . the mdp structure uses a tree representation where the author of the text (author) is the root, and events and other sources are child nodes of the author. the corpora also differ slightly on labels: factbank annotates the factuality of events (alongside their polarities) as ct (certain), pr (probable), ps (possible), uu (unknown), while the mdp corpus annotates events as full positive (pos), partial positive (prt), positive neutral (neut), negative neutral (neutneg), partial negative (prtneg) and full negative (neg). figure 1 : source-and-target factuality represented as a modal dependency structure for the sentence "meteorologists say the weather will get worse because there will be rains." an nlp system's ability to accurately attribute events' factuality according to all sources is vital for downstream tasks that are based on those events. consider the example sentence in figure 1 where we have three events: the say event, the rain event, and the get event. we also have one source that the author mentions, meteorologists. the author is certain (ct+) that the say event happened . however, the author does not tell us (uu) about their view of the factuality of the rains event or the get event because it is being presented by the source meteorologists. according to the source meteorologists, these events are factual (ct+). an information extraction system should extract the specific factuality of these events depending on all sources presenting the event, not only the author. systems and humans can then make a separate judgement about the weather based on their sense of the trustworthiness of the author and of meteorologists. in this paper, we present the source-and-target event factuality prediction task as a linearized tree generation task. we represent both factbank and the mdp corpus as linearized trees, achieving stateof-the-art results for both corpora and beating both our factbank results @xcite and the mdp results from @xcite . this representation format not only performs better, but also allows for a clear and interpretable visualization, which we show in our beleaf system. ## related work author-only factuality all previous approaches to the event factuality prediction task were in the author-only setting, ignoring nested sources. early approaches used rule-based systems and/or lexical and dependency tree based features @xcite . early machine learning work used svms alongside dependency tree and lexical based features @xcite @xcite . neural work includes lstms with multi-task or single-task approaches @xcite or using bert representations alongside a graph convolutional neural network @xcite . jiang and de marneffe (2021) expand on previous work by using other event factuality corpora in multiple training paradigms while also introducing a simpler architecture. these approaches evaluate on pearson correlation and mean absolute error (mae), failing to capture individual label performance and assuming events are given. we @xcite provided the first end-to-end evaluation using f-measure and improve on factbank. source and target factuality one of the main corpora experimented on in this paper, which annotates all events introduced in a corpus of exclusively newswire text is the factbank corpus @xcite .the factbank corpus not only annotates the factuality presented by the author of a text towards an event, but also the factuality of events according to their presentation by sources mentioned inside of the text. @xcite were the first to investigate and perform experiments on the source and target annotations in factbank. their evaluation was not end-to-end and was given manual annotations, so it is therefore not comparable to our results on factbank. we @xcite were the first to represent the event factuality prediction task as a generation task using flan-t5 while also accounting for source and target factuality. however, our previous model did not account for the full nesting structure of the source since our model only generated the last nested source. our new system generates the full nesting structure, and is therefore not comparable to our previous factbank results as that task was far easier and incomplete. @xcite also propose a source-and-target corpus (mdp corpus) and @xcite improve on their previous results by using a prompt-based approach where they treat factuality prediction as a bio tagging task, finetuning on xlm-roberta @xcite . following the modal dependency structure from @xcite , their corpus annotates events, sources, and credibility of sources throughout a whole document. the top level source is always the author of the text. while similar to factbank in some ways, there are some key differences (which we describe in section 3.1), making the corpora incompatible for joint experiments with factbank. qian et al. (2019) are the first to present the document level factuality task, but again in the author-only setting. their work is expanded by @xcite , and more recently @xcite . in this task the input is a document and a factuality target, and the output is the label representing the factuality attributed by the author to the provided target. our task is different, which is to find all sources and targets of factuality assessments. ## beleaf: system description in this section, we present our beleaf system which leverages our generated tree structure. our system is split into three parts: a generalized api for querying our flan-t5 model with either sentences or documents, a preprocessing pipeline where we improve on the document level event factuality/belief task from @xcite by accounting for coreference, and a postprocessing pipeline accounting for syntactic spans with a tree visualization tool. our system is shown in figure 2 . ## beleaf: output and visualization in this section, we provide examples for both sentence level and document level belief, alongside their corresponding tree outputs and json representations. ## conclusion we propose a linearized tree generation model for the source-and-target event factuality task prediction using flan-t5 and gpt-3. we evaluate the model on factbank and the mdp corpus, and achieve results for both. with our new representation and state of the art flan-t5 system, we present beleaf, a system for both sentence and document level factuality. we provide a preprocessing pipeline that accounts for coreference to create true document level representations of factuality. an inference api is then made which feeds to a postprocessing pipeline that creates syntactic spans from head words for users to see the full event contexts. finally, we merge everything into a tree visualization software that also includes a data download option.