Factuality Assessment as Modal Dependency Parsing (2021)

Abstract: As the sources of information that we consume everyday rapidly diversify, it is becoming increasingly important to develop NLP tools that help to evaluate the credibility of the information we receive. A critical step towards this goal is to determine the factuality of events in text. In this paper, we frame factuality assessment as a modal dependency parsing task that identifies the events and their sources, formally known as conceivers, and then determine the level of certainty that the sources are asserting with respect to the events. We crowdsource the first large-scale data set annotated with modal dependency structures that consists of 353 Covid-19 related news articles, 24,016 events, and 2,938 conceivers. We also develop the first modal dependency parser that jointly extracts events, conceivers and constructs the modal dependency structure of a text. We evaluate the joint model against a pipeline model and demonstrate the advantage of the joint model in conceiver extraction and modal dependency structure construction when events and conceivers are automatically extracted. We believe the dataset and the models will be a valuable resource for a whole host of NLP applications such as fact checking and rumor detection.

## acquiring a modal dependency data set in this section we first provide additional detail for the modal dependency structure, and then present our strategy of decomposing the modal dependency structure into subtasks that are suitable for crowdsourcing. we also evaluate the quality of our annotated data set, and provide statistics relevant for training mds parsing models. ## neural modal dependency parsing in this section, we introduce our parser for modal dependency parsing. our modal dependency parser is inspired by @xcite , who introduce a ranking model for temporal dependency parsing. as the temporal dependency tree used to train their model is similar in structure to our modal dependency tree, it is reasonable to adopt their model as the starting point. our model is also inspired by @xcite , who extend zhang and xue (2018a) by replacing the bi-lstm encoder with contextualized neural language models, such as bert @xcite . specifically, our modal dependency parser constructs a modal dependency tree by incrementally identifying the parent node for each child node in textual order. for each child node, the parser ranks the candidate parent nodes and selects the one with the highest score as its parent node. since the nodes in a modal dependency tree are events or conceivers, to parse a text into a modal dependency tree, we need to first extract the events and conceivers, then build the modal dependency structure. since zhang and xue (2018a)'s pipeline system suffers from error propagation, we adopt a multi-task learning approach that jointly trains the event and conceiver extraction and structure building components. ## related work factuality annotation while there is a significant amount of research on annotating factuality or modality @xcite @xcite @xcite @xcite , factuality and opinions @xcite , senses of modal verbs @xcite , and credibility in social media @xcite , a few of them are particularly related to our work. our annotation is closely related to factbank @xcite in that both annotate the level of certainty that the source asserts over an event, but factbank does not explicitly represent their relations in a hierarchical structure and is annotated by expert annotators. like our work, @xcite also annotate event factuality via crowdsourcing, but they only annotate the level of certainty from the perspective of the author, to the exclusion of non-author conceivers. our work is based on @xcite , who first came up with the model dependency annotation scheme. however, they only annotate 377 events from 6 documents in a pilot effort with expert annotators. we have shown that it is feasible for crowd-workers to annotate modal dependency structures with considerable consistency and produce modal dependency annotation at scale. automatic factuality assessment existing work typically casts factuality assessment as a classification or regression problem. for example, saurí and pustejovsky (2012) and prabhakaran et al. (2015) adopt rule-based and feature-based machine learning approaches to factuality classification. more recently, qian et al. (2018) predict event factuality via a generative adversarial networks based model. rudinger et al. (2018) design two lstm based models, and pouran ben veyseh et al. (2019) use a graph-based neural network model for event factuality prediction. our work departs from previous practice and recasts factuality assessment as modal dependency parsing to simultaneously predict the source and its level of certainty over an event, and exposes both for downstream applications. ## conclusion and future work in this paper, we proposed a novel approach to factuality assessment by casting it as a modal dependency parsing problem. we first built a large data set annotated with modal dependency structures via crowdsourcing, and demonstrated the quality of this data set with a careful evaluation of each aspect of the annotation. we then developed the first modal dependency parser, adopting a joint learning approach to alleviate error propagation, and demonstrated its advantage over the pipeline approach in an end-to-end evaluation. future work involves further improving the accuracy of the parser and applying the parser to perform large-scale factuality assessments of events in news media.