Cross-document Misinformation Detection based on Event Graph Reasoning (2022)

Abstract: For emerging events, human readers are often exposed to both real news and fake news. Multiple news articles may contain complementary or contradictory information that readers can leverage to help detect fake news. Inspired by this process, we propose a novel task of cross-document misinformation detection. Given a cluster of topically related news documents, we aim to detect misinformation at both document level and a more fine-grained level, event level. Due to the lack of data, we generate fake news by manipulating real news, and construct 3 new datasets with 422, 276, and 1,413 clusters of topically related documents, respectively. We further propose a graph-based detector that constructs a cross-document knowledge graph using cross-document event coreference resolution and employs a heterogeneous graph neural network to conduct detection at two levels. We then feed the event-level detection results into the document-level detector. Experimental results show that our proposed method significantly outperforms existing methods by up to 7 F1 points on this new task.

## introduction the dissemination of fake news has become an important social issue. for emergent complex events, human readers are usually exposed to multiple news documents, where some are real and others are fake. news documents from different sources naturally form a cluster of topically related documents. we notice that articles about the same topic may contain conflicting or complementary information, which can benefit the task of misinformation detection. an example is shown in figure 1 . as shown in the knowledge graph, the death of rosanne boyland in 2021 us capitol attack is a shared event across all four documents. each document is internally consistent, making it difficult to 1 codes and data are at https://github.com/ shirley-wu/cross-doc-misinfo-detection . most of the existing work on fake news detection is limited to judging each document in isolation. in contrast, we propose a novel task of crossdocument misinformation detection that aims to detect fake information from a cluster of topically related news documents. we perform the task at both the document level and event level. each event describes a specific type of real-world event mentioned in the text (e.g., the death of boyland in figure 1 ), and usually involves certain participants to represent different aspects of the event (e.g., the cause of death and the victim of the death event). document-level detection aims to detect fake news documents. event-level detection is a more fine-grained task that aims to detect fake events, thereby pinpointing specific fake information in news documents. existing work on fine-grained misinformation detection detects triplets of false knowledge @xcite . however, we focus on identifying false events instead of relations or entities, because events are more important for storytelling and easier to compare across multiple documents through cross-document coreference resolution. to the best of our knowledge, there are no fake news detection datasets with clusters of topically related documents. therefore, we construct 3 new benchmark datasets based on existing real news corpus with such clusters. following @xcite , we train a generator that generates a document from a knowledge graph (kg), and feed manipulated kgs into the generator to generate fake news documents. tracking the manipulation operations, we also obtain supervision for event-figure 1 : an example of cross-document misinformation detection, including the texts and knowledge graphs for four news documents. the three real news documents complement each other, while the fake news contradicts the other news. news 1 falsely speculates that boyland was crushed to death, but it admits that the cause of death was not yet verified. news 2 and 3 complete the story by reporting that boyland died of drug overdose. the fake news claims that boyland was killed by police, which contradicts the other news. additionally, the fake news states that the police attacked boyland, which is inconsistent with news 3's claim that the police was trying to help her. ## task formulation given a cluster of documents about the same story, the task of cross-document misinformation detec-tion aims to detect the fake information included in the cluster. formally, let @xmath0, • • • , d n } be the document cluster, and @xmath1. some documents in s are real, while others are fake. from each document @xmath2, we extract events e(d) = {e 1 , • • • , e m }, where @xmath3. in an extracted event set e(d), some events are real and others are fake. we conduct the task of misinformation detection at two levels, document level and event level. document-level detection aims to predict whether each document @xmath4. event-level detection is a more fine-grained task that aims to predict whether each event @xmath5, @xmath6. in the example in figure 1 , the die event in the fake news is fake, since it falsely describes boyland being killed by the police, but she actually died of drug overdose. ## approach an overview of our approach is shown in figure 2 . given a cluster of documents, we first construct a within-document kg for each document using an ie system @xcite , and then connect the within-document kgs into a cross-document kg using cross-document event coreference resolution. based on the cross-document kg, we use a hetereogeneous @xcite to perform detection. we further incorporate the results of event-level detection to help the document-level detector. ## dataset construction currently, there are no existing resources for crossdocument misinformation detection. we propose to construct datasets based on real news datasets with clustering information. for each cluster, we randomly sample 50% real news and replace them with manipulated fake news. figure 3 shows an on the real news and its ie output, we select a high-frequency "detonateexplode" event and replace its argument entity "device" with "machine guns". we then generate the fake news from the manipulated kg. in the kg of generated fake news, the manipulated entity "guns" is an argument of the "detonateexplode" event, so we consider the event as fake. following @xcite , we train a kg-totext generator from the real news in our datasets, and generate fake news from manipulated kgs. the main differences between @xcite 's method and ours in terms of manipulating kg are: we record the manipulation operations, and use a heuristic rule to obtain supervision for event-level detection as explained below. in a fake document, if an event involves manipulated entities as arguments, we consider this event as fake. ## experiments 6.1 data we constructed three new benchmark datasets based on three datasets that naturally have clusters of topically related documents. ied is a complex event corpus, where each complex event refers to a real-world story (e.g., boston bombing) and is described by multiple documents @xcite . therefore, a complex event can be considered as a document cluster. tl17 and crisis are two timeline summarization datasets containing multiple ## conclusions and future work we are the first to study the new task of crossdocument misinformation detection. we conduct the task at two levels, document level and the more fine-grained event level, and construct three new datasets to handle the lack of training data. we further propose a graph-based cross-document detector that conducts reasoning over a cross-document knowledge graph and feed the event-level detection results into document-level detector. the ex-perimental results show that our proposed method significantly outperforms existing methods. for future work, we intend to extend our method to conduct cross-document reasoning over more types of information (e.g., entities and relations) in addition to events. we also plan to extend our method to multi-media news including texts, images, audios and videos, which requires the construction of cross-document multi-modal knowledge graphs. finally, a challenging but important task is to construct a large-scale fake news detection corpus with human-written fake news containing document clusters and study our method in this scenario.