I nfo S urgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection (2021)

Abstract: To defend against machine-generated fake news, an effective mechanism is urgently needed. We contribute a novel benchmark for fake news detection at the knowledge element level, as well as a solution for this task which incorporates cross-media consistency checking to detect the fine-grained knowledge elements making news articles misinformative. Due to training data scarcity, we also formulate a novel data synthesis method by manipulating knowledge elements within the knowledge graph to generate noisy training data with specific, hard to detect, known inconsistencies. Our detection approach outperforms the state-of-the-art (up to 16.8% accuracy gain), and more critically, yields fine-grained explanations.

## introduction in recent years, generative neural network models in natural language processing @xcite and computer vision @xcite have become the frontier for malicious actors to controllably generate misinformation at scale. these realistic-looking ai-generated "fake news" have been shown to easily deceive humans, and it is, thus, critical for us to develop robust verification techniques against machine-generated fake news @xcite @xcite . current misinformation detection approaches mainly focus on document-level fake news detection using lexical features and semantic embedding representations @xcite @xcite . however, fake news is often generated based on manipulating (misusing, exaggerating, or falsifying) only a small part of the true information, namely the knowledge elements (kes, including entities, relations and events). moreover, recent news oftentimes makes claims that do not have verified evidence yet, and evaluating the truthfulness of these real-time claims depends more on their consistency with other information conveyed in other data modalities. in this paper, we propose a new task: finegrained, knowledge element-level cross-media information consistency checking. the task involves treating the entire multimedia news article as one whole interconnected claim, where the goal is to detect misinformative kes across the image, caption, and body text, as revealed by inconsistencies with respect to itself, or to background knowledge. this ke-level detection approach directly points out the fake pieces of information in the news, allowing for better explainability. figure 1 shows an example where both the text and image provide complementary information about key argument roles of an event. we present the information surgeon (infosurgeon) model, which takes full advantage of state-of-theart multimedia joint knowledge extraction techniques to analyze fine-grained event, entity, and relation elements, as well as whether these extracted kes align consistently across modalities and background knowledge. we propose a novel probabilistic graphical neural network model to fuse the outputs from these indicators. a major challenge to performing ke level misinformation detection is the lack of training data. hence, we additionally propose a novel approach to generate noisy training data automatically since existing fake news generators @xcite do not track the specific pieces of information generated that are fake. we take a real news article, extract a multimedia knowledge graph, and replace or insert salient nodes or edges in the graph. we track the specific manipulation operations, and regenerate the manipulated version of the news article figure 1 : the architecture of our fine-grained, ke-level fake news detection system, infosurgeon. our model uses graph-based neural network to aggregate cross-modal and external knowledge in a multimedia kg to determine whether a document is real or fake and provide ke-level explanations. for instance, the document above should ideally be detected as fake due to cross-modal inconsistencies (i.e. the caption @xmath0, located.in, hidden @xmath1, which is inconsistent with the image @xmath2, located.near, visible crowd of @xmath3. <police, blinded, @xmath4, which is not supported by the image or caption. note: the article in this figure is inspired from materials reported in major news outlet that were later taken down due to misinformation. experiment results show that our approach achieves 92%-95% detection accuracy, 16.8% absolute higher than the state-of-the-art approach @xcite . ablation tests demonstrate the effectiveness of our new detection method. the major contributions of this paper are: ## task formulation given a multimedia news article, x, which consists of its body text bt, list of images im 1..i , list of accompanying captions c 1..i , and metadata @xmath5, date, author, headline), our study aims to detect the presence of misinformation at two levels. in document-level detection, we classify each news article as either real or fake, overall. in knowledge element-level detection, we predict the specific set of knowledge elements in the news article conveying misinformation. here, we refer to knowledge elements (kes) comprehensively as the entities, relations, events, and subgraphs/metapaths @xcite in an information network. to detect the misinformative kes, we treat each news article as one ultimate claim represented by a multimedia knowledge graph @xmath6, e) capturing the important information conveyed. the nodes (n ) in the kg consist of entities (t), while the edges (e) in the kg consist of relations (r) or event argument roles (a) connecting the entities. detecting the kes causing a news article to be fake boils down to @xmath7, predicate, object @xmath8, and labeling all of the triplets in which the relation or event between a head entity and tail entity holds false as evidence of misinformation through binary edge classification. figure 2 shows exam- relation: event: events/relations: "hk police" "hidden corner" "hk police" "protesters" physical. located near "arrest" justice. arrest. detainee justice. arrest. jailer "hk police" "people" "shoot" conflict. attack. victim conflict. attack. attacker "real bullets" conflict.attack. instrument @xmath9, @xmath10, t f f f f life.die. victim ples of how kes should be detected if they occur in the kg of a news article. we evaluate document-level fake news detection based on the established metric of prediction accuracy. to evaluate fine-grained ke level fake news detection, we compute the f-score: the harmonic mean of precision and recall across kes. this is an appropriate metric due to the imbalanced nature of fake kes, which usually constitute the minority. ## fake news generation currently, there exists no annotated dataset for ke level misinformation detection. a primary reason may be due to explicitly fake (as opposed to subtly biased) news being edited or taken down by online platforms after initial posting. because manually labeling the misinformative kes in a real-world news corpus is expensive, we aim to create a novel dataset with controlled synthesis of news articles and automatically generated labels for fine-grained ke level explainability in fake news detection. in this section, we propose two novel approaches that generate fake news, and at the same time, automatically label the misinformative knowledge elements. given a set of real news articles, x real , we perform deliberate edit operations on certain salient kes in the new articles' kg to derive a manipulated representation, kg 0 . hence, we can generate a new article conditioned on kg 0 . the corresponding ke level label can then be automatically derived, with the manipulated elements as fake and the unaltered elements as real, while the document-level label for the new generated article is fake. ## conclusions and future work we have demonstrated a novel method for multimedia misinformation detection that can achieve 92%-95% detection accuracy using cross-media information consistency checking and adversarial fake information generation by knowledge graph manipulation. our framework can be used to ingest and assess news articles, while providing finegrained knowledge element-level explanations. as future work, we plan to extend the problem such that any combination of body text, image, video, audio and caption can be "fake". we will also incorporate consistency reasoning across multiple documents and from commonsense knowledge, and extend our approach to open-domain documents from multiple sources, languages and cultures. in the long term, we aim to collect more human-generated data with different types of intent that cause different levels of acceptance by readers, study more types of human manipulations to design additional criteria (e.g., entity novelty, newsworthiness, etc.), jointly detect misinformation and intent, correct detected misinformation, and generate authentic narratives.