Inconsistency Matters: A Knowledge-guided Dual-inconsistency Network for Multi-modal Rumor Detection (2021)

Abstract: Rumor spreaders are increasingly utilizing multimedia content to attract the attention and trust of news consumers. Though a set of rumor detection models have exploited the multi-modal data, they seldom consider the inconsistent relationships among images and texts. Moreover, they also fail to find a powerful way to spot the inconsistency information among the post contents and background knowledge. Motivated by the intuition that rumors are more likely to have inconsistency information in semantics, a novel Knowledge-guided Dual-inconsistency network is proposed to detect rumors with multimedia contents. It can capture the inconsistent semantics at the cross-modal level and the content-knowledge level in one unified framework. Extensive experiments on two public real-world datasets demonstrate that our proposal can outperform the state-of-the-art baselines.

## introduction social media has fostered various false information, including misrepresented or even forged multimedia content, to mislead the readers. the widespread rumors may cause significant adverse effects. for example, some offenders use rumors to guide public opinion, damage the credibility of government, and even interfere with the general election @xcite . therefore, it is urgent to automatically detect and regulate rumors to promote trust in the social media ecosystem. most of existing rumor detection methods focus on textual data to extract distinctive features @xcite @xcite . with multimedia technology development, visual contents have become an important part of rumors to attract and mislead the consumers for more credible storytelling and rapid diffusion @xcite . detecting multimedia rumor posts is a double-edged * * corresponding author figure 1 . a real-world example of a fake multimedia tweet. it is suspicious to see sharks appear in a subway. such abnormality should be captured and serve as an essential clue for rumor identification. apart from cross-modal inconsistency, rumor detection can also benefit from knowledge graph (kg), which can provide faithful background knowledge to verify the semantic integrity of post contents. previous works @xcite used kg to complement the post contents by various data fusion methods. however, they ignore the inconsistent information that may exist between the contents and the background knowledge. for example, in fig 1 , it would be a great help to judge the the truthfulness of the post, given the background knowledge that sharks are very unlikely to occur in a subway. we use content-knowledge inconsistency to describe the uncommon co-occurring entities foot_0 spotting in the multi-modal post contents, such as "shark" and "subway" example in fig 1 . in this paper, we consider both cross-modal inconsistency and content-knowledge inconsistency, which are also referred as dual-inconsistency. we analyze the data and find that the above dualinconsistency shows a statistically significant distinction between rumor and non-rumor posts (see details in sec. 4.2). such findings in data analysis highlight that the dual-inconsistency can be indicative of the news veracity and should be considered when modeling. however, it is challenging to build models to capture such dual-inconsistency for two reasons. first, text, image and kg data have different structures, which can not be directly integrated. second, there is no straightforward way to capture their various semantic relationships, especially the inconsistent relationships. to address these issues, we propose a novel knowledge-guided dual-inconsistency network to capture the inconsistency information at the crossmodal level and the content-knowledge level simultaneously. note that our framework does not require both types of inconsistency to be present to effectively detect rumors. in other words, either type of inconsistent information can be a strong feature to infer a piece of tweet is a rumor. our framework mainly consists of two sub neural networks: one is to extract cross-modal differences between images and texts, excluding their modalshared information; the other is to identify the abnormal entity pairs that co-occur in the post contents through measuring their kg representation distances. the two sub neural networks are tightly coupled to achieve the best performance. the contributions of our paper are three-fold: ## related work textual and social contextual rumor detection. ## experiments in this section, we present the experiments to evaluate the effectiveness of our proposal. ## conclusion we have revealed the necessity of capturing the inconsistent semantics for detecting rumors. we thus propose the knowledge-guided dual-inconsistency network, which involves the cross-modal inconsistency and content-knowledge inconsistency information in one unified framework. we have demonstrated our proposal's effectiveness in capturing and fusing both types of inconsistency features to achieve the best performance. note that the inconsistency features can be easily plugged into other rumor detection frameworks to further improve the performance. in future work, we plan to explore more effective inconsistency features and devise a more explainable model.