Interpretable Multimodal Misinformation Detection with Logic Reasoning (2023)

Abstract: Multimodal misinformation on online social platforms is becoming a critical concern due to increasing credibility and easier dissemination brought by multimedia content, compared to traditional text-only information. While existing multimodal detection approaches have achieved high performance, the lack of interpretability hinders these systems’ reliability and practical deployment. Inspired by Neural-Symbolic AI which combines the learning ability of neural networks with the explainability of symbolic learning, we propose a novel logic-based neural model for multimodal misinformation detection which integrates interpretable logic clauses to express the reasoning process of the target task. To make learning effective, we parameterize the symbolic logical elements using neural representations, which facilitate the automatic generation and evaluation of meaningful logic clauses. Additionally, to make our framework generalizable across diverse misinformation sources, we introduce five meta-predicates that can be instantiated with different correlations. Results on three public datasets (Twitter, Weibo, and Sarcasm) demonstrate the feasibility and versatility of our model.

## introduction misinformation refers to incorrect or misleading information 2 which includes fake news, rumors, satire, etc. the enormous amount of misinformation emerged on online social platforms is attributed to users' reliability on the information provided by the internet and the inability to discern fact from fiction @xcite . moreover, widespread misinformation can have negative consequences for both societies and individuals. there-1 https://github.com/less-and-less-bugs/ logicmd 2 https://www.merriam-webster.com/dictionary/ misinformation fore, there is an urgent need to identify misinformation automatically. while numerous posts are in multimodal style (i.e., text and image) on social media, this work concentrates on multimodal misinformation detection. multimodal approaches, which either fuse text and image features @xcite @xcite or investigate discrepancies between the two modalities @xcite , have been used for misinformation detection with some success. however, these methods often lack interpretability because of the black-box nature of the neural network. some frameworks have been proposed to solve this challenge. as depicted in fig. 1 , methods based on attention maps, such as those outlined in @xcite and @xcite , have been employed to identify highly correlated text or image content (referred to here as "where") according to attention weights, while multi-view based methods, such as those described in @xcite and @xcite , have been utilized to highlight the most contributive perspectives foot_0 (referred to here as "how"). however, the explainability of the fusion of such attention or views has yet to be fully established @xcite , and these methods cannot concurrently illustrate both the "where" and "how" of the reasoning process. such interpretability is crucial for ensuring trust, reliability, and adoption of deep learning systems in real-world applications @xcite @xcite , particularly when it comes to detecting misinformation @xcite . to address the aforementioned limitations, owing to neural-symbolic learning @xcite , we propose to incorporate logic reasoning into the misinformation detection framework to derive human-readable clauses. as shown in fig. 1d , the clause b 1 ((v 1 , v 2 ), rumor)∧ b 2 ((t 1 , t 2 ), rumor) ⇒ h((t, i)), rumor) is induced from the text-image pair where constants v 1 , v 2 , t 1 , t 2 are crucial visual patches and textual tokens for predication, corresponding to "where". body predicates b 1 and b 2 indicate relationships between patches and tokens for misinformation identification, corresponding to "how". we propose to automatically learn these logic clauses which explicitly express evident features and their interactions to promote interpretability and improve the final performance, which has not been explored by previous work. however, given the intrinsic complexity and diversity of multimodal context, it is hard to explicitly predefine the exact relationships as logic predicates. to this end, we introduce five general perspectives relevant to the task of misinformation detection as meta-predicates for clause formulation. these perspectives include suspicious atomic textual content, visual content, relationships between text tokens, visual patches and both modalities. each meta-predicate can be instantiated with different correlations between contents of the text-image pair and target labels (e.g., (t 1 , t 2 ) and rumor in fig. 1d ), aiming to cover a wide range of aspects leading to misinformation. for instance, the fifth perspective implicates exploiting cross-modal contents to debunk misinformation while cross-modal ambiguity learning @xcite , incon-sistency between news contents and background knowledge @xcite and entities misalignment @xcite are candidate correlations to achieve this goal. building upon these definitions, we propose a logic-based multimodal misinformation detection model (logicdm). logicdm first extracts embeddings for text tokens and image patches using corresponding encoders and then generates cross-modal object embeddings for different predicates using a multi-layer graph convolutional network (gcn). we then propose to parameterize meta-predicates by weighing the importance of each correlation. when combined with different object constants, these meta-predicates are softly selected to produce interpretable logic clauses defining the target predicate. the whole framework can be trained end-to-end with differentiable logic operators and probabilistic logic evaluations. to summarize, the contributions of this work include: 1) we propose an explainable neural-symbolic approach capable of automatically generating logic clauses instantiated with multimodal objects via differentiable neural components. 2) we define five meta-predicates building upon existing misinformation detection perspectives and introduce an adaptive mechanism to represent these predicates using soft selections over multiple pre-defined correlations. 3) we provide comprehensive evaluations of our model on three benchmark datasets. ## methodology this section introduces the proposed logicbased multimodal misinformation detection model (logicdm), which offers a more explicit reason-ing process and better performance than existing approaches. the model consists of four main components: feature extraction, cross-modal object generation, clause generation, and clause evaluation. feature extraction generates representations for text tokens and image patches using encoders. cross-modal object generation constructs a crossmodal graph and applies a multi-layer graph convolutional neural network to generate multi-grained representations that constitute cross-modal objects as logic constants. clause generation produces dynamic embeddings for predicates (see ## conclusion we propose an interpretable multimodal misinformation detection model logicdm based on neuralsymbolic ai. we predefine five meta-predicates and relevant variables evolved from corresponding misinformation detection perspectives. and we propose to dynamically represent these predicates by fusion of multiple correlations to cover diversified online information. moreover, we differentiate reasoning process to smoothly select predicates and cross-modal objects to derive and evaluate explainable logic clauses automatically. extensive experiments on misinformation detection task demonstrate the effectiveness of our approach and external experiments on sarcasm detection task reveal the versatility.