M 3 D : M ulti M odal M ulti D ocument Fine-Grained Inconsistency Detection (2024)

Abstract: Fact-checking claims is a highly laborious task that involves understanding how each factual assertion within the claim relates to a set of trusted source materials. Existing approaches make sample-level predictions but fail to identify the specific aspects of the claim that are troublesome and the specific evidence relied upon. In this paper, we introduce a method and new benchmark for this challenging task. Our method predicts the fine-grained logical relationship of each aspect of the claim from a set of multimodal documents, which include text, image(s), video(s), and audio(s). We also introduce a new benchmark (M3DC) of claims requiring multimodal multidocument reasoning, which we construct using a novel claim synthesis technique. Experiments show that our approach outperforms other models on this challenging task on two benchmarks while providing finer-grained predictions, explanations, and evidence.

## introduction misinformation poses serious societal risks by perpetuating narratives that incite fear, sow discord, and affect public health and safety @xcite . despite significant efforts towards developing automated factchecking techniques @xcite @xcite , existing methods face several limitations. first, real-world claims may include assertions that require consulting multiple documents and modalities to verify or refute the claim. existing approaches either assume a single document setting @xcite or perform retrieval across documents to obtain relevant evidence, which is then treated as a single document @xcite , potentially losing important surrounding context. secondly, some methods only predict when claims conflict with relevant knowledge but ignore ambiguous cases where no supporting or refuting information is available @xcite . lastly, most of the existing methods fail to provide the fine-grained analysis needed for users to understand what is inconsistent in a claim or to make revisions to be more factual @xcite @xcite . simply flagging an entire claim as false without pinpointing the specific inaccurate parts provides limited utility. in contrast, we propose an approach for predicting the logical relationship of each piece of a claim with respect to a set of multimodal sources. we perform a semantic dissection of claims into semantic pieces and leverage a hierarchical transformer that operates across multimedia documents to make fine-grained predictions. as illustrated in figure 1 our model ingests the claim along with associated multimedia, preserving the context. it then fuses the cross-document representations into a graph initialized with the claim's abstract meaning representation (amr) @xcite . entailment relations are then predicted for each node (e.g., entities, actions) and tuple (e.g., relations) within the graph. because no prior work has explored making fine-grained claim predictions from a set of multimodal documents, we also introduce a new dataset of claims that contains fine-grained labels for this task called m 3 dc (multimodal multi-document claims). we build our dataset on top of the newsstories @xcite dataset, which includes sets of news articles, images, and videos across multiple topics. we retrieve textual, visual, and audio data from each set to build a robust multimodal multidocument knowledge graph for each set of related documents. next, we develop a claim synthesis method in order to generate claims that require multisource knowledge to verify, which uses a fine-grained claim manipulator model to generate claims manipulated at the sub-claim level. our major contributions are as follows: ## approach we develop a model to predict sample-level and fine-grained entailment labels for a claim and its multimedia evidence (premise). the sample-level label (entailment, neutral, or contradiction) indicates the overall claim's relationship with the premise. fine-grained labels detail entailment relationships for specific claim parts, such as entities and events, based on the claim's amr tree. we first describe our methodology for constructing m 3 dc. we then detail our model architecture, which makes fine-grained claim predictions using multimodal multidocument sets of evidence. ## conclusion we address the challenge of predicting the logical consistency of claims with multimodal sources. our method analyzes claims within a multimodal multidocument context, including text, visual content, and audio. our method is able to reason in a fine-grained manner over complex information across media and modalities. we further introduce a dataset, m 3 dc, created through a unique synthesis technique that produces claims requiring crossdocument, cross-media reasoning for verification. our contributions aim to mitigate the impact of misinformation and enhance the reliability of automated fact-checking systems, thus supporting informed decision-making and fostering a factually accurate public dialogue.