Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation (2023)

Abstract: Mis- and disinformation online have become a major societal problem as major sources of online harms of different kinds. One common form of mis- and disinformation is out-of-context (OOC) information, where different pieces of information are falsely associated, e.g., a real image combined with a false textual caption or a misleading textual description. Although some past studies have attempted to defend against OOC mis- and disinformation through external evidence, they tend to disregard the role of different pieces of evidence with different stances. Motivated by the intuition that the stance of evidence represents a bias towards different detection results, we propose a stance extraction network (SEN) that can extract the stances of different pieces of multi-modal evidence in a unified framework. Moreover, we introduce a support-refutation score calculated based on the co-occurrence relations of named entities into the textual SEN. Extensive experiments on a public large-scale dataset demonstrated that our proposed method outperformed the state-of-the-art baselines, with the best model achieving a performance gain of 3.2% in accuracy.

## introduction the proliferation of mis-and disinformation 1 and its erosion on democracy, justice, and public trust have increased the need for detection and intervention @xcite . while such information consists of primarily text, the increasing popularity of non-textual data on online platforms such as images and short videos has led to more and more mis-and disinformation in different modalities beyond text @xcite . in web pages such as news articles, texts and images often co-exist but they may not be correctly associated, therefore leading to the so-called out-of-context (ooc) mis-and disinformation where readers may be misled to believe some false narratives caused by such false or inaccurate text-image associations @xcite . ooc mis-and disinformation have become a common phenomenon on many online platforms, and have received attention from many researchers recently @xcite @xcite . a major area of research on ooc mis-/disinformation is about development of automated detection methods and construction of datasets for evaluating and comparing performances of different methods. operations such as random matching @xcite , manipulating named entities @xcite and strategic matching @xcite have been used to automatically generate ooc mis-/disinformation samples. rather than relying on static datasets and detection models' intrinsic ability to detect ooc mis-/disinformation (often via cross-modal inconsistencies), @xcite and @xcite also leverage available evidence on the internet to help detect ooc mis-/disinformation. however, their methods do not adequately perform effective modeling of stance relations between evidence and the claim being checked. this can miss important information useful for detecting ooc mis-/disinformation, since a given piece of evidence supporting or refuting the claim can help detect mis-alignment between texts and images. an example can be found in figure 1 . in this paper, to fill the aforementioned research gap on the lack of consideration of stance analysis for ooc mis-/disinformation detection, we propose a unified framework that aims to comprehensively incorporate the stances of multiple pieces of evidence towards claims during the detection process. more specifically, for image-based claim and evidence, text-based claim (image captions in our work) and evidence, we utilize different but independent stance extraction networks (sens) with a similar structure, which allow for cluster-specific presentations of evidence semantics and can extract and fuse multiple stances. in the textual sen, we further emphasize the stance relationship through a support-refutation score calculated based on the co-occurrence relationship of named entities to replace the binary named entity indicator (nei) commonly used in the literature @xcite . our analyses based on a public large-scale dataset showed that the score follows a significantly different distribution for pristine information and ooc mis-/disinformation. further experiments demonstrated that our sens are more efficient at mining useful information in evidence that can be used to verify claims. the major contributions of this paper are summarized below. the rest of the paper is organized as follows. in the next section, we overview some related work. section 3 explains our methodology, and experimental setup and results are covered in section 4. section 5 concludes the paper, with the two sections after it discuss limitations and ethical considerations. ## experiments in this section, we explain the datasets we used, the experimental setup, and the experimental results regarding the performance of our proposed method. we also conduct qualitative analysis of the results to provide an intuitive illustration of our method. ## conclusion we propose a unified framework capable of incorporating stance relation of evidence for automatic out-of-context mis-/disinformation detection. the support-refutation score calculates a stance related score for each textual evidence based on the cooccurrence relationship of named entities, while stance extraction networks are able to uniformly consider the stance of all evidences. our proposed method outperforms the existing baselines and provides a new benchmark for evidence-assisted veracity assessment of image-caption pair task. furthermore, our method offers good interpretability, allowing for a clear understanding of the role of each evidence and how it contributes to the overall decision-making process.