MM -Claims: A Dataset for Multimodal Claim Detection in Social Media (2022)

Abstract: In recent years, the problem of misinformation on the web has become widespread across languages, countries, and various social media platforms. Although there has been much work on automated fake news detection, the role of images and their variety are not well explored. In this paper, we investigate the roles of image and text at an earlier stage of the fake news detection pipeline, called claim detection. For this purpose, we introduce a novel dataset, MM-Claims, which consists of tweets and corresponding images over three topics: COVID-19, Climate Change and broadly Technology. The dataset contains roughly 86000 tweets, out of which 3400 are labeled manually by multiple annotators for the training and evaluation of multimodal models. We describe the dataset in detail, evaluate strong unimodal and multimodal baselines, and analyze the potential and drawbacks of current models.

## introduction the importance of combating misinformation was once again illustrated by the coronavirus pandemic, which came along with a lot of "potentially lethal" misinformation. at the beginning of the covid-19 pandemic, the united nations (un) (dgc, 2020) started even using the term "infodemic" for this phenomenon of misinformation and called for proper dissemination of reliable facts. however, tackling misinformation online and specifically on social media platforms is challenging due to the variety of information, volume, and speed of streaming data. as a consequence, several studies have explored different aspects of covid-19 misinformation online including sharing patterns @xcite , platform-dependent engagement patterns @xcite , web search behaviors @xcite , and fake images @xcite . we are primarily interested in claims on social media from a multimodal perspective (figure 1 ). claim detection can be seen as an initial step in fighting misinformation and as a precursor to prioritize potentially false information for fact-checking. traditionally, claim detection is studied from a linguistic standpoint where both syntax @xcite and semantics @xcite of the language matter to detect a claim accurately. however, claims or fake news on social media are not bound to just one modality and become a complex problem with additional modalities like images and videos. while it is clear that a claim in the text is denoted in verbal form, it can also be part of the visual content or as overlaid text in the image. even though much effort has been spent on the curation of datasets @xcite @xcite and the development of computational models for multimodal fake news detection on social media @xcite @xcite , hardly any research has focused on multimodal claims @xcite . in this paper, we extend the definitions of claims and check-worthiness from previous work @xcite to multimodal claim detection and introduce a novel dataset called multimodal claims (mm-claims) curated from twitter to tackle this critical problem. while previous work has focused on factually-verifiable check-worthy @xcite or general claims (i.e., not necessarily factually-verifiable, e.g., @xcite ) on a single topic, we focus on three different topics, namely covid-19, climate change and technology. as shown in figure 1 , mm-claims aims to differentiate between tweets without claims (figure 1a ) as well as tweets with claims of different types: claim but not check-worthy (figure 1b ), check-worthy claim (figure 1c ), and check-worthy visually relevant claim (figure 1d ). our contributions can be summarized as follows: the remainder of the paper is structured as follows. section 2 describes the related work on unimodal and multimodal approaches for claim detection. the proposed dataset and the annotation guidelines are presented in section 3. we discuss the experimental results of the compared models in section 4, while section 5 concludes the paper and outlines areas of future work. ## mm-claims tasks and dataset this section describes the problem of multimodal claim detection (section 3.1), the data collection (section 3.2), the guidelines for annotating multimodal claims (section 3.3), and the annotation process (section 3.4) to obtain the new dataset. ## experimental setup and evaluation in this section, we describe the features, baseline models, and the comprehensive experiments using our novel dataset. we test a variety of features and recent multimodal state-of-the-art models. ## conclusions in this paper, we have presented a novel mm-claims dataset to foster research on multimodal claim analysis. the dataset has been curated from twitter data and contains more than 3000 manually annotated tweets for three tasks related to claim detection across three topics, covid-19, climate change, and technology. we have evaluated several baseline approaches and compared them against two state-of-the-art fake news detection approaches. our experimental results suggest that the fine-tuning of pre-trained multimodal and unimodal architectures such as albef and bert yield the best performance. we also observed that the overlaid text in images is important in information dissemination, particularly for claim detection. to this end, we evaluated a couple of strategies to incorporate ocr text into our models, which yielded a much better trade-off between identifying visuallyrelevant and visually-irrelevant (text-only) claims. in the future, we will explore other and novel architectures for multimodal representation learning and other information extraction techniques to incorporate individual modalities better. we also plan to investigate fine-grained overlaps of concepts and meaning in image and text, and expand the dataset to covid-19 related sub-topics and specific climate change events.