MMM : An Emotion and Novelty-aware Approach for Multilingual Multimodal Misinformation Detection (2022)

Abstract: The growth of multilingual web content in low-resource languages is becoming an emerging challenge to detect misinformation. One particular hindrance to research on this problem is the non-availability of resources and tools. Majority of the earlier works in misinformation detection are based on English content which confines the applicability of the research to a specific language only. Increasing presence of multimedia content on the web has promoted misinformation in which real multimedia content (images, videos) are used in different but related contexts with manipulated texts to mislead the readers. Detecting this category of misleading information is almost impossible without any prior knowledge. Studies say that emotion-invoking and highly novel content accelerates the dissemination of false information. To counter this problem, here in this paper, we first introduce a novel multilingual multimodal misinformation dataset that includes background knowledge (from authentic sources) of the misleading articles. Second, we propose an effective neural model leveraging novelty detection and emotion recognition to detect fabricated information. We perform extensive experiments to justify that our proposed model outperforms the state-of-the-art (SOTA) on the concerned task.

## introduction fast adoption of social media platforms have promoted people to knowingly or unknowingly subscribe, create and share misleading, fake, and irrelevant information which consists of various attributes like title, text information, visual information, etc. these attributes may contain false or misleading information. the news or stories having false information is called misinformation. in recent years, we observe substantial advancements in automatic fake news detection. however, most of these are targeted to resource-rich language like english. when it comes to the scenario of (relatively) low-resource indian languages like hindi, bengali and tamil, the amount of research is insignificant, primarily due to the unavailability of data and other associated resources. with the advancement of multimedia news on the internet, news containing same (non-novel) image with different (novel) text influences the fake news on social media to mislead the newsreaders. since the image looks authentic and aligns with the new text, it becomes very challenging to detect this category of fake news. the implication of misinformation detection with novelty detection and emotion recognition was first presented by mit scholars foot_0 . novelty refers to the extent to which news readers encounter unfamiliar news, which may include some element of surprise. in this work, we take forward the misinformation work on the shoulder of novelty detection via entailment task with emphasis to textual similarity measures. literature also suggest that novel and emotion invoking contents in the news articles act as fuel for the rapid dissemination @xcite , @xcite and @xcite . although people have performed an extensive investigation in different dimensions of misinformation detection, however, a very few mechanisms have focused on novelty and emotion aware misinformation detection with background knowledge for the relatively low-resource languages. we make an attempt to address these challenges by creating important resources and effective baseline. we first introduce a novel multilingual multimodal misinformation dataset for the indian languages like hindi, bengali and tamil. the instances (here, in-stance means a single claim may be real or fake) in each language are different, meaning the same instance will not be present in more than one language. during training the model, we mix the instances of all the three languages which makes it multilingual. we further design a deep learningbased misinformation detection model using novelty detection and emotion recognition as the assisting tasks. the major contributions offered in this article are as follows: ## related work the concept of misinformation or fake news detection has started in early 2010, as social media started to have an immense impact on people's views. @xcite has introduced one of the first extensive studies for misinformation detection on social media. it has described the fact-checking methodologies as verification of the hypothesis made in a news article to judge if the claim is true or not. the work presented in fakedetector @xcite introduces a deep diffusive network model to detect fake news by learning the representations of news articles, creators, and subjects simultaneously. @xcite introduced a sentence-comment co-attention sub-network which learns and captures check-worthy sentences and user comments jointly to explain why a particular news piece is detected as fake. people have organized many chal-lenges for fake news detection that introduce several novel mechanisms. a competition the fake news challenge (fnc) foot_1 introduced a few works (slovikovskaya and attardi, 2020), @xcite for stance detection which are useful to understand attitudes expressed in texts. stance detection means the detection of relative perspective of two text fragments. the stance detection justifies whether the news article agrees, disagrees, discusses or is unrelated to the news title. if the news article disagrees or is unrelated to the news title, it indicates a high probability of the news to be fake. few notable works, such as @xcite and @xcite verify the human generated claims as fake or real. the mechanism presented by @xcite depicts a word attention-based deep learning model for automatic fake news detection. the work explored in @xcite combines the textual, visual, and social context features using an attention mechanism for fake news prediction. in continuation to it, eann @xcite , vae @xcite and spotfake @xcite have introduced deep learning-based models and justified that the model is efficient in handling newly emerged events better than the existing methods. the research explored in @xcite @xcite have given attention to feature fusion along with the feature extraction mechanisms and proved that the model's performance also depends upon the semantic interaction between different modalities. the method explored in @xcite has taken the first step to find the credibility of previously published news articles on the same events as the background knowledge by introducing the supervised contrastive learning (scl) @xcite . one of the promising works (abonizio et al., 2020) in multilingual misinformation detection explores language-independent fake news detection, which successfully differentiate fake, satirical, and legitimate news across three different languages. another multilingual work presented in @xcite uses the convolutional neural network (cnn) to detect fake news with satire on a multilingual dataset. the works presented in @xcite are the major contributors for multilingual multimodal misinformation detection. they first introduced a dataset which includes the instances in languages other than english. our work is different from the prior works in the perception that (i). we create a multilingual multimodal misinformation (mmm) dataset with background knowledge for relatively low-resource indian languages which includes the data instances in hindi, @xcite . we design a novelty and emotion aware multimodal multilingual framework for misinformation detection. ## data description and analysis several resources like twitter @xcite , weibo @xcite , ti-cnn @xcite , fauxtography @xcite , fakeddit @xcite , newsbag @xcite , etc. are very eminent to study multimodal misinformation detection problems. people have introduced coaid (cui and lee, 2020), mmcovar @xcite and recovery @xcite to tackle the misinformation during covid-19 infodemic. aforesaid datasets are only available in english language. very few datasets such as arcov-19 @xcite and checked @xcite are available in the languages other than english. mm-covid @xcite , @xcite and factdril @xcite are the multilingual multimodal misinformation datasets. however, these datasets do not include background information (where and in which context the news has been published first) of the news articles, which are crucial for misleading misinformation detection. therefore, we prepare a novel multilingual multimodal misinformation (mmm) dataset which includes 10,473 samples. the developed dataset contains the instances from three different indian languages viz. hindi, bengali and tamil. each instance of the dataset is in the form of sourcetarget pair. target is the combination of multimodal hindi, bengali and tamil language instances which claim any information or news. the source is the related background information extracted from different websites corresponding to the target. ## proposed model in this section, we present a brief description of the proposed framework. the overall model is shown in figure 1 which consists of three components: novelty detection, image emotion prediction, and misinformation detection. below, we discuss all these three components in details. ## experiments and results this section presents experimental setup, baseline, results, case studies and error analysis. ## conclusion in this paper we solve the problem of multilingual multimodal misinformation detection in three indian languages, hindi, bengali and tamil. nowa-days, same image is used in different textual context to mislead the reader. to address this problem, first, we have created our multilingual multimodal misinformation dataset and then we have performed experiments on multilinguality, background knowledge, emotion, multimodality and novelty to see the effect. we have built a novel framework based on novelty and emotion which outperform all the baseline and state-of-theart models. further, we want to extend our current work in following direction to prevent the spread of misinformation: (i). by including additional low-resources language; (ii). by addressing crosslingual and code-mixed based resources and mechanisms; and (iii). by incorporating explainability in the model.