How does fake news use a thumbnail? CLIP -based Multimodal Detection on the Unrepresentative News Image (2022)

Abstract: This study investigates how fake news use the thumbnail image for a news article. We aim at capturing the degree of semantic incongruity between news text and image by using the pretrained CLIP representation. Motivated by the stylistic distinctiveness in fake news text, we examine whether fake news tends to use an irrelevant image to the news content. Results show that fake news tends to have a high degree of semantic incongruity than general news. We further attempt to detect such image-text incongruity by training classification models on a newly generated dataset. A manual evaluation suggests our method can find news articles of which the thumbnail image is semantically irrelevant to news text with an accuracy of 0.8. We also release a new dataset of image and news text pairs with the incongruity label, facilitating future studies on the direction.

## introduction we have been suffering from the infodemic as well as the coronavirus pandemic @xcite . the proliferation of fake news during the pandemic has been a significant threat to the world by inducing hate crimes against east asians, reinforcing the wrong beliefs of anti-vaxxers, etc. fake news is defined as "fabricated information that mimics news media content in form but not in organizational process or intent" @xcite . motivated by the fact that unreliable sources generate most false articles, a line of research has attempted to understand the distinct characteristics of fake news sources. a notable study is @xcite , which focused on textual patterns of news articles and identified that overall title structure and the use of proper nouns in titles are significant markers that differentiate fake news from general news. similarly, from consumption and spreading patterns on social media, @xcite found that fake news spreads faster, deeper, and broader than general news. other researchers showed that the reliability of news media could be predicted by various media-level features, including web traffic toward a news website @xcite . in this study, we investigate the use of images in fake news articles; in particular, we focus on a thumbnail, an image displayed as a preview to a news article. when a news article is shared on social media, its title and thumbnail image are the only visible information before a user clicks the link. since many readers skim news without carefully checking the content @xcite , the visuals can mislead users into having a wrong impression if the thumbnail does not represent the news content. fake news sources are less likely to follow the journalistic standard but tend to employ undesirable techniques such as clickbait headlines @xcite . therefore, we hypothesize that unreliable sources may use a less relevant image for the thumbnail to the news text to attract clicks and promote false beliefs. to examine the hypothesis, we propose using clip @xcite , a deep multimodal representation that allows representing image and text in the same embedding space. across three datasets, we measure image-text similarity over the clip embedding and confirm that the fake news media tend to use the semantically less relevant photograph in news content than trustworthy sources. going further, we test clip's ability to detect the incongruity between news image and text. ## limitation and future direction this study bears several limitations. first, the findings were observed from the dataset of nine news media. even though they are well-balanced against political bias and trustworthiness, the findings could not represent general patterns and thus should be carefully interpreted. future studies could examine the hypothesis using more extensive data. second, since this study employs clip as a backbone, our results are subject to unknown biases which clip might learn from training. future studies could adopt pretraining tasks to mitigate the issues. third, we focused on news titles as a proxy of news content. the method could be invalid for some cases where the news title is incongruent with the main text @xcite . future studies could develop a method that exploits body text as a reference, which contains more fruitful information yet is more challenging to be analyzed. ## conclusion this paper examined the usage of news thumbnails and asked whether fake news sources exhibit distinct patterns. by applying clip to the pair of news title and image, we identified the difference between fake news and trustworthy media sources in the image-text similarity: fake news tends to use a less similar thumbnail picture to the news text than general news. next, we tackled the article-level detection problem that targets fake news articles in which the thumbnail picture does not represent the news content. to the end, we generated a paired dataset of 24,669 image-text pairs, each image of which is semantically (in-)congruent to the text. evaluation experiments showed that clip-based models could detect news articles with an unrepresentative thumbnail with high accuracy. these observations highlight the potential of clip for identifying these misinformed articles in the real world. to the best of our knowledge, this is one of the initial attempts to understand fake news characteristics in the use of thumbnail and focus on its semantic representativeness to news content. we hope our methodology and dataset can not only make an impact on the ongoing efforts to curtail fake news dissemination, but also contribute to broader research communities on vision and language.