N ews CLIP pings: A utomatic G eneration of O ut-of- C ontext M ultimodal M edia (2021)

Abstract: Online misinformation is a prevalent societal issue, with adversaries relying on tools ranging from cheap fakes to sophisticated deep fakes. We are motivated by the threat scenario where an image is used out of context to support a certain narrative. While some prior datasets for detecting image-text inconsistency generate samples via text manipulation, we propose a dataset where both image and text are unmanipulated but mismatched. We introduce several strategies for automatically retrieving convincing images for a given caption, capturing cases with inconsistent entities or semantic context. Our large-scale automatically generated the NewsCLIPpings Dataset: (1) demonstrates that machine-driven image repurposing is now a realistic threat, and (2) provides samples that represent challenging instances of mismatch between text and image in news that are able to mislead humans. We benchmark several state-of-the-art multimodal models on our dataset and analyze their performance across different pretraining domains and visual backbones.

## introduction misinformation has reached new heights as sophisticated ai-based tools have come into the spotlight. for instance, it has become easy to generate images of people who "do not exist" foot_0 and create realistic deepfakes of existing people @xcite . recent language models have become better at fooling people into believing that generated texts are from real people @xcite . however, simple and cheap image repurposing remains one of the most widespread and effective forms of misinformation @xcite . specifically, real images of people and events get reappropriated and used out of context to illustrate false events and misleading narratives by misrepresenting who is in the image, what is the context in which they appear, or where the event takes place. this method is effective since augmenting a story with an image has been shown to increase user engagement and make false stories seem true @xcite . here, we explore whether such a threat can be automated. we show that real world images can be automatically matched to captions to generate false but compelling news stories, a threat scenario that may lead to larger-scale image repurposing. while synthetic media conceptually could be detected by doing unimodal analysis (e.g. a detector for gan-generated images), in our case both the text and the image are real. thus, determining whether an image-caption pair is pristine or falsified requires joint multimodal analysis of the image and text (consider figure 1 and make your guess). prior work has proposed several datasets related to our problem statement. one line of work obtains out-of-context image-text pairs by manipulating the named entities within the text @xcite . we find that in practice this may lead to linguistic inconsistencies, providing sufficient signal for a text-only model to distinguish between pristine and falsified descriptions without looking at the images. one recent work on detecting out-of-context images @xcite focuses on a scenario where an image is accompanied by two captions (from two distinct news sources), and one has to establish whether the two captions are consistent. here, we do not manipulate textual descriptions as we aim to minimize unimodal bias in our task. we do not assume that two captions are available per image, rather we focus on classifying each image-caption pair as pristine or falsified. specifically, we propose a large-scale automatically constructed dataset with real and out-ofcontext news based on the visualnews @xcite corpus. we consider several threat scenarios, designing matches based on: (a) caption-image similarity, (b) caption-caption similarity, where we retrieve an image with similar semantics to a given caption while the named entities between the source and the target are disjoint, (c) person match, where we retrieve an image that depicts a person mentioned in the source caption but pictured in a different context, and (d) scene match, where we retrieve an image that has the same scene type as the source image but depicts a different event foot_1 . we use the recent powerful multimodal model clip @xcite and other image and text models to construct the newsclippings dataset. to make our dataset more challenging, we introduce an adversarial filtering technique based on clip. we benchmark several state-of-the-art multimodal models and analyze their performance on the newsclippings dataset. we investigate the impact of the pretraining domains and various visual backbones. we conduct a human evaluation that shows humans find it challenging to distinguish between pristine and falsified samples from our dataset. we also perform a qualitative analysis with the help of visual salience to shed light onto the useful cues discovered by the models trained on our dataset. our dataset is publicly available here: https://github.com/g-luo/news_clippings foot_2 . ## related work we review several most relevant datasets in detail. some earlier proposed datasets for detecting multimodal misinformation are multimodal information manipulation dataset (maim) @xcite and multimodal entity image repurposing (meir) @xcite . maim naively matches images to captions from other random images to create their falsified versions. meir introduces swaps over named entities for people, organizations and locations. one of their assumptions is that for each image-caption "package" there is an unmanipulated related package (geographically near and semantically similar) in the reference set. this allows verifying the integrity of the query package by first retrieving a related package and then comparing the two. this problem statement is different from ours, as we do not assume availability of a perfect reference set. a more recent work has proposed tamperednews @xcite , a dataset where named entities specific to people, locations and events are swapped to other random foot_3 named entities within the article body. we show that such text manipulations lead to significant linguistic biases and the corresponding tasks can be solved without looking at the images (see section 4 for more details). another recent work @xcite aims to detect when images are used out of context, somewhat similar to meir above. they collect a dataset where each image appears in two distinct news sources and thus is associated with two captions. most of the collected data is not labeled, but a small subset has been manually annotated as in-or out-ofcontext. their problem statement (analyzing image and two captions) is again different from ours. one other work @xcite tackles neural news generation by replacing real articles with grover @xcite ) generated text and real captions with synthetic ones. they do not mismatch the images, which remain relevant to the article's content. the impact of image analysis on this task is rather limited, while analyzing the captions and the article body is key to the best detection performance. finally, some work focuses on human-made fake news detection, such as fakenewsnet @xcite and fakeddit @xcite , etc. while these datasets contain important real world examples of fake news, our focus is on exploring an automated threat scenario, where an image is automatically retrieved to match a given caption. ## the newsclippings dataset the objective of this work is to explore techniques for creating challenging, non-random imagecaption matches that require fine-grained semantic and entity knowledge. as seen in figure 2 foot_4 , misinformation in the wild is often extremely subtle and much more difficult than the random matches provided in prior synthetic datasets. in fact, general models that were not specifically trained or finetuned on the news domain can "solve" random news matches. we found that clip was able to achieve 97.39% top-1 accuracy on a captionimage retrieval task with news images foot_5 . for comparison, a recent method trip (thomas and kovashka, 2020) reports a top-1 accuracy of 73.78% on a similar task. as a result, we construct several splits that model specific threat scenarios seen in the real world, and we use clip vit-b/32 off-theshelf to filter out the less challenging samples. in the following, we assume we have a pristine query pair (img 1 , cap 1 ) and retrieve another pair (img 2 , cap 2 ) to form a falsified pair (img 2 , cap 1 ). preprocessing our dataset is derived from visual-news @xcite , a large-scale corpus which contains image-caption pairs from four news agencies (the guardian, bbc, usa today, and the washington post). we use spacy ner @xcite to label named entities in captions and the radboud entity linker (rel) @xcite to link them to their wikipedia 2019 entries. we compute text embeddings using sbert-wk @xcite and clip @xcite . we compute image embeddings with faster r-cnn @xcite and clip. we use a resnet50 classifier trained on the places365 dataset @xcite to get scene embeddings from images. we ensure that all matched samples are at least 30 days apart and that (cap 1 , cap 2 ) have no overlapping named entities identified by spacy and rel to prevent true matches, with the exception of the person split, where we expect at least one "person" entity to match. query for semantics our first split models a threat scenario that queries for specific semantic content, with the intent to portray the subjects of the image as certain other named entities, see figure 2 (i, ii). we consider two ways of getting the matches. query for person this split models a threat scenario that queries for a specific person, with the intent to portray them in a false context, as in figure 2 (iii). we ensure that the person of interest is pictured: all considered samples must have "per-son" entities in their captions and a person related faster-rcnn bounding box detected in the image. to avoid cases where the query person is mentioned but unlikely to be pictured, we filter captions where the person is in the possessive form, the object of the sentence, or modify a noun as determined by spacy's dependency parser. we ensure that the context is distinct: the places365 resnet similarity must be less than 0.9. finally, we found that there were a number of unsolvable falsified samples where the caption could be plausibly matched with any image of the person of interest. we minimize the number of such "generic" captions: we finetune a bert @xcite model on a small labelled subset of our training data to filter these captions from our matching process. (c) sbert-wk text-text: we match samples that mention the query person based on the lowest semantic similarity measured by their sbert-wk score, a text-only sentence embedding. see example (c) in figure 1 . query for scene this split models a threat scenario that queries for a specific scene, with the intent to mislabel the event, see figure 2 (iv). all samples must have no "person" named entities in the captions. this aims to filter headshots and other images with little scene information. (d) resnet place: we match samples with the highest places365 image similarity, as determined by the dot product of their resnet embeddings. see example (d) in figure 1 . merged split this split mixes samples from all the splits to model a more realistic case where a variety of methods are used to generate out-of-context images, i.e. all types of mismatch may be encountered at test time. we merge the splits such that there is an equal number of samples from every split, and the captions and images across splits are disjoint. ## experiments we start by describing our experimental setup and then present the results of our benchmarking study. ## additional analysis in this section, we gain further insights into the quality of our dataset via human evaluation and saliency map analysis. with the human evaluation, we assess whether our dataset could fool humans and pose a realistic threat. we also assess whether our dataset may have "unsolvable" true matches that in fact do not misrepresent anything. with our qualitative saliency map analysis, we investigate if the automatic models are learning to leverage high level semantic or entity cues after training on our dataset. human performance here, we estimate the difficulty of the proposed task for humans, aiming to assess how convincing our automatically matched images and captions are. we randomly select a set of 200 samples from the merged/balanced split, with an equal number of samples from all types (50 from each split, where 25 are pristine and 25 are falsified). we conduct our evaluation on amazon mechanical turk 9 . for each image-caption pair we ask 5 workers the following three questions: (a) "could this image belong to the given caption?" (yes/no), (b) "how confident are you in your answer?" (1: very, 2: somewhat, 3: not at all), (c) "would it help to use a search engine to be more confident?" (yes/no). note, that we specifically instruct the workers not to use search engines, to prevent them from discovering the original news articles on the web. the key takeaways from the evaluation are as follows. qualitative analysis finally, we analyze clip vit saliency maps and prediction using the method presented in @xcite . we select examples from the 200 samples used in our human evaluation. as seen in figure 6 , finetuning on our dataset often forces clip to focus on salient objects mentioned in the caption beyond the person of interest, for example expanding its attention from david cameron to "entourage." clip also has a number of capabilities off-the-shelf that require minimal finetuning, for example sign reading and logo recognition in the case of distinguishing "tigerair australia" from "germanwings." we also present a failure case where clip focuses on the two people in the foreground when the caption talks about "marches." evidently the model does find one point of support -the photo looks like it could have been taken in 1965 -but it fails to identify the absence of a crowd which would have been a "red flag" for a human. similar failure cases, such as a falsified caption that mentions mitt romney and rand paul at a rally but only pictures romney, highlight how our dataset can be particularly challenging because pristine news is an ambiguous domain that often mentions entities but does not picture them. ## conclusion we introduced newsclippings, a large-scale automatically constructed dataset for classifying news image-caption pairs as real or out-of-context. we found clip to be effective for the dataset construction and recognition of mismatches. by design, unimodal models cannot solve our task, while multimodal ones require named entity and semantic knowledge to do well on our diagnostic splits. our merged set aims to model the more realistic diversity of image-caption mismatches in the wild. from our experimental results, we find that the resnet backbone offers a modest performance boost compared to a vit-b/32 model. we find that a clip vit model is able to match human performance on a small subset of our merged / balanced split, and that our task is generally difficult with an average 66% human accuracy. our training data could be used to augment and increase the training data size of human-made falsified news, which often lack ground truth labels or sufficient scale. overall, we show that it is possible to automatically match plausible images for given input captions, and we present a challenging benchmark to foster the development of defenses against large-scale image repurposing.