M i RAG e N ews: Multimodal Realistic AI -Generated News Detection (2024)

Abstract: The proliferation of inflammatory or misleading “fake” news content has become increasingly common in recent years. Simultaneously, it has become easier than ever to use AI tools to generate photorealistic images depicting any scene imaginable. Combining these two—AI-generated fake news content—is particularly potent and dangerous. To combat the spread of AI-generated fake news, we propose the MiRAGeNews Dataset, a dataset of 12,500 high-quality real and AI-generated image-caption pairs from state-of-the-art generators. We find that our dataset poses a significant challenge to humans (60% F-1) and state-of-the-art multi-modal LLMs (< 24% F-1). Using our dataset we train a multi-modal detector (MiRAGe) that improves by +5.1% F-1 over state-of-the-art baselines on image-caption pairs from out-of-domain image generators and news publishers. We release our code and data to aid future work on detecting AI-generated content.

## introduction diffusion models @xcite have shown remarkable advancements in generating hyperrealistic images. particularly, models like midjourney can produce images that even graduate cs students cannot distinguish (sec 2.3). this capability has profound implications, especially with regard to the dissemination of disinformation. recently, there has been a noticeable surge in ai-generated news images on social media @xcite . when coupled with the proficiency of large language models (llms) in generating grammatically and contextually appropriate captions, the potential for ai-driven disinformation campaigns becomes an increasingly critical concern. recent work on detecting ai-generated images has shown impressive performance on images generated by models such as stable diffusion (rom-1 https://github.com/nosna/miragenews real news? fake news? news image + caption miragenews dataset ## detectors baselines. we compare our detectors against recent baselines in three different settings: imageonly, text-only, and multimodal detection. for image-only, we compare to de-fake @xcite , dire @xcite , and knn @xcite . for text-only, we compare with the text bottleneck model (tbm) @xcite , and for multimodal, we compare with hammer @xcite . we also test simple linear models in each modality and benchmark state-of-the-art mllms on the image-only and multimodal detection tasks. see appendix a for a more detailed discussion on each of the detectors tested. mirage-img trains a linear layer on top of the outputs from two models: (1) a linear model trained using eva-clip @xcite image embeddings and (2) an object-class concept bottleneck model (cbm) containing 300 object-class classifiers trained on crops of different objects from owl v2 @xcite . the interpretable object-class cbm focuses on regional anomalies, while the linear model captures the global features. mirage-txt trains a linear layer on top of the outputs from two models: (1) a linear model that is trained using clip text embeddings @xcite early fusion detector uses both image and text features together for classification while (b) late fusion detector uses outputs from previously trained unimodal detectors. 2021 ) and ( 2 ) a text bottleneck model @xcite that extracts 18 textual concepts in the caption. similar to mirage-img, we incorporate the interpretable concept-based approach to capture the auxiliary signals that the linear models missed. mirage fuses mirage-img and mirage-txt together for multimodal generated news detection. we apply two fusion techniques as illustrated in figure 4 : early fusion involves concatenating the outputs from mirage-img and mirage-txt before training a linear layer for classification, while late fusion computes a final prediction from the outputs of these two models with no extra training. we train all models until the evaluation loss plateaus and apply the classification threshold that gives the highest evaluation accuracy in testing. as for our design decisions, we conducted a detailed ablation study for each part of the mirage detector in appendix b. due to the results of these ablations, we chose the late fusion model as our final mirage detector. 4 results ## related work multimodal fake news datasets. there are many datasets for detecting generated images @xcite @xcite and generated text @xcite @xcite . however, there are relatively few publicly available datasets for detecting generated image-text pairs-especially in the news domain. the twitter mediaeval dataset @xcite ) contains a corpus of 17,000 tweets on two events with 514 real or fake images. weibo @xcite collects real and rumor news posts that are verified by the authoritative debunking system with mostly real images. fakenewsnet @xcite collects real and fake news from politifact and gossipcop and contains 17,214 human-written news articles with images and 1,986 fake news articles with images. the more recent dgm4 dataset @xcite offers 230k news samples with image and text, which contain 77k pristine pairs and 152k manipulated pairs from small regional manipulations on image and/or text. while previous datasets offer the foundation of multi-modal fake news, our dataset aims to address the more dangerous forms of fake news, namely the ones that have convincingly deceptive visuals fully generated by diffusion models. ## conclusion in this paper, we introduce miragenews, a dataset designed to facilitate the development and benchmarking of detection methods for aigenerated news. our dataset is the first of its kind to include high-quality images from modern generators along with misleading or harmful captions, and our results highlight the significant challenges faced by humans and current state-of-the-art multimodal language models in detecting such news content. we show that classifiers trained on our data achieve high accuracy on the most difficult-todetect images while still showing strong generalization performance on out-of-domain generators and news sources.