Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News (2020)

Abstract: Large-scale dissemination of disinformation online intended to mislead or deceive the general population is a major societal problem. Rapid progression in image, video, and natural language generative models has only exacerbated this situation and intensified our need for an effective defense mechanism. While existing approaches have been proposed to defend against neural fake news, they are generally constrained to the very limited setting where articles only have text and metadata such as the title and authors. In this paper, we introduce the more realistic and challenging task of defending against machine-generated news that also includes images and captions. To identify the possible weaknesses that adversaries can exploit, we create a NeuralNews dataset which is comprised of 4 different types of generated articles as well as conduct a series of human user study experiments based on this dataset. Coupled with providing a relatively effective approach based on detecting visual-semantic inconsistencies, the valuable insights gleaned from our user study experiments and, consequently, this paper will serve as an effective first line of defense and a valuable reference for future work in defending against machine-generated disinformation.

## introduction the rapid progression of generative models in both computer vision @xcite @xcite and natural language processing @xcite @xcite has led to the increasing likelihood of realistic-looking news articles generated by artificial intelligence (ai). the malicious use of such technology could present a major societal problem. @xcite report that humans are easily deceived by its ai-generated propaganda. by manipulating such technology, parliament was scheduled to reconvene on oct 9, but mr. johnson said he planned to extend its break. in september, voters overwhelming rejected a plan from prime minister theresa may's team for the united kingdom to stay in the european union. on march 29, britain will officially exit the union after years of campaigning and serious negotiations. the eu's chief brexit negotiator, michel barnier, has warned that there could be no future trade deals with the united kingdom if there is a "no deal." the transition period will allow the united kingdom and the european union to work out a new plan for their relationship. but we may not know ... article photo caption model human or machine-generated? figure 1 : we propose an approach for detecting news articles generated by machines. prior work uses only the article content and metadata including title, date, domain, and authors. however, news articles often contain photos and captions as well. we propose to leverage possible visual-semantic inconsistency between the article text, images, and captions, such as missing or inconsistent named entities (underlined in red). in this paper, we present the first line of defence against neural fake news with images and captions. to the best of our knowledge, we are the first to address this challenging and realistic problem. premised on the assumption that the adversarial text generator is unknown beforehand, we propose to evaluate articles based on the semantic consistency between the linguistic and visual components. while state-of-the-art approaches in bidirectional image-sentence retrieval @xcite have leveraged visual-semantic consistency to great success on standard datasets such as mscoco @xcite and @xcite , we show in appendix d they are not able to reason effectively about objects in an image and named entities present in the caption or article body. this is due to discrepancies in the distribution of these datasets, as captions in the standard datasets usually contain general terms including woman or dog as opposed to named entities such as mrs betram and a golden retriever, which are commonly contained in news article captions. moreover, images are often not directly related to the articles they are associated with. for example, in figure 1 , the article contains mentions of the british prime minister. yet, it only contains an image of the united kingdom flag. to circumvent this problem, we present didan, a simple yet surprisingly effective approach which exploits possible semantic inconsistencies between the text and image/captions to detect machinegenerated articles. for example, notice that the article and caption in fig. 1 actually mention different prime ministers. besides evaluating the semantic relevance of images and captions to the article, didan also exploits the co-occurrences of named entities in the article and captions to determine the authenticity score. the authenticity score can be thought of as the probability that an article is human-generated. we adopt a learning paradigm commonly used in image-sentence retrieval where models are trained to reason about dissimilarities between images and non-matching captions. in this instance, negative samples constitute articles and non-corresponding image-caption pairs. not only is this a reasonable approach when the adversarial generative model is unknown, we show empirically that it is crucial to detecting machinegenerated articles with high confidence even with access to machine-generated samples during training. more importantly, this means that didan is easily trained on the abundance of online news articles without additional costly annotations. to study this threat, we construct the neural-news dataset which contains both human and machine-generated articles. these articles contain a title, the main body as well as images and captions. the human-generated articles are sourced from the goodnews @xcite dataset. using the same titles and main article bodies as context, we use grover @xcite to generate articles. instead of using gan-generated images which are easy to detect even without exposure to them during training time @xcite , we consider the much harder setting where the articles are completed with the original im-ages. we include both real and generated captions which are generated with the sota entity-aware image captioning model @xcite . we present results and findings from a series of empirical as well as user study experiments. in the user study experiments, we use 4 types of articles including real and generated news to determine what humans are most susceptible to. the insights derived from these findings help identify the possible weaknesses that adversaries can exploit to produce neural fake news and serve as a valuable reference for defending against this threat. last but not least, our experimental results provide a competitive baseline for future research in this area. in summary, our contributions are multi-fold: ## neuralnews dataset collection to facilitate our endeavor of studying this threat, we introduce the neuralnews dataset which consists of human and machine-generated articles with images and captions. it provides a valuable testbed for ai-enabled disinformation that adversaries can exploit presently and yet, is the hardest to detect. the human-generated articles are sourced from the goodnews (biten et al., 2019) dataset, which consists of new york times news articles spanning from 2010 to 2018. each news article contains a title, the main article body as well as image-caption pairs. note that we source original images from real articles since machine-generated images are relatively easy to detect @xcite . in our dataset, we restrict the number of image-caption pairs to be at most 3 per article. the entire dataset used in the empirical and user study experiments in total, we collect about 32k samples of each article type (resulting in about 128k total). to collect machine-generated news articles, we use grover @xcite to generate fake articles using original titles and articles from the goodnews dataset as context. type c articles are completed by incorporating the original image-caption pairs. in type b and d articles, we use the entity-aware image captioning model @xcite to generate fake image captions based on the articles. we believe that this dataset presents a realistic and challenging setting for defending against neural news. dataset statistics. ## user study experiments we endeavor to determine the susceptibility of humans to different types of neural fake news through several user studies. to this end, we conduct a series of user study experiments based on the neu-ralnews dataset. the user study results provide vital information to help us identify salient points which adversaries can take advantage of. qualified amazon mechanical turk workers with required english proficiency are used in all of our experiments. we briefly describe the experimental setups below. see appendix a for a template of our prompts and response options for each setting. article article only naive participants trained participants type article caption accuracy accuracy accuracy no imgs generated -68.8% -no imgs real -49.2% --a real real -64.0% 70.7% b real generated -34.0% 78.7% c generated real -42.7% 56.7% d generated generated -44.0% 55.3% average --59.0% 46.2% 67.8% article only user predictions: given articles with titles which do not contain images and captions, can humans detect if they are generated or not?we ask participants to predict if an article is fake when they only contain their titles and main article bodies. in this variant, participants are provided with hints to pay attention to possible inconsistencies between the text and title. this is done with the purpose of understanding the importance of visual-semantic cues provided by image-caption pairs in this task in the following experiments. ## didan: detecting cross-modal in our task, the goal is to detect machine-generated articles that also include images and captions. the example in figure 1 points towards an inherent challenge: identifying indirect relationships between the image and the text. due to the common need to measure visual-semantic similarity, an intuitive first step would be to base one's approach on image-sentence similarity reasoning models which are commonly used in sota bidirectional imagesentence retrieval. we hypothesize, from their dismal performance ( an illustrative overview of didan is shown in figure 3 . an article a consists of a set of sentences s where the article is also comprised of a set of image-caption pairs where each image i is represented by a set of regional object features {o 1 , •••, o i } and each caption c contains a sequence of words @xmath0, • • •, w i }. spacy's named entity recognition model @xcite is used to detect named entities in both articles and image captions. d t , d i and d vse are used to denote the initial dimensions of the text and image representations as well as the hidden dimension respectively. each sentence is tokenized and encoded with a bert model @xcite ) that is pretrained on bookscorpus @xcite and english wikipedia. ## experiments given a news article from our neuralnews dataset, our goal is to automatically predict whether it is human or machine-generated. we compare di-dan to several baselines, evaluating performance based on how often an article was correctly labeled. note that in our experiments, only type a and c articles are used. this is due to the fact that generated captions often contain repeated instances of named entities without any stop words, which is not challenging for humans to detect (see ## summary of exploitable weaknesses and defense directions while this is not entirely representative of all the future challenges presented by neural fake news, we believe that this comprehensive study will provide an effective initial defense mechanism against arti-