Neural Deepfake Detection with Factual Structure of Text (2020)

Abstract: Deepfake detection, the task of automatically discriminating machine-generated text, is increasingly critical with recent advances in natural language generative models. Existing approaches to deepfake detection typically represent documents with coarse-grained representations. However, they struggle to capture factual structures of documents, which is a discriminative factor between machine-generated and human-written text according to our statistical analysis. To address this, we propose a graph-based model that utilizes the factual structure of a document for deepfake detection of text. Our approach represents the factual structure of a given document as an entity graph, which is further utilized to learn sentence representations with a graph neural network. Sentence representations are then composed to a document representation for making predictions, where consistent relations between neighboring sentences are sequentially modeled. Results of experiments on two public deepfake datasets show that our approach significantly improves strong base models built with RoBERTa. Model analysis further indicates that our model can distinguish the difference in the factual structure between machine-generated text and human-written text.

## introduction nowadays, unprecedented amounts of online misinformation (e.g., fake news and rumors) spread through the internet, which may misinform people's opinions of essential social events @xcite @xcite . recent advances in neural generative models, such as gpt-2 @xcite we can observe that the factual structure of entities extracted by named entity recognition is inconsistent. previous works on deepfake detection of text are dominated by neural document classification models @xcite @xcite . they typically tackle the problem with coarse-grained document-level evidence such as dense vectors learned by neural encoder and traditional features (e.g., tf-idf, word counts). however, these coarsegrained models struggle to capture the fine-grained factual structure of the text. we define the factual structure as a graph containing entities mentioned in the text and the semantically relevant relations among them. as shown in the motivating example in figure 1 , even though machine-generated text seems coherent, its factual structure is inconsistent. our statistical analysis further reveals the difference in the factual structure between humanwritten and machine-generated text (detailed in section 3). thus, modeling factual structures is essential for detecting machine-generated text. based on the aforementioned analysis, we propose fast, a graph-based reasoning approach uti-lizing factual structure of text for deepfake detection. with a given document, we represent its factual structure as a graph, where nodes are automatically extracted by named entity recognition. node representations are calculated not only with the internal factual structure of a document via a graph convolution network, but also with external knowledge from entity representations pre-trained on wikipedia. these node representations are fed to produce sentence representations which, together with the coherence of continuous sentences, are further used to compose a document representation for making the final prediction. we conduct experiments on a news-style dataset and a webtext-style dataset, with negative instances generated by grover @xcite and gpt-2 @xcite respectively. experiments show that our method significantly outperforms strong transformer-based baselines on both datasets. model analysis further indicates that our model can distinguish the difference in the factual structure between machine-generated text and human-written text. the contributions are summarized as follows: ## task definition we study the task of deepfake detection of text in this paper. this task discriminates machinegenerated text from human-written text, which can be viewed as a binary classification problem. we conduct our experiments on two datasets with different styles: a news-style dataset with fake text generated by grover @xcite and a large-scale webtext-style dataset with fake text generated by gpt-2 @xcite . the newsstyle dataset consists of 25,000 labeled documents, and the webtext-style dataset consists of 520,000 labeled documents. with a given document, systems are required to perform reasoning about the content of the document and assess whether it is "human-written" or "machine-generated". ## factual consistency verification in this part, we conduct a statistical analysis to reveal the difference in the factual structure between human-written and machine-generated text. specifically, we study the difference in factual structures from a consistency perspective and analyze entitylevel and sentence-level consistency. through data observation, we find that humanwritten text tends to repeatedly mention the same entity in continuous sentences, while machinewritten continuous sentences are more likely to mention irrelevant entities. therefore, we define entity consistency count (ecc) of a document as the number of entities that are repeatedly mentioned in the next w sentences, where w is the sentence window size. sentence consistency count (scc) of a document is defined as the number of sentences that mention the same entities with the next w sentences. for instance, if entities mentioned in three continuous sentences are "a and b; a; b" and @xmath0, then @xmath1. @xmath2. we use all 5,000 pairs of human-written and machine-generated documents from the newsstyle dataset and each pair of documents share the same metadata (e.g., title) for statistical analysis. we plot the kernel density distribution of these two types of consistency count with sentence window size @xmath3, 2}. as shown in figure 2 , human-written documents are more likely to have higher entity-level and sentence-level consistency count. this analysis indicates that human-written and machine-generated text are different in the factual structure, thus modeling consistency of factual structures is essential in discriminating them. ## methodology in this section, we present our graph-based reasoning approach, which considers factual structures of documents, which is used to guide the reasoning process for the final prediction. figure 3 gives a high-level overview of our approach. with a document given as the input, our system begins by calculating the contextual word representations by roberta ( § 4.1). then, we ## related work recently, fake news detection has attracted growing interest due to the unprecedented amount of fake contents propagating through the internet @xcite . spreading of fake news arises public concerns @xcite as it may influence essential public events like politic elections @xcite . online reviews can also be generated by machines, and advances in generative models have promoted the development of detection methods. previous studies in the field of deepfake detection of generated text are dominated by deep-learning based document classification models and studies about discriminating features of generated text. grover @xcite detects generated text by a fine-tuned model of the generative model itself. @xcite fine-tune the bert model for discrimination and explore how sampling strategies and text excerpt length affect the detection. gltr @xcite develops a statistical method of computing per-token likelihoods and visualizes histograms over them to help deepfake detection. @xcite and @xcite study language distributional features including n-gram frequencies, text coherence and syntax features. @xcite study the effectiveness of different numeric representations (e.g., tfidf and word2vec) and different neural networks (e.g., anns, lstms) for detection. @xcite tackle the problem as a ranking task and study about the cross-architecture and cross-corpus generalization of their scoring functions. @xcite indicate that simple provenance-based detection methods are insufficient for solving the problem and call for development of fact checking systems. however, existing approaches struggle to capture fine-grained factual structures among continuous sentences, which in our observation is essential in discriminating human-written text and machine-generated text. our approach takes a step towards modeling finegrained factual structures for deepfake detection of text. ## conclusion in this paper, we present fast, a graph-based reasoning approach utilizing fine-grained factual knowledge for deepfake detection of text. we represent the factual structure of a document as a graph, which is utilized to learn graph-enhanced sentence representations. sentence representations are further composed through document-level aggregation for the final prediction, where the consistency and coherence of continuous sentences are sequentially modeled. we evaluate our system on a news-style dataset and a webtext-style dataset, whose fake instances are generated by grover and gpt-2 respectively. experiments show that components of our approach bring improvements and our full model significantly outperforms transformer-based baselines on both datasets. model analysis further suggests that our model can distinguish the difference in the factual structure of machine-generated and human-written text.