Threat Scenarios and Best Practices to Detect Neural Fake News (2022)

Abstract: In this work, we discuss different threat scenarios from neural fake news generated by state-of-the-art language models. Through our experiments, we assess the performance of generated text detection systems under these threat scenarios. For each scenario, we also identify the minimax strategy for the detector that minimizes its worst-case performance. This constitutes a set of best practices that practitioners can rely on. In our analysis, we find that detectors are prone to shortcut learning (lack of out-of-distribution generalization) and discuss approaches to mitigate this problem and improve detectors more broadly. Finally, we argue that strong detectors should be released along with new generators.

## introduction during the covid-19 pandemic, an overabundance of inaccurate information made it hard for people to find reliable guidance when they needed it, creating the first global infodemic. this widespread presence of online fake news poses a risk for the principles of the global information and communication space, which is considered to be a public good @xcite . widely available and reliable information is necessary for the function of a democratic state and is seen by unesco as vital to ensure "public participation and civic space". 2 neural generation systems unlock new possibilities for disinformation campaigns to generate large quantities of targeted content @xcite @xcite . recent progress in pretrained language models enables the generation of fluent, original text that can be easily confused with humanwritten text. progressively more complex use of such language models can be achieved depending on the technical expertise and resources of the users (i.e., access to the language model, capability to run existing systems, to finetune them, or to develop a new generator and train it). this is a pressing problem, and it is reported that certain actors spend substantial monetary resources to spread misinformation and influence public opinion, for example during the covid 19 pandemic foot_2 and the us presidential elections of 2016 @xcite . although the majority of online disinformation is currently manually written @xcite , the rapid progress of language generation systems along with their increased availability could attract these actors for whom the financial barriers do not pose problems. to prevent the spread of artificial and potentially malicious content online, it is necessary to develop automated detection systems that can distinguish human and generated text reliably. in this work, we discuss the problem of machine generated text detection @xcite , by identifying and analyzing potential scenarios of malicious text generation (i.e., threat scenarios) based on (1) state-of-the-art models in neural language generation, and (2) previous work that describes how these techniques can be used to forge content @xcite @xcite . our analysis focuses on the interaction of generators and detectors for each scenario across their technical capabilities and associated costs. knowing the limitations of current detection systems can help policymakers identify the threats posed by different actors by assessing their technical expertise and resources. the contributions of this work are: (1) identifying and assessing threat scenarios for the use of pretrained language models to spread disinforma-tion, based on associated costs and availability of generators, (2) analyzing the performance of current methods -discriminators trained to defend against these threat scenarios, and (3) identifying challenges and future research directions to mitigate potential threats. ## problem description in this section we introduce the problem of detecting generated text and describe state-of-the-art methods for both generators and detectors. ## threat scenarios a given generator can be used in various ways, based on domain expertise and budget, resulting in different styles of generated text. we describe threat scenarios by identifying possible uses of pretrained lms and their associated costs (technical expertise and monetary resources required). interactive playgrounds and paid apis most language models have been open-sourced and the model parameters can be downloaded. however, many also have free online playgrounds that sim-plify the user-interaction 4 and require little to no programming expertise. the latest models coming from the private sector (e.g., gpt3 @xcite ) are usually not available publicly or are only accessible through paid apis and online playgrounds. with some delay, equally powerful models are often released publicly (e.g., opt @xcite ). these playgrounds are either free or very cheap. even the paid gpt3 playground's most powerful model only costs $0.06 for 1000 tokens. generating text through the playgrounds is time consuming and requires heavy intervention by the user who needs to write prompts and ensure that the generation is sensible. however, this method could be used to speed up the process of writing fake content by offloading some portion of the writing to the language model. apis extend the playgrounds by allowing generation to be performed programmatically. this increases the potential scale of the generation process. targeted generations through prompting still need careful handcrafting. ## experimental setup here we describe the generators, detectors, and datasets that we use in our experiments. @xcite , and their indomain detection performance when training and testing with nucleus sampling (@xmath0.96). ## results our experiments simulate the different threat scenarios identified in the previous section and evaluate their associated risks. borrowing from the field of security @xcite , we describe how "attackers" can employ generators and "defenders" train effective detectors under varying assumptions. the first three scenarios (online playgrounds, paid apis, and execution of lm) can be studied together as they involve using an existing generator without any domain specific tuning of the model parameters. users can still tailor the generation style by changing decoding hyperparameters. next, we explore the scenario of generator finetuning, focusing on the covid domain. lastly, we simulate the training of new generators by looking at the performance of detectors when they are trained and tested on different generators. throughout our experiments, we choose to isolate and study a specific phenomenon to determine its impact of the detection performance in isolation. ## ethical considerations, limitations, and recommendations although in this work we discuss the problem of detecting generated text with a particular focus on generated fake news, we would like to point out that not all generated text is fake news and most generated text does not have a malicious intent. language models are widely used in nlp for both conditional language generation tasks like summarization, translation, and dialogue @xcite , as well as unconditional generation tasks like story generation @xcite . in this work, by presenting results on the detection performance of different generators, we inevitably provide information that could inform malicious actors about how to improve generators and where our detection system fail. however, we believe that the experimental evidence in this paper points to research directions which can help improve systems for the detection of generated text beyond our current capabilities. we also would like to point out that the findings of this work are limited to the domains we focused on but the methodology is general and can be applied to other domains. in particular, the outof-domain performance will change depending on the distance between domains. for example, we did not include any experiments with other dialects of english or other languages. one limitation of this work is that we do not experiment with controllable generation systems @xcite @xcite . this is partly due to the high risk of dual use in developing systems that controllably generate malicious content. we point to prior work which experimented detecting generations from these systems @xcite . finally, based on the observations made in this paper, we recommend that language models should be released along with strong detectors. our experiments showed that in-domain detection performance is reasonably high (> 90% acc.) even for the largest generator gpt-3, while the out-ofdomain performance is significantly lower (64.8% acc.). we also pointed out that variations in the domain between generated data and human data used to train discriminators can lead to shortcut learning. this suggests that the requirement for authors of new generators should be to release data sampled from the training dataset along with sample generations to ensure the two splits of the data have matching domains. ## conclusion we provide an assessment of the current landscape of generated text detection and identify three primary threat scenarios. through extensive experimentation simulating the identified threats, we establish the minimax strategies that minimize the worst case scenario for the detector. we argue that these strategies constitute best practices for practitioners. we find that, when confronted with an adversary capable of training a new generator, the worst case detection performance could be as low as 64.8%. we then discuss observed detection challenges related to shortcut learning, point to several avenues for future work, and provide recommendations for the community to release detectors along with new models. we use the default causal language modeling hyperparameters of the huggingface toolkit to adapt the generators to the covid domain. we maximize the batch size that fits on a gpu. for the larger models we used deepspeed stage 3 and 4 a40 gpus. while previous work did not comment on generation hyperparameters, we found that generation quality is highly dependent on the hyperparameters of the generation process. in the covid domain, after adapting the generators to the nela gt 2020 dataset, we found the use of beam search to be necessary to obtain fluent text. in section 6, we discuss ideas which could help future work perform hyperparameter tuning more extensively and efficiently.