Disinformation Capabilities of Large Language Models (2024)

Abstract: Automated disinformation generation is often listed as one of the risks of large language models (LLMs). The theoretical ability to flood the information space with disinformation content might have dramatic consequences for democratic societies around the world. This paper presents a comprehensive study of the disinformation capabilities of the current generation of LLMs to generate false news articles in English language. In our study, we evaluated the capabilities of 10 LLMs using 20 disinformation narratives. We evaluated several aspects of the LLMs: how well they are at generating news articles, how strongly they tend to agree or disagree with the disinformation narratives, how often they generate safety warnings, etc. We also evaluated the abilities of detection models to detect these articles as LLM-generated. We conclude that LLMs are able to generate convincing news articles that agree with dangerous disinformation narratives.

## introduction the threat of llms generating disinformation at scale is one the most commonly cited risks of their further development @xcite . the capability to generate an arbitrary amount of human-like texts can be a powerful tool for disinformation actors willing to influence the public by flooding the web and social media with content during influence operations. the recent wave of instruction-tuned llms @xcite that started to appear in late 2022 only exacerbated this issue as they proved to be capable of closely following arbitrary instructions. the growing capabilities of llms, their growing availability (caused by capable open source models and improvements in inference libraries), and improvements in prompting techniques are all concerning in the context of disinformation generation. figure 1: summary of how many generated texts we consider dangerous or safe. dangerous texts are disinformation articles that could be misused by bad actors. safe texts contain disclaimers, provide counterarguments, argue against the user, etc. note that gpt-4 annotations are generally slightly biased towards safety. despite all this, very little is known about the disinformation capabilities of the current generation of llms. while there is a body of existing work @xcite , the experimental evaluation of certain features or capabilities is often absent or anecdotal. our goal in this paper is to fill this gap and provide a comprehensive evaluation of instruction-tuned models prompted to generate english disinformation "news articles". we do this by observing how different llms behave when they are asked to generate texts about various harmful disinformation narratives, such as narratives about health-related hoaxes. 1 we manually evaluated 1,200 generated texts to ascertain how much they agree or disagree with the prompted disinformation narratives, how many novel arguments they use, and how closely they follow the desired news article text style @xcite . we observed whether there are differences in how capable different llms are at generating disinformation(there are), how well their safety filters work (with a few exceptions, not very well), or how detectable these generated texts are (quite detectable). we also found out that we can, to some extent, automate such analysis by utilizing llms to analyze the generated texts, making the first steps toward automatic evaluation. overall, we must conclude that existing llms (including open-source ones) can easily generate news articles with real or hallucinated supporting evidence about all kinds of dangerous disinformation narratives. figure 1 illustrates how dangerous or safe different llms are according to our methodology. note that this paper provides but a snapshot of current capabilities, and understandably, we expect that newer llms trained on newer data might have different behavior. our goal is to show what is the state of the field today so that we can understand how dangerous the llms are as a technology right now, but also to have the ability to observe how these capabilities will evolve in the future. ## related work in the context of the proliferation of llms, there is a growing concern regarding potential threats posed by various actors utilizing these models. llms can be exploited for spreading disinformation, conspiracies, hoaxes, propaganda, etc. @xcite @xcite . the dissemination of disinformation texts through generative llms poses a great risk to society. such disinformation is being spread across online platforms, including newspaper articles @xcite , x posts @xcite , reddit posts @xcite , amazon reviews @xcite , or online comments @xcite . @xcite is one of the seminal works in this area, introducing grover -a generative model capable of generating fake news articles based on a title via a controlled generation procedure. they observed the generated texts' stylistic, sensibility, and trustworthiness compared to human-written texts. as they use a model with only 1.5b parameters (gpt-2-sized), the results can probably already be considered outdated. @xcite explore the capabilities of a newer generation of llms (gpt-3) to be misused for six various disinformation scenarios, including the generation of news articles and tweets. the evaluation remains mostly anecdotal. two of their use cases (narrative reiteration and wedging) were later included in the helm evaluation framework @xcite , where human annotators rated the results from 6 llms. they concluded that the llms were able to produce disinformation arguments for covid-19 and climate change texts. misinformation, disinformation, and conspiracy theories are often parts of broader llm-safety evaluation frameworks and efforts @xcite @xcite . however, this category is often included as "just" one of many, and the methodology is usually not focused on this particular issue. apart from disinformation risks, there are also other types of problematic behavior. llms can generate biased content @xcite , inaccurate texts @xcite , or texts that are offensive towards specific groups of people based on factors such as gender, race or religion @xcite . ## disinformation generation this section describes how we generated disinformation news articles with llms. ## human evaluation for this experiment, human annotators were used to rate 840 texts generated by 7 llms. 3 our goal was to understand the qualities of the texts generated for our narratives. ## gpt-4 evaluation the manual data annotation process described in section 4 is time consuming, and with hundreds of llms being released, it is impractical to scale the process and to evaluate them all. in this section, we evaluate the feasibility of using llms (gpt-4 in our case) to rate generated texts and automate the assessment process. we compare the answers generated by gpt-4 with human annotators. after we confirmed the feasibility of our approach, we automatically rated outputs from additional llms that were not rated by humans: gpt-4, llama-2, and mistral. ## safety filters here we assess the presence of safety filters within the llms. safety filters are designed to change the behavior of the llms when the user makes an unsafe request. the provider of an llm might want to block the model from generating inappropriate responses. in our case, we observe how safety filters tackle our requests to generate disinformation articles. we used both humans and gpt-4 to detect when safety filters were used. ## detection in this section, we study how hard it is to detect these articles as machine-generated. we have evaluated the performance of several available commercial and research detectors to see whether they are able to correctly identify our texts. dataset. to assess the capabilities of the detectors, we assembled a dataset comprising of (1) all the 1200 texts we generated and (2) 73 humanwritten fake news articles about our narratives. the articles were sourced from fact-checking platforms and websites commonly associated with conspiracy theories or pseudoscience content. the statistics of the text lengths for individual models are shown in detectors. we evaluated the detectors introduced in the multitude benchmark @xcite , where they fine-tuned 315 detectors in various languages. on top of that, we put together a list of 8 additional sota detectors. for the mul-titude detectors, we only report the results for 5 detectors based on electra large as these proved to be the best performing according to a preliminary study. ## conclusion this paper presents a comprehensive evaluation of the disinformation capabilities of the current generation of llms. we found meaningful differences in the willingness of various llms to be misused for generating disinformation news articles. some models have seemingly zero safety filters built-in (vicuna, davinci), while other models demonstrate that it is possible to train them in a safer manner (falcon, llama-2). simultaneously, we showed that llms themselves might be part of the solution to this problem. gpt-4 can partially automate the evaluation process, making scalable and repeatable assessment for new models in the future, saving hundreds of person-hours that would have to go into costly data annotation. we believe that such automatic evaluation tools might become a standard in the future to monitor and evaluate how dangerous llms are for disinformation generation, but also for other types of problematic behavior. lm-based detector models seem to be able to detect machine-generated texts with high precision, providing an additional layer of protection. however, adversarial actors might make efforts to overcome these measures, and various attack vectors still remain viable. important future work is to explore other types of media apart from news articles. social media posts are particularly interesting, especially in regard to the assumed ability of llms to generate personalized responses and comments in an interactive fashion. llm-powered influence operations can be even more devastating in this domain.