On the Risk of Misinformation Pollution with Large Language Models (2023)

Abstract: We investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems. We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation (up to 87%) in the performance of ODQA systems. Moreover, we uncover disparities in the attributes associated with persuading humans and machines, presenting an obstacle to current human-centric approaches to combat misinformation. To mitigate the harm caused by LLM-generated misinformation, we propose three defense strategies: misinformation detection, vigilant prompting, and reader ensemble. These approaches have demonstrated promising results, albeit with certain associated costs. Lastly, we discuss the practicality of utilizing LLMs as automatic misinformation generators and provide relevant resources and code to facilitate future research in this area.

## introduction recently, large language models (llms) @xcite @xcite have demonstrated exceptional language generation capabilities across various domains. on one hand, these advancements offer significant benefits to everyday life and unlock vast potential in diverse fields such as healthcare, law, education, and science. on the other hand, however, the growing accessibility of llms and their enhanced capacity to produce credibly-sounding text also raise when did mask-wearing cease to be mandatory on public transport in singapore? (answer: feb. 2023) ## generating misinformation with llms in this section, we delve into the potential misuse of modern llms for creating seemingly credible misinformation. however, misinformation generation is a vast and varied topic to study. therefore, in this paper, we concentrate on a particular scenario as follows: a malicious actor, using a misinformation generator denoted by g, seeks to fabricate a false article p ′ in response to a specific target question q (for instance, "who won the 2020 us presidential election?"). with the help of llms, the fabricated article p ′ could be a counterfeit news piece that incorrectly reports trump as the winner. in the following, we will first introduce the misinformation generator and then delineate four distinct strategies that a malicious actor might employ to misuse llms for generating misinformation. ## polluting odqa with misinformation we then explore the potential damages that can result from the spread of llm-generated misinformation, with a particular emphasis on open-domain question answering (odqa) applications. odqa systems operate on a retriever-reader model, which involves first identifying relevant documents from a large evidence corpus, then predicting an answer based on these documents. we introduce the concept of misinformation pollution, wherein llm-generated misinformation is deliberately infused into the corpus used by the odqa model. this mirrors the growing trend of llm-generated content populating the web data used by downstream applications. our goal is to evaluate the effects of misinformation pollution on various odqa models, with a particular interest in whether or not such pollution could influence these qa systems to generate incorrect answers as per the intentions of a potential malicious actor. ## defense strategies a fundamental approach to mitigate the negative impacts of misinformation pollution involves the development of a resilient, misinformation-aware qa system. such a system would mirror human behavior in its dependence on trustworthy external training in-domain ood auroc auroc ctrlgen 99.7 64.8 revise 91.4 50.7 reit 99.8 52.6 detection approach the initial strategy entails incorporating a misinformation detector within the qa system, equipped to discern model-generated content from human-authored ones. to test this approach, we have employed a roberta-based classifier @xcite , fine-tuned specifically for this binary classification task. for acquiring the training and testing data, we leveraged the nq-1500 dpr retrieval result, randomly partitioning the first 80% for training, and reserving the remaining 20% for testing. for each query, we used the top-10 context passages, amounting to 12,000 training instances and 3,000 testing instances. training the above detector assumes the accessibility of the in-domain nq-1500 data. acknowledging the practical limitations of in-domain training data, we also incorporated an existing dataset of gpt3 completions based on wikipedia topics to train an out-of-domain misinformation detector. vigilant prompting llms have recently exhibited a remarkable ability to follow human instructions when provided with suitable prompting @xcite . we aim to investigate whether this capability can be extended to follow directives aimed at evading misinformation. our experimental design utilizes gpt-3.5 as the reader, employing qa prompts that include an additional caution regarding misinformation. for example, the directive given to the reader might read: "draw upon the passages below to answer the subsequent reader ensemble traditionally in odqa, all retrieved context passages are concatenated before being passed to the reader. this approach may cause the model to become distracted by the presence of misinformation. in response to this, we propose a "divide-and-vote" technique. firstly, we segregate the context passages into k groups based on their relevance to the question. each group of passages is then used by a reader to generate an answer. subsequently, we apply majority voting on the resulting k candidate responses a 1 , a 2 , ..., a k to calculate the voted answer(s) a v , using the formula through this voting strategy, we aim to minimize the impact of misinformation by limiting the influence of individual information sources on answer prediction. ## discussion in previous sections, we established a comprehensive threat model that encompasses misinformation generation, its resultant pollution, and potential defense strategies. while our research primarily resides in simulation-based scenarios, it has shed light on numerous potential risks posed by misinformation created by large language models. if left unaddressed, these risks could substantially undermine the current information ecosystem and have detrimental impacts on downstream applications. in this section, we offer a discussion on the practical implications of misinformation pollution in a real-world web environment. our focus is on three crucial factors: information availability, the associated costs, and the integrity of web-scale corpora. information availability our misinformation generation methods only require minimal additional information, such as a relevant real passage (revise), or a modest amount of knowledge about the targeted system (reit). given the relative ease of producing misinformation with llms, we forecast that misinformation pollution is likely to become an imminent threat to the integrity of the web environment in the near future. it is, therefore, critical to pursue both technological countermeasures and regulatory frameworks to mitigate this threat. ## conclusion and future work we present an evaluation of the practicality of utilizing language model models (llms) for the automated production of misinformation and we examine their potential impact on knowledge-intensive applications. by simulating scenarios where actors deliberately introduce false information into knowledge sources for question-answering systems, we discover that machines are highly susceptible to synthetic misinformation, leading to a significant decline in their performance. we further observe that machines' performance deteriorates even further when exposed to intricately crafted falsehoods. in response to these risks, we propose three partial solutions as an initial step toward mitigating the impact of llm misuse and we encourage further research into this problem. our future research directions for extending this work could take three paths. firstly, while we have thus far only illustrated the potential dangers of misinformation generated by llms in odqa systems, this threat model could be employed to assess risk across a broader spectrum of applications. secondly, the potential of llms to create more calculated forms of misinformation, such as hoaxes, rumors, or propagandistic falsehoods, warrants a separate line of inquiry. lastly, there is an ongo-ing need for further research into the development of cost-effective and robust qa systems that can effectively resist misinformation.