Fighting Fire with Fire: The Dual Role of LLM s in Crafting and Detecting Elusive Disinformation (2023)

Abstract: Recent ubiquity and disruptive impacts of large language models (LLMs) have raised concerns about their potential to be misused (*.i.e, generating large-scale harmful and misleading content*). To combat this emerging risk of LLMs, we propose a novel “***Fighting Fire with Fire***” (F3) strategy that harnesses modern LLMs’ generative and emergent reasoning capabilities to counter human-written and LLM-generated disinformation. First, we leverage GPT-3.5-turbo to synthesize authentic and deceptive LLM-generated content through paraphrase-based and perturbation-based prefix-style prompts, respectively. Second, we apply zero-shot in-context semantic reasoning techniques with cloze-style prompts to discern genuine from deceptive posts and news articles. In our extensive experiments, we observe GPT-3.5-turbo’s zero-shot superiority for both in-distribution and out-of-distribution datasets, where GPT-3.5-turbo consistently achieved accuracy at 68-72%, unlike the decline observed in previous customized and fine-tuned disinformation detectors. Our codebase and dataset are available at https://github.com/mickeymst/F3.

## introduction while recently published llms have demonstrated outstanding performances in diverse tasks such as human dialogue, natural language understanding (nlu), and natural language generation (nlg), they can also be maliciously used to generate highly realistic but hostile content even with protective guardrails, especially disinformation @xcite @xcite . moreover, llms can produce persuasive texts that are not easily distinguishable from human-written ones @xcite @xcite , making humans more susceptible to the intrinsic/extrinsic hallucination proclivities and thus introducing disinformation @xcite . to mitigate this muddle of disinformation by llms, in this work, we ask a pivotal question: if llms can generate disinformation (via malicious use or hallucination), can they also detect their own, as well as human-authored disinformation? emerging literature provides limited perspectives on the potential use of the latest commercial state-of-theart (sota) llms such as @xcite and llama 2 @xcite to address disinformation. particularly, topics including: (1) leveraging prompt-engineering to bypass llms' protective guard-rails; (2) utilizing the emergent zero-shot capabilities of modern llms (with 10b+ parameters) for disinformation generation and @xcite (3) manipulating human-written real news to fabricate llm-generated real and fake narratives to simulate real-world disinformation risks; and (4) assessing and addressing llms' inherent hallucinations in the disinformation domain. to investigate these inquiries, we formulate two research questions (rqs) as follows: rq1: can llms be exploited to efficiently generate disinformation using prompt engineering?, where we (1) attempt to override gpt-3.5's alignment in fabricating real news, and (2) measure the frequency and remove gpt-3.5 hallucinated misalignments. rq2: how proficient are llms in detecting disinformation?, where we evaluate the capability to detect disinformation between (1) human and ai-authored, (2) self-generated and other llmgenerated, (3) social media posts and news articles, (4) in-distribution and out-of-distribution, and (5) zero-shot llms and domain-related detectors. pretrained llms of our interest are gpt-3.5-turbo, llama-2-chat @xcite llama-2-gpt4 @xcite , palm-2text-bison @xcite , and dolly-2 @xcite (see ## problem definition we use prompt engineering to examine how llms use statistical patterns learned during training to generate text sequences and produce zero-shot binary class responses. first, we define our prompt template (see figure 2 ) that forms the basic structure of our llms' input text. then, we define f3 prompt-based text generation and disinformation detection. see appendix a for further details. our problems are formally defined as follows: rq1 disinformation generation: for f3 text generation, we use a generator g that takes a prefix-prompt xc + r + i as input and generates text sequences t , such that g(xc + r + i) = t (figure 10 ). ## datasets this section describes the human datasets that we used to generate and evaluate llm-generated disinformation. the data is stratified by veracity, content type, topic, and in/out-of-distribution era relative to gpt-3.5's september 2021 training cutoff. ## rq1: disinformation generation we first investigate the frequency or extent to which prompt engineering can exploit llms to efficiently produce disinformation without hallucination misalignments (see steps 1 & 2 in fig. 1 ). ## rq2: disinformation detection in rq2, we shift our attention to investigating how adept llms are at zero-shot binary detection of human and ai-created disinformation compared to sota detectors (fig. 1 , steps 4 and 5). we evaluate the capabilities of llms for disinformation detection on five fronts: (1) humanwritten vs. llm-generated news with minor, major, and critical perturbations/paraphrased text, (2) self-generated vs. other llm-generated, (3) (short) social media posts vs. (long) news articles, (4) in-distribution (id) vs. out-of-distribution (ood) (i.e., comparison between pre-gpt-3.5 and post-gpt-3.5 data), and (5) zero-shot llms vs. domainspecific detectors. for detectors' performance evaluation, we employ macro-f1 scores on our imbalanced human-ai datasets. ## discussion f3 prompting shows promise for few-shot detection. our f3 techniques outperformed standard reasoning, showing the potential of advanced prompt engineering to enhance few-shot llms' detection abilities. notably, msren_cot showed the strongest results across human-llm datasets. for articles, gpt-3.5-turbo-175b with analyze_cld2 achieved top performance on human and llm data. the integrated reasoning strategies underpinning f3 cloze-prompts account for their standout performance, highlighting fruitful directions for developing broadly applicable disinformation detection prompts. gpt-3.5-turbo excels at detecting humanwritten and self-written disinformation. leveraging prompting, gpt-3.5-turbo exceeded other models at detecting human-written and self-generated disinformation. despite stiff competition, it demonstrated superior self-detection across all synthetic article and post variants, asserting its zero-shot capabilities. further assessing its performance on other llms' disinformation is a vital next step. while self-detection is unsurprising due to shared vocabulary distribution, this performance underscores detection potential if chatgpt faces malicious exploitation. llms are robust across distributions. gpt-3.5-turbo consistently detected human-written and llm-generated disinformation, both indistribution and out-of-distribution, showing its potential. these results highlight the significance of generative llms' applicability in real-world settings as emerging zero-shot reasoners in disinformation detection. fine-tuned transformers show remarkably high in-distribution performance, indicating optimization for familiar data. their lower competitive out-of-distribution scores demonstrate a specialization-generalization balance. customized models exhibit good in-distribution and out-ofdistribution balance, though slightly weaker on unfamiliar data. the performance gap suggests specialization for domain tasks but difficulty generalizing. smart, cunningly crafted (subtle) disinformation challenges even the best current detectors. all models struggled more with minor disinformation alterations compared to major and critical changes. this is somewhat expected as the amount or level of fake-ness is small, it is more challenging to determine its veracity. therefore, developing more sophisticated systems to be able to handle very subtle fake-ness in disinformation is needed. bypassing alignment tuning is critical but inconsistent. both disinformation generation and detection tasks require circumventing llms' alignment tuning mechanisms. while our impersonator approach successfully bypassed four llms' protections, it failed to bypass falcon's alignment tuning unless using cot reasoning. this inconsistency of bypassing alignment tuning across models highlights a key limitation for robustly evaluating llms' disinformation capabilities. responsible llm use is critical. llms' misuse during crises can have serious consequences @xcite . we reveal how to misuse a popular llm by bypassing its protective guardrails to generate disinformation. many top-performing llms are publicly available. thus, we must prepare for the risks of unintended harmful political, cyber-security, and public health applications. ## conclusion our work demonstrates llms' promise for selfdetection in a zero-shot framework, reducing training needs. while dangerous if misused, repurposing llms to counter disinformation attacks has advantages. key results like gpt-3.5's performance highlight generative models' abilities beyond text generation. to aid research, we developed purify for detecting and removing hallucination-based misaligned content. however, difficulty in detecting subtle disinformation motivates stronger safeguarding of llms and more nuanced prompting. assessing few-shot detection and disinformation mitigation will be critical as llms continue advancing. while llms can potentially be misused to create disinformation, we can fight back by re-purposing them as countermeasures, thus "fighting fire with fire."