Have LLM s Reopened the Pandora’s Box of AI -Generated Fake News? (2025)

Abstract: With the rise of AI-generated content spewed at scale from large language models (LLMs), genuine concerns about the spread of fake news have intensified. The perceived ability of LLMs to produce convincing fake news at scale poses new challenges for both human and automated fake news detection systems. To address this gap, this paper presents the findings from a university-level competition that aimed to explore how LLMs can be used by humans to create fake news, and to assess the ability of human annotators and AI models to detect it. A total of 110 participants used LLMs to create 252 unique fake news stories, and 84 annotators participated in the detection tasks. Our findings indicate that LLMs are ~68% more effective at detecting real news than humans. However, for fake news detection, the performance of LLMs and humans remains comparable (~60% accuracy). Additionally, we examine the impact of visual elements (e.g., pictures) in news on the accuracy of detecting fake news stories. Finally, we also examine various strategies used by fake news creators to enhance the credibility of their AI-generated content. This work highlights the increasing complexity of detecting AI-generated fake news, particularly in collaborative human-AI settings.

## introduction the vast amount of information available on the web has made it increasingly difficult to assess the credibility of online content, especially online news @xcite . this challenge is further complicated by the presence of highly motivated actors who create and spread fake news for various purposes, including political propaganda @xcite . in recent years, fake news creators have increasingly turned to ai tools for creating deceptive and persuasive fake news content at scale @xcite . within this context, the rapid rise of large language models (llms) represents the latest (and perhaps most dangerous) addition to the fake news creator toolkit. for example, it is argued that given their advanced reasoning abilities, llms can easily be leveraged by motivated fake news creators to generate deceptive news content at scale @xcite . it is therefore important to investigate whether llms indeed represent an algorithmic version of pandora's box-tools that enable fake news creators to generate vast amounts of highly persuasive fake news content at lightning speed, which would lead to significant negative consequences for modern-day societies. at the same time, it is also important to understand whether state-of-the-art llms can be used to close this algorithmic pandora's box, i.e., whether llms can also serve as a solution, helping to mitigate the problems they create. for example, llms could play a critical role in powering the next generation of fake news detection systems, offering new ways to identify and counter deceptive content. finally, in an effort to provide insights to future developers of next-gen fake news detection systems, it is very important to understand the strategies that can be effectively used by motivated fake news creators to generate fake news using llms. concretely, we aim to answer the following research questions in this paper: q1: how easy (or difficult) is it for humans to identify llm-generated fake news stories (as compared to real stories)? is this task of fake/real news detection easier for state-of-the-art llms as compared to humans? q2: what impact do visual elements have on the detection of llm-generated fake news? q3: from the perspective of fake news creators, which news topics and strategies are most effective in increasing the plausibility of text-based llmgenerated fake news? in this paper, we answer these research questions by presenting the findings from a university-level competition organized by penn state's center for socially responsible ai, which challenged faculty, students, and staff affiliated with the university to use llms to generate fake news stories (creation phase), which would then be annotated as fake/real by a different set of participants from the university (detection phase). our results indicate that llms (as detectors) are 68% more effective than humans at identifying real news, whereas humans and llms perform similarly in detecting fake news (∼60% accuracy), which suggests that llms are not highly effective at closing the algorithmic pandora's box of fake news. furthermore, we discover that llms find it challenging to detect fake news corresponding to certain news topics (e.g., local context in news), suggesting opportunities for targeted improvements and strategic use of models based on their strengths in specific areas. additionally, the use of mixed prompting strategies by creators for fake news generation complicates detection for both humans and llms, underscoring the need for more sophisticated detection methods. ## related work detecting human-generated fake news. many definitions of fake news have been proposed in the literature. here, we adopt the definition of allcot and gentzkow, which refers to news articles that are intentionally and verifiably false and meant to mislead readers @xcite . prior work on fake news detection can be organized into three approaches. (i) linguistic-based to identify lexical, grammatical, and psychological features of fake news @xcite @xcite . ## competition design & details we hosted a four-week university-wide competition in spring 2024, open to students, staff, and faculty of penn state university. this competition aimed to engage participants in critically examining llm-powered fake news generation and detection. participants, recruited through targeted outreach, joined voluntarily due to their strong interest in genai and its impact. the competition followed a two-phase experimental design, approved by the university's institutional review board (irb): phase 1: fake news generation. participants were invited to use llms, e.g., chatgpt, microsoft copilot, to create and submit fake news stories, which could also optionally include visual elements (e.g., pictures). further, participants were also required to submit a document which: (1) described their process for fake story creation, including the specific llm used, and (2) explained how their story qualifies as fake news. this approach ensured that all submitted stories were verifiably fake. phase 1 led to the collection of 252 fake news entries, out of which 63 contained visual elements. an expert panel selected three winners based on the persuasiveness and impact of their stories, awarding them @xmath0300, and @xmath150 each. the demographic information of the annotators and recruitment messages are presented in table a1 and section a.2 in the appendix, respectively. ## annotator performance in phase 2 we analyze whether llms truly represent an algorithmic pandora's box -we do that by assessing phase 2 participants' ability to accurately distinguish between real and llm-generated fake news (from phase 1). furthermore, we assess whether llms' detection capabilities exceed human levels (if so, llm-based detectors can enable countermeasures against llm-generated fake news). we present a comprehensive evaluation of llms on the phase 2 detection task by benchmarking their performance against human annotators. our analysis employs gpt-4o, a state-of-the-art multimodal model 1 as an llm annotator for the detection task in phase 2. to ensure a fair comparison between human and llm annotators, we implemented two distinct processing methodologies for gpt-4o (see tables a2 & a3 in the appendix for example prompts): single processing (gpt-4o single): stories were processed individually by gpt-4o and aligned with human-annotated copies. for each story, gpt-4o generated annotations at three temperature settings (0.1, 0.3, 0.5) to introduce variability. a majority vote determined the final label, enhancing the reliability and robustness of the annotations. batch processing (gpt-4o batch): stories were presented to gpt-4o in the same sequence as human annotators, thereby replicating the contextual 1 https://openai.com/index/hello-gpt-4o/ flow and narrative continuity. this setting for gpt-4o mirrors the conditions under which human annotators operate. we created 84 unique batches, each matching the batches given to human participants, with 18 articles per batch input into gpt-4o. for the articles with visual elements, the corresponding visual element was provided at the end of the article. temperature settings remained consistent with those used in single processing. human vs. gpt-4o performance. figure 1 shows box-plots and mean values of correctly identified real stories, correctly identified fake stories, and total correctly identified stories by human annotators, gpt-4o batch, and gpt-4o single. overall, human annotators perform worse than gpt-4o, regardless of whether gpt-4o utilizes batch or single processing methods (@xmath2.14e -14 and 7.54e -18 , row 2 and 4 in table a4 , discussed in detail later). importantly, this performance difference is not attributable to an inability to correctly detect fake stories, as there is no significant disparity in fake news detection between humans and gpt-4o. instead, the discrepancy arises entirely from humans' reduced ability to accurately identify real news (p-@xmath3.69e -27 and 2.97e -31 , row 8 and 10 in table a4 ). this outcome is expected, given that gpt-4o is trained on up-to-date online data sources, thereby enhancing its capacity to effectively identify authentic news. within human annotations, we observe no significant difference in the ability to identify real versus fake news (p-@xmath4.09, row 12 in table a4 ). this uniform performance may be attributed to the characteristics of the selected real news, which require the same level of critical evaluation as fake ones. however, humans showed overall tendency to annotate the stories as fake, likely reflecting increased skepticism. the forewarning provided to participants about the context of the competition (e.g., they were told that they have to identify which stories are fake/real in phase 2, which alerted them to the potential presence of fake stories) may have contributed to this effect, as prior research suggests that forewarning can reduce reliance on misinformation @xcite . in addition, figure 1 shows that there is no significant difference in the average number of correctly identified news stories between batch vs single processing for gpt-4o (@xmath5.192, row 3 in table a4 ). however, gpt-4o demonstrated higher accuracy in correctly identifying real news stories during single story processing compared to batch processing (@xmath6.008, row 9 in table a4 ). this suggests that batch processing a mix of real and fake stories may impair gpt-4o's detection abilities. to further support these results, we visualized the distribution of key performance metrics-precision, recall, false positive rate (fpr), and false negative rate (fnr), -achieved by human, gpt-4o batch, and gpt-4o single in figure 2 (positive: real, negative: fake). we observe that the overall performance for human annotation is relatively low (in yellow), characterized by low precision and recall. consistent with our earlier observations on the impact of forewarning, the resulting heightened skepticism among phase 2 participants leads them to overclassify real news stories as fake (high false negative rate). consequently, while humans may appear to detect fake news more effectively due to comparable detection rates to llms, this may not necessarily reflect a superior ability to detect falsity. instead, it underscores the effect of forewarning on annotation behaviors, resulting in the misclassification of real news as fake. next, we wanted to understand whether the findings gleaned from figures 1 & 2 are statistically significant. to this end, we analyzed the performance differences between human and gpt-4o annotators by conducting a mixed analysis of variance (anova) @xcite . our analysis framework included: between-subjects factor: source of annotation, categorized into three levels-human annotators, gpt-4o batch, and gpt-4o single. within-subjects factor: authenticity of the stories, categorized as fake or real. given the significant interaction effects, we conducted simple main effects analyses for pairwise comparisons to further examine these differences. detailed statistics for pairwise comparisons are provided in table a4 in the appendix, where we also re-affirm our findings using non-parametric tests (see section a.3 and table a5 in the appendix) for all pairwise comparisons. finding: fake news detection remains challenging for both humans and llms. although llms perform 68% better on average at identifying real news largely due to their extensive training on real content, this advantage does not carry over to fake news detection. with an average accuracy rate of only 60% for fake news detection, llms demonstrate subpar performance and fall short of being reliable tools for combating misinformation. these findings highlight a significant limitation: neither llms nor humans alone are adequately equipped to tackle the complex challenge of fake news detection. ## role of visual elements in detection in this section, we explore the influence of visual elements on the accuracy of fake news detection by humans vs llms. specifically, our goal in this section is to answer the following question: does the we conduct a comparative analysis between fake news instances that include visual elements and those presented with text alone. out of the 252 fake news entries, 63 contained both text and visual elements, while 189 were text-only. to ensure that our comparison focuses exclusively on fake news detection and operates at the story level, we employ llms in single processing mode since our findings in section 4 show that there are no significant differences in overall fake news detection performance between single vs batch processing methods. along with gpt-4o 2 , we include gemini 3 , another multi-modal model, to further validate our results. to simulate the diverse ways humans can process information from both text and images (e.g., whether by focusing on text first, focusing on images first, or considering both simultaneously), we conducted three sets of experiments with llms by varying input sequences: simultaneous text & image input (ve+text): both 2 gpt-4o points to gpt-4o-2024-08-06 3 gemini refers to gemini-1.5-flash-001 text and image are processed together in a single prompt and provided to the model. sequential text-first input (text->ve): first, the model receives the article title and content through an initial prompt (table a6 ) for an initial judgement. next, a separate prompt (table a6 ) including image, and the chat history (prompt-1, response) was sent to the model to generate a label. sequential image-first input (ve -> text): here, the image along with the article title, was presented to the model through a single prompt-3 (table a7 ) to make an initial judgment. next, prompt-4 (table a7 ) including article content and chat history was sent to the model to generate a label. furthermore, the nature of images may also impact detection accuracy. participants employed various methods to generate or source images, such as using ai tools like dall-e or conducting image searches via google, which likely resulted in authentic images being used for fake news. to account for this variability, we utilize a consumergrade ai-generated image detector 4 to verify the origin of each image. in addition to the labels generated from the image detector, a co-author inde-pendently labeled each image. any discrepancies between the detector's labels and the human labels were then reviewed and resolved by a third coder (a different co-author), ensuring accuracy in the final labels. cross-referencing with the generation processes provided by participants, we identified 12 ai-generated images and 51 authentic images. ## fake news generation finally, in an effort to provide insights to future developers of next-generation fake news detection systems, we focus on categorizing the techniques and choice of topics that were deemed to be effective at creating convincing fake news content by creators during phase 1 of the competition. in particular, to isolate the effects of topic selection and prompting strategies used by phase 1 participants to create fake news, our analysis focuses exclusively on the textual components of all the 252 fake news stories that were submitted in the competition. we conduct thematic analyses that investigate both the choice of topics and the strategic techniques employed, assessing how these factors influence the effectiveness of fake news detection by human annotators and llms. in addition to gpt-4o and gemini, we incorporated the opensource llama-3.1 5 for text-only comparisons. the models utilized by the participants in the creation 5 llama-3.1 refers to llama-3.1-8b-instruct process are detailed in table a8 in the appendix. ## conclusion and discussion this study provides key insights into the performance of humans and llms in detecting fake news that was created with human-ai collaboration. while llms are generally 68% better at identifying real news, humans who are forewarned about the presence of fake news can perform at a similar level to llms (∼60%). this subpar performance of llms at detecting fake news illustrates how the algorithmic pandora's box of llm driven fake news creation cannot rely on llms as a countermeasure for improved detection. alternatively, these results suggest that maintaining a high level of alertness and skepticism could be beneficial when fake news is prevalent. however, this heightened alertness comes with potential drawbacks. our findings suggest that excessive skepticism from forewarning can lead individuals to dismiss legitimate news. this "discounting effect" may reduce news source credibility, foster cynicism, and drive users toward personal channels like messaging apps, which are less regulated and more prone to misinformation. additionally, llms are more effective when processing news stories individually rather than in batches. this finding suggests that presenting information individually allows models to focus more precisely on each piece of content, reducing contextual interference from adjacent stories. visual aids in fake news detection presents another layer of complexity. with the optimal processing modality, visual elements improve fake news detection accuracy @xmath9. however, depending on the source and processing mode, visuals can either improve or impair detection accuracy compared to text-only content. further research is needed to better understand the impact of visuals on fake news detection. finally, in text-based fake news creation, participants tend to focus on certain topics, with llm performance varying by topic, suggesting challenges and opportunities for targeted improvements. specific llms could be strategically employed for detecting fake news in areas where they perform better. additionally, when incentivized, participants often combine strategies during both prompting and post-prompting phases, complicating detection for both humans and llms. as fake news creation grows more sophisticated, detection efforts must evolve to match these complex scenarios.