M isinfo E val: Generative AI in the Era of “Alternative Facts” (2024)

Abstract: The spread of misinformation on social media platforms threatens democratic processes, contributes to massive economic losses, and endangers public health. Many efforts to address misinformation focus on a knowledge deficit model and propose interventions for improving users’ critical thinking through access to facts. Such efforts are often hampered by challenges with scalability, and by platform users’ personal biases. The emergence of generative AI presents promising opportunities for countering misinformation at scale across ideological barriers. In this paper, we introduce a framework (MisinfoEval) for generating and comprehensively evaluating large language model (LLM) based misinformation interventions. We present (1) an experiment with a simulated social media environment to measure effectiveness of misinformation interventions, and (2) a second experiment with personalized explanations tailored to the demographics and beliefs of users with the goal of countering misinformation by appealing to their pre-existing values. Our findings confirm that LLM-based interventions are highly effective at correcting user behavior (improving overall user accuracy at reliability labeling by up to 41.72%). Furthermore, we find that users favor more personalized interventions when making decisions about news reliability and users shown personalized interventions have significantly higher accuracy at identifying misinformation.

## introduction in the last decade, there has been growing concern about the proliferation of misinformation on social media platforms. for example, between 2006 and 2017, sensational "fake news" articles spread rapidly on facebook, diffusing farther and faster than truthful or reputable content @xcite . these sharing trends have been amplified by "filter bubble" algorithms that intentionally create ideological echo chambers, which reinforce existing viewpoints and further facilitate spread of misinformation @xcite . many proposed interventions for combating misinformation focus on tagging unreliable content @xcite or encouraging critical thinking by users @xcite . however, the two major bottlenecks in many such interventions are user bias and scalability. conventional factchecking interventions rely on the assumption that users are rational agents, who will agree upon a common "ground truth" once exposed to enough information. this assumption is often violated in the real world due to the fact users do not process new information neutrally, and are more critical of counter-partisan news and more accepting of pro-partisan news at face value @xcite @xcite . tagging unreliable or suspicious content also requires careful inspection, which is currently performed by professional fact-checking organizations such as snopes. these organizations are constrained both in terms of their financial resources and qualified fact-checkers they can employ. alternatively, platforms have explored a decentralized approach through crowdsourced verification by users. while this approach allows for scale, it is vulnerable to user misuse such as recent reports of disinformation and partisan bias spread through x's community notes @xcite . breakneck advances in large language models (llms) offer a promising avenue for large-scale fact checking, as they provide tools for fast information processing and can detect patterns associated with misleading content @xcite . early evidence suggests that llm-based explanations of veracity can significantly reduce social media users' tendency to accept false claims @xcite . llms also present a potential path to understanding and countering user bias: recent work @xcite @xcite argues llms are capable of very simple forms of world and cognitive modeling. this opens up the possibility of tailored approaches to countering misinformation that target users across diverse backgrounds (e.g. varying education levels or ideological leanings). our agenda is to develop a powerful, automated tool via llms for generating tailored misinformation interventions and a framework for testing the effectiveness of these interventions @xcite . to do this, we have two study phases focused on examining effects of ai interventions with and without personalization: phase i: we conduct an a/b testing experiment using a simulated social media environment that features both true and false content. implications for content moderation: first we expand upon existing research showing that explanation-based credibility indicators can mitigate spread of misinformation. in a large-scale experiment, we show explanation interventions can effectively inform diverse users about unreliable content, improving user accuracy over label-only indicators by at least 34.2% vs. 24.2% for label-only indicators. explanations generated by @xcite perform best at encouraging user flagging of misinformation (preintervention users correctly flag in 3.1% of cases vs. 38.1% post-intervention). this indicates the promise of llm-based explanations for future intervention strategies, corroborating findings from recent and concurrent work @xcite . implications for healthcare: the spread of medical misinformation poses a serious obstacle to healthcare providers if unchecked. some notable recent examples include false claims spread online through websites like x (formerly twitter), youtube and webmd of "alternative cures" for cancer @xcite . such false claims could pose a significant health risk given that alternative medicines can more than double the risk of mortality in cancer patients @xcite . we find that gpt-4 explanations could provide automated support in mitigating medical misinformation, leading to an user accuracy at predicting news reliability of 97.6% (95% ci: [96.0, 99.2]). phase ii: next, we explore how personalization of explanations can further improve their effectiveness. we measure how the degree of personalization affects user-reported helpfulness, and find that explanations that are highly aligned with users' attributes (e.g., education, political ideology, gender) are deemed more helpful by users than explanations without personalization (average helpfulness score of 2.98 vs. 2.71). we also find that users shown personalized explanations have significantly higher accuracy at identifying misinformation than users shown misaligned explanations. our results highlight the powerful persuasive capabilities of llms. our simulated social media feed platform and surveys with over 4,000 diverse participants will be made publicly available to further research on how llm outputs influence users, including potential disinformation risks. to the best of our knowledge, this is the first study to consider tailored llm-based interventions based on specific attributes of social media users. we view it as a first step in this agenda, since general advances in foundation models will increase these tools' capabilities for combating misinformation. we envisage better fine-tuning of explanations as additional information becomes available about users (without violating their privacy). ## related work our work aligns with a growing body of literature, mostly from political science, that examines the effectiveness of mitigating misinformation through user-facing interventions. assuming a known "ground truth" label determined by human or ai fact-checkers, this literature aims to reduce user consumption and interaction with false content by designing effective ways to present this information or to otherwise nudge users to consider them. fact-checking labels that are specifically attributed to ai have been shown to be effective as interventions in reducing user consumption of misinformation @xcite , though earlier studies found they are often less effective than labels attributed to other sources such as professional fact-checkers @xcite @xcite . there is evidence that explaining the mechanics behind how the fact-checking label is generated improves their effectiveness @xcite . concurrently with our work, it has been found that gpt-based explanations of content veracity can significantly reduce social media users' reported tendency to accept false claims @xcite , though they can be equally effective when used with malicious intent to generate deceptive explanations @xcite . there have also been some early works that explore the use of personalization in ai fact-checking systems, such as jahanbakhsh et al. (2023), which examines the effects of a personalized ai prediction tool based on the user's own assessments; and @xcite , with a focus on toxicity in personalized content moderation tools. our work departs from these as we consider the generation of arguments and justifications given a label, rather than predicting the veracity. ## social media platform experiment and study design we recruit human participants to interact with a simulated news feed interface that mimics realworld social media platforms such as facebook and x. the news feed consists of news headlines (claims), and an intervention button ("find out more") that the user may voluntarily click on. upon doing so, they may see an intervention with a veracity label of the headline (true or false) and an explanation of the label (see figure 1 ). users may react to the news item as they normally would on social media, and provide feedback on the intervention. by varying the types of interventions presented to users and comparing their subsequent behavior, we can analyze the impacts of these interventions. details of the interface are given in §3.1. phase i ( §4.1) compares the effectiveness of five non-personalized explanations, while phase ii ( §4.2) directly compares gpt-4 generated explanations with and without personalization. in both phases, the experimental interface consists of four components: (1) a consent form; (2) task instructions; (3) a questionnaire on user demographics and opinions; (4) a simulated newsfeed. ## misinfoeval environment results in 4.1, we discuss best practices for evaluating effectiveness of misinformation interventions. we first compare non-personalized interventions in a simulated social media newsfeed in 4.2. then we assess the effectiveness of personalized interventions in 4.3. ## conclusion in conclusion, we introduced a framework for generating, personalizing and comprehensively evaluating llm-based misinformation interventions (misinfoeval). our findings show a promising direction for social media platforms and policy makers to combat misinformation by improving the presentation of content to users. with the ability of llms to efficiently generate explanations to support veracity judgments with user personalization, they have the potential to serve as key components in designing scalable and powerful interventions. however, it is important to highlight that their success depends on model accuracy at predicting label veracity. this requires further improvements in automated prediction tools, greater coordination with human fact-checkers, or both. however, our observations raise concern that in the future advanced llms like gpt-4 can be misused to create targeted "fake news" campaigns against certain groups or even individuals. the personalization ability of llms is a double-edged sword, and collaboration between policy makers, researchers and engineers is needed to ensure they are used for ethical and desirable intentions.