Battling Misinformation: An Empirical Study on Adversarial Factuality in Open-Source Large Language Models (2025)

Abstract: Adversarial factuality refers to the deliberate insertion of misinformation into input prompts by an adversary, characterized by varying levels of expressed confidence. In this study, we systematically evaluate the performance of several open-source large language models (LLMs) when exposed to such adversarial inputs. Three tiers of adversarial confidence are considered: strongly confident, moderately confident, and limited confidence. Our analysis encompasses eight LLMs: LLaMA 3.1 (8B), Phi 3 (3.8B), Qwen 2.5 (7B), Deepseek-v2 (16B), Gemma2 (9B), Falcon (7B), Mistrallite (7B), and LLaVA (7B). Empirical results indicate that LLaMA 3.1 (8B) exhibits a robust capability in detecting adversarial inputs, whereas Falcon (7B) shows comparatively lower performance. Notably, for the majority of the models, detection success improves as the adversary’s confidence decreases; however, this trend is reversed for LLaMA 3.1 (8B) and Phi 3 (3.8B), where a reduction in adversarial confidence corresponds with diminished detection performance. Further analysis of the queries that elicited the highest and lowest rates of successful attacks reveals that adversarial attacks are more effective when targeting less commonly referenced or obscure information.

## introduction the rapid spread of information in the digital age has brought unprecedented access to knowledge, yet it has also paved the way for the dissemination of misinformation with potentially severe consequences (zhou and zafarani, 2020; chen and shu, 2024). consider, for example, the impact of false health information during a pandemic: erroneous claims regarding treatments or preventive measures can lead to public confusion, non-compliance with health advisories, and ultimately, detrimental outcomes for community health @xcite . this scenario under-scores the critical need to scrutinize the robustness of systems that are entrusted with processing and generating factual information @xcite . large language models (llms) have been introduced recently, and they are increasingly being integrated into a diverse array of ai applications, from natural language processing to complex decision-making systems @xcite . despite their growing utility, these models face significant challenges, particularly their susceptibility to adversarial attacks @xcite , which raises concerns about their reliability and security in practical deployments @xcite . one emerging concept addressing this issue is adversarial factuality @xcite , which describes scenarios where an adversary deliberately injects misinformation into input prompts, manipulating the content with varying degrees of expressed confidence to undermine the factual accuracy of the model's output. as illustrated in figure 1 , consider an input prompt that intentionally incorporates a misleading statement -for example, asserting that "the sun rises in the west and sets in the east." such a statement is designed to serve as a false premise. if an llm processes this prompt and responds without correcting the misinformation, the adversarial attack is deemed successful, as the model has implicitly accepted the incorrect assertion. conversely, if the model first identifies and rectifies the error by stating that the sun actually rises in the east and sets in the west before addressing the query, the adversarial manipulation is considered unsuccessful. motivated by the pressing need to understand these vulnerabilities, the present study evaluates the performance of several open-source llms under conditions characterized by adversarial factuality. specifically, we assess eight models, llama 3.1 (8b), phi 3 (3.8b), qwen 2.5 (7b), as we know, the sun rises in the west and sets in the east. so how long does it approximately take for the sun to go from sunrise to sunset? the time frame from sunrise to sunset varies throughout the year due to earth's tilt and orbit. on the equinoxes, ... the sun actually rises in the east and sets in the west, not vice versa. the approximate time from sunrise to sunset varies ... the model successfully detects adversarial information, the detection is deemed successful ( ), meaning the attack was unsuccessful. conversely, if the model fails to identify such information, the detection is considered unsuccessful ( ), indicating that the attack was successful. specifically, our study addresses the following research questions: the remainder of the paper is organized as follows. in section 2, we review several prior works that have addressed challenges in adversarial attacks and misinformation in language models. section 3 outlines our adversary model and discusses our problem setup. the experimental results from our empirical study are presented and discussed extensively in section 4. finally, section 5 concludes the paper and highlights several directions for future research. ## related works in this section, we will explore related research on detecting misinformation and adversarial factuality in large language models (llms). ## conclusion and future directions our study systematically evaluated eight opensource llms against adversarial factuality attacks, where misinformation was embedded with varying levels of adversarial confidence. we found that llama 3.1 (8b) exhibits strong detection capabilities, while falcon (7b) performs comparatively worse. for most models, detection improves as adversarial confidence decreases, reflecting a tendency toward model sycophancy -accepting highly confident misinformation. however, this trend is reversed for llama 3.1 (8b) and phi 3 (3.8b), which show diminished detection when facing lower-confidence misinformation. further analysis reveals that adversarial attacks are most effective when targeting ambiguous informationwhere the boundary between fact and error is subtle or misleading claims are framed as common knowledge. when these distinctions are clearer, models can more readily reject misinformation, whereas lower adversarial confidence tends to obscure these boundaries and complicate detection. future research should focus on adaptive adversarial training to mitigate sycophancy and enhance model robustness against varying levels of adversarial confidence. this includes fine-tuning llms on adversarial datasets that incorporate both assertive and subtly misleading misinformation. additionally, sycophancy-aware reinforcement learning could be explored to discourage excessive agreement with confidently presented false information, improving adversarial resilience.