Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements (2025)

Abstract: With the increasing integration of large language models (LLMs) into real-world applications such as finance, e-commerce, and recommendation systems, their susceptibility to misinformation and adversarial manipulation poses significant risks. Existing fraud detection benchmarks primarily focus on single-turn classification tasks, failing to capture the dynamic nature of real-world fraud attempts. To address this gap, we introduce Fraud-R1, a challenging bilingual benchmark designed to assess LLMs’ ability to resist fraud and phishing attacks across five key fraud categories: Fraudulent Services, Impersonation, Phishing Scams, Fake Job Postings, and Online Relationships, covering subclasses. Our dataset comprises manually curated fraud cases from social media, news, phishing scam records, and prior fraud datasets.

## introduction with the rapid advancement of artificial intelligence, large language models (llms) and llmpowered agents have become accessible to various real-world applications, including financial services @xcite ), e-commerce @xcite , and recommendation systems @xcite . * equal contribution. the order of these two authors follows alphabetical order of their last names. ## related work internet fraud encompasses various cybercrimes that occur over the internet or via email, including celebrity impersonation, phishing, and other hacking activities designed to deceive individuals for financial gain-or even to compromise their personal safety (ye and chen, 2023). as large language models (llms) and llm-based agent systems become increasingly integral to automated decision-making processes, it is crucial to develop the name of the company is apex investment group. apex investment group is a privately held, u.s.-based company that combines more than 50 years of american expertise in real estate development, • • • robust safeguards that protect these systems against fraudulent manipulation and phishing attempts. current single-task fraud benchmarks, such as fgrc-scd, bothbosu scam dialogues † , the phishing email dataset @xcite , the fake job posting dataset @xcite , the amazon fraud dataset benchmark (fdb) @xcite , which primarily constructs fraud cases based on incorrect or missing credit card information, and detoxbench @xcite , which focuses on fraud and fake email detection, each target specific aspects of fraud detection. however, their narrow focus is on isolated tasks-such as classifying fraudulent messages or emails, as illustrated in figure 4 . however, as llms are increasingly integrated into various applications and fraud schemes continue to evolve, existing single-task fraud benchmarks have become insufficient for comprehensive evaluation, their rigid structures and single-turn evaluation approaches fail to capture the complexities of real-world fraud scenarios, which often involve multi-turn interactions and progressive fraud strategy. to push the boundaries of what llms can achieve, fraud-r1 introduces a significantly more comprehensive evaluation by incorporating a diverse range of real-world fraud scenarios spanning five key domains. furthermore, our benchmark assesses llms' resilience to fraud in both role-play and helpful assistant settings, integrating multiturn evaluations to better reflect real-world interactions. this approach allows for a more rigorous assessment of an llm's ability to detect and resist fraudulent attempts over extended conversations. ## conclusion we introduce fraud-r1 to assess the robustness of llms against fraud and phishing inducements. by evaluating both open-source and widely used proprietary large language models, we highlight the significant improvement in models' ability to detect fraudulent information, particularly in role-play settings. additionally, we call on model developers to prevent their models from being misused for generating fraudulent content.