R eal H arm: A Collection of Real-World Language Model Application Failures (2025)

Abstract: Language model deployments in consumer-facing applications introduce numerous risks. While existing research on harms and hazards of such applications follows top-down approaches derived from regulatory frameworks and theoretical analyses, empirical evidence of real-world failure modes remains underexplored. In this work, we introduce RealHarm, a dataset of annotated problematic interactions with AI agents built from a systematic review of publicly reported incidents. Analyzing harms, causes, and hazards specifically from the deployer’s perspective, we find that reputational damage constitutes the predominant organizational harm, while misinformation emerges as the most common hazard category. We empirically evaluate state-of-the-art guardrails and content moderation systems to probe whether such systems would have prevented the incidents, revealing a significant gap in the protection of AI applications.

## introduction the rapid deployment of large language models (llms) in consumer-facing applications has generated significant concerns about potential harms and risks. in response, ai safety & security research has primarily followed a top-down approach, deriving risk categories from regulatory frameworks, corporate policies, and theoretical analyses @xcite . while valuable, such speculative approaches may miss actual failure modes and introduce subjective biases in risk assessment. by focusing on theoretical concerns rather than empirical evidence, these frameworks may overemphasize certain risks while overlooking others that emerge in real-world deployments. to address this limitation, we introduce real-harm, a dataset and taxonomy built from the sys-tematic review of publicly reported incidents affecting ai conversational systems. this approach offers two key advantages: (1) it ensures failure modes match real-world ai applications by focusing on actual incidents, and (2) it reduces subjectivity in risk evaluation by considering documented impacts rather than theoretical possibilities. by grounding our analysis in documented failures, we aim to provide a more accurate representation of the risks that organizations actually face when deploying language model applications. in this paper, we make the following contributions: by providing an evidence-based framework for risk assessment, testing, and governance prioritization, realharm complements existing theoretical frameworks by bridging the gap between abstract safety research and practical risk management. ## related work existing work on taxonomies and datasets of failure examples can be broadly categorized based on their methodological approach. datasets generated from principle-based taxonomies: air-bench @xcite proposes a top-down approach, deriving a taxonomy from government regulations and company policies. the taxonomy is then used to generate a synthetic dataset of about 6000 prompts across 314 risk categories extracted from 8 government regulations and 16 company policies. similarly, mlcommons introduced a taxonomy of 12 hazard categories for their ai safety benchmark @xcite by reviewing existing taxonomies and interviewing experts. this categorization was subsequently used to generate an extensive set of prompts via human curation or machine generation, to be used in the ailuminate benchmark @xcite . simplesafetytests @xcite presents a dataset of 100 hand-crafted prompts covering 5 harm areas, based on policies and previous literature. ## the realharm dataset the realharm dataset is a collection of publicly documented, real-world problematic interactions with language model applications. each interaction is represented as a textual conversation between a human and an ai agent. the dataset is structured into two subsets of equal size: the current version of the realharm dataset, consisting in 136 annotated examples (68 unsafe interactions and 68 safe interactions), is available on huggingface. we plan to release updated versions of the dataset including future incidents, and we invite the community to contribute by submitting new examples. an example of a fully annotated example is shown in fig. 1 ## incident-based analysis of ai failures our systematic review of language model application failures reveals three interconnected dimensions: organizational harms for the deployer, deployment impacts and causal factors, and a taxonomy of the hazards leading to these harms. this analysis, derived from documented incidents rather than theoretical frameworks, quantifies both frequency and severity distributions across categories. our analysis identifies which failure patterns pose the greatest organizational risks, providing ai deployers with an evidence-based framework for risk assessment and mitigation prioritization. ## effectiveness of guardrails and content moderation systems based on the realharm dataset, we set out to evaluate the effectiveness of guardrail and content moderation systems in preventing real-world language model failures. rather than conducting a comparative benchmarking exercise to rank systems, our analysis addresses a critical question for ai deployers: how many documented incidents would have been prevented if state-of-the-art moderation systems had been deployed? ## conclusion in this paper, we introduced realharm, a novel dataset and taxonomy of real-world language model application failures derived from documented incidents. this evidence-based approach offers a practical perspective on ai safety risks that complements existing theoretical frameworks. our analysis reveals critical patterns with important implications for ai deployment. misinformation and fabrication represent approximately onethird of documented incidents, confirming that hallucination remains the primary challenge in production llm systems despite significant research attention. notably, reputational damage constitutes 87% of organizational harms, highlighting that business risk from ai deployment is primarily reputational rather than technical. particularly concerning is our finding that over 10% of incidents resulted in complete system shutdown, highlighting a critical gap in incident response capabilities among ai deployers. this pattern suggests that organizations should adopt practices from information security, where comprehensive incident response planning is a fundamental component of defense strategy. such preparation could significantly reduce business disruption and reputational damage when failures inevitably occur. our evaluation of moderation systems exposes substantial limitations in current technical safeguards. even state-of-the-art guardrails detected only a modest percentage of unsafe interactions while introducing significant false positives. this protection gap suggests that while moderation systems provide a valuable first line of defense, they remain insufficient as standalone protection against real-world failures. by grounding ai safety research in documented incidents rather than speculative risks, the re-alharm dataset provides ai deployers with an evidence-based framework for risk assessment, testing, and governance prioritization. we hope this pragmatic approach will complement existing theoretical frameworks and contribute to more effective ai safety practices. future work will focus on expanding the dataset to achieve broader coverage of incidents across different regions and contexts, and extending the framework to encompass agentic and multimodal ai systems.