A ra S afe: Benchmarking Safety in A rabic LLM s (2025)

Abstract: We introduce AraSafe, the first large-scale native Arabic safety benchmark for large language models (LLMs), addressing the pressing need for culturally and linguistically representative evaluation resources. The dataset comprises 12K naturally occurring, human-written Arabic prompts containing both harmful and non-harmful content across diverse domains, including linguistics, social studies, and science. Each prompt was independently annotated by two experts into one of nine fine-grained safety categories, including ‘Safe/Not Harmful’, ‘Illegal Activities’, ‘Violence or Harm’, ‘Privacy Violation’, and ‘Hate Speech’. Additionally, to support training classifiers for harmful content and due to the imbalanced representation of harmful content in the natural dataset, we create a synthetic dataset of additional 12K harmful prompts generated by GPT-4o via carefully designed prompt engineering techniques. We benchmark a number of Arabic-centric and multilingual models in the 7 to 13B parameter range, including Jais, AceGPT, Allam, Fanar, Llama-3, Gemma-2, and Qwen3, as well as BERT-based fine-tuned classifier models on detecting harmful prompts. GPT-4o was used as an upper-bound reference baseline. Our evaluation reveals critical safety blind spots in Arabic LLMs and underscores the necessity of localized, culturally grounded benchmarks for building responsible AI systems.

## introduction recent advances in large language models (llms) have shown impressive capabilities across many nlp tasks, but concerns around safety and harmful outputs persist, especially in underrepresented languages like arabic. while english benchmarks for safety and toxicity detection abound, including realtoxicityprompts (gehman et al., 1 warning: this paper includes example data points that could be considered offensive, harmful, or reflect biases. 2020), toxigen @xcite , safety-bench @xcite and implicitly abusive language @xcite , there is a lack of equivalent resources in arabic, exacerbated by the cultural, dialectal, and script-based variability. this paper introduces arasafe, a benchmark for evaluating the safety of arabic llms. the benchmark includes: the contributions of our study involve: (i) we create the first large-scale arabic safety benchmarking dataset originating naturally from diverse arabic users. the prompts in the natural dataset are classified by two experts as safe or into one of eight harmful classes, with all annotation discrepancies resolved. we release the data on github foot_0 ; (ii) to support training safety classifiers, we develop two prompting approaches that we use with gpt-4o to generate harmful synthetic data, and we show the effectiveness of this data when used to fine-tune bert-based classifiers; (iii) we benchmark four prominent arabic-centric instruction-fine-tuned llms: jais-13b, acegpt-v2-8b, allam-7b, and fanar-1-9b, in addition to three capable multilingual chat models with comparable model sizes: llama3.1-8b, gemma-2-9b, and qwen3-8b. we show the vulnerability that most of these models exhibit in detecting unsafe content; and (iv) we analyze model outputs for insights on improving the detection of unsafe responses. while this study focuses on safety detection in arabic, a similar approach can easily be applied to other underrepresented or low-resource languages to assess and improve llm safety. multilingual and low-resource safety evaluation. studies of cross-lingual safety reveal that models often underperform on non-english prompts. rtp-lx (de wynter et al., 2025) performs a toxicity evaluation for llms across 28 languages, finding low agreement between llms and humans on the toxicity of subtle-yet-harmful content. on the other hand, polyguard @xcite and its polyguardmix corpus combine in-the-wild and translated adversarial prompts across 17 languages to strengthen guardrails. crosslingual bypass studies show that translating harmful inputs to arabic can evade english-centric filters, and low-resource languages like arabic see higher rates of unsafe outputs due to limited safety training data (activefence, 2023). arabic-specific moderation resources. in contrast to english and chinese, arabic has only seen narrow or small-scale harmful-content corpora. @xcite presented work on abusive and obscene language detection in arabic social media content, while levantine hate speech and abusive language dataset @xcite only covered 'hate', 'abuse' and 'neutral' labels for tweets. a number of shared tasks on offensive language and hate speech focused on limited unsafe categories @xcite . general evaluation suites like alue @xcite include coarse offensiveness checks but do not address fine-grained safety dimensions. recent efforts such as aratrust @xcite introduce trustworthiness evaluations with a small scale dataset of 522 multiple-choice questions on ethics and safety, with themes like privacy violation, illegal activities and offensive language, while arabic safeguard @xcite localized a chinese safety dataset to arabic, injecting region-specific harm categories to evaluate model bias across social perspectives. to our knowledge, no naturallyoccurring, multi-class arabic safety benchmark exists at scale. arasafe addresses this need. ## conclusion we introduced arasafe, the first large-scale benchmark to evaluate the safety of arabic llms. it includes 12k human-written arabic prompts across a broad range of domains and dialects, annotated into nine fine-grained safety categories by expert annotators. to ensure a rich representation of harmful content, we complemented the dataset with 12k synthetic gpt-4o generated harmful prompts. we conducted a comprehensive benchmarking of a variety of arabic and multilingual models. our findings highlight notable safety vulnerabilities in current llms, emphasizing the importance of localized and culturally aware safety benchmarks for evaluating and improving the behavior of these systems in real-world applications. we outline several future directions for extending this work, such as covering underrepresented arabic dialects to ensure fair evaluation across regions; adopting safety mitigation strategies, including instruction tuning and rlhf tailored for arabic; and integrating trained classifiers into moderation tools and ai safety layers for llm deployment. we propose leveraging llms as initial annotators to identify potentially harmful prompts, followed by human validation. this approach can increase the percentage of harmful content in the dataset.