ELAB : Extensive LLM Alignment Benchmark in P ersian Language (2025)

Abstract: This paper presents a comprehensive evaluation framework for aligning Persian Large Language Models (LLMs) with critical ethical dimensions, including safety, fairness, and social norms. It addresses the gaps in existing LLM evaluation frameworks by adapting them to Persian linguistic and cultural contexts. This benchmark creates three types of Persian-language benchmarks: (i) translated data, (ii) new data generated synthetically, and (iii) new naturally collected data. We translate Anthropic Red Teaming data, AdvBench, HarmBench, and DecodingTrust into Persian. Furthermore, we create ProhibiBench-fa, SafeBench-fa, FairBench-fa, and SocialBench-fa as new datasets to address harmful and prohibited content in indigenous culture. Moreover, we collect extensive dataset as GuardBench-fa to consider Persian cultural norms. By combining these datasets, our work establishes a unified framework for evaluating Persian LLMs, offering a new approach to culturally grounded alignment evaluation. A systematic evaluation of Persian LLMs is performed across the three alignment aspects: safety (avoiding harmful content), fairness (mitigating biases), and social norms (adhering to culturally accepted behaviors). We present a publicly available leaderboard that benchmarks Persian LLMs with respect to safety, fairness, and social norms.

## introduction the rapid advancement of large language models (llms) has raised concerns regarding their alignment with human values, particularly in non-english languages. while research on llm alignment has focused on english, there is a growing need for frameworks applicable to other languages, like persian. persian llms face unique challenges due to linguistic structures, cultural nu-1 leaderboard ances, and ethical considerations that differ from english-speaking contexts. existing alignment frameworks, such as @xcite and harmbench @xcite , are essential for identifying harmful outputs and biases, but they are primarily developed for english. persian llms require a tailored approach due to their gender-inflected grammar and culturally specific norms, such as deference to authority ('taarof') and social dignity ('aberoo') @xcite , which differ from western standards. these unique cultural aspects must be considered to prevent harmful stereotypes and biases in persian llms. this study builds on previous multilingual nlp and ai ethics work by adapting well-known alignment datasets into persian. by creating persian versions of benchmarks like @xcite , advbench @xcite , harmbench @xcite , and decodingtrust @xcite , it lays the foundation for evaluating persian llms' safety, fairness, and alignment. we introduce five new datasets, prohibibench-fa, safebench-fa, fairbench-fa, socialbench-fa, and guardbench-fa, that address ethical issues specific to persian language models, providing a transparent and systematic framework for cross-model comparisons @xcite . also, it bridges the mentioned gaps by (1) adapting safety frameworks: using methodologies from red teaming language models and harmbench @xcite to design persian-specific red-team prompts and refusal evaluations. (2) extending fairness metrics: incorporating bbq's @xcite bias taxonomy while adding persian-centric dimensions (e.g., dialect fairness, gender inflection bias). (3) defining persian social norms: proposing criteria inspired by decodingtrust @xcite but grounded in persian sociology (e.g., politeness hierarchies, familial honor). (4) unifying bench-marks: combining safety (xstest @xcite ), fairness (bbq), and norms into a single framework, addressing interdependencies (e.g., overly strict safety filters exacerbating dialect bias). this work aims to develop a comprehensive framework for evaluating persian llms, focusing on safety, fairness, and social norms. the primary contribution is introducing a culturally grounded alignment evaluation framework for persian llms, which bridges the gap between western-centric frameworks and the unique challenges posed by persian. this work offers a scalable, transparent evaluation method for assessing the safety, fairness, and social norms of persian llms since these three aspects constitute culturally salient dimensions in persian linguistic and cultural frameworks. the detailed contributions of this paper are as follows: ## related works despite notable advancements in persian natural language processing (nlp), progress has largely focused on developing models for specific tasks such as parsbert @xcite and beheshti-ner @xcite or evaluating model performance on benchmarks like parsinlu @xcite and persianmmlu @xcite . however, the critical area of alignment in persian remains unexplored. while various studies on alignment exist, including safetybench @xcite and harm-bench @xcite , they are predominantly designed for english. these approaches face several limitations: (1) reliance on western cultural norms, (2) disregard for the linguistic intricacies of persian, and (3) treating safety, fairness, and norms as independent aspects rather than interconnected factors. prior alignment evaluation frameworks focusing on the english language can be broadly categorized into (1) safety evaluation, (2) fairness evaluation, and (3) social norms and trustworthiness evaluation that are explained below. safety evaluation: @xcite systematizes adversarial testing methods to expose harmful outputs, while safetybench @xcite and salad-bench @xcite provide hierarchical, multi-dimensional safety benchmarks (e.g., misinformation, illegal advice). harmbench @xcite standardizes automated red-teaming and refusal robustness, and xstest @xcite identifies exaggerated safety behaviors (e.g., over-refusals in benign queries). work like @xcite studies jailbreak prompts that bypass safety guardrails, and @xcite demonstrates cross-model exploitability of alignment vulnerabilities. these highlight the need for rigorous safety testing, but their focus on english limits applicability to persian. while safety-bench @xcite and salad-bench @xcite include limited multilingual tasks, they do not address persian-specific challenges. persian llms face unique risks, such as generating harmful content that leverages regional taboos or dialect-specific slang. adversarial attacks from @xcite may not transfer to persian due to script/logical differences. persian hate speech datasets exist but lack llm-focused red-teaming protocols. fairness evaluation: the bbq benchmark @xcite evaluates social biases in questionanswering tasks, measuring stereotyping across demographics. similarly, decodingtrust @xcite assesses fairness as part of a broader trustworthiness evaluation, identifying disparities in gpt's treatment of marginalized groups. while these frameworks quantify bias, they lack adaptations for persian linguistic structures (e.g., genderneutrality challenges) or cultural contexts (e.g., regional dialects). social norms and trustworthiness: decod-ingtrust @xcite also evaluates normative alignment, such as ethical reasoning and compliance with societal expectations. however, social norms are culturally specific (e.g., politeness strategies in persian differ vastly from english), and no existing benchmark explicitly adapts these criteria for non-english languages. existing frameworks like decodingtrust @xcite use english-centric normative anchors (e.g., individualism vs. collectivism), misaligning with persian cultural values (e.g., 'taarof' rituals). no work evaluates how persian llms handle norms like 'aberoo' (social dignity). ## evaluation to evaluate the alignment of large language models (llms), we developed a systematic methodology involving model deployment, response generation, and scoring using the llm-as-a-judge approach. the evaluation process consists of the following steps: additionally, a subset of questions was created specifically to assess cultural alignment with persian cultural subjects. to account for this cultural focus, four distinct system prompts were designed for the llm-as-a-judge method. ## conclusion this work develops a unified framework for evaluating persian llms, integrating safety, fairness, and social norms, and adapting existing benchmarks to persian. moreover, by creating new persian datasets, we ensure culturally relevant evaluations and promote the responsible development of persian llms. this research emphasizes the importance of culturally methods for llm alignment and lays the foundation for future multilingual ai alignment efforts.