Towards Inclusive A rabic LLM s: A Culturally Aligned Benchmark in A rabic Large Language Model Evaluation (2025)

Abstract: Arabic Large Language Models are usually evaluated using Western-centric benchmarks that overlook essential cultural contexts, making them less effective and culturally misaligned for Arabic-speaking communities. This study addresses this gap by evaluating the Arabic Massive Multitask Language Understanding (MMLU) Benchmark to assess its cultural alignment and relevance for Arabic Large Language Models (LLMs) across culturally sensitive topics. A team of eleven experts annotated over 2,500 questions, evaluating them based on fluency, adequacy, cultural appropriateness, bias detection, religious sensitivity, and adherence to social norms. Through human assessment, the study highlights significant cultural misalignments and biases, particularly in sensitive areas like religion and morality. In response to these findings, we propose annotation guidelines and integrate culturally enriched data sources to enhance the benchmark’s reliability and relevance. The research highlights the importance of cultural sensitivity in evaluating inclusive Arabic LLMs, fostering more widely accepted LLMs for Arabic-speaking communities.

## introduction arabic, spoken by over 400 million people, ranks among the world's most widely used languages unesco. despite its global prominence, arabic has received limited attention in nlp research, classifying it as a low-resource language @xcite . consequently, arabic nlp models, particularly large language models, are often evaluated on translated datasets that fail to capture the language's rich cultural context @xcite . this reliance on culturally detached benchmarks has led arabic llms to frequently exhibit biases and misalignment, diminishing their effectiveness and cultural adequacy, especially in areas that require cultural sensitivity. given that culture funda-mentally shapes communication and social norms @xcite , it is essential for llms to authentically reflect these nuances to better serve arabic-speaking communities. the reliance on culturally misaligned benchmarks creates a problematic feedback loop: models trained and evaluated on such data are less likely to handle culturally sensitive or nuanced topics, as they are never adequately assessed for these capabilities. consequently, arabic llms may perform well on technical metrics yet fail to resonate with the cultural values and expectations of their target audience @xcite ; @xcite . this disconnect reduces trust in the model's outputs, limiting its usefulness for arabic-speaking users and decreasing wider acceptance of arabic llms @xcite . bridging this benchmarking gap is essential for creating linguistically accurate and culturally relevant arabic resources. to address these challenges, our study undertakes a comprehensive evaluation of the arabic massive multitask language understanding (mmlu) @xcite , a widely recognized benchmark with multiple arabic versions, including machine-translated using gpt-3.5-turbo model @xcite and humantranslated provided by openai foot_0 . the mmlu benchmark has gained popularity for evaluating llms due to its extensive coverage of 57 topics across various fields, providing a robust framework for assessing a model's general knowledge and adaptability across domains. this study emphasizes the critical need to prioritize cultural alignment in the development and evaluation of arabic llms. by focusing on benchmarks and methodologies that reflect the linguistic and cultural intricacies of arabic-speaking communities, our work aims to advance the creation of more inclusive and contextually accurate language technologies. this approach underscores the importance of moving beyond technical performance metrics to ensure that arabic llms are both culturally resonant and widely trusted by their users. ## related work research on cultural values in ai emphasizes designing systems that respect user cultural contexts for improved social acceptability and effectiveness. studies highlight challenges in culturally aligning language models (llms) trained on english language datasets, which may overlook the values of other cultural contexts. jinnai (2024) explores japanese llms aligned with english datasets, finding limitations in capturing japanese moral frameworks and calling for culturally tailored japanese data. @xcite chinese and english, revealing biases that underscore the need for culturally aware ai design with continuous monitoring. @xcite evaluates cultural bias across major llms, noting they often reflect protestant european cultural norms and proposing "cultural prompting" to enhance alignment with diverse regions, though scarce language data remains a challenge. @xcite introduced arabicmmlu, an arabic dataset with 14,575 questions across 40 tasks to evaluate arabic language models, enhancing comprehension in north african and levantine contexts. @xcite presents juhaina, an arabic-english bilingual llm, paired with camel-eval, a benchmark for assessing cultural relevance in arabic llm responses. @xcite acegpt-v1.5, which improves arabic vocabulary handling through progressive vocabulary expansion, enhancing text comprehension and cultural alignment for arabic users. our study focuses on six culturally misaligned topics-human sexuality, moral disputes, moral scenarios, philosophy, world religions, and professional psychology-where cultural sensitivity is particularly critical. to further enhance the benchmark's cultural relevance, we introduced five additional topics uniquely significant to arabicspeaking communities: islamic religion, old arab history, islamic history, arabic ethics, and arabic educational methodologies. a team of eleven experts reviewed over 2,500 questions across these domains, applying detailed criteria covering fluency, adequacy, cultural appropriateness, bias detection, religious sensitivity, and adherence to social norms. this comprehensive evaluation highlights significant cultural misalignments and biases, prompting the development of annotation guidelines and the incorporation of culturally enriched data sources to improve the benchmark's reliability. ## methodology in this work, we critically examine the arabic mmlu benchmark, focusing on its cultural alignment and relevance for evaluating arabic large language models (llms). the original mmlu benchmark is in english, has since been translated into arabic in two versions: one by gpt-3.5 turbo and another by arabic native human translators, both of which are widely used to assess the capabilities of arabic llms. figure 1 presents the various topics included in the mmlu benchmark, categorized by their level of cultural alignment sensitivity. the identified critical misalignment topics frequently lack alignment with arabic cultural norms and values, potentially leading to inaccurate or culturally insensitive outputs in arabic language models. the arabic mmlu benchmark includes over 700 questions on western-centric topics, such as european and u.s. history and u.s. foreign policy, which lack cultural relevance for arabic-speaking communities, rendering them unsuitable for cultural alignment assessments. to address this, we implemented a comprehensive evaluation framework encompassing linguistic and cultural dimensions. linguistic metrics include fluency (naturalness and grammatical correctness) and adequacy (faithfulness in conveying the source text's meaning), both rated on a 1-5 scale. for cultural alignment, we introduced four metrics: cultural appropriateness (sensitivity to cultural nuances), bias detection (presence of various bias types), religious sensitivity (respect for religious beliefs), and social norms (adherence to societal values), each carefully scored or annotated. alongside human evaluation metrics, we employed several automated metrics to quantify translation quality and similarity. these include @xcite , @xcite , @xcite , chrf @xcite , bertscore zhang et al. (2019), and comet rei et al. (2020), which provide insights into linguistic accuracy and fluency. by combining these automated metrics with hu-profession-psychology world religions moral scenarios human sexuality moral disputes philosophy educationa-methodolog-old arab history islamic religion islamic history islamic ethics loading [mathjax]/extensions/mathmenu.js figure 1: arabic mmlu benchmark topics: general, excluded, added, and culturally misalignment topics man evaluations, we established a rigorous and multidimensional framework to support a comprehensive analysis of the benchmark's cultural and linguistic suitability for arabic-speaking communities. this approach allows us to identify key areas of misalignment and provides valuable insights for enhancing arabic nlp models' cultural sensitivity and reliability. lastly, to facilitate a standardized evaluation for arabic llms, we created the index for language models for arabic assessment on multitasks (ilmaam) foot_1 , a dedicated leaderboard that benchmarks performance on the refined arabic mmlu, excluding culturally sensitive topics assessed for alignment. ilmaam serves as a reliable measure of model accuracy across non-critical topics, providing transparency and consistency in arabic llm evaluation. the refined dataset addresses linguistic and cultural misalignments identified in the arabic mmlu benchmark. the updated version, which includes culturally enriched questions, is publicly available on hugging face foot_2 . ## annotation process the annotation methodology involved eleven trained arabic-language experts who independently assessed question subsets to ensure coverage and consistency. three annotators evaluated each topic, with quality checks by researchers to uphold accuracy and guideline adherence. this approach promoted high inter-annotator reliability, minimizing subjectivity for robust evaluations. the cultural alignment assessment was structured to identify subtle and overt cultural misalignments through a multi-step procedure. annotators evaluated fluency, adequacy, cultural appropriateness, and sensitivity using predefined metrics, with regular consensus meetings to refine interpretations. this framework systematically captured cultural biases, offering a comprehensive cultural assessment. for detailed guidelines, see appendix a. ## results our evaluation of the arabic mmlu benchmark identifies key issues in three areas: cultural, methodological and structural, and linguistic (figure 5 ). cultural issues include deficiencies in representing philosophical and ethical foundations and language and expressions, leading to content that may feel culturally misaligned or insensitive for arabic-speaking users. methodological and structural issues reveal inadequacies in structural design and source relevance, affecting content clarity and coherence. linguistic issues highlight translation problems, including grammatical and stylistic errors that reduce readability and authenticity. these findings emphasize the need for a culturally aligned evaluation framework and refined translation methods. subsequent sections provide detailed analyses, including translation scores, similarity metrics, and reviewer assessments. ## discussion the evaluation of the arabic mmlu benchmark highlights foundational challenges across three key areas: linguistic, cultural, and methodological/structural issues. these challenges underscore the limitations of directly translating westerncentric benchmarks for arabic-speaking audiences, emphasizing the urgent need for a more culturally aligned and linguistically coherent approach to developing nlp resources for arabic llms. figure 5 summarizes the primary issues identified, serving as a basis for the discussions and recommendations presented in this study. linguistic issues were prevalent throughout the corpus, impacting clarity and coherence. translafigure 4: distribution of questions in newly added culturally relevant topics cultural issues are evident where western concepts, values, and figures are presented without adaptation, assuming universality and disregarding their relevance to arabic-speaking communities. the corpus includes frequent references to western laws, systems, and historical figures, while notable arab figures and culturally significant examples are notably absent. this lack of cultural resonance weakens the benchmark's relevance for arabic users, as it fails to reflect the linguistic and cultural heritage central to the arabic-speaking world. moreover, the reliance on western terms, examples, and expressions, without providing classical or colloquial arabic alternatives, distances the corpus from arabic cultural and linguistic authenticity. additionally, the inclusion of references to foreign ethnic groups and lineages, as well as cul-model name parameters average score model type qwen/qwen2.5-72b-instruct 72b 73.45 instruction-tuned cohereforai/aya-expanse-32b 32b 63.87 pretrained qwen/qwen2.5-32b-instruct 32b 60.27 instruction-tuned cohereforai/c4ai-command-r-08-2024 32.2b 59.85 pretrained google/gemma-2-9b-it 9b 57.73 pretrained qwen/qwen2.5-7b-instruct 7b 55.57 instruction-tuned methodological and structural issues were also observed, indicating a lack of organization and clear source attribution within the corpus. misplaced questions and a lack of references to arabic sources and statistical research limit the benchmark's relevance and accuracy in arabic contexts. without properly cited sources or organized content, the text may feel less credible, as it does not ground its questions or assumptions in resources or research relevant to the arabic-speaking world. this lack of structural coherence undermines the benchmark's utility, as it risks presenting information or perspectives that may not be applicable or accurate in an arabic cultural framework. ## conclusion this study provides a comprehensive evaluation of the arabic mmlu benchmark, highlighting critical issues in linguistic coherence and cultural alignment that hinder its effectiveness for arabic. results reveal cultural misalignments stemming from an over-reliance on western concepts and a lack of clear arabic source references, all of which reduce the benchmark's cultural relevance and usability. furthermore, the large volume of questions across varied topics poses a challenge for thorough cultural review, as addressing this comprehensively requires a larger team and extended time commitment. these insights underscore the need for a refined benchmark with culturally aligned topics. future work should focus on evaluating arabic llms on culturally tailored benchmarks to assess their performance when engaging with content that resonates with arabic social, historical, and ethical perspectives.