JAWAHER : A Multidialectal Dataset of A rabic Proverbs for LLM Benchmarking (2025)

Abstract: Recent advancements in instruction fine-tuning, alignment methods such as reinforcement learning from human feedback (RLHF), and optimization techniques like direct preference optimization (DPO), have significantly enhanced the adaptability of large language models (LLMs) to user preferences. However, despite these innovations, many LLMs continue to exhibit biases toward Western, Anglo-centric, or American cultures, with performance on English data consistently surpassing that of other languages. This reveals a persistent cultural gap in LLMs, which complicates their ability to accurately process culturally rich and diverse figurative language, such as proverbs. To address this, we introduce *Jawaher*, a benchmark designed to assess LLMs’ capacity to comprehend and interpret Arabic proverbs. *Jawaher* includes proverbs from various Arabic dialects, along with idiomatic translations and explanations. Through extensive evaluations of both open- and closed-source models, we find that while LLMs can generate idiomatically accurate translations, they struggle with producing culturally nuanced and contextually relevant explanations. These findings highlight the need for ongoing model refinement and dataset expansion to bridge the cultural gap in figurative language processing.

## introduction instruction fine-tuning @xcite has significantly enhanced the creativity and customizability of llms, while alignment techniques, such as rlhf @xcite and dpo @xcite , have improved the ability of these models to align with user preferences. the vast reservoir of cultural knowledge embedded within llms, combined with the potential of these alignment techniques, has sparked interest in how llms * equal contribution figure 1 : overview of the jawaher benchmark, featuring arabic proverbs along with their idiomatic translations and explanations in both english and arabic; for instance, the libyan proverb whose literal meaning is "the son of a snake is a serpent", is explained as conveying the idea that "like father, like son in terms of bad behavior and twisted character". figurative (i.e., non-literal) language @xcite @xcite rich with implicit cultural references, varies significantly across cultures and relies on shared cultural knowledge. differences in figurative expressions between languages often arise from unique cultural values, historical context, and other regional factors. thus, understanding figurative language depends on grasping culturally significant concepts and their associated sentiments @xcite . to ensure llms are inclusive and deployable across regions and applications, it is essential for these models to function adequately in diverse cultural contexts @xcite . among common types of figurative language, proverbs are a key form of cultural expression, encapsulating diverse areas of human knowledge and experience @xcite . despite their widespread use, proverbs are linguistically complex due to their cultural significance and structure, exhibiting notable lexical and syntactic variations @xcite , making their automatic identification in text challenging for nlp systems. recent research aiming to study cultural representation and inclusion in llms has found that many models remain strongly biased towards western, anglo-centric, or american cultures, with performance on english data still surpassing that of other languages @xcite . this highlights the existence of significant 'culture gaps' @xcite in llms, which further complicate their ability to accurately handle culturally rich and diverse forms like proverbs. in this paper, we make several contributions toward closing the cultural gap in llms by introducing jawaher, which facilitates the building and assessment of models designed to understand figurative language, specifically arabic proverbs, as follows: ## related works figurative language. prior work on figurative language understanding has covered a wide range of topics, including simile detection and generation @xcite @xcite . results show that, although language models can often recognize non-literal language and may attend to it less, they still struggle to fully capture the implied meaning of such phrases @xcite . this has led to more recent studies shifting toward tasks focused on comprehending figurative language @xcite @xcite . efforts in dataset and task development have focused on diverse tasks such as recognizing textual entailment, multilingual understanding, and figurative language beyond text @xcite @xcite . however, the lack of comprehensive datasets focusing on figurative language, particularly arabic proverbs, limits the ability to thoroughly evaluate llms' cultural awareness and their true understanding of culturally embedded non-literal expressions, further motivating our work. more details regarding dataset and task development on figurative language are provided in appendix a. ## jawaher jawaher consists of arabic proverbs paired with their idiomatic or literal english translations, along with explanations in both arabic and english, covering 20 different arabic varieties. below, we outline the analysis of jawaher, focusing on its coverage of dialects, themes, cultural context, and the tasks it facilitates. ## experimental setup we evaluate jawaher using both open-and closed-source state-of-the-art multilingual llms (mllms) to assess their abilities across our proposed tasks. the models are tested in a zero-shot setting @xcite , allowing us to evaluate their inherent capacity to interpret, explain, and contextualize arabic proverbs. to achieve this, we create a universal prompt template in english: (1) we set the role of the model as a language expert with deep knowledge of arabic proverbs, cultural history, and literary meanings. (2) we deliver the test input to the model to produce the output. (3) we provide the name of the task that needs to be performed. (4) we specify the context, asking to include any relevant background stories or cultural context that could be helpful for explanations on the tasks. (5) finally, we define what should be the expected outcome of the model. the prompt used to evaluate mllms can be found in figure 5 . ## results & discussion ## conclusion we introduced jawaher, a new benchmark for figurative language understanding in arabic proverbs. our manually curated dataset features proverbs from various arabic dialects, paired with idiomatic translations and explanations, facilitating the development and evaluation of models capable of translating and explaining figurative language. our experiments show that while both open-and closedsource mllms are able to provide idiomatically correct translations, they struggle with the deeper challenge of comprehending figurative language, particularly when generating detailed explanations that capture the stories behind the proverbs, dialectspecific nuances, and historical or cultural contexts discussed in our dataset. these findings highlight the need for continuous model improvements and dataset enrichment to address the complexities of fully understanding proverbs. we hope our work will inspire further interest in figurative language understanding, particularly across diverse arabic dialects.