P er C ul: A Story-Driven Cultural Evaluation of LLM s in P ersian (2025)

Abstract: Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios.Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut. We evaluate several state-of-the-art multilingual and Persian-specific LLMs, establishing a foundation for future research in cross-cultural NLP evaluation. Our experiments demonstrate a 11.3% gap between best closed source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. You can access the dataset from here:https://huggingface.co/datasets/teias-ai/percul

## introduction effective interactions between users from diverse backgrounds and llms are contingent on outputs that are culturally relevant @xcite . as the use of generative artificial intelligence increases to expedite and automate personal and professional tasks, the cultural values embedded in ai models may inadvertently bias people's authentic expression and perpetuate the dominance of certain cultures @xcite , particularly western culture, which is over-represented in englishdominated training data @xcite . this highlights the importance of creating culture-specific benchmarking tools to assess * equal contribution, ordered randomly the extent to which llms encapsulate knowledge about particular cultures. despite the numerous benchmarks that evaluate various aspects of llms @xcite , a significant gap remains in assessing their knowledge of culture across many non-english languages such as persian. although some efforts have been made to create llm benchmarks for the persian language, focusing on reading comprehension and scientific knowledge @xcite @xcite , cultural benchmarks specifically tailored for persian are limited, either concentrating on specific aspects such as social norms @xcite or being constrained by size @xcite . to address the gap in evaluating the sensitivity of llms to persian culture, we introduce percul, a carefully curated dataset featuring multiple-choice questions. in percul, cultural concepts are subtly embedded in short stories (see figure 1 ). cultural phenomena often manifest implicitly through interactions between individuals, which can be effectively conveyed through the medium of a short story @xcite . unlike previous benchmarks, percul is specifically curated for persian demographic, avoiding irrelevant or overly generalized concepts shared by other cultures @xcite . furthermore, the dataset is resistant to use translation as a proxy @xcite . unlike other similar datasets that rely on llms for generation @xcite , percul leverages input from diverse native persian annotators, ensuring broader knowledge coverage. llms are used only for generating storylines based on our handcrafted data, with human editing involved to ensure factual accuracy and to prevent hallucinations. to establish baselines, we evaluate several recent models from different families, including meta llama 3.x @xcite , openai gpt (team, 2024), anthropic claude @xcite , as well as state-of-the-art persian-specific models, namely persianmind-v1.0 @xcite and dorna-llama3-8b-instruct @xcite . our experiments reveal a gap between models understanding of persian culture and layperson baseline. we also demonstrate that translating per-cul results in a significant drop in model's performance. furthermore, we observe that persian finetuned llms perform worse than their respective multilingual base models, which may result from a low-quality, small training set. lastly, our comprehensive error analysis highlights a limitation in llms: they often rely on surface-level details rather than synthesizing contextual clues when it comes to identifying specific cultural concepts. most existing persian benchmarks focus on language understanding tasks such as textual entailment and question answering @xcite @xcite , or the evaluation of factual/scientific knowledge of llms @xcite . for instance, the khayyam-challenge @xcite proposes a set of 20k persian questions divided into 38 tasks, but these tasks are mainly schoollevel examinations, primarily covering mathematical and scientific subjects. although this is useful for evaluating the capabilities of llms to solve scientific problems in persian, it fails to assess llms' understanding of persian culture. this also applies to the work of @xcite which introduces two new datasets to evaluate llm abilities in solving persian mathematical and scientific questions. among works on persian culture, psn (saffari et al., 2024) provides pairs of social norms and contexts along with a label for each pair describing the appropriateness of each pair. however, it is limited to social norms, leaving out other important aspects such as visible behavior or rituals. blend @xcite and @xcite are multi-cultural datasets that despite the inclusion of certain questions about persian culture, present crucial limitations. blend primarily features questions that focus on non-persian cultural events and traditions, such as thanksgiving or christmas, making it less relevant for assessing cultures where these events are not celebrated, such as persian. culturalbench, while contains question relevant to persian culture, is small in size. ## percul construction the process of creating percul consists of multiple steps, as shown in figure 2 . briefly, the creation process begins (1) by identifying cultural categories based on hall's triad of culture @xcite . (2) then, native annotators generate descriptive facets, seed topics, and metadata for these categories. (3) using this metadata, llms produce storylines. (4) these storylines undergo rigorous human correction and selection. (5) llms also create comprehension options guided by careful human-crafted heuristic rules, followed by human correction and selection. the resulting dataset features culturally relevant persian story comprehension questions in multiple-choice format. these questions subtly incorporate cultural elements from various categories, informed by human-generated metadata, without directly referencing the cultural concepts. ## experiments we perform a comprehensive series of evaluations on our dataset using state-of-the-art closedsource and open-weight models, as depicted in figure 4. we also assess two persian open-weight llms, namely persianmind-v1.0 @xcite and partai dorna-llama3-8b-instruct (par-tai, 2024), which are aimed to enhance persian language and cultural understanding by further pretraining & fine-tuning on corpora with dominant persian data. to ensure the reproducibility of our experiments, all models utilize zero temperature and allowed to generate up to their maximum generation length. we employ the same prompts for each question across all models, which can be found in appendix c. 1 the results are presented in accuracy and model size. figure 4 shows the performance variation of models with their size (in terms of parameters). as can be seen, there is a clear positive correlation between the number of parameters and the accuracy of models within each model family. however, this relationship does not hold across different families. for instance, llama 3.1 405b, while being the top model in the llama family, its performance is close to command-r-plus, despite being nearly four times larger. similarly, gemma 2 9b's, which is comparable in accuracy to models 10x and 40x larger in terms of parameter count. these differences imply that architectural advancements and data efficiency might have a more substantial impact than mere size. as persian was not the target language for any of these multilingual llms, this outcome is expected due to the unknown quantity and quality of persian data in their training set. persian (fine-tuned) models. another interesting observation, as depicted in figure 6 , is the negative impact of fine-tuning on persian models. both persian models exhibited lower performance compared to their corresponding base models. although it's fair to note that persianmind refused to answer most of the questions by stating "the answer is not available in the provided options". this could be attributed to the quality of the finetuning data which may have introduced noise or caused overfitting, resulting in a decline in the models' ability to generalize effectively. this finding prompts further investigation into the quality and relevance of the fine-tuning data, which we propose as a direction for future research. ## conclusion in this paper, we introduced percul, a carefully curated dataset designed to assess llms' sensitivity towards persian culture. our dataset is nontrivial, as it employs implied concepts within conversations or story scenarios, rendering translation ineffective for solving our benchmark. the experiments demonstrated a significant performance gap between open-weight and closed-source llms for persian culture. we also showed that the knowledge of persian culture in llms is not dependent on the number of parameters when comparing interfamily models, whereas parameter count plays a crucial role in intra-family models. lastly, our experiments revealed that current state-of-the-art persian-specific llms fall short and even degrade in performance, when compared to their original base models, emphasizing the need for more effective methods, models, and higher-quality datasets to train persian-specific llms. for future research, we suggest studying llms based on the final level of culture, namely informal level, where categories are more subjective. one potential approach could involve assigning personalities to each llm and observing their behavior in a simulated environment to evaluate the third level of cultural understanding.