Matina: A Culturally-Aligned P ersian Language Model Using Multiple L o RA Experts (2025)

Abstract: Large language models (LLMs) are powerful tools for a variety of applications, but to interact effectively with users, they must align with the cultural values and linguistic nuances of their audience. However, existing LLMs often fall short in adequately modeling underrepresented languages and cultures, such as Persian, limiting their applicability and acceptance. To address this, we construct diverse, high-quality datasets specifically tailored to Persian linguistic and cultural contexts, ensuring a more authentic and context-aware training process. Using these datasets, we develop Matina, a Persian-focused multi-expert model designed to embody Iranian cultural values and linguistic structures. Matina is trained by fine-tuning LLaMA3.1 8B-Instruct models across five domains: culinary, tourism, socio-culture, translation, and summarization. These experts are combined using a classifier to create a unified multi-expert system. By leveraging culturally aligned datasets, Matina outperforms baseline models in both task performance and user satisfaction, demonstrating the importance of data-driven cultural adaptation in LLM development.

## introduction large language models (llms) play a vital role in everyday life, assisting users by understanding their intentions and aligned with their demographics, beliefs, and culture. however, studies have shown that models like gpt and llama often reflect western @xcite and american values @xcite , largely due to the predominance of english in their training data @xcite and development in english-speaking regions @xcite . we define cultural alignment as ensuring ai systems reflect user values and norms, drawing from sociological and anthropological perspectives @xcite . this alignment helps prevent biases that lead to misrepresentation or cultural insensitivity, particularly in moral values, legal frameworks, and traditions. despite advancements in multilingual llms, persian remains underrepresented, even though it is spoken by over 110 million people foot_0 . more than just a language, persian carries a rich cultural heritage shaped by centuries of literature and traditions. however, the lack of high-quality, culturally relevant data limits llms' ability to capture its nuances, reducing their effectiveness in persian-speaking contexts. to bridge the gap between llms and persian culture and language, we introduce matina, a persian-focused language model specifically designed to reflect iranian values and linguistic nuances. matina is trained on a large dataset of 264,980,480 tokens and over 42,097 cultural documents for pretraining, supervised fine-tuning, and human alignment, ensuring that persian cultural and linguistic structures are deeply embedded in its foundation. our work focuses on three representative cultural areas--culinary domain, tourism, and sociocultural contexts--while also targeting essential language tasks for persian users, including english-persian translation and text summarization. to build our datasets, we employ a multi-faceted approach, including translating existing high-quality resources, collecting domain-specific persian documents, generating culturally relevant content using gpt, and applying data augmentation techniques to ensure diversity and inclusively. we fine-tune the llama3.1 8b-instruct model on these datasets in various experimental configurations and evaluate performance using llm-based evaluations and human assessments. our results show significant improvements in cultural under-standing, task-specific performance, and consistency in persian-language outputs compared to baseline models, confirming the efficacy of our strategy. the rest of the paper is organized as follows: section 2 summarizes previous research on cultural and linguistic alignment approaches and datasets. section 3 describes the dataset preparation and training strategies used. section 4 presents evaluation results, and section 5 discusses limitations and future work. additional details are in the appendix. ## related work large language models (llms) have progressed from monolingual @xcite to multilingual systems @xcite @xcite , meeting a wide range of linguistic needs. however, cultural knowledge frequently falls behind linguistic proficiency, with models reflecting biases and stereotypes in training data @xcite @xcite . this is particularly problematic for conversational agents, which may produce culturally insensitive responses in multilingual regions.in this section, we look at alignment techniques, dataset construction, and advancements in the focal domains. ## methodology as mentioned earlier, our work focuses on three culture-representative domains and two task-based domains that could benefit the persian-speaking community. we first define each domain in section 3.1, then detail the data creation methods employed for each area, followed by an overview of the training methodologies applied. ## model evaluation evaluating cultural alignment in language models is challenging due to the lack of statistical bench-marks and established standards. to address this, we employed three evaluation methods: (1) human as a judge, (2) llm as a judge, and (3) human satisfaction in multi-experts interaction. these methods aim to assess how effectively the models reflect the target culture's knowledge, norms, and values, with human preference serving as a central measure of performance compared to baseline models. as described in section 3.3, we used llama3.1-instruct 8b as the base model for training the experts, making it our primary baseline. to validate our datasets and training procedures, we also finetuned llama3.1-instruct 70b and incorporated it into the same classifier-based system, establishing it as an additional baseline. results for fine-tuning gemma2 @xcite are also provided in ## conclusion in this paper, we introduced matina, a persianfocused multi-expert language model designed to bridge the gap between llms and persian linguistic and cultural representation. our approach involved creating a diverse instruction-tuning and preference optimization dataset across five key ar-eas: culinary, tourism, socio-culture, translation, and summarization. to ensure cultural alignment, our data curation focused on culturally rich documents. we applied this dataset to train models such as llama-3.1-instruct-8b and gemma-2-9b through a three-stage process: pretraining, supervised fine-tuning, and preference optimization. additionally, we leveraged a multi-expert architecture to enhance inference accuracy and memory efficiency. evaluation results demonstrate that matina outperforms baseline models in both task performance and user satisfaction, underscoring the effectiveness of our approach in developing culturally aligned ai models.