Empowering P ersian LLM s for Instruction Following: A Novel Dataset and Training Approach (2025)

Abstract: Instruction-tuned large language models have demonstrated remarkable capabilities in following human instructions across various domains. However, their proficiency remains notably deficient in many low-resource languages. To address this challenge, we begin by introducing FarsInstruct: a comprehensive instruction dataset designed to enhance the instruction-following ability of large language models specifically for the Persian language—a significant yet underrepresented language globally. FarsInstruct encompasses a wide range of task types and datasets, each containing a mix of straightforward to complex manual written instructions, as well as translations from the Public Pool of Prompts, ensuring a rich linguistic and cultural representation. Furthermore, we introduce Co-CoLA, a framework designed to enhance the multi-task adaptability of LoRA-tuned models. Through extensive experimental analyses, our study showcases the effectiveness of the FarsInstruct dataset coupled with training by the Co-CoLA framework, in improving the performance of large language models within the Persian context. As of the current writing, FarsInstruct comprises 197 templates across 21 distinct datasets, and we intend to update it consistently, thus augmenting its applicability.

## introduction the modern era of artificial intelligence is marked by numerous breakthroughs, among which is the rise of large language models (llms), such as gpt4 @xcite , llama3 @xcite and palm @xcite . instruction-tuning emerges as a vital technique in the evolution of language models, involving training a model on a wide range of tasks described through natural language instructions. this method diverges from traditional task-specific fine-tuning and adapts the model's behavior to respond to user queries with relevant and helpful answers. this technique offers a more generalized and versatile approach to model training, thus contributing significantly to the advancement of llms. despite the steady progress of instruction-tuned language models, a persistent limitation remains: their difficulty in capturing the nuanced complexities of low-resource languages. this critical challenge stems from the significant gap in the availability of high-quality instruction datasets tailored to these languages. @xcite highlights this concern, demonstrating that datasets lacking sufficient multilingual diversity can cause models to lose previously learned multilingual capabilities, leading to performance degradation. moreover, translating english-centric datasets offers only partial solutions due to several inherent limitations @xcite @xcite . while efforts have been made to compile extensive multilingual instruction datasets @xcite @xcite , gaps remain in creating diverse and complex prompts for languages like persian compared to other languages. in this study, we propose farsinstruct, a comprehensive human-annotated instruction dataset created from existing persian nlp datasets. it includes a mixture of manually written instructions ranging from basic to proficient language levels, alongside translations from the public pool of prompts (p3) @xcite , which is a collection of prompted english datasets. to ensure the diversity and representativeness of farsinstruct, we developed 197 prompt templates derived from 21 distinct public datasets. each prompt template comprises an input template and a target template, both of which function to extract relevant data fields from their respective datasets and reformat them into a unified structure designed for the instruction-tuning figure 1 : an example of the prompts utilized in the training process. the persian version of the prompt is employed for training purposes, while the translated english version is provided to enhance comprehension. the instruction component is highlighted in black, the data fields are marked in orange, and the target answer is indicated in gray. in appendix d, this example is shown in the promptsource environment. the collected public datasets encompass ten different task categories: text summarization, textual entailment, text classification, sentiment analysis, word sense disambiguation, query paraphrasing, question answering, reading comprehension, named entity recognition (ner), and translation. figure 1 depicts an instance of a prompt within our dataset after applying its respective template. a detailed overview of the farsinsturct dataset is provided in section 3. additionally, parameter-efficient fine-tuning (peft) methods, such as low-rank adaptation (lora) @xcite , not only face challenges in multi-task settings but are also prone to catastrophic forgetting @xcite @xcite . to address these issues, we propose co-cola, a novel integration of cola @xcite with rehearsal training @xcite . more specifically, we adopt an iterative optimization framework that merges learned low-rank matrices into the model parameters and reinitializes optimization for new lora modules. at each iteration, we retrain a subset of data from previously learned tasks, mixing it with the current task's data during training. this periodic revisiting of earlier tasks ensures that the model retains performance across both old and new tasks, all while preserving computational efficiency. section 4 presents an in-depth explanation of the co-cola method. in summary, our contributions to advancing persian instruction understanding are threefold: (1) we present farsinstruct, a comprehensive humanannotated instruction dataset for persian, covering varied and representative tasks for different categories such as text summarization, named entity recognition, and translation. (2) we introduce co-cola, a method that combines cola with rehearsal training to mitigate catastrophic forgetting in multi-task learning. (3) we release farsinstruct as an open-source resource, with a commitment to its continued expansion to include a broader range of tasks and modalities foot_0 , foot_1 . ## related work instruction-tuning: instruction tuning refers to the process of training language models using specific input-output pairs derived from diverse data sources. this approach enhances the ability of a pre-trained llm to interpret and respond to a wide range of human requests expressed in natural language. instruction datasets used for this purpose are typically created in one of three ways: parameter effecient fine-tuning: conventional full-parameter fine-tuning becomes computationally impractical as model size and the number of downstream tasks increase. to address this challenge, recent advancements in peft methods advocate for training only a small subset of parameters while leaving the majority of pre-trained model parameters intact. one of the most widely utilized paradigms in peft is low-rank adaptation (lora) @xcite . lora modifies only a small, low-rank portion of the model's weights by incorporating low-rank matrices into the model's weights during the training process. despite the significant computational advantage of lora, it falls short in multi-task adaptation, additionally, @xcite demonstrated that peft techniques, including lora, remain vulnerable to catastrophic forgetting, where models lose previously acquired knowledge when fine-tuned on new tasks. multilora @xcite addresses the limitations of lora by reducing the dominance of top singular vectors, horizontally scaling lora modules, and altering the initialization of adaptation matrices, which leads to improved performance across multiple tasks with minimal additional parameters. mixlora @xcite introduces multiple lora-based experts within a frozen pre-trained model using a top-k routing strategy to efficiently distribute tasks, independently configure attention layer adapters, and apply auxiliary load balance loss, significantly enhancing performance while reducing gpu mem-ory consumption and training latency. further, cola @xcite introduces an iterative optimization framework designed to improve the finetuning of llms by employing multiple iterations of lora. in this paper, we design co-cola to address the issue of catastrophic forgetting, while ensuring an effective multi-task adaption. ## farsinstruct dataset with about 130 million foot_2 speakers, persian -also referred to as farsi in iran -is an important language in the middle east and central asia. farsinstruct represents a project to provide a comprehensive public instruction dataset for the persian community. as of this writing, farsinstruct has 197 carefully designed and created prompt templates for 21 already-published public datasets and some translations from existing prompted datasets. unlike multilingual collections focusing on common tasks such as text summarization and question answering, farsinstruct introduces more task types, including named entity recognition and word sense disambiguation. the creation procedure, statistics, task augmentation, and quality of the dataset are covered in detail in the following subsections. additional illustrations and tables are provided in the appendix b, c, d. ## methodology and experimental setup to maintain our model's robustness and generalization capabilities, we integrate the cola framework @xcite with continual learning @xcite . this section offers a thorough overview of the training procedure and evaluation setup. ## results to investigate the applicability of farsinstruct, we instruction-tuned ava-a llama-3-based persian llm-using the co-cola framework across a suit of tasks. the results were compared with monolingual and multilingual instruction-tuned models, using quantitative evaluations. for a comprehensive overview of the training configuration, please refer to the appendix a. ## discussion figure 5 provides a detailed breakdown of the overall performance reported in ## conclusion this study aims to address the limitations in instruction-following capabilities of language models for persian, an important but underrepresented language, by introducing a novel instruction dataset and a training approach specifically designed to enhance the instruction comprehension of large language models. farsinstruct presents a carefully curated dataset that combines human-annotated instruction data with translations from englishcentric instruction datasets, featuring tasks in different forms and from varying levels of difficulty. further, co-cola leverages the strengths of cola with rehearsal training to mitigate catastrophic forgetting and improve multi-task adaptation, through its iterative optimization framework. our results demonstrate that this allows for sustained model performance over diverse tasks while optimizing computational resources. we hope our dataset fills the critical gap and serves as a valuable resource to the multilingual nlp community.