P ersian MCQ -Instruct: A Comprehensive Resource for Generating Multiple-Choice Questions in P ersian (2025)

Abstract: We present PersianMCQ-Instruct, a comprehensive resource that includes a dataset and advanced models for generating multiple-choice questions (MCQs) in standard Iranian Persian, a low-resource language spoken by over 80 million people. This resource features three state-of-the-art models for Persian MCQ generation: PMCQ-Gemma2-9b, PMCQ-Llama3.1-8b, and PMCQ-Mistral-7B. Inspired by the Agent Instruct framework and GPT-4o, we created the dataset by curating over 4,000 unique Persian Wikipedia pages, resulting in three MCQs per page and a total of over 12,000 questions. To ensure the quality of this dataset, we conducted human evaluations and model fine-tuning, both of which demonstrated significant performance improvements in Persian MCQ generation. The dataset and models are publicly available, offering valuable tools for researchers and educators, with particular benefits for advancing Persian-language educational technology.

## introduction generating high-quality multiple-choice questions is essential for various applications, including educational assessments, language learning tools, and automated tutoring systems. these questions efficiently evaluate comprehension and knowledge retention. in natural language processing (nlp), large language models (llms) have significantly enhanced automated text generation and comprehension. this paper introduces a novel method for generating persian multiple-choice questions (mcqs) by fine-tuning llms. inspired by the agent instruct framework @xcite , we developed a high-quality persian multiple choice question (mcq) dataset named persianmcq-instruct to address the shortage of educational resources for the persian language, a low-resource language in the field of nlp. this dataset is crafted to meet rigorous educational standards and supports the growing need for persian language resources in educational technology. persianmcq-instruct, which includes corresponding texts sourced from prominent persian-language wikipedia pages. we then fine-tuned the llm using this dataset. in this study, we significantly contribute to persian mcq generation by introducing the persianmcq-instruct dataset, a comprehensive collection of farsi articles from wikifarsi paired with welldesigned mcqs. this pioneering resource enables the development and evaluation of mcq generation models tailored specifically to the persian language. drawing from the aforementioned framework, we employ a three-step approach for mcq generation from text: content transformation, mcq generation, and mcq refinement. each phase ensures high-quality educational questions. we used the gpt-4o model, resulting in the high-quality persianmcq-instruct dataset. to demonstrate persianmcq-instruct quality and enhance persian mcq generation, we fine-tuned several llmsgemma2-9b-it, llama3.1-8b-instruct, and mistral-7b-instruct-v0.3using persianmcq-instruct dataset. this resulted in improved models: pmcq-gemma2-9b, pmcq-llama3.1-8b, and pmcq-mistral-7b, designed to automate persian mcq and answer generation from educational content. we evaluated the models' performance before and after fine-tuning using the persianmcq-instruct dataset, demonstrating marked improvements and underscoring the effectiveness of the fine-tuning process. finally, we made the persianmcq-instruct dataset and all fine-tuned models publicly available to foster further research and practical application, advancing persian mcq generation and enhancing educational resources. the paper is structured as follows: section 2 reviews relevant literature. section 3 details our methodology. section 4 analyzes experimental results. section 5 concludes and section 6 discusses the limitations of this study. ## related work question generation (qg) is a crucial task in natural language processing that involves automatically creating questions from a given sentence or paragraph. this task is challenging as it requires identifying key statements within the context and generating questions based on them. qg can be categorized into answer-aware and answer-agnostic types @xcite . multiple-choice questions (mcqs) are particularly important in educational settings, where they are widely used in assessments and exams. for languages like persian, where resources are limited, developing mcqs is especially crucial to enhance educational tools and materials. qg and question answering (qa) are interconnected tasks that require reasoning between questions and answers. datasets originally created for qa tasks, such as sciq, @xcite @xcite , are also used in qg research @xcite @xcite . specialized datasets for qg include learn-ingq, khanq, and eduqg, which cover various subjects and educational levels @xcite @xcite . early qg methods relied on rule matching, but advancements have led to the use of seq2seq models with attention, linguistic feature integration, multi-modal models, multi-task learning, reinforcement learning, and language models like bert and gpt-3 @xcite @xcite @xcite @xcite @xcite @xcite . researchers have proposed encoding answers with context, utilizing answer positions, or using text summaries to incorporate answer information in both answer-aware and answer-agnostic qg @xcite @xcite . agentinstruct is an extensible agentic framework designed to automatically generate large volumes of diverse and high-quality synthetic data. this framework leverages raw data sources such as text documents and code files as seeds to create both prompts and responses. the process involves three main stages: content transformation flow, where raw text is transformed into structured content like argument passages or api lists; seed instruction generation flow, where this transformed content is used to generate a comprehensive set of seed instructions; and instruction refinement flow, where these instructions are iteratively refined to enhance quality, diversity, and complexity. by automating these steps, agentinstruct aims to facilitate generative teaching, enabling powerful models to teach new skills or behaviors to other models efficiently @xcite . this framework can significantly aid in generating multiple-choice questions by creating a wide range of high-quality questions and options, thereby enhancing the training data for mcq generation models. in educational contexts, controlling question difficulty is crucial for effective education, with methods assessing difficulty based on answerability, inference steps needed, or learners' abilities @xcite @xcite . aligning questions with the syllabus is important for test focus, leading to studies training classifiers or ranking models to determine question relevance @xcite . personalized education requires generating customized questions for students, prompting the development of knowledgetracking models based on student answer histories or few-shot knowledge-tracking models incorporating sequences of student states and questions @xcite . previous works in developing educational tools with llms have also focused on this aspect @xcite , @xcite and @xcite . significant advancements have been achieved in educational technology for specific languages. notably, a turkish mcqs generator has been successfully developed @xcite . additionally, inspired by self-instruct methods, several works have explored various languages, including turkish, arabic, english, and italian. @xcite , @xcite and @xcite . in the realm of persian natural language processing, notable works include parsinlu, which covers challenges in reading comprehension, multiplechoice question-answering, textual entailment, sentiment analysis, question paraphrasing, and machine translation @xcite . persian-mind achieved state-of-the-art results on the persian subset of the belebele benchmark and the parsinlu multiple-choice qa task @xcite . efforts to develop qa systems for persian have involved translating english qa datasets, but this approach often fails to produce high-quality annotated data due to translation imperfections. there are few open-domain qa datasets for persian. for instance, abadani et al. @xcite translated squad into persian, creating parsquad, and kazemi et al. @xcite developed persianquad, a native dataset where annotators pose questions and specify answers within paragraphs. despite these advancements, there remains a critical gap in the generation of text augmentation and multiple-choice questions with answers in persian. existing works have not adequately addressed the need for comprehensive and high-quality resources in this area. to fill this void, our study introduces the persianmcq-instruct dataset, specifically designed for generating multiple-choice questions in persian. furthermore, we present several fine-tuned models, including pmcq-gemma2-9b, pmcq-llama3.1-8b, and pmcq-mistral-7b, tailored for generating mcqs from text in persian. this work is essential for advancing natural language processing in the persian language, offering valuable resources for educational applications and language assessment tools. by addressing the current limitations, our contributions aim to significantly enhance the quality and availability of persian mcq datasets, thereby fostering further research and development in this field. ## methodology in this study, we introduce the development of an advanced persian educational multiple-choice question (mcq) generator, leveraging state-of-theart large language models (llms). we have curated an extensive dataset, named persianmcq-instruct, which includes multiple-choice questions derived from persian texts. to generate and evaluate persian mcqs utilizing the persianmcq-instruct dataset, we fine-tuned a variety of llms across multiple scenarios, focusing on the multiplechoice format. the models optimized in this process included llama3.1-8b-instruct, gemma2-9b-it, and mistral-7b-instruct-v0.3. this section outlines the methodologies employed for dataset generation and model fine-tuning, providing a detailed account of the procedures followed to establish an effective persian mcq generator. figure 1 presents the comprehensive methodology applied in this study. ## experiments this section outlines the experimental procedures used to fine-tune large language models 17 llama3 github repository 18 mistral github repository 19 gemma2 github repository (llms) to improve their performance in generating multiple-choice questions (mcqs) in persian. the persianmcq-instruct dataset, created as detailed in section 3, served as the foundation for this process. we employed this dataset to fine-tune three distinct llms: llama-3.1-8b-instruct, mistral-7b-instruct-v0.3, and gemma2-9b-it. the dataset was divided into two subsets for the fine-tuning phase. the first subset was allocated for training, consisting of 12,000 mcqs and 4,000 unique texts (as previously noted, we generated three different questions from each text). the second subset, used for evaluation, consisted of 476 mcqs and 159 texts. this set was assessed using both automated metrics and human evaluation methods. the specific objective was to evaluate each model's capability to generate mcqs from persian texts. ## conclusion in summary, this paper introduces the persianmcq-instruct dataset, a comprehensive collection containing over 4000 unique texts and more than 1200 multiple-choice questions (mcqs) in persian. this dataset provides both text content and corresponding mcqs in persian. we rigorously evaluated the quality of this dataset through human assessment and automatic metrics, validating its reliability and effectiveness. moreover, we fine-tuned three different small-sized language models (llms) ranging from 7 billion to 9 billion parametersincluding gemma2-9b-it, llama3.1-8b-instruct, and mistral-7b-instruct-v0.3using this dataset. the resulting models, pmcq-gemma2-9b, pmcq-llama3.1-8b, and pmcq-mistral-7b, demonstrated a significant improvement in generating high-quality persian mcqs. this underscores the dataset's utility and potential impact. our models and dataset are publicly available, paving the way for various educational applications in the persian language. in this work, we also help tackle low-resource languages, improving the persian language model. looking ahead, we plan to expand the dataset with a greater focus on educational content across diverse subjects such as mathematics, physics, and history. additionally, we aim to extend this initiative to other languages, broadening its applicability and impact.