Matina: A Large-Scale 73 B Token P ersian Text Corpus (2025)

Abstract: Text corpora are essential for training models used in tasks like summarization, translation, and large language models (LLMs). While various efforts have been made to collect monolingual and multilingual datasets in many languages, Persian has often been underrepresented due to limited resources for data collection and preprocessing. Existing Persian datasets are typically small and lack content diversity, consisting mainly of weblogs and news articles. This shortage of high-quality, varied data has slowed the development of NLP models and open-source LLMs for Persian. Since model performance depends heavily on the quality of training data, we address this gap by introducing the Matina corpus, a new Persian dataset of 72.9B tokens, carefully preprocessed and deduplicated to ensure high data quality. We further assess its effectiveness by training and evaluating transformer-based models on key NLP tasks. Both the dataset and preprocessing codes are publicly available, enabling researchers to build on and improve this resource for future Persian NLP advancements.

## introduction since the introduction of the transformer architecture @xcite , natural language processing (nlp) has advanced rapidly, transforming many language-related tasks. transformer-based models, like bert @xcite and gpt-2 @xcite , initially focused on tasks like sentiment analysis, translation, and summarization. however, with the development of large-scale language models (llms), such as gpt-3 (brown, 2020) and later models @xcite @xcite , the research shifted towards more complex tasks, including generalization, creative problem-solving, and critical thinking. while english dominates nlp research, there has been a growing effort to curate multilingual datasets @xcite @xcite and develop models capable of understanding multiple languages @xcite @xcite . despite persian being widely spoken, it remains underrepresented in nlp research. although both conventional models and llms can process persian, their performance is often suboptimal, mainly because of the limited availability and poor quality of existing data. persian text data is predominantly sourced from news websites and blogs, which often lack formal or factual content. moreover, no standardized preprocessing pipeline exists to ensure the high quality of persian datasets at the same level as those available for other languages. to address this gap, we introduce the matina corpus, a 72.9 billion token persian dataset designed for training language models. unlike other persian datasets @xcite , the matina corpus has undergone a rigorous and well-designed preprocessing pipeline and a comprehensive deduplication process to ensure its high quality. the dataset includes not only publicly available persian datasets but also introduces newly collected sources to ensure greater diversity and the inclusion of factual information. the diverse sources in the dataset make it suitable both for training large language models and for a variety of downstream tasks that require clean, high-quality persian data. ## related work the scope of our dataset encompasses two key dimensions: (1) the preprocessing steps involved in creating large-scale corpora and (2) the development of extensive text corpora in persian. accordingly, we divide this section into two parts. first, we review notable large-scale corpora available in languages other than persian, along with the preprocessing techniques applied to these datasets. then, we examine and analyze the current state of publicly available persian corpora. ## matina corpus the matina corpus is built from a variety of data sources, each of which is processed based on its specific content characteristics. although these sources are grouped into three main categories, the overall preprocessing pipeline remains consistent, as depicted in figure 1 , with variations primarily in hyperparameters. certain sources, however, demand additional cleaning steps, which are detailed in their respective sections. figure 2 visualizes the distribution of token counts across documents from each source, using a box plot to illustrate the variance in document length. these three categories-web-based crawled data, crawled books and papers, and social media-form the core of our dataset, each with distinct preprocessing requirements. in this section, we describe the data collection process, the preprocessing techniques applied, and the rationale behind the decisions made throughout these steps. ## assessing the impact of the matina corpus a large-scale persian corpus has numerous applications in nlp, including training transformer-based models for tasks such as summarization, sentiment analysis, emotion detection, question answering, sentence embeddings, and text retrieval. additionally, such corpora play a crucial role in pretraining large language models (llms) and generating instructions for llm post-training. to assess the effectiveness of the matina corpus, we conducted experiments on transformer-based model training and continued pretraining of llms. this section provides a detailed discussion of these experiments and their outcomes. ## conclusion in conclusion, the matina corpus provides a crucial resource for advancing persian nlp by addressing the limitations of existing datasets in terms of scale and diversity. with 72.9 billion tokens, it enables the training of more advanced and accurate models for tasks such as machine translation, summarization, and large-scale language modeling. we further demonstrate its effectiveness by training and evaluating transformer-based models on key nlp tasks as well as llm pretraining, highlighting the benefits of high-quality persian data. by making both the dataset and preprocessing tools publicly available, we aim to support further research and foster collaboration in the development of open-source tools and models for persian.