F a BERT : Pre-training BERT on P ersian Blogs (2025)

Abstract: We introduce FaBERT, a Persian BERT-base model pre-trained on the HmBlogs corpus, encompassing both informal and formal Persian texts. FaBERT is designed to excel in traditional Natural Language Understanding (NLU) tasks, addressing the intricacies of diverse sentence structures and linguistic styles prevalent in the Persian language. In our comprehensive evaluation of FaBERT on 12 datasets in various downstream tasks, encompassing Sentiment Analysis (SA), Named Entity Recognition (NER), Natural Language Inference (NLI), Question Answering (QA), and Question Paraphrasing (QP), it consistently demonstrated improved performance, all achieved within a compact model size. The findings highlight the importance of utilizing diverse corpora, such as HmBlogs, to enhance the performance of language models like BERT in Persian Natural Language Processing (NLP) applications.

## introduction recently, we've seen the rise of sophisticated language models like bert @xcite , transforming the understanding of languages, including persian. whether designed for multiple languages or specifically for persian, these models have been employed across various applications in persian natural language processing (nlp). their training encompassed a diverse range of textual sources, including websites like wikipedia and social media platforms such as twitter, as well as news articles and academic journals. more recently, large language models (llms) with a substantial increase in parameters have significantly reshaped the landscape of nlp, excelling in a myriad of tasks. despite their significant contributions, finely-tuned lms such as bert still demonstrate robust performance, achieving comparable results or, in many cases, even outperforming llms in traditional natural language understanding (nlu) tasks, including natural language inference (nli), sentiment analysis, text classification, and question answering (qa) @xcite . encoder-only models like bert remain the workhorses of practical language processing, with applications ranging from content moderation to information retrieval systems. additionally, llms often come with the drawback of slower response times and increased latency compared to smaller models. moreover, the use of llms typically demands advanced hardware, creating accessibility challenges for many users. privacy concerns may also emerge when employing llms online. notably, encoder models like bert have found crucial roles in supporting llm deployments, serving as efficient filters for content safety @xcite , performing rapid document retrieval in rag systems @xcite , and enabling cost-effective preprocessing of large-scale data @xcite . their compact size and efficient architecture make them particularly suitable for edge devices and mobile applications, where computational resources and power consumption are constrained. recent studies @xcite highlight the value of incorporating informal text into training corpora, as it improves a model's ability to handle colloquial language and social media content, leading to better performance on diverse linguistic tasks. our motivation is to develop fabert, a persian bert model exclusively pre-trained on persian blogs, to enhance performance in traditional nlu tasks and enable efficient processing of both formal and informal texts in the language. blogs, which have not previously been utilized for pre-training persian lms, serve as a rich source of colloquial language with flexible sentence structures, idiomatic expressions, and informal lexicons inherent in everyday persian communication. while recent models have demonstrated commendable capabilities, there still remains room for improvement, particularly in tasks involving informal persian text. blog content includes diverse and evolving language variations such as cultural references, informal lexicons, and slang in persian, which have been user-generated across different demographics over a long period, contributing to fabert's robust performance. our findings reveal that pre-training on the hm-blogs corpus from persian blogs enhances the model's performance, leading to state-of-the-art results across various downstream tasks. the main contributions of this paper are: the subsequent sections of the paper are structured as follows: section 2 provides an introduction and comparison of various bert models employed for persian nlp. section 3 delves into the details of our corpus, model, and its pre-training procedure. section 4 compares fabert's performance in downstream tasks with other models. finally, section 5 concludes the paper by summarizing our findings. ## related work bert that stands as bidirectional encoder representations from transformers, has demonstrated its exceptional abilities across a wide range of natural language understanding tasks. unlike traditional language models that process text in a unidirectional manner (left-to-right or right-to-left), bert considers both the left and right context of words. bert's pre-training involved two training objectives: masked language modeling (mlm) and next sentence prediction (nsp). mlm randomly masks words in a sentence, and the model learns to predict the missing words based on context, enhancing its ability to grasp the semantic meaning and relationships between words within sentences. on the other hand, in the nsp task, the model has to predict whether sentence b logically succeeds sentence a. mlm and nsp are designed for the model to learn a language representation, which can then be used to extract features for downstream tasks. continuing the discussion, we will present a selection of persian-language bert models. the most well-known persian language model is parsbert @xcite . it was pretrained using both mlm and nsp tasks, utilizing a training corpus collected from 8 different sources. parsbert has become the preferred choice for persian nlp tasks, thanks to its outstanding performance. ariabert @xcite is another persian language model that follows roberta's enhancements @xcite and utilizes byte-pair encoding tokenizer. its diverse training dataset, exceeding 32 gigabytes, includes conversational, formal, and hybrid texts. additionally, many multilingual language models have been released since, and few of them include persian. multilingual bert, also known as mbert, was introduced by @xcite . it was trained with nsp and mlm tasks on the wikipedia pages of 104 languages with a shared word-piece vocabulary. mbert has shown impressive zero-shot cross-lingual transfer and is effective in utilizing task-specific annotations from one language for fine-tuning and evaluation in another. although mbert has shown solid performance across different languages, monolingual bert models outperform mbert in most downstream tasks. similarly, xlm-r @xcite , an extension of the roberta model by facebook ai, is designed for cross-lingual understanding. this model was pre-trained with the mlm objective on a vast corpus comprising more than 2 terabytes of text from 100 languages and outperformed mbert in many downstream tasks. the models previously reviewed adhere to the architecture introduced by the original bert-base model, featuring 12 layers and 12 attention heads. while maintaining this consistency, there are variations in vocabulary size among these models. a larger vocabulary facilitates the capture of more unique tokens and their relationships, but it comes at the expense of an increased number of parameters. this, in turn, necessitates more extensive training data for learning embeddings. conversely, smaller vocabularies may struggle to capture all the details of language, potentially causing information and context to be lost. an instance is found in the multilingual model mbert, which supports 100 different languages with a vocabulary size of only 100,000. despite the broad language coverage, this choice leads to a limited set of tokens for each language. consequently, sentences are transformed into a greater number of tokens, potentially exceeding the maximum supported sequence length and resulting in the loss of information. ## experiments and results in this section, we assess the fabert model across four different categories of downstream tasks. for nli and question paraphrasing, sentence pairs are processed to generate labels based on their relationship. in ner, entities within single input sentences are labeled at the token level. sentiment analysis and irony detection involve processing individual sentences and assigning corresponding labels. in question answering, models utilize a given question and the provided paragraph to generate tokenlevel spans for answers. for each task, we fine-tuned fabert and compared its performance to other state-of-the-art models, such as parsbert @xcite , ari-abert @xcite , and multilingual models like mbert @xcite and xlm-r @xcite . lastly, we analyze the effectiveness of fabert's tokenizer and compare it with other bert models. to ensure a fair comparison, all models were fine-tuned on the same datasets using consistent train/validation/test splits. for each model and dataset pair, we performed a grid search over hyperparameters, selecting the configuration that achieved the best validation score. the scores reported in this paper correspond to the test set results obtained under these optimal conditions. details of the grid search ranges and dataset splits are provided in appendix a. ## conclusion in this paper, we pre-trained fabert, a bert-base model from scratch exclusively on the diverse hm-blogs corpus, consisting solely of raw texts from persian blogs. notably, our model's smaller vocabulary size resulted in a more compact overall size compared to competitors. fabert performed exceptionally well in 12 different datasets, outperforming competitors in nine of them. in the remaining tasks where it did not secure the top position, it consistently ranked among the top performers, closely following the highest-performing model. our results indicate that texts with diverse writing styles, both formal and informal, found in persian blogs can significantly contribute to the high-quality pre-training of language models, including bert. the effectiveness of the hmblogs corpus in the performance of our bert model in downstream tasks demonstrates its potential for being used in pre-training both language models and large language models alongside other relevant persian corpora. this success aligns with the broader trend in nlp where encoder-only models continue to prove their value, particularly in scenarios requiring efficient processing of large-scale text data while maintaining high performance standards. the practical advantages of our approach -combining the efficiency of bert's architecture with rich, diverse training data -position fabert as a valuable tool for persian nlp applications, especially in resource-constrained environments where larger models may be impractical. this work not only advances persian language processing capabilities but also reinforces the continuing relevance of carefully designed encoder models in the evolving landscape of natural language processing.