Transformer-Based Language Models for B ulgarian (2023)

Abstract: This paper presents an approach for training lightweight and robust language models for Bulgarian that mitigate gender, political, racial, and other biases in the data. Our method involves scraping content from major Bulgarian online media providers using a specialized procedure for source filtering, topic selection, and lexicon-based removal of inappropriate language during the pre-training phase. We continuously improve the models by incorporating new data from various domains, including social media, books, scientific literature, and linguistically modified corpora. Our motivation is to provide a solution that is sufficient for all natural language processing tasks in Bulgarian, and to address the lack of existing procedures for guaranteeing the robustness of such models.

## introduction natural language processing has witnessed significant advancements in recent years, driven by the development of large-scale pre-trained language models (lms) such as bert and gpt-2,3,4. however, such models suffer from biases in the data, which can lead to unfair or discriminatory outputs. bulgarian language, like many other languages, also lacks robust language models that are not biased towards gender, political views, race, or other factors. the rapid advancement in the field, especially in recent years, has brought forth unprecedented leaps in the development of high-performance models for various language understanding tasks. however, despite these noteworthy achievements, the nlp research community still faces significant challenges, one of which lies in the scarcity of comprehensive and diverse datasets for pre-training transformer models in less-resourced languages, including bulgarian. this limitation greatly hinders the otherwise promising potential of these state-of-the-art models to make a profound impact across multiple sectors and geographies. recognizing the need for a robust and representative dataset for the bulgarian language is pivotal in addressing this challenge. an ideal dataset should capture the breadth and depth of linguistic diversity, encompassing variations in dialects, registers, and domains. beyond the level of linguistic parsing, the dataset also needs to reflect the cultural subtleties and local phenomena that enrich the texture of the language. additionally, this dataset must be constructed in a manner that is free from the perils of bias, hate speech, and other problematic elements that would not only undermine the scientific integrity of the research, but potentially lead to harmful real-life consequences. against this drawback, the primary objective of this paper is to present an initial dataset (see section 3) for pre-training transformer models in bulgarian language, carefully crafted to meet the aforementioned criteria. this dataset serves as a starting point for fine-tuning and experimentation, advancing the state-of-the-art in the area of bulgarian language understanding tasks. our hope is that the development of such a dataset will not only pave the way for further innovation in bulgarian nlp, but also inspire similar research endeavours for other under-resourced languages. furthermore, this paper also outlines the first set of bulgarian models trained on this initial dataset -section 4. by offering a transparent account of the methodologies, data pre-processing and augmentation techniques as well as evaluation metrics employed during the process, we aim to offer a replicable and extensible blueprint for future research efforts in bulgarian nlp. the paper contributes to the nlp research community's ongoing commitment to create robust and inclusive language understanding models, capable of unlocking the potential of ai technologies in diverse linguistic, cultural, and regional contexts. it is our hope that the introduction of this bulgarian dataset and the first models trained on it will serve as a catalyst for future developments in the global nlp landscape. in the next section we present some related work. then in section 3 we describe the preprocessing of the first version of the dataset for the training of transformer language models. in section 4 we present the training of two transform language models: bert-web-bg and gpt-web-bg. we performed two types of evaluation: ## related work there are various directions in training and using llm for less-resourced languages. for example, @xcite propose an unsupervised approach for improving the cross-lingual representations of low-resource languages. this is realized through bootstrapping word translation pairs from monolingual corpora and using them to improve language alignment in pre-trained language models. authors work with 9 languages among which macedonian. evaluation includes zero-shot ner that showed an improved cross-lingual quality. another idea on improving the usage of the pre-trained models for less-resourced languages is the exploitation of transfer learning and backtranslation as described in maali tars and tättar (2022). the authors use data from other finno-ugric languages to improve results for english-livonian translation directions. @xcite present an approach of using llms in improving semantic parsers across several languages. @xcite explore language models for west slavic languages with the aim to evaluate the potential of these language models for low-resource languages like upper sorbian and kashubian. the authors show that low-resource languages in the west slavic family can profit from the language models of the other related languages. in @xcite authors report on training four language models for ancient greek with the help of roberta and t5. the benchmarking models include a monolingual one for this language as well as a multilingual one that includes latin and english. the aim is to support research within the field of classical philology. @xcite demonstrate that applying knowledge distillation techniques for filtering language-specific models from a large multilingual model often outperform the multilingual model. in particular, two languages have been considered with respect to the proposed setup -slovene and swahili. biases found in llm is also discussed lately from various points of view. for example, @xcite propose a specific structured causal model (scm) whose parameters are easier to estimate. the evaluation on relation extraction task shows improvement on roberta and gpt-3.5. in @xcite the social bias evaluation is approached as software testing. in @xcite the authors discuss an approach for overcoming the stereotypical biases in pre-trained language models. thus, they introduce a specially developed large-scale natural english dataset to measure stereotypical biases in four domains: gender, profession, race, and religion. the authors show that well known models like bert, gpt-2, roberta and xlnet exhibit strong stereotypical biases. abubakar abid and zou (2021) demonstrate that gpt-3 shows persistent muslimviolence bias. in our work we use monolingual trained models with available bulgarian data only. we would like also to test whether our models show bias and if yes, to what extent. ## work on the initial bulgarian dataset for pre-training transformers the process of creating a diverse, bias-proof, and ethically fair dataset requires a meticulous and effective approach to clean the raw text data extracted from the internet. to address this challenge, we propose a specialized, multi-step procedure organized into the following stages: we emphasize on ensuring an adequate balance between topics and sentiment, as an imbalanced dataset can lead to biased results. to guarantee diverse subject matter and reduce the risk of topic bias, topic classification is employed to categorize the texts based on their content. a diverse set of classes is identified using supervised and unsupervised techniques. the identified topics and subtopics are further balanced in the data ensuring equal and diverse distribution of the content. sentiment classification is essential to understanding the emotional tone and polarity of the text. through the categorization of the texts into positive, negative, and neutral sentiment categories we target the diversitiy of the dataset towards different opinions and expressions of the reality in bulgaria in the covered period. carefully redistributing instances across topics and sentiment categories results in a more representative and inclusive dataset for language modelling, a statement we test in our evaluations further. ## training of bert-web-bg and gpt-web-bg transformers in the scope of the initial experiments conducted using the refined dataset, we set out to pre-train two popular language models, namely bert and gpt-2, training the proper tokenizers on the bulgarian web dataset. bert (bidirectional encoder representations from transformers) is a pre-trained architecture, initially proposed by @xcite . the model uses a masked language modeling (mlm) objective to predict missing tokens in a given sequence, allowing it to process textual data bidirectionally. this results in a deeper understanding of linguistic contexts from both directions. in our experiments, we train the original bert model with preserved parameters, employing a tokenizer designed specifically for the bulgarian dataset. the tokenizer is trained on the dataset to segments the input text into subwords, obtaining a bulgarian tokenizer with vocabulary of size 50000. gpt-2 (generative pre-trained transformer 2) is a large-scale generative model developed by ope-nai @xcite . the gpt-2 framework utilizes causal language modeling, which relies on the context to the left of the mask during text generation. this approach helps the model better understand and predict tokens based upon preceding sequences. in our experiments, we train gpt-2 using the original parameters provided by the authors, adapting its tokenizer for the bulgarian dataset. similar to bert, the gpt-2 tokenizer is trained on the dataset with vocabulary of size 50000, ensuring efficient and accurate representation of the language. both bert-web-bg and gpt-web-bg are pre-trained from scratch using the described tokenizers to segment the bulgarian dataset. by training these architectures, we aim to gain insights into the impact of the dataset on the performance and the generalization capabilities of these two popular language models as well as its potential to contribute to the upcoming advanced model architectures. the aim is to assess the performance of these well-known architectures on a dataset that has been thoughtfully crafted to address shortcomings related to bias and hateful content. furthermore, the training of these smaller, domain-specific models in a particular language offers distinct advantages, including a reduced carbon footprint and budget-friendly requirements, making them more accessible to nlp communities. in our experiments, we utilized a single nvidia v100 gpu with 2x32g cores. bert-web-bg took approximately 78 hours to complete 5 epochs on the dataset, while gpt-web-bg required approximately 800 hours for the same number of epochs. these resource requirements are highly favorable for research laboratories, especially when compared to the considerably greater demands necessitated by the training of more general models. domain-specific language models are a valuable choice due to their numerous advantages related to data specificity, efficiency, and cost-effectiveness @xcite . there are several reasons why these models might be more appropriate compared to more general-purpose models: ## fine-tuning for named entity recognition and text classification the approach of fine-tuning the bert-web-bg model on the bsnlp ner dataset @xcite and achieving comparable or better performance to state-of-the-art models contributes to the development of cost-effective and robust domainspecific models. in 1 we compare our fine-tuned model with the multilingual bert and another bulgarian bert model from huggingface models hub, unfortunately not sufficiently documented, and the stateof-the-art on this dataset reported by @xcite . we fine-tune both models with the same data under the same conditions and parameters to be able to compare them. the findings clearly indicate that multilingual models may not be suitable for low-resource languages with rich morphology. therefore, utilizing datasets like ours becomes essential for ensuring the success of these models in downstream tasks, such as named entity recognition. recent research lai et al. (2023) compares the zero-shot capabilities of general language models like gpt-3/4 to the alternative of fine-tuning smaller language specific models. the comprehensive experimental findings from the authors reveal that chatgpt underperforms in various nlp tasks and languages, which highlights the need for additional research to enhance model development and comprehension in multilingual learning. our results align with these findings. additionally, we demonstrate that fine-tuning these multilingual models may not be significantly beneficial, likely due to the uneven representation of languages, such as bulgarian, in the dataset utilized for pre-training the bert-basemultilingual model. thus using a model pre-trained on bulgarian language for fine-tuning on the downstream tasks looks like the best alternative for underrepresented languages at this time. the second model that we compare ours to, is found in the huggingface models hub - https://huggingface.co/rmihaylov/bert-base-bg and it was trained by adapting the multilingual bert for the bulgarian language using chintanka, oscar and wikipedia data. despite this adaptation and the fine-tuning performed by us, the model struggles to achieve comparable results on the same bsnlp dataset under the same conditions as bert-web-bg. the performance of the fine-tuned bert-web-bg model which, from now on will be called bert-bsnlp-ner-bg is on par with the current state-of-art model by @xcite , while offering the benefit of being computationally lightweight compared to the model present there. furthermore bert-web-bg and gpt-web-bg are used and reported as best results by @xcite in their paper looking for traces of textual deepfakes in bulgarian on social media. gpt-web-bg is used to generate fluent bulgarian texts for their dataset for detecting deepfake in social media together with chatgpt. bert-web-bg is fine-tuned on this dataset to classify whether the message is generated by large language model bot or written by human and is reported to have accuracy of 0.97 for the binary task and 0.94 for the multiclass task significantly outperforming the other text classifiers that they evaluate. ## evaluation of bias and limitations similar to the studies conducted by @xcite and @xcite , we examine whether the models inherit gender and racial stereotypes. to assess this, we created a small dataset comprising sentences that include gender or racespecific terms. by masking the occupation or other related words, we prompt the models to make decisions, allowing us to evaluate their tendency for bias. some examples are given below: gpt-web-bg gender and professions test example 1: prompt: "по професия той е " (he is by profession ) generated: "по професия той е строителен работник." (he is a construction worker by profession.) example 2: prompt: "по професия тя е " (she is by profession) generated: "по професия тя е специалист по математика." (by profession, she is a specialist in mathematics.) example 3: in examples 1 through 4, we present a selection of prompts designed to evaluate whether the model exhibits gender preferences when generating content related to professions. in examples 1 and 2, gpt is prompted to complete sentences that begin with "he/she is working as a " for the "he" prompt, the model generates "he is working as a construction worker," while for the "she" prompt, it produces "she is working as a mathematics specialist." these responses suggest that the model may associate certain professions with specific genders, which is evident from the stereotypical allocation of a man to a construction worker position and a woman to a mathematics specialist role. this highlights the importance of examining further potential gender biases in the model training data and refining its adaptability to prevent such biases from influencing generated content. in examples 3 and 4, the model is prompted to generate an adjective to describe "her" and "his" profession. in both cases, it classifies their professions as "very important." these responses indicate that, despite potential biases observed in examples 1 and 2, the model has been trained on a well-designed dataset that emphasizes balancing polarity and ensuring gender equality, resulting in unbiased adjectives. this outcome demonstrates the importance of carefully curating a dataset that represents the diversity of human experiences, thoughts, and attitudes. ## conclusions and future work in this paper we present a dataset for training of transformer-based language models and the two trained models -gpt-2 and bert for bulgarian. we evaluated the two models with respect to bulgarian ner task and to biases learned from the dataset. the promising results obtained through the use of domain-specific dataset for training language models underscore the importance and potential of continuing this line of research. to facilitate the development of more robust and accurate models for the bulgarian language, there is a clear need for expanding and diversifying the available datasets. in the future, we plan to focus on the following aspects: diverse domains: the creation and utilization of datasets from various sources, such as books, wikipedia, scientific and legal literature, and instructional materials, will ensure a more comprehensive representation of the bulgarian language. this will lead to models with a broader understanding of contexts and better performance across tasks. data quality: emphasis will be put on curating high-quality datasets, which will play a critical role in addressing issues such as noise, inconsistencies, and inaccuracies in the data. by refining the data, we expect to see further improvements in model performance. multimodal data: incorporating different types of data, such as images, audio, and video, along with textual information will enable us to explore multimodal learning approaches. this will pave the way for creating more versatile and efficient models that can handle a wide range of tasks. bias and fairness: future research should also concentrate on identifying and mitigating biases related to gender, race, and other demographic factors in the models. creating inclusive, balanced, and diverse datasets will contribute to the development of more equitable and responsible ai systems. adaptation fine-tuning: recent studies hu et al. (2021), @xcite introduce low-rank adaptation (lora), a method that keeps the pretrained model weights fixed while incorporating trainable rank decomposition matrices into each layer of the transformer architecture. this approach significantly reduces the number of trainable parameters required for downstream tasks and is a natural extension in our future work. by focusing on these aspects in our future work, we aim to advance the state-of-the-art in the development of bulgarian language models, ensuring they become more comprehensive, accurate, efficient accelerated and optimized. this will, in turn, enhance the impact and applicability of these models in various domains and applications.