Open foundation models for A zerbaijani language (2024)

Abstract: The emergence of multilingual large language models has enabled the development of language understanding and generation systems in Azerbaijani. However, most of the production-grade systems rely on cloud solutions, such as GPT-4. While there have been several attempts to develop open foundation models for Azerbaijani, these works have not found their way into common use due to a lack of systemic benchmarking. This paper encompasses several lines of work that promote open-source foundation models for Azerbaijani. We introduce (1) a large text corpus for Azerbaijani, (2) a family of encoder-only language models trained on this dataset, (3) labeled datasets for evaluating these models, and (4) extensive evaluation that covers all major open-source models with Azerbaijani support.

## introduction large language models (llms) have seen a sudden rise in popularity in recent years. both open-source and proprietary models have seen wide adoption across various industries. this boost has not been shared equally across different regions, however, mostly due to the slow osmosis of these technologies into low-resource languages. azerbaijani language falls on the "other" side of this barrier, with its 24 million speakers worldwide. while some models have a limited understanding of the azerbaijani language, only paid models * equal contribution offered by openai have seen some level of adoption in the industry. open-source models are being created with multilingual or azerbaijani-only capabilities, but the community is not as keen to adopt them. this is possibly due to the limited exploration of these models' potential. this paper encompassed several lines of work that share a common goal -promoting open-source foundational models for azerbaijani. our contributions are as follows: ## previous works the use of neural networks for language modeling can be traced back to the early 2000s. @xcite and @xcite had created neural networks that outperformed traditional state-of-the-art model. @xcite uses neural networks for machine translation. these models and their derivatives were taskspecific. the idea of creating a foundational language model that could later be adapted (i.e., finetuned) to specific tasks was popularized only after the introduction of the transformer architecture by a cyrillic alternative. azerbaijan started the transition to an updated latin alphabet in 1991, which was completed in 2001. @xcite . the earliest foundational language model that gained wide adoption was bert by @xcite and later variations like roberta @xcite . bert was an encoder-only model, therefore more suitable for problems that could be formulated as a subset of the classification problem. generative foundation models came out around the same time, in the example of gpt-1 @xcite , gpt-2 @xcite , and t5 @xcite . while the gpt series continued with closed-source, enterprise models, other alternatives quickly emerged with superior performance. the most famous of these was the llama series, which directly or indirectly resulted in the development of hundreds of open-source language models. @xcite . early foundation models were trained on english text, but multilingual models quickly emerged. google had released multilingual bert alternatives, and mgpt by @xcite ) was an early variation of the gpt architecture for multiple languages. xlm-roberta by @xcite was a larger and more successful alternative to mgpt and was quickly adopted worldwide. xlm-roberta was also one of the first (if not the first) foundation models that supported azerbaijani. we are aware of only one academic work that has concentrated on the development of foundational language models for azerbaijani. @xcite have trained a roberta model on the azerbaijani split of the oscar dataset @xcite . this work is a first of its kind for azerbaijani and a very valuable starting point. however, it does not concentrate on the development of a foundation model. its main focus is improving model performance by text augmentation. therefore, they do not perform a systematic evaluation of the model. they have released one roberta model, without different sizes, which is yet another limiting factor in the adoption of the work. unfortunately, this model has not been included in our evaluation benchmarks because they have not released a tokenizer that is compatible with their model. there have also been some community attempts to create such open-source models. a series of roberta models were developed by continuing the pre-training phase on a small azerbaijani dataset @xcite . alas development center has developed a series of decoder-only llms for azerbaijani foot_2 , but they offer no explanation regarding their approach, and the models failed to pass initial sanity checks. ## text corpus a large text corpus is a prerequisite for training a large language model. for reference, gpt-2 and roberta both were trained on openwebtext @xcite , consisting of 13.5 billion tokens, which is roughly equivalent to 10 billion words. original bert models were trained on 3.3. billion words. while these numbers have exploded in recent years, the success of these models suggests that similarly effective models can be trained on similarly sized datasets. the largest corpora that existed at the beginning of our work were oscar, which contained 316 million words in azerbaijani, and colossal clean crawled corpus (c4) with 1.7 billion words. introduced by @xcite , c4 is one of the most widely used datasets in the pretraining stage of llms. c4 is labeled by language and contains 1.83 million documents tagged as azerbaijani. upon further inspection, however, we discovered a significant portion of this text is not only in different languages, but also in different alphabets (armenian, georgian, and cyrillic). in addition, the c4 dataset contains a significant amount of informal text. this can be a valuable resource, but it is outside the scope of our work. considering all of these points, we decided against using it. oscar @xcite dataset is also derived from commoncrawl. it suffers from the same problems, so it was not included in our corpus either. due to these limitations, we decided to curate a new dataset specifically for pre-training llms that understand azerbaijani. this new corpus is called dollma (dataset for open large language models in azerbaijani). 4 the first and current version of this dataset contains azerbaijani wikipedia, translated english wikipedia (incomplete), news, blogs, books, and azerbaijani laws. this dataset contains about 651.1 million words. 5 of dollma will incorporate the common crawl data. books. we attempted to create a large book corpus but faced several challenges. most of the available electronic books in azerbaijani are scanned copies. publishers rarely offer electronic books that are suitable for text extraction. as of 9 may 2024, qanun publishing, the largest publishing house in azerbaijan, offers 52 pdfs or epubs on its website. the remaining books, which were sampled from the azerbaijan national library foot_5 , children's library foot_6 , and other sources, are all scanned copies that have occasionally passed through an ocr model. for ocr, tesseract @xcite was chosen due to its multilingual support and open-source availability. we scanned thousands of books and manually sampled and analyzed them. tesseract failed to capture guillemets, which is widespread in older azerbaijani books. it also mixed up "m" with "rn" in scanned books. this happened often enough to decrease the quality of the text substantially. due to these limitations, we decided against using ocr output altogether as training data. instead, we opted for two datasets: wikipedia. we used dumps provided by the wikimedia foundation to create a new version of azerbaijani wikipedia. both the data (allma @xcite and cleaning scripts foot_7 are publicly available. bhos ai team leads another initiative where they are using open-source translation models to translate english wikipedia into azerbaijani (bhos ai r&d center, 2024). while this dataset offers little in terms of linguistic variety, it provides an invaluable knowledge base to train the models. therefore, it was included in the final corpus. news. there is an abundance of news datasets for azerbaijani. however, we decided against using a very large news corpus, since it offers little variety in terms of language. in our experience, models trained on news datasets do not learn the language comprehensively, possibly because the news contains little to no creative writing, firstand second-person narration, and dialogue. due to these limitations, only two news datasets were included. one contains text scraped from several news platforms, and the other contains news and updates from azerbaijan national library. the bhos ai team provided both datasets. blogs. another data source was blog posts collected from various websites. instead of scraping a large number of websites for their blogs, several blogs were manually picked due to their highquality text and informative content. laws. the last part consisted of azerbaijani laws, all of which are publicly available. we have also released this as an independent text corpus (allma @xcite . you can see a summary of these sources and their accompanying upscaling ratios in a major shortcoming of dollma is imbalanced domain distribution. while the dataset contains a substantial amount of text on azerbaijani laws, it is lacking in terms of first-person narrative, and stem fields. it is also heavily azerbaijancentric, which may or may not be an issue depending on the final goal. deduplication has not been performed since none of the sources has the potential of overlapping with another (i.e., wikipedia and news, or books and laws). however, the addition of a deduplication stage is important if this corpus is to be expanded further. later versions of dollma will include several major changes: ## pre-training using dollma, we have developed a series of foundational language models called allma (a large language model for azerbaijani). allma has been trained in three sizes: small, base, and large. base and large correspond to the original bert models bert base and bert large @xcite . small architecture was borrowed from @xcite . architectural details of these models can be found in we recognize two alternative approaches to the problem of modeling a low-resource language: the pre-training task was only masked language modeling. the next sentence prediction task constitutes one of our benchmarks but is not included in the pre-training stage. training loss of allma-small and allma-base models can be found in figure 1 . one major limitation of the original bert paper was static masking. if tokens are masked before the training process, then even with multiple epochs, the model will always have to predict the same token. we borrow the idea of dynamic masking from @xcite . instead of masking tokens before the training, tokens are masked on demand. this results in various masking patterns on the same text samples. since our model is trained from scratch on an azerbaijani-only dataset, using existing multilingual tokenizers offered no advantages. a wordpiece tokenizer foot_11 was trained on a weighted version of dollma, with a vocabulary size of 64k. we have not performed a systematic evaluation to find the optimal vocabulary size. (kaya and tantug, 2024) have researched the impact of vocabulary size on the performance of turkish language models. since both azerbaijani and turkish are ## benchmarks this section presents the tasks that were used to evaluate the natural language understanding capabilities of foundation models in azerbaijani. all of these tasks are a form of classification since the models are encoder-only. we created three new datasets -text classification (aze-sci), closedbook multiple-choice questions (cb-mcq), and next-sentence prediction (aze-nsp) as a part of this project. four more datasets (wikiann, translated mrpc, translated squad, and ldquad) were borrowed from the open-source community. for each task, all models were trained with the same hyperparameters (learning rate, number of epochs, etc.). in almost all cases, models were undertrained -the project had hardware and time constraints and we were trying to get comparative results rather than functioning models. the source code for all experiments is being released, and the reader can generate better-performing models by simply training longer. benchmarks have been summarized in ## results initial tests were performed on dozens of foundation models and some were deliberately left out of the final analysis due to their inferior performance. the final benchmark includes four model categories: multilingual foundation models. bert-base-multi is a multilingual version of the original bert model. xlm-roberta-base and xlm-roberta-large are some of the bestperforming multilingual models @xcite . mdeberta-v3 is a multilingual version of deberta v3 model @xcite ). multilingual models further pre-trained for azerbaijani. bert-base-aze @xcite , roberta-base-aze @xcite , and mdeberta-v3-aze (hajili, 2024c) have been further pre-trained on a small and highquality azerbaijani dataset. their base models are roberta-base, bert-base-multi, and deberta-base, respectively. models pre-trained from scratch. allma-small, allma-base, and allma-large are the only monolingual azerbaijani models. baseline models. the original english-only bert-base was added as a baseline for the multilingual models. bert-scratch refers to the models trained on a specific task without pre-training weights. it functions as a baseline for all models in the benchmark. you can find the results in bert-scratch performs particularly well on aze-sci, mrpc, and wikiann tasks. we believe this has two explanations. the first is that these tasks can be solved partially with statistical information from the input text, while this is not possible with the other tasks. the second is that the random baseline in these tasks is relatively high, while squad and ldquad have very low random baselines. these results demonstrate several points regarding foundation models for low-resource languages: it is still possible that we have missed some high-quality models and we are open to feedback regarding this. our work can be strengthened by finding or creating new benchmarks. we hope that this work will lay the foundations for such developments. ## conclusion despite some academic and community attempts to create a foundation model for azerbaijani, this problem has not received systemic treatment. we tackle this issue by introducing a new family of foundation models for the language and benchmarking these models and other existing alternatives. to compensate for the lack of datasets suitable for benchmarking llms in azerbaijani, we introduce text classification, closed-book questionanswering, and next-sentence prediction datasets. this work can be extended in several ways. the simplest improvement would be training larger models on larger corpora. our project does not achieve this due to time and hardware limitations. allma models are not a final product, but an early prototype. a larger training corpus, more advanced hardware, and a better-optimized training process will certainly result in more robust foundation models for azerbaijani. a more urgent work, however, is extending the benchmarks by creating more labeled taskspecific datasets and adding other existing models to the leaderboards. including the next-sentence prediction task in the pre-training phase can increase the performance of allma models further. another ambitious direction would be using our corpus to develop a generative foundation model. this paper concentrated on encoder-only models because it is a simpler problem to solve and it has more immediate applications. nevertheless, generative language models have wide-ranging industrial applications and demand a systemic treatment.