On the Path to Make U krainian a High-Resource Language (2025)

Abstract: Recent advances in multilingual language modeling have highlighted the importance of high-quality, large-scale datasets in enabling robust performance across languages. However, many low- and mid-resource languages, including Ukrainian, remain significantly underrepresented in existing pretraining corpora. We present Kobza, a large-scale Ukrainian text corpus containing nearly 60 billion tokens, aimed at improving the quality and scale of Ukrainian data available for training multilingual language models. We constructed Kobza from diverse, high-quality sources and applied rigorous deduplication to maximize data utility. Using this dataset, we pre-trained Modern-LiBERTa, the first Ukrainian transformer encoder capable of handling long contexts (up to 8192 tokens). Modern-LiBERTa achieves competitive results on various standard Ukrainian NLP benchmarks, particularly benefiting tasks that require broader contextual understanding or background knowledge. Our goal is to support future efforts to develop robust Ukrainian language models and to encourage greater inclusion of Ukrainian data in multilingual NLP research.

## introduction recent progress in large language models (llms) has been strongly driven by the scale and quality of pre-training data. while english enjoys massive, high-quality corpora, many other languages -including ukrainian -remain significantly underrepresented in the datasets used for multilingual model training @xcite @xcite . as a result, it often receives less attention during training, leading to suboptimal performance on ukrainian inputs in otherwise powerful multilingual models. to change this, we believe it is essential to make highquality data widely available and to encourage its inclusion in future multilingual training pipelines. to support this goal, we present kobza, a new large-scale ukrainian text corpus containing nearly 60 billion tokens. to our knowledge, this is the largest publicly available ukrainian corpus to date. kobza is designed to be easily integrated into multilingual data mixtures for llm training, and we hope it will help raise the share of ukrainian in such efforts. alongside the dataset, we pre-train modern-liberta, a long-context transformer encoder that supports input sequences of up to 8,192 tokens. the model builds on the modernbert large @xcite architecture, originally designed for efficient, high-throughput processing on modern hardware. modern-liberta is the first ukrainian-language model capable of handling such long contexts, enabling improved performance on tasks that require document-level understanding. finally, we outline a broader initiative to support the development of ukrainian natural language processing. in the future work, we plan to expand kobza to at least 100 billion tokens and to build lightweight tools for filtering and scoring document quality in ukrainian. although this part lies beyond the scope of this paper, it is a key component of our long-term vision: to elevate ukrainian to a highresource language in the era of large-scale language technologies. our main contributions can be defined as follows: ## related work recent best-performing language models share the same trait -the scale of their training data. leading english and multilingual models, such as modern-bert (trained on 2 trillion tokens, @xcite , neobert (600 billion, breton et al., 2025), and eurobert (5 trillion, @xcite , exemplify this trend. these models benefit not only from the vast availability of high-quality english data @xcite , but also from a mature ecosystem of tools for data curation @xcite and synthesis @xcite . in contrast, many other languages, particularly mid-and low-resource ones, lag behind in terms of data availability and tooling. ukrainian is a prime example. while it is spoken by tens of millions and supported by active linguistic and technological communities, the scale and quality of data available for large-scale pre-training still remains limited. ## kobza in this section, we describe the collection and preparation of the kobza corpus -a large-scale ukrainian text dataset. ## modern-liberta this section outlines how we adapted modern-bert @xcite , originally trained exclusively on english data, for use with ukrainian. we describe the training corpus, model architecture, tokenizer, initialization approach, and training setup, including the extension to long-context sequences. ## evaluation in this section, we evaluate the performance of modern-liberta across a range of language understanding benchmarks for ukrainian. we focus on two aspects: (1) intrinsic language modeling quality, measured via masked language modeling (mlm) perplexity, and (2) performance on a set of standard downstream tasks, in comparison to existing ukrainian and multilingual models. ## conclusion in this paper, we introduced kobza, the largest publicly available ukrainian text corpus, containing nearly 60 billion tokens collected from diverse, high-quality sources. using this dataset, we trained modern-liberta, the first ukrainian language model capable of processing long input sequences of up to 8,192 tokens. our evaluation demonstrates that modern-liberta achieves competitive results on ukrainian nlp benchmarks, especially benefiting tasks that rely on background knowledge. we consider this work an important step toward elevating ukrainian from its current status as an underrepresented language in multilingual models to a high-resource language. by releasing kobza and modern-liberta, we aim to facilitate further advancements in ukrainian nlp research and development. we encourage future multilingual modeling efforts to incorporate more ukrainian data to enhance model performance and support richer linguistic diversity in nlp technologies.