Introducing U ber T ext 2.0: A Corpus of M odern U krainian at Scale (2023)

Abstract: This paper addresses the need for massive corpora for a low-resource language and presents the publicly available UberText 2.0 corpus for the Ukrainian language and discusses the methodology of its construction. While the collection and maintenance of such a corpus is more of a data extraction and data engineering task, the corpus itself provides a solid foundation for natural language processing tasks. It can enable the creation of contemporary language models and word embeddings, resulting in a better performance of numerous downstream tasks for the Ukrainian language. In addition, the paper and software developed can be used as a guidance and model solution for other low-resource languages. The resulting corpus is available for download on the project page. It has 3.274 billion tokens, consists of 8.59 million texts and takes up 32 gigabytes of space.

## introduction in this paper, we introduce ubertext 2.0, which is the new and extended version of ubertext, a corpus of modern ukrainian texts designed to meet various nlp needs. modern development of word embeddings @xcite , transformers @xcite , neural machine translators @xcite , speech-to-text models @xcite , and question answering systems @xcite opens new horizons for natural language processing. most of the models mentioned above rely heavily on the availability of corpora for a target language. while it is not usually a problem to obtain such a dataset for languages such as english, chinese or spanish, for low-resource languages, the absence of publicly available corpora is a severe barrier for researchers. different approaches can be used to overcome this problem. researchers might use multilingual transformers to achieve sub-optimal performance for low-resource languages @xcite . alternatively, they might rely on the publicly available multilingual corpora, such as wikipedia @xcite , common crawl @xcite , or oscar @xcite , or collect their own corpus using web crawling technologies. the latter approach requires a lot of data engineering to combat noisy data and extract relevant texts in the target language. while many authors follow this path, it shifts the attention from the target task, requires specific skills, and takes time to collect the data rather than make use of it. to enable researchers to work on large language models or perform a data mining on texts, we release a high-quality corpus for the ukrainian language at scale and a model solution that can be applied to other low-resource languages. the core concept of our corpus is that the same data, once collected and processed, can be later used to produce various deliverables suitable for different computational linguistics tasks. the corpus size, the additional layers (like pos tags and lemmas), and its availability for direct download make it an invaluable dataset. at the same time, the data model behind it and its flexible architecture allows exporting the corpus version pinpointed to a particular task or research need. the pipeline behind the corpus simplifies data collection, pre-and post-processing, and export of the deliverables, helping set up a regular release cycle so that end users can use the fresh copy of the data or update their models built on the previous versions when needed. such deliverables can include: ## background the ukrainian language is a morphologically rich language of the synthetic type, spoken by more than 40 million people. historically, it took shape in different centers, which influenced modern ukrainian as we know it. while it is one of the most widely spoken slavic languages, it can still be considered a low-resource language and is underrepresented in modern nlp research. the reason is the lack of publicly available corpora tailored to different needs. it can be speculated with a high degree of confidence that similar issues exist in other languages. we want to address this gap for the ukrainian language and propose a model solution that can be reused for other languages. existing corpora are scattered across quite a wide range. on one end, we have relatively small, well-balanced corpora such as brown @xcite , bruk @xcite , or any national corpus collected by a dedicated team. on the other end, we have gigantic corpora, such as oscar @xcite and common crawl foot_0 , which have been collected fully automatically. in between these two extremes, there are many corpus projects that may be used either as the main data source or as supplementary material, depending on the task at hand. in our opinion, each corpus should have a clear contract with the end user that specifies the guarantees and promises it fulfills, the availability of the data, the functionality offered on top of the data (e.g., a corpus manager or extra layers), frequency of updates, and the methodology behind the data collection and processing. this will allow the researcher to pick the right tool for the job and understand the limits of this tool. to meet the requirements of modern computational linguistics, we establish the following contract for the corpus: most existing corpora for the ukrainian language do not meet all the criteria outlined above, particularly when it comes to the scale and availability of the data for direct download. corpora unavailable for download: ## the corpus to address the issues of availability and scale and allow researchers to train large language models for ukrainian, we release a new version of ubertext. the new version shares some sources and texts with ubertext 1.0, but all of them were re-crawled and pre-processed. the total size of the corpus after post-processing and filtering is: ## intended usage and cooperation we successfully used developers' preview of the corpus in various tasks: we also share the texts of the corpora with the grac project 29 to improve the coverage of this vital corpus and make modern texts accessible to linguists, translators, and students through a userfriendly corpus manager foot_29 . ## conclusions and future work to build a massive corpus of high-quality texts for a low-resource language, researchers must have a clear contract of what the corpus guarantees and does not guarantee, a methodology, data sources, and a clear pipeline. proper pipeline implementation will allow for updating the corpus and its deliverables with minimum manual labor. while implementation of such a pipeline and required infrastructure is more related to data engineering and programming rather than to nlp, the impact on the natural language processing for a target language can be enormous. when collected and made available, a good corpus is a solid foundation for myriads of computational linguistics tasks, multiplying the impact on the industry. corpora for low-resource languages can also be included in the datasets used to train multilingual word embedding models, such as xlm-roberta proposed by @xcite . to continue the effort made for ubertext, we are planning to: