Creating a POS Gold Standard Corpus of M odern U krainian (2023)

Abstract: This paper presents an ongoing project to create the Ukrainian Brown Corpus (BRUK), a disambiguated corpus of Modern Ukrainian. Inspired by and loosely based on the original Brown University corpus, BRUK contains one million words, spans 11 years (2010–2020), and represents edited written Ukrainian. Using stratified random sampling, we have selected fragments of texts from multiple sources to ensure maximum variety, fill nine predefined categories, and produce a balanced corpus. BRUK has been automatically POS-tagged with the help of our tools (a large morphological dictionary of Ukrainian and a tagger). A manually disambiguated and validated subset of BRUK (450,000 words) has been made available online. This gold standard, the biggest of its kind for Ukrainian, fills a critical need in the NLP ecosystem for this language. The ultimate goal is to produce a fully disambiguated one-million corpus of Modern Ukrainian.

## introduction ukrainian has a growing ecosystem of nlp datasets and tools. still, it falls into the category of low-resource languages, despite increasing interest in the language and the development of multiple resources and tools over the past couple of years. most general-purpose corpora that are available for ukrainian, such as the general regionally annotated corpus of ukrainian (grac) by @xcite , zvidusil by @xcite , and the ukrainian language corpus (kum) by @xcite and her team, are only accessible via a web user interface. among downloadable ukrainian corpora, one project that stands out here thanks to its size and thoroughness is ubertext 2.0 by @xcite . however, one of the missing resources is a reliable, balanced, and disambiguated corpus of sufficient size. until recently, the only such resource was the treebank created within the universal de-pendencies project by natalia kotsyba, bohdan moskalevskyi, and mykhailo romanenko. 1 with the overall size of some 120,000 tokens, it is comprised of fiction (24%), essays (8%), legal acts (7%), fairytales (7%), analytical articles (6.5%), news (6%), commentary (5%), textbooks (5%), wikipedia articles (5%), scholarly works (4%), letters (3%), and some other types. 2 the creators made a laudable effort to include a wide variety of texts, and their resource has been invaluable for ukrainian nlp. nevertheless, some aspects require improvement. for one thing, the texts in the ud ukrainian treebank come both from modern sources (past 15-20 years) and the first half of the 20th century, which does not make the entire treebank representative of any one period. second, the proportions of text types are far from reflecting either the production or the consumption of texts in modern ukrainian society. for example, news is significantly more popular than its share in this treebank would suggest. third, a bigger corpus would help achieve better quality of nlp models. furthermore, the small proportions of all types, except fiction, in the treebank complicate the task of training or fine-tuning models for a specific type. finally, the development of this treebank seems to have come to a halt several years ago. ## corpus design perceiving the need for a more balanced and larger disambiguated corpus, we have developed the ukrainian brown corpus (bruk) foot_2 modeled on the original brown university corpus. the brown university standard corpus of present-day american english @xcite has been an indispensable resource for the development of computational linguistics. it has given rise to an entire family of brown corpora, including the lancaster-oslo/bergen corpus (lob) @xcite , the freiburg-brown corpus of american english (frown) @xcite and freiburg-lob corpus of british english (flob) @xcite . similar corpora have also been constructed for other languages @xcite and successfully used for training nlp models. in order to establish the categorial structure of bruk, we have used the same method of an expert poll with averaged results as did henry kucera and w. nelson francis and kept the overall split into informative and imaginative types. however, further subdivision into categories is different as it is aimed at reflecting the prevalence of each category of texts in modern ukrainian society. this is in line with the established practice as corpora derived from the brown university corpus include modifications on the original design and adjustments to account for the specific features of the language and country in question. the categories thus established for bruk are as follows (percentages represent proportions of the total size): a. press, 25%. while bruk has no formal subdivision into reportage, editorials, and reviews, a special effort has been made to represent these subcategories and ensure topical diversity (politics, society, finances, sports, culture, and environment). this category includes texts selected from national, regional, and local (city or districtlevel) mass media outlets in both printed and electronic form. b. religion, 3%. importantly, texts representing different religions have been included. c. skills and hobbies, 7%. popular topics, such as household, crafts, farming, gardening, and construction, are represented. d. essays, biography, memoirs, etc., 7%. this is a catch-all category for informative texts that do not fit elsewhere, including forewords, personal letters, and literary and art criticism. e. administrative documents, 3%. laws, government regulations, reports, and official letters comprise this category. f. popular science, 5%. experts agreed that these texts required a separate category due to their linguistic characteristics. g. science, 10%. a balanced selection of texts in natural sciences and the humanities has been made. h. textbooks, 15%. this sizable category reflects the important role such texts play in ukraine, where a wide audience of students reads them. i. fiction, 25%. while no formal subdivision has been adopted, variety is ensured by selecting works of different lengths (from short stories to novels) and genres. in filling each category in the corpus with texts, we employed random sampling through crowdsourcing: more than a hundred individuals were involved in sample selection. submitted samples were verified and filtered by corpus creators, for example, to remove duplicates and avoid overrepresentation of a particular newspaper, author, or topic. each text fragment in the corpus is supplied with metadata identifying the author(s), title, book/journal title (if applicable), place and year of publication, publisher, page range, length in tokens, orthography (official or alternative), and detected errors. metadata information is stored separately from texts in a .csv file available for download and processing. each file containing a text fragment is given a name that begins with a letter (a-i) for the respective category, enabling users to quickly separate the necessary category from the entire corpus. ## text requirements texts in bruk must meet a set of requirements, some of which mirror those for the original brown university corpus, while others represent a conscious departure from its model to match the realities of modern ukrainian better: the primary challenge here was to weed out texts surreptitiously translated from russian (a common practice among some publishers and mass media outlets in ukraine) and products of machine translation. in doubtful cases, we opted to err on the side of exclusion. detailed annotation guidelines foot_5 have been used by all contributors to bruk. ## pos tagging its current version (6.1.1) comprises over 418,000 lemmas from which more than 6.5 million wordforms are generated. the dictionary achieves 97-99% word coverage on non-encyclopedic texts. moreover, the tagtext tagger includes a dynamic tagging component to recognize and tag words not found in vesum, reaching 95% accuracy on these out-of-vocabulary items @xcite . this combination of tools has been successfully utilized to tag successive iterations of grac, a large reference corpus of ukrainian (shvedova, 2020) @xcite . second, unlike other morphological dictionaries of ukrainian, vesum includes numerous proper nouns and nonstandard lemmas, such as alternative spellings, slang, deprecated lexical items, dialectal words, and substandard wordforms, which are not to be found in other lexicographic resources. these linguistic items occur in modern texts and need to be duly recognized. ## disambiguation ukrainian is a highly inflected language with ubiquitous lexical and morphological ambiguity. in bruk, an ambiguous word may have from 2 to over 30 homonymic readings. as of this writing, ambiguity has been resolved for 450,000 ukrainian words (560,000 tokens), making the disambiguated subset of bruk the biggest such resource for ukrainian. this part comprises 80,000 ukrainian types and over 37,000 lemmas. morphological ambiguity (58% of the words in the disambiguated subset of bruk) is much more prevalent in ukrainian than lexical ambiguity (13%), and a ukrainian word has 2.88 homonym interpretations on average. after automatic tokenization, lemmatization, and pos tagging (all performed by tagtext), bruk texts were subjected to a two-stage (in some cases, three-stage) disambiguation process. initially, ambiguity was resolved by trained individuals (students), and these results were then verified by an expert linguist. another expert was consulted in difficult cases. the nuances of tagging were communicated to students during training, and a number of challenging cases are explained in tagging guidelines foot_10 . the outcome of this process is a set of disambiguated texts in which each token has one correct and verified reading. ## conclusions and future work the ukrainian brown corpus (bruk) is a onemillion balanced corpus of modern ukrainian covering 2010-2020. it is loosely modeled on the original brown university corpus and consists of small fragments (mostly up to 1,000 but no longer than 2,000 words of running text) divided between 9 categories. the creators have made a concerted effort to ensure variety in the corpus along different dimensions. the corpus has been automatically tokenized, lemmatized, and pos-tagged. a subset of bruk (450,000 words) has been manually disambiguated, validated through a multi-stage process, and made available for download. several factors make bruk a unique resource compared to other ukrainian corpora: it is a balanced downloadable corpus comprised of modern ukrainian text samples that vary along several dimensions and is currently the largest corpus representing a pos gold standard for ukrainian. bruk has the potential to become a key resource in solving the foundational problem of pos disambiguation for a wide variety of practical projects. other applications are also possible, such as testing spellchecking systems, ner models, and so on. bruk has been used to build a stochastic model for pos tagging, generating a strong baseline. on the theoretical side, bruk provides insights into ukrainian morphology that have already helped us improve its formal description for the purposes of nlp and computational linguistics research. our immediate plans include the semiautomatic disambiguation of the rest of bruk (550,000 words). it is desirable to complement bruk with later publications to cover a rapidly growing number of texts about the unprovoked war russia unleashed against ukraine on 24 february 2022. further plans include adding a dependency annotation layer to the corpus. another line of activity is training language models. even if trained on the released subset of bruk rather than the entire corpus, they can be instrumental in solving various computational linguistics and nlp tasks, bringing the ukrainian language a step closer to the status of a mid-resource language.