Introducing Q u BERT : A Large Monolingual Corpus and BERT Model for S outhern Q uechua (2022)

Abstract: The lack of resources for languages in the Americas has proven to be a problem for the creation of digital systems such as machine translation, search engines, chat bots, and more. The scarceness of digital resources for a language causes a higher impact on populations where the language is spoken by millions of people. We introduce the first official large combined corpus for deep learning of an indigenous South American low-resource language spoken by millions called Quechua. Specifically, our curated corpus is created from text gathered from the southern region of Peru where a dialect of Quechua is spoken that has not traditionally been used for digital systems as a target dialect in the past. In order to make our work repeatable by others, we also offer a public, pre-trained, BERT model called QuBERT which is the largest linguistic model ever trained for any Quechua type, not just the southern region dialect. We furthermore test our corpus and its corresponding BERT model on two major tasks: (1) named-entity recognition (NER) and (2) part-of-speech (POS) tagging by using state-of-the-art techniques where we achieve results comparable to other work on higher-resource languages. In this article, we describe the methodology, challenges, and results from the creation of QuBERT which is on par with other state-of-the-art multilingual models for natural language processing achieving between 71 and 74% F1 score on NER and 84–87% on POS tasks.

## introduction with the availability of online digital resources for computation and data storage, the capability for executing natural language processing (nlp) tasks such as named-entity recognition (ner), partof-speech (pos) tagging, and machine translation (mt) on low-resource languages, languages with few digital resources available, has increased. the processing power and data available for experimentation are unsurpassed in history and research @xcite has shown that in the current decade we are on track to overcome previous methods, such as moore's law @xcite , for predicting computing time of experiments. this finding is better observed on high-resources languages like english and french where the amount of data that exists is more than enough to take advantage of the latest computing architectures. unfortunately, for other low-resource languages like quechua, an indigenous language spoken by millions in peru, south america, it is more difficult to create statistically significant nlp models due to the amount of data needed (typically on the order of millions of sentences). therefore, it is critical to create publicfacing mechanisms for low-resource languages like quechua to help provide research collaboration which will improve the quality for low-resource language nlp systems. we aim to improve the digital resources available for quechua by curating a large monolingual corpus for southern quechua, a dialect of quechua spoken in the southern region of peru not commonly found in most literature. the initiative we present in this article can be considered a major contribution and advancement as means to improve the quality of nlp tasks for the quechua language. we outline the multiple innovations and contributions provided below. in order to cover our innovations and contributions, we highlight the details in several sections. first, in section 2, we describe the latest work on quechua and other techniques related to lowresource nlp tasks such as the ones we introduce on ner and pos. next in section 3, we provide more background on the quechua language by covering morphological, phonological, and other important grammatical details. then, we describe how we curated our corpus in section 4. in section 5, we provide details on the parameters and configuration for our models and tokenization techniques which leads way to the experimental evaluation and results from the ner and pos tasks in section 6. finally, we wrap up with a few proposed lines of future work and a conclusion in section 7. ## related work in this section we present several works that can be considered state-of-the-art at this time for quechua. since we are introducing several new contributions, we briefly cover the most recent work and how it related to each contribution mentioned. first, concerning the introduction of the corpus, we discuss work where corpora have been introduced for public use. like many low-resource nlp projects, one of the several corpora that is often used is the opus foot_1 @xcite corpus. it contains text similar to ours in southern quechua (quechua ii, see more details on quechua variants in section 3); however, it contains biblical text only. other work @xcite introduced the jw300 corpus @xcite ; their corpus was for one domain also. the corpus we present contains entries from several diverse sources while at the same time including opus and the jw300. ortega et. al @xcite also presented a magazine selection known as hinantin which contained 250 non-biblical quechua-spanish sentences found on-line foot_2 . while the hinantin magazine was a more diverse domain than other quechua corpora previously introduced, our corpus is the largest and most diverse compiled currently available. our second contribution consists of a normalization technique used in previous work @xcite @xcite . the work presented in this article uses the same normalization technique (described further in section 5) but, to our knowledge, this is the first time that the normalization technique has been used on a corpus of this size for southern quechua. thirdly, for quechua, there has not been a tokenization comparison similar to the one presented here. there are two works @xcite ) that present approaches called bpe-guided and prpe separately but their work did not compare on such a varied corpus for namedentity recognition or part-of-speech tasks, both of their works for the machine translation task only. the fourth, fifth, and sixth contributions are all related to the first-time presentation of a deep learning transformer model for quechua that is used for ner and pos classification tasks. one of the works that presented deep learning approaches for quechua is a shared task @xcite ) from the first workshop on nlp for indigenous languages of the america @xcite . another work called indt5 @xcite used an encoder-decoder model transformers based on t5 @xcite . both models were mainly used for translation and the data did not contain nearly as much quechua-spanish text as ours. @xcite applied a deep learning approach where quality was low due to the use of the opus corpus for training and hinantin for testtheir deep learning approach was for machine translation also. other work @xcite has presented large corpora with trans-former architectures but did not include quechua as one of the low-resource languages. the one work that can be considered closest to ours in size and technique is the work by wongso et. al @xcite , they pre-trained mono-lingual models on gpt-2 @xcite , bert @xcite , and roberta @xcite . like our work, they used a monolingual corpus which consisted of a variety of text and evaluated the models on a sentiment classification task for sudanese. the main difference between their work and our work is that our tasks are slightly different and are based on quechua. in order to better understand why nlp tasks for quechua can be more complex than for other languages, we present more details in the next section on the language. ## quechua language quechua is an indigenous language native to several regions in south america, mainly peru, ecuador, and bolivia, and is spoken by nearly 8 million people. it is known @xcite @xcite to be a highly inflective language based on its suffixes which agglutinate. due to its morphology, quechua has been found to be similar to other languages like finnish @xcite @xcite . linguistically, quechua can be considered a unique and even complex language due to the highly polysynthetic nature and phonology. slight changes in morphemes (small sub-word units) can modify a word's meaning drastically. since quechua is the south american language with the highest amount of native speakers and those speakers tend to introduce diverse accentuated tones on different words depending on the locality, one can assume that the combination of morphological and tonal rules that cause inflection can make tasks like the ones presented in this article (ner and pos) difficult due to the high likelihood of non-common meanings for sub-words and letters. for example, by adding an accent to the letter 'o' in quechua, words become plural. quechua synthesis, or the synthetic index (greenberg, 1963) -the average number of morphemes per word, is about two times larger than english. english typically has around 1.5 morphemes per word and quechua has about 3 morphemes per word. this high morphological complexity has been described in detail in the past @xcite ; few have been able to overcome the challenges that low-resource languages like quechua present for digital processing. quechua's phonology uses three vowels for the most part: a, i, and u. consonants, on the other hand, are numerous and depending on the region where it is spoken, quechua can have up to 14 constants @xcite . generally speaking, lexemes are monosyllabic or bi-syllabic having two vowels (vv) or two consonants (cc) that do not concur in the same syllable. from a phonological perspective, the scheme of any quechua root is: (c)v(c)-cv(c) @xcite . the region where quechua is spoken can be considered important. alfredo torero @xcite reported that there are two main divisions of the language (quechua i and quechua ii). quechua ii is mostly spoken in regions such as ayacucho, peru and is considered a "southern" language. there are several more dialects spoken and others @xcite report several divisions for quechua ii; but, in this article we focus specifically on the southern version at a high-level. a lot of the quechua morphology has been documented in previous works @xcite @xcite @xcite ; however, there is not a clear consensus to resolve all morphology issues that may arise. in order to statistically determine which branch of morphemes a verb phrases falls under can be difficult with quechua since there are so few resources. a short example sentence of how complex morpheme determination can be is depicted in ## results the results presented in this section show how well qubert performs on two main tasks: ner and pos. we feel that the contributions presented in section 1 are sufficient to warrant wider use of our work; however, it is our intention to show that the corpus, model, and experiments could provide easy access for future work. we cover each task (ner and pos) as separate sections below in order to provide better insight into how the model performs in different scenarios, specifically for the different tokenization and normalization (called "norm." in ## conclusion and future work in this article, we have introduced a novel monolingual corpus, curated and compiled for southern quechua. we have shown that the corpus can be used for downstream tasks such as ner and pos tagging by creating and releasing a deep learning model based on bert @xcite called qubert. additionally, we experimented with the state-of-the-art tokenization techniques for preprocessing and normalization in order to achieve results similar to those found on high-resource languages. in the future, we would like to experiment with other model architectures for more complex ner tasks such as those presented at semeval-2022 @xcite , of particular interest is the work from @xcite . we would like to include more native quechua speaking annotators in order to improve the data set even more. the introduction of two or more annotators will allows us to introduce models for tasks such as machine translation, question-answering, and topic modeling where the reference data is even more important. we believe that our work can serve as a baseline for future work and invite other researchers to use the contributions presented here for further investigative lines such as the ones we are considering: online tools for native quechua speakers and human interaction.