K inya BERT : a Morphology-aware K inyarwanda Language Model (2022)

Abstract: Pre-trained language models such as BERT have been successful at tackling many natural language processing tasks. However, the unsupervised sub-word tokenization methods commonly used in these models (e.g., byte-pair encoding - BPE) are sub-optimal at handling morphologically rich languages. Even given a morphological analyzer, naive sequencing of morphemes into a standard BERT architecture is inefficient at capturing morphological compositionality and expressing word-relative syntactic regularities. We address these challenges by proposing a simple yet effective two-tier BERT architecture that leverages a morphological analyzer and explicitly represents morphological compositionality.Despite the success of BERT, most of its evaluations have been conducted on high-resource languages, obscuring its applicability on low-resource languages. We evaluate our proposed method on the low-resource morphologically rich Kinyarwanda language, naming the proposed model architecture KinyaBERT. A robust set of experimental results reveal that KinyaBERT outperforms solid baselines by 2% in F1 score on a named entity recognition task and by 4.3% in average score of a machine-translated GLUE benchmark. KinyaBERT fine-tuning has better convergence and achieves more robust results on multiple tasks even in the presence of translation noise.

## introduction recent advances in natural language processing (nlp) through deep learning have been largely enabled by vector representations (or embeddings) learned through language model pre-training @xcite @xcite @xcite . language models such as bert @xcite are pre-trained on large text corpora and then fine-tuned on downstream tasks, resulting in better performance on many nlp tasks. despite attempts to make multilingual bert models @xcite , research has shown that models pre-trained on high quality monolingual corpora outperform multilingual models pre-trained on large internet data @xcite . this has motivated many researchers to pretrain bert models on individual languages rather than adopting the "language-agnostic" multilingual models. this work is partly motivated by the same findings, but also proposes an adaptation of the bert architecture to address representational challenges that are specific to morphologically rich languages such as kinyarwanda. in order to handle rare words and reduce the vocabulary size, bert-like models use statistical sub-word tokenization algorithms such as byte pair encoding (bpe) @xcite . while these techniques have been widely used in language modeling and machine translation, they are not optimal for morphologically rich languages @xcite . in fact, sub-word tokenization methods that are solely based on surface forms, including bpe and character-based models, cannot capture all morphological details. this is due to morphological alternations @xcite and non-concatenative morphology @xcite that are often exhibited by morphologically rich languages. for example, as shown in ## morphology-aware language model our modeling objective is to be able to express morphological compositionality in a transformerbased @xcite language model. for morphologically rich languages such as kinyarwanda, a set of morphemes (typically a stem and a set of functional affixes) combine to produce a word with a given surface form. this requires an alternative to the ubiquitous bpe tokenization, through which exact sub-word lexical units (i.e. morphemes) are used. for this purpose, we use a morphological analyzer which takes a sentence as input and, for every word, produces a stem, zero or more affixes and assigns a part of speech (pos) tag to each word. this section describes how this morphological information is obtained and then integrated in a two-tier transformer architecture (figure 1 ) to learn morphology-aware input representations. ## experiments in order to evaluate the proposed architecture, we pre-train kinyabert (101m parameters for kinyabert adr and 105m for kinyabert asc ) on a 2.4 gb of kinyarwanda text along with 3 baseline bert models. the first baseline is a bert model pre-trained on the same kinyarwanda corpus and with the same position encoding @xcite , same batch size and pre-training steps, but using the standard bpe tokenization. we call this first baseline model bert bp e (120m parameters). the second baseline is a similar bert model pretrained on the same kinyarwanda corpus but tokenized by a morphological analyzer. for this model, the input is just a sequence of morphemes, in a similar fashion to @xcite . we call this second baseline model bert m orp ho (127m parameters). for bert m orp ho , we found that predicting 30% of the tokens achieves better results than using 15% because of the many affixes generated. the third baseline is xlm-r (conneau et al., 2020) (270m parameters) which is pretrained on 2.5 tb of multilingual text. we evaluate the above models by comparing their performance on downstream nlp tasks. ## related work bert-variant pre-trained language models (plms) were initially pre-trained on monolingual highresource languages. multilingual plms that include both high-resource and low-resource languages have also been introduced @xcite @xcite . however, it has been found that these multilingual models are biased towards high-resource languages and use fewer low quality and uncleaned low-resource data @xcite . the included low-resource languages are also very limited because they are mainly sourced from wikipedia articles, where languages with few articles like kinyarwanda are often left behind @xcite . @xcite classify the state of nlp for kinyarwanda as "scraping-by", meaning it has been mostly excluded from previous nlp research, and require the creation of dedicated resources and models. kinyarwanda has been studied mostly in descriptive linguistics @xcite @xcite . few recent nlp works on kinyarwanda include morphological analysis @xcite ), text classification @xcite , named entity recognition @xcite @xcite , pos tagging @xcite @xcite @xcite , and parsing @xcite . there is no prior study on pre-trained language modeling for kinyarwanda. there are very few works on monolingual plms for african languages. to the best of our knowledge there is currently only afribert (ralethe, 2020) that has been pre-trained on afrikaans, a language spoken in south africa. in this paper, we aim to increase the inclusion of african languages in nlp community by introducing a plm for kinyarwanda. differently to the previous works (see recently, there has been a research push to improve sub-word tokenization by adopting characterbased models @xcite . while these methods are promising for the "language-agnostic" case, they are still solely based on the surface form of words, and thus have the same limitations as bpe when processing morphologically rich languages. we leave it to future research to empirically explore how these characterbased methods compare to morphology-aware models. ## conclusion this work demonstrates the effectiveness of explicitly incorporating morphological information in language model pre-training. the proposed twotier transformer architecture allows the model to represent morphological compositionality. experiments conducted on kinyarwanda, a low resource morphologically rich language, reveal significant performance improvement on several downstream nlp tasks when using the proposed architecture. these findings should motivate more research into morphology-aware language models.