S wah BERT : Language Model of S wahili (2022)

Abstract: The rapid development of social networks, electronic commerce, mobile Internet, and other technologies, has influenced the growth of Web data. Social media and Internet forums are valuable sources of citizens’ opinions, which can be analyzed for community development and user behavior analysis. Unfortunately, the scarcity of resources (i.e., datasets or language models) become a barrier to the development of natural language processing applications in low-resource languages. Thanks to the recent growth of online forums and news platforms of Swahili, we introduce two datasets of Swahili in this paper: a pre-training dataset of approximately 105MB with 16M words and annotated dataset of 13K instances for the emotion classification task. The emotion classification dataset is manually annotated by two native Swahili speakers. We pre-trained a new monolingual language model for Swahili, namely SwahBERT, using our collected pre-training data, and tested it with four downstream tasks including emotion classification. We found that SwahBERT outperforms multilingual BERT, a well-known existing language model, in almost all downstream tasks.

## introduction nowadays, online social networking has revolutionized interpersonal communication. the influence of social media in our everyday lives, at both a personal and professional level, has led recent studies to language analysis in social media @xcite . especially, natural language processing (nlp) tools are often used to analyze textual data for various real-world applications; mining social media for information about health @xcite , diseases analysis (e.g., covid-19 @xcite , ebola @xcite ), identifying sentiment and emotion toward products and services, and developing dialog systems @xcite . language models have recently drawing much attention as they are known to be effective in many nlp tasks (e.g., text classification, entailment, sequence labeling), but they commonly require a huge amount of data for pre-training and fine-tuning; some models are designed for few-shot learning that does not require much labeled data for fine-tuning, though they still require plenty of pre-training data. as it is expensive and difficult to get the labeled and unlabeled data, the majority of the data are in high-resource languages (hrls) (e.g., english, spanish). unfortunately, other than about 20 hrls languages, approximately 7,000 low-resource languages (lrls) in the world are left behind, where most of lrls are spoken and little written @xcite . africa and india are the main hosts of lrls, where some languages are spoken by more than 20 million people (e.g., hausa, oromo, zulu, and swahili). as more data on social media in lrls, qualified datasets, and publicly available language models will bring many advantages in various fields, such as education @xcite , healthcare (de las @xcite , entertainment @xcite , and business. swahili, a bantu language, is one of the two official languages (the other being english) of the east african countries such as tanzania @xcite , kenya, and uganda. it has been widely spread in african countries not only as a lingua franca but also as a second or third language across the african continent and broadly in education, administration, and media. with the rapid development of social networks, electronic commerce, mobile internet, and other technologies, swahili is also spreading in online places that result in the growth of web data. for example, jamiiforum is a popular online platform in tanzania, and it provides a place to discuss different issues, including political, business, educational, and lifestyle; this means more collected textual data of swahili is available. by making use of the online textual data of swahili, there are several studies for different tasks (e.g., sentiment classification @xcite @xcite , news classification @xcite ). recently, language models have drawn much attention from the industry and academic world, as the language models brought much better performance (e.g., accuracy) than other existing models. there are few studies that employed the language models for swahili: named entity recognition (ner) @xcite and sentiment classification @xcite . although these studies have shown successful results, they are limited in that they just borrow the language models (e.g., multilingual bidirectional encoder representations from transformer (mbert) @xcite , cross-lingual model-roberta (xlm-r) @xcite ) pre-trained with other resources (i.e., other languages); in other words, their language models are pre-trained for multiple languages but not dedicated for swahili. although such multilingual language models have shown great generalization power across multiple languages, several studies @xcite @xcite reported that the monolingual models often outperform these multilingual models. there was no study that proposed a monolingual language model for swahili (i.e., swahili-specific language model), and the main reason is that swahili is one of the lrls, so the existing studies commonly suffered from lack of available data. in this paper, we focus on the swahili language. to the best of our knowledge, this is the first study that collects a pre-training dataset and uses it for pre-training of the swahili-specific language model. we also provide a manually annotated dataset for the emotion classification task. the contributions are summarized as follows. ## background most african countries have minority languages that are used by specific ethnic groups (approx. 158 in tanzania foot_0 ). however they speak different national and official languages of their countries, including native and colonial that can be used in public services such as education, politics, and the media. swahili is a bantu language widely spoken in sub-saharan africa and acts as the common tongue for most east african @xcite . many swahili vocabularies are derived from loanwords, the vast majority from arabic, but also english, hindi, portuguese, and other bantu languages foot_1 . as the language grows, new formal and informal vocabularies emerge. the formal vocabularies are used in official documents, whereas the informal vocabularies are mostly used by young adults and on social media platforms @xcite . structurally, it is considered an agglutinative language with polysemous features. its morphology depends on prefixes and suffixes which are syllables @xcite . a single word is generated with morphemes (i.e., stem, prefixes, and affixes) that will have corresponding inflectional forms. nouns are divided into classes on the basis of their singular and plural prefixes. despite its popularity, a limited amount of textual data is available and it is one of the low-resource languages (lrls). although there have been a few studies that illustrate the value of nlp @xcite @xcite , they commonly suffered from the lack of available data. ## dataset in this section we describe our collected dataset for pre-training and the downstream task of emotion classification. we open our datasets for future use in various studies foot_4 . ## swahbert with the collected dataset, we pre-trained the monolingual bert for swahili, namely swahbert 11 . the swahbert basically has the same architecture as the original bert. this section describes the process of pre-training and fine-tuning of swah-bert. ## experiments we tested our model on downstream tasks and compared with other models. we put an additional linear layer and an output layer on top of the pretrained language models, where all models are im- vocabulary tokenization mbert wa ##nan ##chi wa ##nata ##raj ##ia fur ##sa ke ##dek ##ede swahbert(32k) wananchi wanatarajia fursa ke ##de ##ke ##de swahbert(50k) wananchi wanatarajia fursa kede ##ke ##de swahbert(70k) wananchi wanatarajia fursa kedekede ## discussion we constructed two datasets for the low-resource swahili language: for the downstream task of emotion classification and pre-training of the new swahili-specific bert model. for pre-training purposes, we managed to collect a corpus of about 105 mb. although the size of our corpus is quite smaller than that of rich-resource languages (e.g., english), the pre-trained swahbert has shown great improvement on the downstream tasks. this result is consistent with other previous studies. for example, @xcite found that wellperforming language models can be obtained with a little size of corpora of 100mb. similarly, in @xcite , experimental results with the language models pre-trained with the 4gb dataset were comparable to those pre-trained with 138gb dataset. however, we believe that plenty of qualified datasets will help to increase the power of language models. as demonstrated in the experimental results, swahbert is generally superior to mbert in almost all downstream tasks. we believe that our tokenizer with swahili vocabulary has the biggest contribution to the results. the tokenizer of mbert works by sharing vocabulary over multiple languages, and this tokenizer tends to split the words without taking into account morphological boundaries (e.g., stem, prefix, and postfix), as shown in we examined the characteristics of the datasets by frequency histograms of vocabulary words in the same order as depicted in figure 2 . the emotion ## conclusion in this study, we introduced our pretraining corpus and annotated dataset for the emotion classification task. the emotion classification dataset contains 7 emotion classes, including neutral, and has approximately 13k instances. we also performed pretraining of monolingual bert for swahili, namely swahbert, and experimentally compared it with the multilingual bert (mbert). the swahbert outperformed the mbert in almost all downstream tasks, where the downstream tasks include emotion classification, news classification, sentiment classification, and ner. although swahbert exhibited superior performance with a relatively smaller pre-training corpus, a more qualified pre-training corpus will definitely contribute to the development of better language models. therefore, with the growth of the digital platforms for swahili, we will continue to use the available sources, including native swahili speakers as annotators, and collect more data from different domains. we hope that this study will facilitate the development of other methodologies and pre-trained language models (e.g., xlm-r) and also aid in social services (e.g., user emotion analysis on forum texts).