M asakha NER 2.0: A frica-centric Transfer Learning for Named Entity Recognition (2022)

Abstract: African languages are spoken by over a billion people, but they are under-represented in NLP research and development. Multiple challenges exist, including the limited availability of annotated training and evaluation datasets as well as the lack of understanding of which settings, languages, and recently proposed methods like cross-lingual transfer will be effective. In this paper, we aim to move towards solutions for these challenges, focusing on the task of named entity recognition (NER). We present the creation of the largest to-date human-annotated NER dataset for 20 African languages. We study the behaviour of state-of-the-art cross-lingual transfer methods in an Africa-centric setting, empirically demonstrating that the choice of source transfer language significantly affects performance. While much previous work defaults to using English as the source language, our results show that choosing the best transfer language improves zero-shot F1 scores by an average of 14% over 20 languages as compared to using English.

## introduction many african languages are spoken by millions or tens of millions of speakers. however, these languages are poorly represented in nlp research, and the development of nlp systems for african languages is often limited by the lack of datasets for training and evaluation @xcite . additionally, while there has been much recent work in using zero-shot cross-lingual transfer @xcite @xcite to improve performance on tasks for low-resource languages with multilingual pretrained language models (plms) @xcite , the settings under which contemporary transfer learning methods work best are still unclear @xcite @xcite . for example, several methods use english as the source language because of the availability of training data across many tasks @xcite , but there is evidence that english is often not the best transfer language @xcite @xcite , and the process of choosing the best source language to transfer from remains an open question. there has been recent progress in creating benchmark datasets for training and evaluating models in african languages for several tasks such as machine translation @xcite @xcite @xcite , and sentiment analysis @xcite . in this paper, we focus on the standard nlp task of named entity recognition (ner) because of its utility in downstream applications such as question answering and information extraction. for ner, annotated datasets exist only in a few african languages @xcite , the largest of which is the masakhaner dataset @xcite ) (which we call masakhaner 1.0 in the remainder of the paper). while masakhaner 1.0 covers 10 african languages spoken mostly in west and east africa, it does not include any languages spoken in southern africa, which have distinct syntactic and morphological characteristics and are spoken by 40 million people. in this paper, we tackle two current challenges in developing ner models for african languages: to address the first challenge, we create the masakhaner 2.0 corpus, the largest humanannotated ner dataset for african languages. masakhaner 2.0 contains annotated text data from 20 languages widely spoken in sub-saharan africa and is complementary to the languages present in previously existing datasets (e.g., @xcite . we discuss our annotation methodology as well as perform benchmarking experiments on our dataset with state-of-the-art ner models based on multilingual plms. in addition, to better understand the effect of source language on transfer learning, we extensively analyze different features that contribute to cross-lingual transfer, including linguistic characteristics of the languages (i.e., typological, geographical, and phylogenetic features) as well as data-dependent features such as entity overlap across source and target languages @xcite . we demonstrate that choosing the best transfer language(s) in both single-source and cotraining setups leads to large improvements in ner performance in zero-shot settings; our experiments show an average of a 14 point increase in f1 score as compared to using english as source language across 20 target african languages. we release the data, code, and models on github 1 ## related work african ner datasets there are some humanannotated ner datasets for african languages 1 https://github.com/masakhane-io/ masakhane-ner/tree/main/masakhaner2.0 such as the sadilar ner corpus @xcite covering 10 south african languages, lorelei @xcite , which covers nine african languages but is not open-sourced, and some individual language efforts for amharic @xcite , yorùbá @xcite , hausa @xcite , and tigrinya @xcite . closest to our work is the masakhaner 1.0 corpus @xcite , which covers 10 widely spoken languages in the news domain, but excludes languages from the southern region of africa like isizulu, isixhosa, and chishona with distinct syntactic features (e.g., noun prefixes and capitalization in between words) which limits transfer learning from other languages. we include five languages from southern africa in our new corpus. cross-lingual transfer leveraging crosslingual transfer has the potential to drastically improve model performance without requiring large amounts of data in the target language @xcite but it is not always clear from which language we must transfer from @xcite . to this end, recent work investigates methods for selecting good transfer languages and informative features. for instance, token overlap between the source and target language is a useful predictor of transfer performance for some tasks @xcite . linguistic distance @xcite , word order @xcite and script differences @xcite , and syntactic similarity @xcite have also been shown to impact performance. another research direction attempts to build models of transfer performance that predicts the best transfer language for a target language by using some linguistic and data-dependent features @xcite . ## cross-lingual transfer the success of cross-lingual transfer either in zero or few-shot settings depends on several factors, including an appropriate selection of the best source language. several attempts at cross-lingual transfer make use of english as the source language due to its availability of training data. however, english is unrepresentative of african languages and transfer performance is often lower for distant languages (adelani et al., 2021b). ## conclusion in this paper, we present the creation of masakhaner 2.0, the largest ner dataset for 20 diverse african languages and provide strong baseline results on the corpus by fine-tuning multilingual plms on in-language ner and multilingual datasets. additionally, we analyze cross-lingual transfer in an africa-centric setting, showing the importance of choosing the best transfer language in both zero-shot and few-shot scenarios. using english as the default transfer language can have detrimental effects, and choosing a more appropriate language substantially improves fine-tuned ner models. by analyzing data-dependent, geographical, and typological features for transfer in ner, we conclude that geographical distance and entity overlap contribute most effectively to transfer performance.