M asakha NER : Named Entity Recognition for A frican Languages (2021)

Abstract: We take a step towards addressing the under- representation of the African continent in NLP research by bringing together different stakeholders to create the first large, publicly available, high-quality dataset for named entity recognition (NER) in ten African languages. We detail the characteristics of these languages to help researchers and practitioners better understand the challenges they pose for NER tasks. We analyze our datasets and conduct an extensive empirical evaluation of state- of-the-art methods across both supervised and transfer learning settings. Finally, we release the data, code, and models to inspire future research on African NLP.1

## introduction africa has over 2,000 spoken languages @xcite ; however, these languages are scarcely represented in existing natural language processing (nlp) datasets, research, and tools @xcite . ∀ (2020) investigate the reasons for these disparities by examining how nlp for low-resource languages is constrained by several societal factors. one of these factors is the geographical and language diversity of nlp researchers. for example, of the 2695 affiliations of authors whose works were published at the five major nlp conferences in 2019, only five were from african institutions @xcite . conversely, many nlp tasks such as machine translation, text classification, partof-speech tagging, and named entity recognition would benefit from the knowledge of native speakers who are involved in the development of datasets and models. in this work, we focus on named entity recognition (ner)-one of the most impactful tasks in nlp @xcite . ner is an important information extraction task and an essential component of numerous products including spell-checkers, localization of voice and dialogue systems, and conversational agents. it also enables identifying african names, places, and organizations for information retrieval. african languages are underrepresented in this crucial task due to lack of datasets, reproducible results, and researchers who understand the challenges that such languages present for ner. in this paper, we take an initial step towards improving representation for african languages for the ner task, making the following contributions: other ner datasets for african languages include sadilar @xcite for ten south african languages based on government data, and small corpora of fewer than 2k sentences for yorùbá @xcite and hausa @xcite . additionally, the lorelei language packs @xcite include some african languages (yorùbá, hausa, amharic, somali, twi, swahili, wolof, kinyarwanda, and zulu), but are not publicly available. ner models popular sequence labeling models for ner include the crf @xcite , cnn-bilstm (chiu and nichols, 2016), bilstm-crf @xcite , and cnn-bilstm-crf @xcite . the traditional crf makes use of hand-crafted features like part-of-speech tags, context words and word capitalization. neural ner models on the other hand are initialized with word embeddings like @xcite , glove @xcite , and fasttext @xcite . more recently, pretrained language models such as bert @xcite , roberta @xcite , and luke (yamada et al., 2020) have been applied to produce state-of-the-art results for the ner task. multilingual variants of these models like mbert and xlm-roberta @xcite ) make it possible to train ner models for several languages using transfer learning. language-specific parameters and adaptation to unlabeled data of the target language have yielded further gains @xcite . ## focus languages ## data and annotation methodology our data were obtained from local news sources, in order to ensure relevance of the dataset for native speakers from those regions. the dataset was annotated using the elisa tool @xcite by native speakers who come from the same regions as the news sources and volunteered through the masakhane community. 3 annotators were not paid but are all included as authors of this paper. the annotators were trained on how to perform ner annotation using the muc-6 annotation guide. 4 we annotated four entity types: personal name (per), location (loc), organization (org), and date & time (date). the annotated entities were inspired by the english conll-2003 corpus @xcite . we replaced the misc tag with the date tag following @xcite as the misc tag may be ill-defined and cause disagreement among non-expert annotators. we report the number of annotators as well as general statistics of the datasets in a key objective of our annotation procedure was to create high-quality datasets by ensuring high annotator agreement. to achieve high agreement scores, we ran collaborative workshops for each language, which allowed annotators to discuss any disagreements. elisa provides an entity-level f1-score and also an interface for annotators to correct their mistakes, making it easy to achieve 3 https://www.masakhane.io . 4 https://cs.nyu.edu/ ∼grishman/muc6.html. ## conclusion and future work we address the ner task for african languages by bringing together a variety of stakeholders to create a high-quality ner dataset for ten african languages. we evaluate multiple state-of-the-art ner models and establish strong baselines. we have released one of our best models that can recognize named entities in ten african languages on huggingface model hub. 7 we also investigate cross-domain transfer with experiments on five languages with the wikiann dataset, along with cross-lingual transfer for low-resource ner using the english conll-2003 dataset and other languages supported by xlm-r. in the future, we plan to use pretrained word embeddings such as glove @xcite and fasttext @xcite instead of random initialization for the cnn-bilstm-crf, increase the number of annotated sentences per language, and expand the dataset to more african languages.