Employing W ikipedia as a resource for Named Entity Recognition in Morphologically complex under-resourced languages (2021)

Abstract: We propose a novel approach for rapid prototyping of named entity recognisers through the development of semi-automatically annotated datasets. We demonstrate the proposed pipeline on two under-resourced agglutinating languages: the Dravidian language Malayalam and the Bantu language isiZulu. Our approach is weakly supervised and bootstraps training data from Wikipedia and Google Knowledge Graph. Moreover, our approach is relatively language independent and can consequently be ported quickly (and hence cost-effectively) from one language to another, requiring only minor language-specific tailoring.

## introduction named entity recognition (ner) is the task of identifying proper names and assigning them to one of several named entity (ne) classes, such as person (per), location (loc) or @xcite , which is a crucial processing step for many nlp tasks, but also for many applications in the digital humanities where information about the entities involved (e.g. names of emperors or archaelogical sites) is often particularly important. while stateof-the-art systems obtain good results for standard ne inventories and general purpose english @xcite , annotated data sets for the development of named entity taggers are not readily available for most of the world's languages. 1 in this paper, we focus on semi-automatically generating annotated data and bootstrapping ne recognisers for under-resourced languages (cf. @xcite ), i.e., languages for which manually annotated data as well as pre-processing tools, such as part-of-speech taggers, are typically hard to come by. to this end, we propose a weakly supervised approach that bootstraps the training set from wikipedia (in the target language) and google knowledge graph (in english), requiring no manual annotation and no pre-processing apart from the language-specific tweaking of our matching heuristics. this approach is therefore in principle suitable for any language for which wikipedia articles exist. 2 because the manual effort is limited, systems can be quickly ported to new languages, while still obtaining reasonable results. we demonstrate this by developing the system for malayalam and then porting it to isizulu. these two languages were chosen because they are agglutinating and morphologically complex, making the task considerably more challenging than for many indo-european languages where nes are only minimally inflected. while our target languages are both agglutinating, they are also structurally quite different in other respects and exhibit different degrees of "under-resourcing", with noticeably fewer resources being available for isizulu (see sect. 3) 2 related work wikipedia has been employed for ner in three main ways: in a monolingual setting, early studies used it to extract gazetteer lists which were then used as features in (typically supervised) ner systems. one of the first studies taking this approach was by @xcite , who extract gazetteers by matching the first sentence of a wikipedia article heuristically against the wordnet @xcite noun hierarchy to identify the category of the entity described. this was followed by a number of similar approaches @xcite @xcite . going one step further, some researchers used wikipedia not only for extracting gazetteers but also for bootstrapping annotated training data. for example, @xcite exploit hyperlinks to annotate the sentences containing them with category information, which is extracted from the article the hyperlink links to. as not all mentions of an entity in an article are hyperlinked, they extend the data set by finding verbatim repetitions of the hyperlink's anchor text in the article. finally, they use the data to train an ne tagger. the system requires hand-labelling of seed data that maps information extracted from articles to ne classes. wikipedia has also been used in a multilingual setting to obtain ne taggers for languages other than english, e.g. by exploiting cross-lingual links between articles @xcite @xcite . this approach has also been applied to under-resourced languages @xcite . @xcite go one step further and construct entity type mappings for the english wikipedia before projecting across wikipedia language links. bouamor et al. @xcite propose employing wikipedia as a resource for creating domain-specific lexicons for machine-translation. they demonstrate their approach for english-french and english-romanian translation tasks. mayhew et al. @xcite combine lexicon-based translation of training data from a source to a target language with features generated from wikipedia and show that this approach can be applied to under-resourced languages. studies that address ner for our target languages are very limited. to our knowledge the first ner system for malayalam was proposed by @xcite , who use supervised machine learning utilising a variety of features complemented with a finite-state automaton to deal with complex words. @xcite propose a hybrid approach that combines rules with supervised machine learning. @xcite tackle named entity extraction from social media and combine supervised machine learning (svms) with skipgram features. @xcite propose another supervised approach based on the tnt tagger @xcite and maximum entropy models. a neural network approach is proposed by @xcite who use word embeddings of context words and morphs of the target word as features. a similar system but with a different neural architecture (rnn-lstm) has also been proposed @xcite . to our knowledge, the only ner system for isizulu was proposed by @xcite , who used linear-chain conditional random fields (crfs) for the classification of the named entities. the features included gazetteer lists and graphemic information (capitalization, punctuation, numerals). ## the target languages: malayalam and isizulu we test our system on two agglutinating languages: malayalam and isizulu. we hypothesise that inflection and agglutination will make the task particularly challenging, as one token can correspond to several linguistic words (see sec. 3.1 and 3.2). however, malayalam and isizulu also differ in several aspects: they use different writing systems (brahmic vs. latin) and while the former tends to make extensive use of suffixes the latter tends to favour prefixes to encode grammatical information. from a practical perspective, while both languages are under-resourced, isizulu is so to a greater extent, in particular its wikipedia version is more than an order of magnitude smaller (see sec. 4). we thus believe that these two languages pose sufficiently heterogeneous use cases. ## data sets and resources for bootstrapping training data, we utilise the malayalam and isizulu wikipedias. the former is significantly larger (65,000 vs. 2,701 articles as of june 2020) and therefore also gives rise to a larger training set. in order to find appropriate named entity tags for wikipedia articles (see sect. 5.2) we employ the google knowledge graph (gkg) @xcite . we test our system on two external data sets for malayalam (arnekt and cusat) and one for isizulu (nchlt ii) as well as part of our bootstrapped data: ## bootstrapping the training data as our focus is on under-resourced languages, we do not assume that a manually labelled training set is available. instead we bootstrap from wikipedia and gkg. utilising wikipedia has a number of advantages: first, as it is communitydriven, many under-resourced languages have a version of wikipedia. second, wikipedia articles cover a wide range of subjects and often refer to named entities. third, wikipedia has a number of features that help with bootstrapping entity labels (see sect. 2). finally, it has been shown that additional training data bootstrapped from wikipedia can also improve the performance of taggers trained on other sources, especially if they are applied out-of-domain @xcite . we employ a 4-step pipeline to bootstrap ne labelled data (fig. 1 ): first, we extract a list of titles from wikipedia dumps in the target language. second, we use the wikipedia language links to look up their english counterparts. third, we employ the gkg to extract candidates for named entity tags. finally, we use the title list to annotate wikipedia articles. the distribution of the different ne tags for both data sets is shown in ## experiments and machine learning setup for comparison, we use two baseline systems. one rule-based baseline annotation system and a neural network baseline. the rule-based baseline directly annotates the data sets with the title list generated in section 5.1. a bi-gram search is used to annotate titles that have two words. this procedure does not account for inflections and annotates perfect matches in the corpus. the rule-based baseline is therefore language independent. this system is used to evaluate the importance of accommodating inflection and agglutination when compiling an ner data set for morphologically complex languages. the deep learning baseline for ner is implemented using keras @xcite . it is a recurrent lstm network with the following layers: the model is trained using the rmsprop optimizer with a learning rate of 0.001 for 10 epochs. we use xlm-roberta @xcite ) to build the ner system. it is a pre-trained multilingual transformer model which has successfully been applied to low resourced languages such as swahili and urdu. the model is trained in the xlm-roberta-base configuration using decoupled weight decay @xcite and layer-wise decaying learning rates @xcite . the embedding layer is frozen to avoid overfitting. we train the model on a tpu in google colab foot_9 using bfloat16 mixed precision training and the following hyperparameters: • weight decay factor: 0.99 the data sets are split into training, testing and validation sets by a 80:10:10 ratio. ## analysis of results since testing on wikiml can be regarded as indomain, we focus on the analysis of errors on the external data sets. figures 4 and 5 display the confusion matrices of the results obtained when testing our model before and after fine tuning. in both cases, the biggest source of error is observed to be misclassification into the "other" category. this is expected in languages with morphological complexities, since named entities are concealed within agglutinations and suffixes. it should be noted that for arnekt, the performance errors do not necessarily originate from the model. to our knowledge, arnekt was not manually annotated, but created with rule based annotation procedures and word lists. consequently, annotation errors can be observed within the data set. in some cases, the wikiml model is seen to predict correct named entity tags for tokens wrongly annotated in arnekt. two examples are presented in (4) and (5). wrong annotations have been highlighted in red. fine tuning clearly improves the impact of errors involving the "other" category significantly. arnekt: prediction: u»m place organization hnä¿oi other other fnhm }xç°o ther other bwm other other bn°o ther person so°_±tp°o ther person figure 5: confusion matrix for arnekt before (above) and after (below) fine tuning jan tinbergen is a dutch economist (5) tokens: arnekt: prediction: }fpd¹°o ther place }io•êm other person nco}x¿otd person other gêeqw other other aoi» other other xncºjotdncnjnwm other other aqcjo person person murali is one of the best players in the history of sri lankan cricket for cusat, the presence of out-ofdomain/unseen words is clearly the cause of most errors in the vanilla model. once fine-tuned with a portion of the data set, this is reduced significantly. disregarding the "other" class, the model seems to confuse "person" and "organization" entites with "place" entities in both data sets. this is almost always observed with multi worded entities that have places embedded in their names. cases include people with places attached to them (as explained in section 5.3.1) and organizations with the same characteristic (e.g. "new york public library"). the "new york" portion in such entities can be thought of as a place entity embedded in an organization entity, or can be viewed simply as an organization entity without taking into account embedded entity classes. for one-worded entities, errors can often be seen to arise from annotation variations between the ground truth and the automatically generated dataset. for example, words such as "library" and "college" are mapped as "place" entities by the google knowledge graph during the generation of title lists. subsequently, instances of such words are labeled as "place" by our vanilla model trained on the wikiml dataset. however, the external datasets label them as "organization" entities in some cases, which indirectly translates to mistakes during evaluation. ## conclusion we demonstrated the implementation of a fully automated pipeline for the creation of a named entity tagged data set with freely available resources. we showed how the pipeline can be adaptated for two morphologically complex, agglutinating languages. finally, we propose an easily portable, weakly supervised ner system for malayalam and isizulu based on this pipeline. the system can be developed quickly: we spent 2 weeks on developing the initial system for malayalam and 2 days for porting it to isizulu. we tested in-and out-of-domain on a number of publicly available data sets, with encouraging results, especially for malayalam.