Building Low-Resource NER Models Using Non-Speaker Annotations (2021)

Abstract: In low-resource natural language processing (NLP), the key problems are a lack of target language training data, and a lack of native speakers to create it. Cross-lingual methods have had notable success in addressing these concerns, but in certain common circumstances, such as insufficient pre-training corpora or languages far from the source language, their performance suffers. In this work we propose a complementary approach to building low-resource Named Entity Recognition (NER) models using “non-speaker” (NS) annotations, provided by annotators with no prior experience in the target language. We recruit 30 participants in a carefully controlled annotation experiment with Indonesian, Russian, and Hindi. We show that use of NS annotators produces results that are consistently on par or better than cross-lingual methods built on modern contextual representations, and have the potential to outperform with additional effort. We conclude with observations of common annotation patterns and recommended implementation practices, and motivate how NS annotations can be used in addition to prior methods for improved performance.

## introduction work in low-resource languages is not only academically compelling, breaking from popular use of massive compute power on unlimited english data, but also useful, resulting in improved digital tools for under-resourced communities. two common strategies for low-resource nlp include (a) building cross-lingual models, and (b) annotating data in the target language. cross-lingual approaches -in which models are trained on some high-resource language, and applied to the target language -have been 1 for more details, see: http://cogcomp.org/page/publication_view/941 shown to be surprisingly effective @xcite . however, in common circumstances, such as when working with languages with insufficient training corpora or those far from the available source languages, cross-lingual methods suffer @xcite . absent sufficient crosslingual methods, conventional wisdom suggests that only native (or fluent) speakers of a language can provide useful data to train nlp models. but in low-resource scenarios, fluent speakers may not be readily available. to address this limitation, we hypothesize that the search for annotators can be extended beyond fluent speakers. in this work, we propose an unconventional approach for low-resource named entity recognition (ner) by getting annotations from annotators with no familiarity in the target language, referred to as "non-speaker" (ns) annotation. we posit that annotators are able to use phonetic, syntactic, and even semantic information from their languages of fluency to inform recognition. one example of how phonetic information can be used for ner annotation is shown in figure 1 . we test our hypothesis in a carefully controlled annotation experiment, comparing the performance of non-speaker (ns) annotators to that of fluent speakers (fs) in indonesian, russian, and hindi. our findings are summarized in two key takeaways: (1) non-speaker annotators are able to produce useful annotations despite having no experience annotating or learning the target language; and (2) non-speaker annotations are on par or better than cross-lingual methods built on modern contextual representations. we conclude with observations over factors that can influence ns annotation quality, such as availability of a good romanization system, or presence of capitalization in the target language. ## related work named entity recognition (ner) has been studied for many years @xcite @xcite , with most focus on english and a few other european languages @xcite . recently, there has been growing interest in low-resource nlp, with work in part-of-speech tagging @xcite , parsing (rasooli and collins, 2017), machine translation @xcite , and other fields. low-resource ner has seen work using wikipedia @xcite , self attention @xcite , and multilingual contextual representations @xcite . there has been a small amount of work using non-speaker annotations @xcite , but mainly as an application of a technique, falling short of the exhaustive study in this paper. several interfaces exist for non-speaker annotations in ner, including talen @xcite , which we use, elisa ie @xcite , and dragonfly @xcite , which performed small-scale experiments with non-speaker annotators. a similar approach has been proposed for machine translation @xcite and speech recognition @xcite . in the former case (assuming the translation direction is foreign-to-english), it is often sufficient to translate several of the most important content words, then reconstruct the most likely sentence that uses these. in speech recognition, it is possible to listen to a language one does not speak, and produce a phonetic transcriptions that can be aggregated with others into a reasonable transcription, a process referred to as mismatched crowdsourcing. language script capitalization example indonesian latin yes amerika russian cyrillic yes америка hindi devanagari no अमे रका ## experimental setup our experiment consisted of a series of trials, typically attended by 1-5 participants. each trial ran for four hours and consisted of three tasks: language selection we chose three target languages: indonesian, russian, and hindi. these languages were chosen based on availability of gold-annotated data and fluent speakers, and language difficulty. the constraint of available fluent speakers for annotation, which we use as a point of comparison on non-speaker annotation performance, led us to choose mid-to high-resource languages for evaluation. to read accounts of similar techniques used on true low-resource languages, see the applications section ( §4.3). we define language difficulty as the taskspecific difficulty experienced by an english speaker creating ner annotations in the target language. in practice, this difficulty mainly depends on script and capitalization, but may also depend on other factors such as language family and number of english loanwords. under this task-specific definition and relevant properties summarized in participant selection in total, there were 30 participants involved in the study, selected largely through a network of friends and acquaintances at the university of pennsylvania. all participants were uniformly paid $10/hour for their time and were preliminarily screened for language exposure. we chose not to use crowd-sourcing platforms, such as mechanical turk, to allow flexibility in administration format and recruitment strategy. the methodology for the study was approved by the institutional review board at the university. ## experiments & analysis this section describes the analysis done on the gathered fs and ns annotations, through the setup of our models and metrics used and key experimental takeaways. ## discussion while section 4 showed quantitative outcomes of experimental processes, this section explores the many factors that can contribute to obtaining high quality ns annotations. ## conclusion we demonstrate the effectiveness of using nonspeaker annotations as an alternative to crosslingual methods for building low-resource ner models. a qualitative exploration of the resulting data provides insights about what makes ns annotators so unintuitively successful. one avenue for future exploration is with active learning @xcite , which has been shown to help in low-resource situations @xcite . further work may also explore optimal ways to combine ns annotators with fs annotators, should they be available.