Do “ E nglish” Named Entity Recognizers Work Well on Global Englishes? (2023)

Abstract: The vast majority of the popular English named entity recognition (NER) datasets contain American or British English data, despite the existence of many global varieties of English. As such, it is unclear whether they generalize for analyzing use of English globally. To test this, we build a newswire dataset, the Worldwide English NER Dataset, to analyze NER model performance on low-resource English variants from around the world. We test widely used NER toolkits and transformer models, including models using the pre-trained contextual models RoBERTa and ELECTRA, on three datasets: a commonly used British English newswire dataset, CoNLL 2003, a more American focused dataset OntoNotes, and our global dataset. All models trained on the CoNLL or OntoNotes datasets experienced significant performance drops—over 10 F1 in some cases—when tested on the Worldwide English dataset. Upon examination of region-specific errors, we observe the greatest performance drops for Oceania and Africa, while Asia and the Middle East had comparatively strong performance. Lastly, we find that a combined model trained on the Worldwide dataset and either CoNLL or OntoNotes lost only 1-2 F1 on both test sets.

## introduction most of english named entity recognition (ner) uses american or british english data, with less attention paid to low-resource english contexts. multiple problems may occur in low-resource ner settings; for example, named entities with regionspecific meanings can be confused for common words. indeed, the japanese diet is a governmental body, but ner models focused on us and british english may incorrectly interpret this entity as a medical term. among many ner datasets released in recent years, 1 the most widely used datasets are conll 1 a collection of ner references is available at https: //github.com/juand-r/entity-recognition-datasets 2003 @xcite and ontonotes @xcite , which focus on british and american english, with significant european parliament coverage. other recently created ner datasets study the medical domain, such as the n2c2 challenges @xcite , historical english @xcite , or music recommendation terminology @xcite , still using american and british english. the lack of regional variety in these datasets suggests that models trained on these datasets might not accurately recognize entities from more global contexts. furthermore, the lack of test data for other regions makes it difficult to even measure this phenomenon. in this work, we evaluate the performance of a variety of ner tools, including flair and spacy on this dataset. we then retrain two commonly used ner models, those of stanza @xcite and @xcite , to see whether their performance improves when the worldwide english dataset is combined with their usual training data. using these toolkits allows for easy retraining while comparing the performance losses between models using categorical features, word embeddings, and transformers. when using stanza, in order to compare the generalization of different language models, we retrain with each of roberta-large @xcite , electra-large @xcite , and just the conll 2017 word vectors @xcite . we find that, as expected, there is a large dropoff in performance when the models are applied to regions not well covered by the training data in conll or ontonotes. this is true whether the model uses categorical features, word vectors, or transformer language models. when considering specific regions around the world, we find that indigenous oceania and africa had the worst performance due to a disproportionately large number of unrecognized tokens. we believe these results reflect the impact of minimal discussions of the global south in u.s. and u.k. newswire. across all regions, models trained on conll or ontonotes struggled to accurately identify cultural and currency names. furthermore, we find that retraining the models in question on the training split of the new dataset improves their performance on those regions, but has the opposite effect of hurting the performance on us and british english texts. when retrained on both an existing dataset and the new dataset at the same time, however, the models maintain their previous performance on conll or ontonotes while drastically improving their performance on other worldwide regions. we make our new dataset public to better represent the diversity in written english around the world. foot_0 2 related work ner has been explored in non-english languages with great success, with one example being stanford nlp's open-source stanza ner model @xcite , which covers many languages such as chinese and arabic, along with including several other english medical models. there has also been research on low-resource foreign languages with notable success using neuralbased approaches @xcite . however, low-resource contexts of english have not been recently examined. @xcite showed that western-english trained models performed poorly in south african contexts of english due to the presence of unknown words such as "xabanisa", or alternative uses of words like "peace", a common south african name. however, their model was a simple bayesian network, so their results are not indicative of how modern neural models might perform, especially with pretraining. @xcite revealed that nameregularity bias still exists in neural models, even when contextual clues are present. for example, in the sentence "obama is located in southwestern fukui prefecture.", state-of-the-art ner models labeled "obama" as a person rather than a location. ghaddar et al. did not examine contexts of english, so further research remains important. outside the ner realm, @xcite explored dialectic differences in machine translation and question answering, but did not address ner. ## experiments using stanza we first examine stanza models @xcite on conll and our worldwide english; we consider other ner systems and ontonotes in section 7. ## analysis to better understand model behavior in error cases, we examine the sentences with incorrect predictions, generalizing error patterns across all regions and examining region-specific challenges. ## alternate explanations although our hypothesis is that regional variations are the largest contributor to performance gaps between models, several other possibilities could be valid. we address some possible explanations here. dataset domain one possible explanation of the scores would be different domains. however, conll is trained entirely on newswire, and the majority of ontonotes is also newswire. therefore, aside from regional or temporal differences, the domains of the datasets should be very similar. differences in annotation style could also be a culprit. however, except where noted for the date label and the collapsing of location and gpe, we used conll and ontonotes as guidelines for the annotation team to follow. temporal drift a concerning problem would be temporal drift in annotations. we use a dataset similar to conll to demonstrate that temporal drift alone does not account for the differences seen @xcite . this dataset uses conll style annotations, but with news articles from 2020. excluding the asia portion of the training data. we find that although the asia dataset is a fraction of the entire dataset, training on just conll and asia is more accurate than the larger subsets. these results are summarized in data leakage another explanation would be that of data leakage from a regional training sets to the test set, especially in the case of a news story with worldwide coverage. for example, if an article from latin america covered an event concerning an american presidential election in the training data, an article from a news source in asia covering the same event might be randomly assigned to the test set, giving an unfair advantage to a model trained on the worldwide dataset. the data collection emphasized stories of regional interest, however, and so there should not be data leakage of international news stories. alternate embeddings an embedding with more worldwide coverage might also show less degradation. for example corenlp includes a model trained on conll, with the caveat that it does not use b-and i-tags, but rather assigns labels as either part of a class or o for outside. this is rarely a practical issue in english, with exceptions such as lists without commas. the model is a crf which uses text features of the words and their neighbors. for example, the word "corenlp" would itself be a feature, along with the prefixes "c", "co", . . . , and the suffixes "p", "lp", . . . . similarly, "corenlp" and other categorical features of the word would be used for features for the neighboring words. the baseline conll model included with corenlp has an f1 of 89.27% on conll03. @xcite testing the included model on the worldwide dataset shows a very large drop in performance, 70.47% f1. the largest drop was on organization, which went from 85.63% to 58.75% f1. we retrained the model on the worldwide dataset, achieving an f1 of 82.65%. the worldwide trained model also suffered a very large drop when tested on the conll dataset, falling to 63.67% f1. again, organization was the largest drop, going from 75.78% to 38.28% f1. a likely explanation for the large drop in performance is that categorical features can provide conperson and location, however, are much more robust when tested across datasets. for person, especially, even when the last names are different, many regions have similar first names. for example, anne, anthony, and tom are common first names in the indigenous test set. finally, we retrained the model on a combination of the conll and worldwide datasets. this model mostly combined the strengths of the two previous models, achieving 0.8304 f1 on the worldwide dataset and 0.8886 on conll. the number of features in the trained models increased from 658k for the conll model and 1m for the worldwide model to 1.4m for the combined model; this indicates there is very little overlap between the features for the two models, resulting in the model effectively learning the two datasets in parallel. when compared on a regional basis, we find that each model shows the same drop on indigenous regions, with the original corenlp conll model especially inaccurate on the worldwide indigenous section. considering there is some middle eastern text in the original conll, the corenlp model also does surprisingly poorly on the middle east region. ## conclusion we introduce an ner dataset composed of newswire from outside britain and america, designed to act as an "adversarial" evaluation set for english ner models solely trained on american or british contexts. we find limitations in stanza's ner when trained solely on conll or ontonotes and tested in global contexts of english (10 percent drop in f1 test scores). the model becomes most confused when contextual clues, which often disambiguate named entities, are missing. however, context clues are not a panacea-we also find that unknown tokens from the worldwide english contexts can cause confusion despite clear sentence context. these effects persist with the addition of pre-trained word embeddings from word2vec, roberta, and electra, suggesting that large models are not a catch-all solution to the challenge of classifying regional contexts. with that said, we find that the american and british english-trained model performs better on global texts with the addition of bert embeddings than without them. one takeaway from this study is that one way to achieve stronger ner performance across a diverse set of english contexts is to train on more diverse sets of english data. indeed, we find that the combined model demonstrates strong performance on both datasets. therefore, we propose that existing stateof-the-art models would do well to combine their data with similarly sourced global english corpora.