Sources of Transfer in Multilingual Named Entity Recognition (2020)

Abstract: Named-entities are inherently multilingual, and annotations in any given language may be limited. This motivates us to consider polyglot named-entity recognition (NER), where one model is trained using annotated data drawn from more than one language. However, a straightforward implementation of this simple idea does not always work in practice: naive training of NER models using annotated data drawn from multiple languages consistently underperforms models trained on monolingual data alone, despite having access to more training data. The starting point of this paper is a simple solution to this problem, in which polyglot models are fine-tuned on monolingual data to consistently and significantly outperform their monolingual counterparts. To explain this phenomena, we explore the sources of multilingual transfer in polyglot NER models and examine the weight structure of polyglot models compared to their monolingual counterparts. We find that polyglot models efficiently share many parameters across languages and that fine-tuning may utilize a large number of those parameters.

## introduction multilingual learning-using data from multiple languages to train a single model-can take many forms, such as adapting a model from a highresource to low-resource language @xcite @xcite @xcite , taking advantage of beneficial multilingual features or datasets @xcite @xcite , and unsupervised representation learning @xcite . we adopt the term "polyglot" from @xcite to refer to models that are trained on and applied to multiple languages. there are several advantages to training a single polyglot model across languages. single models ease production requirements; only one model need be maintained. they can be more efficient, using fewer parameters than multiple monolingual models. additionally, they can enable multilingual transfer @xcite @xcite . however, a key goal of polyglot learning concerns producing a single model that does better on each language than a monolingual model. in the context of named entity recognition, we may expect aspects of the task to transfer across languages. for example, since entity names tend to be transliterated or directly used across languages, even distant languages may see benefit from training a single model, e.g. "apple" (company) is rendered as such in french rather than as "pomme." intuitively, the more similar and the larger the set of languages, the more we should expect to see a benefit from considering them jointly. these polyglot models can take advantage of different sets of labeled corpora in different languages @xcite . nevertheless, progress towards this goal remains mixed; polyglot models often do not improve results in each language @xcite @xcite . models trained across all languages come close but typically fail to outperform monolingual models. thus, while multilingual learning can benefit low resource languages through transfer and simplify models by sharing one across all languages, it fails to realize a key goal: improving results in each language. our experiments in §4 confirm this negative result in two different multilingual settings for 4 different neural ner models. our first contribution is a technique in which a polyglot ner model can be adapted to a target language by fine-tuning on monolingual data. a similar continued training approach to transfer has been explored for domain adaptation in neural machine translation @xcite ; we show that it works with polyglot models for ner, improving performance by up to 3 f 1 over monolingual baselines. our second contribution is an explanation of the surprising effectiveness of this technique through an extensive empirical study of polyglot models for ner. we compare several types of neural ner models, including three character (or byte) level architectures, and evaluate transfer across a small (4) and large (10) set of languages. in particular, we find that: to our knowledge, ours is the first systematic study of polyglot ner models. ## related work there is a long history of multilingual learning for ner @xcite @xcite . this work has is driven by an interest in learning ner models for many languages @xcite and the relative lack of data for many languages of interest @xcite . polyglot models @xcite and @xcite showed that a single neural mt model could benefit from being trained in a multilingual setting. @xcite showed similar results for ner, presenting a model that benefited from learning to perform ner on 4 languages at once. we find that other polyglot ner models are rarely better than monolingual models in terms of absolute performance. @xcite showed that polyglot language model pretraining can help improve performance on ner tasks, although polyglot ner training hurts. however, multilingual bert @xcite , when compared to monolingual bert performance on ner, shows that polyglot pretraining is not always beneficial for downstream tasks. finally, most recently, @xcite showed how to train a single model on 75 languages for dependency parsing while retaining competitive performance or improving performance, mostly on low-resource languages. this work is closely related to ours, although we are predominantly interested in how we can leverage polyglot learning to improve performance across all languages. cross-lingual models cross-lingual transfer leverages labeled data from different source languages to augment data for a target language. @xcite do this on a massive scale for ner, leveraging over 40 languages for crosslingual transfer. @xcite employed selfattention to combat word-order differences when transferring parameters from high-resource languages to low-resource. much work in this space has looked at how to leverage a mixture of shared features and language-specific features @xcite , similar to domain adaptation techniques @xcite . recently, a lot of this work has focused on using adversarial models to force models to learn language-agnostic feature spaces @xcite . these works show, similar to our work, that it is possible to leverage multilingual data to increase performance across languages. ## models we evaluate three polyglot ner neural models. foot_0 ## experiments each lorelei language has less than half the data of a conll language, but in total, the two datasets are roughly equal in size. the conll setting consists of european languages in the same alphabet, and prior work has shown beneficial transfer in this setting @xcite . lorelei is more challenging because it contains more distantly related languages. we train a monolingual ner model for each language (14 models) and two polyglot models: conll and lorelei. for polyglot training we concatenate each annotated language-specific dataset into one combined corpus. because our language-specific datasets are comparable in size we do not correct for minor size differences. 6 all models were trained over 5 random seeds, with the best model selected by development performance. for polyglot models, we select the best model using the average development performance across all languages. results the exception to these observations is the bts model, which showed significant improvements in the polyglot settings, matching the conclusion of @xcite . however, our implementation failed to match the higher numbers of the original paper, and so the model is significantly worse overall compared to the other ner models. perhaps the unique architecture of bts enables it to improve in the polyglot setting. however, if bts requires more training data to achieve results similar to the other models, the polyglot improvements may not hold up. conclusion polyglot ner models fail to improve over their monolingual counterparts, despite using 4 (conll) or 10 (lorelei) times more labeled data. discrepancies of label priors between languages do not, by themselves, account for this. ## how do polyglot models learn? we now turn our attention towards understanding how polyglot models are transferring information across languages. we examine the types of errors made in each setting, as well as how polyglot models efficiently use parameters and how parameter weights are shared across languages. ## conclusions we explore the benefits of polyglot training for ner across a range of models. we find that, while not all models can benefit in performance from polyglot training, the parameters learned by those models can be leveraged in a language-specific way to consistently outperform monolingual models. we probe properties of polyglot ner models, and find that they are much more efficient than monolingual models in terms of the parameters they require, while generally maintaining a competitive performance across all languages. we show that the high amount of parameter sharing in polyglot models partially explains this, and additionally find that language-specific fine-tuning may use a large portion of those shared parameters. in future work, we will explore whether the observed trends hold in much larger polyglot settings, e.g. the wikiann ner corpus @xcite . finally, regarding the sharing of weights between languages in polyglot models, our key conclusion is that standard training objectives are unable to find an optimum which simultaneously achieves high task performance across all languages. with this in mind, exploring different training strategies, such as multi-objective optimization, may prove beneficial @xcite . on the other hand, when the objective is to maximize performance on a single target language it may be possible to improve the proposed fine-tuning approach further using methods such as elastic weight consolidation @xcite .