Language Clustering for Multilingual Named Entity Recognition (2021)

Abstract: Recent work in multilingual natural language processing has shown progress in various tasks such as natural language inference and joint multilingual translation. Despite success in learning across many languages, challenges arise where multilingual training regimes often boost performance on some languages at the expense of others. For multilingual named entity recognition (NER) we propose a simple technique that groups similar languages together by using embeddings from a pre-trained masked language model, and automatically discovering language clusters in this embedding space. Specifically, we fine-tune an XLM-Roberta model on a language identification task, and use embeddings from this model for clustering. We conduct experiments on 15 diverse languages in the WikiAnn dataset and show our technique largely outperforms three baselines: (1) training a multilingual model jointly on all available languages, (2) training one monolingual model per language, and (3) grouping languages by linguistic family. We also conduct analyses showing meaningful multilingual transfer for low-resource languages (Swahili and Yoruba), despite being automatically grouped with other seemingly disparate languages.

## introduction large transformer language models @xcite have shown impressive progress on tasks across different languages, including joint multilingual learning. many works have focused on cross-lingual transfer from highto low-resource languages in a zero-or few-shot setting @xcite . however recent work has also highlighted that small amounts of data may be available for some low-resource languages, and even very few examples for fine-tuning on a target language can be effective @xcite . given these insights and the scarcity of studies that present a middle ground between monolingual and multilingual learning, we investigate methods for clustering languages to boost multilingual performance on named entity recognition (ner). one transformer model that has shown particularly strong performance on multilingual tasks is xlm-roberta @xcite , a variant of the roberta model @xcite that adapts the multilingual training regime of xlm @xcite to a commoncrawl corpus containing 100 languages. this model can be adapted to tasks in multiple languages, and we take this as the base model for ner fine-tuning. additionally, inspired by work in multilingual neural machine translation (nmt) @xcite , we investigate a method for grouping similar languages using an automated clustering method. we provide a focused evaluation of this method on 15 languages from the wikiann corpus @xcite following the train-test splits from @xcite and show that ner models trained on language clusters largely outperform (a) individual monolingual models trained for each language, (b) multilingual models trained on languages that are grouped by linguistic family, and (c) a single multilingual model trained on all available languages. ## clustering languages for multilingual ner while many of the works above provide insight into multilingual ner performance in both broad and narrow contexts, many focus on zero-or few-shot transfer, or linguistically similar language groups. our work seeks to fill a gap by studying multilingual ner performance for several diverse languages where data is available (though not evenly distributed) to understand how to best group languages for multilingual ner training. here we present our proposed automatic clustering approach to address this problem. to obtain input representations for a clustering algorithm, we use a pre-trained xlm-r model. 1 for each sentence in our corpus, we obtain a single vector for the sentence as the output from xlm-r. we then input these vectors to a clustering algorithm to obtain cluster-assigned labels for each sentence. to obtain the final cluster label for an entire language, we simply compute the majority vote of the clustering labels for all sentences within a language. while the base xlm-r model provides a good starting point for downstream tasks, we found that when clustering in this model's embedding space most languages were assigned to the same cluster regardless of the number of desired clusters. 2 thus, we fine-tune xlm-r on a language identification task where the model is trained to classify sentences into one of the 15 languages in the dataset. we then use the [cls] token embedding that is fed to the classification layer during finetuning as the input for clustering. this language identification model is fine-tuned for 3 epochs on the wikiann training set with a batch size of 20, and achieves an overall accuracy of 90% across all languages. figure 1 shows qualitative evidence of strong grouping of languages such as overlap between chinese and japanese that is reflected in assigned clusters in section 4 below. to automatically group languages, we follow @xcite in choosing bottom-up agglomerative clustering, which assigns each data point its own cluster and iteratively merges clusters such that the sum of squared distances between points within all clusters is minimized. similar to k-means, agglomerative clustering uses a k hyperparameter for the number of clusters, and after experimentation with @xmath0, 4, 5, 6} and noting sub-optimal groupings for many values of k, we set this parameter to 4. ## experimental setup for training ner models with this method, we group all sentences from languages that are assigned the same cluster and train and evaluate on these languages from the wikiann dataset. we compare these models against monolingual models for each language, a single multilingual model trained on all languages, and another set of grouped models using linguistic family as the assigned group. language groupings for the automated clustering method and the linguistically-informed method are shown in tables 1 and 2 respectively. we note several observations from these groupings. first, several languages appear in their own individual clusters when grouped by linguistic family (ja, ko, zh) or our clustering method (ar). in these cases results for grouped models are identical to those for monolingual models. second, we note differences between the automated and linguistic grouping methods, most notably the inclusion of yoruba and swahili in an otherwise indo-european cluster. this may be the result of few examples for these two languages in this dataset foot_2 , however we show in section 5 that this grouping is beneficial to these languages in our experiments despite being counter-intuitive from a linguistic perspective. finally, we note the grouping of chinese and japanese under the automatic clustering method, consistent with qualitative evidence from overlap in semantic space of the fine-tuned language classifier discussed above. ## results additional information about these results is plotted in figure 2 below. 4 here we use box plots to show the distribution of the class-averaged f1 we also note drastic performance improvement for swahili and yoruba when trained in a single multilingual model compared to monolingual training, consistent with previous findings for lowresource languages in multilingual settings @xcite @xcite . however, we observe best performance for these two languages when grouped using our proposed clustering method, which is somewhat surprising given the counterintuitive grouping with mostly european languages, though this grouping is also observed in previous work @xcite . this raises a question as to whether this improvement is due to effective learning of shared multilingual representations or whether it is primarily due to availability of more data of any kind. to test this, we evaluate ner models in a zero-shot framework where we train a multilingual model on all languages in cluster 2 with swahili and yoruba removed and evaluate this model on these two heldout languages. these results are presented in finally, as a test of generalization of our method we evaluate on the english test set of the connll-2003 ner dataset. we present results from training on four different training sets: the wikiann training set containing languages in cluster 2 (denoted wikiann in our results), the conll-2003 english training set @xcite , the combination of all wikiann and conll-2003 training data (denoted all), and finally the combination of the wikiann language cluster 2 training set with the conll-2003 training set (denoted as cluster combined). we train each model for a single run with the same settings described above, and present our results in we first note poor performance from the model trained solely on wikiann data, which is unsurprising given the domain mismatch and idiosyncrasies in each of the datasets. performance improves substantially in all cases where conll training data is used, with best performance noted in the "cluster combined" model, which slightly outperforms using all available training data from both datasets. this suggests that even in a new domain the multilingual representations of closely related languages may be helpful, and that utilizing related languages is more useful than simply combining all available multilingual training data as in the "all" setting. we finally note that, despite not being extensively tuned on this dataset, we achieve results within 3.5 f1 points of previously reported state of the art results on this test set @xcite . ## conclusion we have presented a simple data-driven clustering technique for improving performance on multilingual ner, and showed that this technique largely outperforms naive combination of all languages studied here within a single model, as well as outperforming monolingual models and models for languages grouped by linguistic family. we further tested whether improved performance for lowresource languages in the niger-congo family was solely the result of more available data and showed evidence of multilingual transfer via a focused zeroshot experiment. we believe this straightforward method can be easily applied to other multilingual settings as has been shown in previous work in nmt.