On the Choice of Auxiliary Languages for Improved Sequence Tagging (2020)

Abstract: Recent work showed that embeddings from related languages can improve the performance of sequence tagging, even for monolingual models. In this analysis paper, we investigate whether the best auxiliary language can be predicted based on language distances and show that the most related language is not always the best auxiliary language. Further, we show that attention-based meta-embeddings can effectively combine pre-trained embeddings from different languages for sequence tagging and set new state-of-the-art results for part-of-speech tagging in five languages.

## introduction state-of-the-art methods for sequence tagging tasks, such as named entity recognition (ner) and part-of-speech (pos) tagging, exploit embeddings as input representation. recent work suggested to include embeddings trained on related languages as auxiliary embeddings to improve model performance: catalan and portuguese embeddings, for instance, help ner models on spanish-english code-switching data @xcite . in this paper, we analyze whether auxiliary embeddings should be chosen from related languages, or if embeddings from more distant languages could also help. for this, we revisit current language distance measures @xcite and adapt them to the embeddings and training data used in our experiments. we investigate the question, whether we can predict the best auxiliary language based on those language distance measures. our results suggest that no strong correlation exists between language distance and performance and that even less related languages can be a good choice as auxiliary languages. in our experiments, we explore both available monolingual and multilingual pre-trained byte-pair encoding @xcite and flair embeddings @xcite . for combining monolingual subword embeddings from different languages, we investigate two different methods: the concatenation of embeddings and the use of attention-based meta-embeddings @xcite . we perform experiments on conll and universal dependency datasets for ner and pos tagging in five languages and show that meta-embeddings are a promising alternative to the concatenation of additional auxiliary embeddings as they learn to decide on the auxiliary languages in an unsupervised way. moreover, the inclusion of more languages is often beneficial and meta-embeddings can be effectively used to leverage a larger number of embeddings and achieve new state-of-theart performance on all five pos tagging tasks. finally, we propose guidelines to decide which auxiliary languages one should use in which setting. language distance measures. @xcite proposed to measure distances between languages by using the perplexity of language models trained on one language and applied to another language. campos et al. (2019) used a similar method to retrace changes in multilingual diachronic corpora over time. another popular measure of similarity is based on vocabulary overlap, assuming that similar languages share a large portion of their vocabulary @xcite . ## sequence tagging model following @xcite , we use a bidirectional long short-term memory network (bil-stm) @xcite followed by a conditional random field (crf) classifier @xcite (see figure 1a ). ## conclusion in this paper, we investigated the benefits of auxiliary languages for sequence tagging. we showed that it is hard to predict the best auxiliary language based on language distances. we further showed that meta-embeddings can leverage multiple embeddings effectively for those tasks and set the new state of the art on part-of-speech tagging across different languages. finally, we proposed a guide on how to decide which method of including auxiliary languages one should use.