M ul DA : A Multilingual Data Augmentation Framework for Low-Resource Cross-Lingual NER (2021)

Abstract: Named Entity Recognition (NER) for low-resource languages is a both practical and challenging research problem. This paper addresses zero-shot transfer for cross-lingual NER, especially when the amount of source-language training data is also limited. The paper first proposes a simple but effective labeled sequence translation method to translate source-language training data to target languages and avoids problems such as word order change and entity span determination. With the source-language data as well as the translated data, a generation-based multilingual data augmentation method is introduced to further increase diversity by generating synthetic labeled data in multiple languages. These augmented data enable the language model based NER models to generalize better with both the language-specific features from the target-language synthetic data and the language-independent features from multilingual synthetic data. An extensive set of experiments were conducted to demonstrate encouraging cross-lingual transfer performance of the new research on a wide variety of target languages.

## introduction named entity recognition (ner) aims to identify and classify entities in a text into predefined types, which is an essential tool for information extraction. it has also been proven to be useful in various downstream natural language processing (nlp) tasks, including information retrieval @xcite , question answering @xcite ) and text summarization @xcite . however, except for some resource-rich languages (e.g., english, german), training sets for most of the other languages are still very limited. moreover, it is usually expensive and time-consuming to annotate such data, particularly for low-resource languages @xcite . therefore, zero-shot cross-lingual ner has attracted growing interest recently, especially with the influx of deep learning methods @xcite @xcite . existing approaches to cross-lingual ner can be roughly grouped into two main categories: instance-based transfer via machine translation (mt) and label projection @xcite , and model-based transfer with aligned cross-lingual word representations or pretrained multilingual language models @xcite @xcite @xcite . recently, @xcite unify instance-based and model-based transfer via knowledge distillation. these recent methods have demonstrated promising zero-shot cross-lingual ner performance. however, most of them assume the availability of a considerable amount of training data in the source language. when we reduce the size of the training data, we observe significant performance decrease. for instance-based transfer, decreasing training set size also amplifies the negative impact of the noise introduced by mt and label projection. for model-based transfer, although the large-scale pretrained multilingual language models (lm) @xcite have achieved state-of-the-art performance on many cross-lingual transfer tasks, simply fine-tuning them on a small training set is prone to over-fitting @xcite @xcite . to address the above problems under the setting of low-resource cross-lingual ner, we propose a multilingual data augmentation (mulda) framework to make better use of the cross-lingual generalization ability of the pretrained multilingual lms. specifically, we consider a low-resource setting for cross-lingual ner, where there is very limited source-language training data and no targetlanguage train/dev data. such setting is practical and useful in many real scenarios. our proposed framework seeks the initial help from the instance-based transfer (i.e., translate train) paradigm @xcite . we first introduce a novel labeled sequence translation method to translate the training data to the target language as well as to other languages. this allows us to finetune the lm based ner model on multilingual data rather than on the sourcelanguage data only, which helps prevent over-fitting on the language-specific features. one commonly used tool for translation is the off-the-shelf google translate system foot_1 , which supports more than 100 languages. alternatively, there are also many pretrained mt models conveniently accessible, e.g., more than 1,000 @xcite models have been released on the hugging face model hub. 3 note that the instance-based transfer methods add limited semantic variety to the training set, since they only translate entities and the corresponding contexts to a different language. in contrast, data augmentation has been proven to be a successful method for tackling the data scarcity problem. inspired by a recent monolingual data augmentation method @xcite , we propose a generation-based multilingual data augmentation method to increase the diversity, where lms are trained on multilingual labeled data and then used to generate more synthetic training data. we conduct extensive experiments and analysis to verify the effectiveness of our methods. our main contributions can be summarized as follows: ## mulda: our multilingual data augmentation framework we propose a multilingual data augmentation framework that leverages the advantages of both instance-based and model-based transfer for crosslingual ner. in our framework, a novel labeled sequence translation method is first introduced to translate the annotated training data from the source language s to a set of target languages @xmath0, . . . , t n }. then language models are trained on {d s , d t 1 , ..., d tn } to generate multilingual synthetic data, where d s is the sourcelanguage training data, and d t i is the translated data in language t i . finally, we post-process and filter the augmented data to train multilingual ner models for inference on target-language test sets. ## experiments we conduct experiments to evaluate the effectiveness of the proposed multilingual data augmentation framework. firstly, we compare our labeled sequence translation method with the previous instance-based transfer (i.e., translate train) methods. following that, we show the benefit of adding multilingual translations. then we continue the transformer decoder is trained to generate labeled sequences autoregressively. following the mbart pretraining tasks, we add language tokens at the end of masked sequences when feed them into encoder. ## related work cross-lingual ner there has been growing interest in cross-lingual ner. prior approaches can be grouped into two main categories, instancebased transfer and model-based transfer. instancebased transfer translates source-language training data to target language, and then apply label projection to annotate the translated data @xcite . instead of mt, some earlier approaches also use parallel corpora to construct pseudo training data in the target language @xcite . to minimize resource requirement, @xcite and @xcite design frameworks that only rely on word-to-word/phrase-to-phrase translation with bilingual dictionaries. besides, there are also many studies on improving label projection quality with additional feature or better mapping methods @xcite . different from these methods, our labeled sentence translation approach leverages placeholders to determine the position of entities after translation, which effectively avoids many issues during label projection, such as word order change, entity span determination, noise-sensitive similarity metrics and so on. model-based transfer directly applies the model trained on the source language to the targetlanguage test data @xcite @xcite , which heavily relies on the quality of cross-lingual representations. recent methods have achieved significant performance improvement by fine-tuning large scale pretrained multilingual lms @xcite @xcite . besides, there are also some approaches that combine instance-based and model-based transfer @xcite . compared with these methods, our approach leverages mt models and lms to add more diversity to the training data, and prevents over-fitting on language-specific features by fine-tuning ner models on multilingual data. data augmentation data augmentation @xcite adds more diversity to training data to help improve model generalization, which has been widely used in many fields, such as computer vision @xcite , speech @xcite , nlp @xcite and so on. for nlp, back translation @xcite is one of the most successful data augmentation approaches, which translates target-language monolingual data to the source language to generate more parallel data for mt model training. other popular approaches include synonym replacement @xcite , random deletion/swap/insertion @xcite , generation @xcite , etc. data augmentation has also been proven to be useful in the cross-lingual settings @xcite @xcite @xcite , but most of the exiting methods overlook the better utilization of multilingual training data when such resources are available. ## conclusions we have proposed a multilingual data augmentation framework for low resource cross-lingual ner. our labeled sequence translation method effectively avoids many label projection related problems by leveraging placeholders during mt. our generation-based multilingual data augmentation method generates high quality synthetic training data to add more diversity. the proposed framework has demonstrated encouraging performance improvement in various low-resource settings and across a wide range of target languages.