Extending Multilingual BERT to Low-Resource Languages (2020)

Abstract: Multilingual BERT (M-BERT) has been a huge success in both supervised and zero-shot cross-lingual transfer learning. However, this success is focused only on the top 104 languages in Wikipedia it was trained on. In this paper, we propose a simple but effective approach to extend M-BERT E-MBERT so it can benefit any new language, and show that our approach aids languages that are already in M-BERT as well. We perform an extensive set of experiments with Named Entity Recognition (NER) on 27 languages, only 16 of which are in M-BERT, and show an average increase of about 6% F1 on M-BERT languages and 23% F1 increase on new languages. We release models and code at http://cogcomp.org/page/publication_view/912 .

## introduction recent works @xcite have shown the zero-shot cross-lingual ability of m-bert @xcite on various semantic and syntactic tasks -just fine-tuning on english data allows the model to perform well on other languages. cross-lingual learning is imperative for low-resource languages such as somali and uyghur, as obtaining supervised training data in these languages is particularly hard. however, m-bert is not pre-trained with these languages, thus limiting its performance on them. languages like oromo, hausa, amharic and akan are spoken by more than 20 million people, yet m-bert does not cover them. indeed, there are about 4000 foot_1 writ- figure 1 : comparison between m-bert and our proposed approach e-mbert: we report average zero-shot ner performance on 16 languages that are already in m-bert and 11 new language that are out of m-bert; m-bert performance with supervised ner data is also reported as an upper-bound. in both languages in m-bert and out of m-bert, our method e-mbert performs better than m-bert. to accommodate a language not in m-bert, we propose an efficient approach, extend. extend works by first enlarging the vocabulary of m-bert to accommodate the new language and then continuing pre-training on this language. our approach trains for less than 7 hours on a single cloud tpu. we perform comprehensive experiments on cross-lingual ner on the lorelei dataset (strassel and tracey, 2016) with 27 languages of which 11 languages are not present in m-bert. as shown in figure 1 , our approach significantly outperforms m-bert when the target language is not in the 104 languages in m-bert and it is superior to m-bert even for the high-resource languages that are already in it. the key contributions of our work are (i) extend, a simple yet novel approach to add a new language to m-bert, (ii) experiments that show extend improves m-bert for languages that are in m-bert as well as those that are not, (iii) results showing that extend provides performance and efficiency improvements, in most cases, over b-bert. ## related works cross-lingual learning has seen increased interest in nlp, with such works as bicca @xcite , laser @xcite and xlm @xcite . although these models have been successful, they need cross-lingual supervision such as bilingual dictionaries or parallel corpora @xcite , which are particularly challenging to obtain for low-resource languages. our work differs in that we do not require such supervision. while other approaches like muse @xcite and vecmap @xcite can work without any cross-lingual supervision, m-bert alone often outperforms these approaches @xcite . @xcite has a continuing training setting that is similar to ours. however, their approach focuses on comparing between whether b-bert (jointpair) learns cross-lingual features from overlapping word-pieces, while ours aims at improving m-bert on target languages, and addresses the problem of missing word-pieces. we show that our extend method works well on m-bert, and is better than b-bert in several languages, whereas their method (monotrans) has a similar performance as b-bert. this implies that our extend method benefits from the multilinguality of the base model (m-bert vs bert). a recent work on multilingual bert (wu and dredze, 2020) reveals that a monolingual bert underperforms multilingual bert on low-resource cases. our work also identifies this phenomenon in some languages (see appendix), and we then present an effective way of extending m-bert to work even better than multilingual bert on these low-resource languages. ## our method: extend in this section, we discuss our training protocol extend which incorporates the target language by extending the vocabulary, encoders and decoders, and then continues pre-training. let m-bert's vocabulary be v mbert and let the extended new vocabulary be v new . throughout the paper, we fix the size of |v new | = 30, 000. the training goes as following: 1. extend the vocabulary, encoder, and decoder to accommodate v new . that is, let |v extra | = |v new -v mbert |, and increase the dimension of size |v mbert | to |v mbert | + |v extra |. 2. initialize all the new weights with m-bert's default weight initialization. 3. continue pre-training with monolingual data of the target language. we call the trained model e-mbert. ## experiments the goal of our experiments is to establish the extent to which our method extend and the resulting model e-mbert (i) improves over m-bert, (ii) does not necessary require additional monolingual data to continue training on, and (iii) is both effective and efficient compared to b-bert. ## conclusions and future work we proposed extend, an efficient method that extends m-bert to deal with languages that were originally outside it. our method has shown greatly improved performance across several languages comparing to m-bert and b-bert. while extend deals with one language each time, it would be an interesting future work to extend on multiple languages at the same time. furthermore, instead of randomly initializing the embeddings of a new vocabulary, we could possibly use alignment models like muse or vecmap with bilingual dictionaries to initialize. we could also try to apply our approach to better models like roberta @xcite . xuezhe ma and eduard hovy. 2016. end-to-end sequence labeling via bi-directional lstm-cnns-crf. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1064-1074, berlin, germany. association for computational linguistics. telmo pires, eva schlinger, and dan garrette. 2019. how multilingual is multilingual bert? in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4996-5001, florence, italy. association for computational linguistics. sebastian schuster, sonal gupta, rushin shah, and mike lewis. 2019. cross-lingual transfer learning for multilingual task oriented dialog. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 3795-3805, minneapolis, minnesota. association for computational linguistics. stephanie strassel and jennifer tracey. 2016. lorelei language packs: data, tools, and resources for technology development in low resource languages. in proceedings of the tenth international conference on language resources and evaluation (lrec 2016), pages 3273-3280. shyam upadhyay, manaal faruqui, chris dyer, and dan roth. 2016. cross-lingual models of word embeddings: an empirical comparison. in acl. shijie wu and mark dredze. 2019. beto, bentz, becas: the surprising cross-lingual effectiveness of bert. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 833-844, hong kong, china. association for computational linguistics. shijie wu and mark dredze. 2020. are all languages created equal in multilingual bert? arxiv preprint arxiv:2005.09093.