Discovering Representation Sprachbund For Multilingual Pre-Training (2021)

Abstract: Multilingual pre-trained models have demonstrated their effectiveness in many multilingual NLP tasks and enabled zero-shot or few-shot transfer from high-resource languages to low-resource ones. However, due to significant typological differences and contradictions between some languages, such models usually perform poorly on many languages and cross-lingual settings, which shows the difficulty of learning a single model to handle massive diverse languages well at the same time. To alleviate this issue, we present a new multilingual pre-training pipeline. We propose to generate language representation from multilingual pre-trained model and conduct linguistic analysis to show that language representation similarity reflects linguistic similarity from multiple perspectives, including language family, geographical sprachbund, lexicostatistics, and syntax. Then we cluster all the target languages into multiple groups and name each group as a representation sprachbund. Thus, languages in the same representation sprachbund are supposed to boost each other in both pre-training and fine-tuning as they share rich linguistic similarity. We pre-train one multilingual model for each representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.

## introduction the use of pre-trained models is considered a milestone in the development of nlp research. though early works @xcite on monolingual pre-training (pre-training one model for one language) significantly boosts the performance on the target language, monolingual pre-training can hardly be generalized to multilingual settings because of high training cost and insufficient corpora resources for many languages. multilingual pre-training was proposed to resolve this issue. by using shared vocabulary across languages and pre-training with corpora from multiple languages, multilingual pre-trained models handle cross-lingual tasks in one model. large scale multilingual pre-trained models provide powerful representation for languages worldwide, enabling significant advances in various multilingual tasks. however, existing widely used multilingual pre-trained models @xcite @xcite perform poorly on many languages and some cross-lingual tasks like zero/few-shot cross-lingual transfer. e.g. the performance of zero shot transfer on xnli task from english data to urdu language is 15%+ lower than to english @xcite . such a huge performance gap is the result of cross-lingual contradictions and differences. this phenomenon is also recognized as negative transfer in transfer learning @xcite . many existing works in cross-lingual transfer @xcite @xcite and machine translation @xcite have shown that cross-lingual transfer works best between typologically similar languages. we believe that utilizing similarity between languages is potentially beneficial for large-scale multilingual pre-training. motivated by this, we propose a new multilingual pre-training pipeline. first, we design a fully data-driven end-to-end way to generate language representation for all languages (108 languages) based on massive multilingual corpora. we represent each language as a 768-dimension vector and use cosine similarity as their similarity measure. with the similarities between languages quantified by their language representation, we automatically divide all languages into a small number of representation sprachbunds. sprachbund is a linguistic terminology in german that refers to a group of close languages (sprach means language and bund means federation in german, so literally it is "language federation" in english). a representation sprachbund is defined as a group of languages with similar language representation. we conduct extensive linguistic analysis and show that languages with similar representation are similar and related from many linguistic perspectives, including language family, geographical sprachbund, lexicostatistics and syntax typology. we believe that training with similar languages in pre-training and fine-tuning are beneficial as they share similar linguistic properties. second, we train multiple multilingual pre-trained models. each model is trained with corpora from one representation sprachbund. when handling downstream tasks in one specific language, we fine-tune the model pre-trained with the corresponding representation sprachbund corpora. we conduct experiments on 8 representative cross-lingual tasks from xglue @xcite and xtreme @xcite including sentence classification, structure prediction, question answering and sentence retrieval. experiment results show that our model significantly outperforms strong baselines. our contributions can be summarized as follows: i) we propose a way to automatically generate language representation from multilingual pre-trained models and massive multilingual corpora. ii) we conduct extensive analysis to show language representation and representation sprachbunds can reflect linguistic language similarity and relatedness from multiple perspectives, therefore they can be considered as new paradigm for clustering similar languages in linguistics. iii) we use representation sprachbunds in multilingual pre-training to alleviate the cross-lingual contradiction and differences, and obtain significant improvements compared with strong baselines. ## related work our approach presents a new pipeline of multilingual pre-training. our representation sprachbund is inspired by linguistic language clustering. our analysis is closely related to methodology in linguistics. multilingual pre-training multilingual pretraining was proposed to pre-train a single model with hundreds of languages. many works use a large amount of multilingual data (e.g., mc4 @xcite , ccnet @xcite ) to pretrain large multilingual models like xlm (lam-ple and conneau, 2019), xlm-r @xcite , unicoder @xcite and mt5 @xcite . several benchmarks are proposed to evaluate the cross-lingual ability of multilingual pre-trained models, including xglue @xcite , xtreme @xcite and xtreme-r @xcite language clustering in linguistics the linguists propose to classify languages in several ways from different perspectives. there are two main kinds of language clustering: genealogical clustering and typological clustering. in genealogical clustering, languages are clustered into language families @xcite by their genetic relatedness. languages in the same language family have the same ancestral language. in typological clustering, languages are clustered by their typological features, like word order, morphology @xcite and lexicostatistics @xcite . geographical sprachbund @xcite is also a typological clustering method as it groups languages according to their similar areal features coming from geographical proximity. ## approach our proposed pipeline on representation sprachbund for multilingual pre-training is illustrated in figure 1 . our approach can be divided into two stages: first, we quantify the similarity between languages with generated language representation, and cluster all languages in our corpora into k clusters based on their similarity. each cluster is called one representation sprachbund. k is a hyper-parameter in our clustering algorithm. second, we separate our corpora into k parts based on their corresponding representation sprachbund, and pre-train one model with corpora from each representation sprachbund. ## conclusion to reduce the cross-lingual contradictions in pretraining one model for all languages, we propose to merge similar languages into a representation sprachbund and pre-train one model for each representation sprachbund. results show that our approach outperforms strong baselines in various settings and tasks. we also identify the relationship between our representation sprachbund with linguistic theories. applications of our representation sprachbund as a paradigm for clustering languages in linguistics will be explored in subsequent work.