Beyond E nglish-Centric Bitexts for Better Multilingual Language Representation Learning (2023)

Abstract: In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing state-of-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical applications. We show that going beyond English-centric bitexts, coupled with a novel sampling strategy aimed at reducing under-utilization of training data, substantially boosts performance across model sizes for both Electra and MLM pre-training objectives. We introduce XY-LENT: X-Y bitext enhanced Language ENcodings using Transformers which not only achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands, is also competitive across bands. Our XY-LENT XL variant outperforms XLM-R XXL and exhibits competitive performance with mT5 XXL while being 5x and 6x smaller respectively. We then show that our proposed method helps ameliorate the curse of multilinguality, with the XY-LENT XL achieving 99.3% GLUE performance and 98.5% SQuAD 2.0 performance compared to a SoTA English only model in the same size band. We then analyze our models performance on extremely low resource languages and posit that scaling alone may not be sufficient for improving the performance in this scenario

## introduction recent advancements in natural language processing (nlp) have been a direct consequence of leveraging foundational models @xcite , pretrained on a large text corpora in a selfsupervised fashion. this has also been the case for multilingual nlp where pre-trained models like multilingual bert (mbert) @xcite , xlm @xcite , xlm-roberta @xcite , xlm-electra @xcite and mt5 (xue et al., 2021) * equal contribution. have all shown non-trivial performance gains, especially in the setup of zero-shot transfer, and have been the work-horse for a diverse number of multilingual tasks. given their ubiquitous applicability in zero-shot downstream scenarios, improving the quality and enabling their usage in resourceconstrained applications is also an important vein of research which we explore in this paper. a source of improvement for these models has been leveraging bitext data for better representation learning @xcite . most prior work, however, has focused on leveraging english-centric (en-x) bitext data. contemporaneously, the related area of massively multilingual machine translation (a single model for translating between different pairs of languages, eg: @xcite ; @xcite @xcite ) has shown tremendous progress, with @xcite showing that a crucial aspect of this improvement has been moving beyond en-x parallel corpora and leveraging web-based mined x-y bitexts spanning 1000s of translation directions @xcite @xcite . this makes a compelling case to explore if leveraging x-y bitexts can also improve multilingual representation learning. in this work, we introduce xy-lent (pronounced as "excellent"): x-y bitext enhanced language encodings using transformers. we first identify problems with using the commonly used sampling strategy proposed in @xcite , showing that it induces sparse sampling distributions leading to under-utilization of data, and thus propose a novel strategy to mitigate this issue ( §3.2). we then propose leveraging x-y bitexts in conjunction with the improved sampling strategy, as well as a vocap @xcite style sentencepiece vocabulary re-construction for improving multilingual representation learning ( §3.1). we show that our proposed method improves performance across all model size bands ( §6). furthermore, these performance gains hold for both masked language models (mlm) and electra style models. our approach results in an almost 12x speedup in training for mlm model training ( §6.2). we systematically analyse the impact of model scaling with respect to the curse of multilinguality @xcite to observe that the gap between current english only sota models and multilingual models can be considerably reduced ( §6.3). our analysis reveals that xy-lent improves performance across language families ( §6.4) and helps reduce the cross-lingual transfer gap in multilingual tasks ( §6.5). we then demonstrate that the training dynamics of such models can be used to better understand the underlying datasets and use it to find interesting defects in them ( §6.6). finally, we show some limitations of such multilingual representational models vis-à-vis extremely low resource languages, identifying potential shortcomings that are not addressed with scaling of such models, as well as issues around catastrophic forgetting in the way current models are used for domain adaptation. in doing so, we establish state of the art on 5 multilingual downstream tasks (xnli, paws-x, ty-diqa, xquad and mlqa) within a model size band, and achieve competitive performance across size bands, thereby showing for the first time (to the best of our knowledge) an interesting notion of parameter efficiency: xy-lent xl outperforms xlm-r xxl @xcite and performs competitively with mt5 xxl @xcite , whilst being 5x and 6x smaller respectively (figure 1 ). furthermore, our proposed model reduces the gap for english specific tasks: xy-lent xl achieves 99.3% glue performance and 98.5% squad 2.0 performance compared to a sota english only model in the same size band. ## related work large scale self-supervised learning has emerged as a prominent way of building cross-lingual language models that can be adapted for numerous multilingual downstream applications. especially for building multilingual encoder transformer @xcite models, two popular paradigms have been masked language modeling (mlm; devlin et al. (2019); conneau et al. (2020)) and pre-training encoders as discriminators (elec-tra; clark et al. (2020b); chi et al. (2022)), with the latter showing considerable compute efficiency. these approaches can further be improved by leveraging parallel corpora in different ways: conneau and lample (2019) propose a translation language modeling task (tlm) wherein the model predicts masked tokens in concatenated translation pairs, chi et al. (2022) propose a translation replaced token detection (trtd) task, an analogous task for electra-style models. other approaches include using bitexts to construct code-switched sequences as inputs during pre-training (alm; yang et al. (2020)) and for contrastive learning (infoxlm; chi et al. (2021a) ), or using token-level alignments in parallel data to improve cross-lingual modeling @xcite . however, all the aforementioned works rely on englishcentric bitexts. @xcite show that moving beyond en-x bitexts for massively multilingual machine translation affords substantial improvements over approaches that rely solely on english-centric data @xcite . the primary factor responsible for this improvement has been the curation of x-y aligned bitext data, constructed by mining bitexts from publicly available web data @xcite @xcite . the dataset construction either follows a local mining approach (first aligning documents using heuristics, and then mining parallel bitexts from the aligned documents; used in ccaligned (el-kishky et al., 2020)), or a global mining approach (all bitexts are embedded in a common vector space, and then aligned candidates are found by looking at the normalized nearest neighbors; used in ccmatrix @xcite ). @xcite also propose a sampling strategy for leveraging the x-y bitexts, wherein the marginals are constrained to be similar to what is used for en-x bitexts, and show their proposed method improves over uniform sampling. however, as we show in ( §3.2), their proposed strategy has the undesirable artefact of inducing extremely sparse solutions, thereby resulting in data wastage. ## pretraining details we follow the xlm-e @xcite pretraining approach and only introduce a few architectural changes to improve the overall performance of the model. we use the transformer model @xcite trained with electra @xcite style of replace token detection (rtd) on both monolingual (mrtd) and bitext (trtd) data. in the current setup of training, we use two transformer encoders in conjunction: a generator g and a discriminator d, where the generator g is trained with masked language modeling objective (mlm; @xcite ) and the discriminator is trained on replaced token detection objective (rtd; @xcite on all the tokens passing through the generator. in addition to using the gated relative position bias introduced in @xcite , we do not mask the [cls] token and flip bitext language order with probability @xmath0.5 for the trtd task. pretraining data: for our monolingual data, we follow @xcite and use the cc-100 dataset foot_1 @xcite which contains texts in 100 languages collected from common crawl. as mentioned in ( §3.1), we explore the utility of the ccmatrix and the mul-ticcaligned x-y aligned bitext data. ccmatrix consists of 1015 language pairs (97 unique languages) foot_2 ; while the multiccaligned dataset consists of 2959 language pairs (92 unique languages) foot_3 . we contrast this against only using en-x bitexts (ccaligned, el-kishky et al. ( 2020 )). model size bands: while our base and large models have more parameters when compared with xlm-r, most of the additional parameters come from the increased vocabulary size ( §3.3). concretely, our base model has 12 layers and 768 hidden states, while the large model has 24 layer and 1024 hidden states, which is identical to xlm-r base and xlm-r large respectively. however, even with the increased parameter count, the computation cost on a text classification task is roughly the same within a model size family (since mapping tokens to an embedding is a lookup operation). finally, it is noteworthy that even with the increased vocabulary size, the number of parameters for xy-lent xl is less compared to the xl and xxl variants of both xlm-r and mt5. pretraining setup: for the base model, we train for 125k steps with a batch size of 8192 for mrtd task and for the large model, we train the model for 500k steps with a batch size of 2048. finally for the xl model, we train for 150k steps with a batch size of 8192. we use a dynamic batch size for trtd task which is based on original length of the translated bi-text pair. please refer appendix a for additional details. we adopt the standard practice of using a linear warmup schedule for the learning rate and use the adam (kingma and ba, 2014) optimizer for all the models. following @xcite , we do not apply any dropout to the generator. cross-lingual downstream evaluation: for evaluating the cross-lingual understanding of the model, we consider 5 multilingual evaluation benchmarks. we consider 2 classification tasks and 3 question answering tasks. for classification, we evaluate on the cross-lingual natural language inference dataset (xnli; @xcite al. (2018) for cross-lingual question answering, we consider mlqa @xcite , xquad @xcite and tydiqa-goldp @xcite . for all the aforementioned tasks, we perform the evaluation in zero-shot setting, i.e. only using the english data for fine-tuning. to further assess the model's performance when translated data is available, we evaluate the model on the translate-train setup for the classification tasks. english downstream evaluation: to further assess xy-lent's performance on english and see how the curse of multilinguality impacts the model, we also assess the model's performance on the commonly used glue benchmark @xcite , comprising of 8 tasks: mnli @xcite , sst-2 @xcite , qnli @xcite , mrpc @xcite , cola @xcite , qqp , sts-b @xcite and rte. additionally, we also evaluate the english performance of our model on a question answering task, using the squad 2.0 dataset @xcite . please refer to appendix b for additional details on the datasets. ## conclusion in this work, we introduced a family of models which achieve sota performance over 5 multilingual benchmarks compared to other models belonging to similar model size bands and are competitive across the bands. our xy-lent xl model outperforms xlm-r xxl and is competitive with mt5 xxl being 5x and 6x smaller respectively. furthermore, the xl model variant also achieves 99.3% and 98.5% of the current best performant models on glue and squad 2.0 respectively, thereby aiding in reducing the curse of multilinguality. the performance gains are consistent across language families.