Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages (2022)

Abstract: Scaling multilingual representation learning beyond the hundred most frequent languages is challenging, in particular to cover the long tail of low-resource languages. We move away from the popular one-for-all multilingual models and focus on training multiple language (family) specific representations, but most prominently enable all languages to still be encoded in the same representational space. We focus on teacher-student training, allowing all encoders to be mutually compatible for bitext mining, and enabling fast learning of new languages. We also combine supervised and self-supervised training, allowing encoders to take advantage of monolingual training data.Our approach significantly outperforms the original LASER encoder. We study very low-resource languages and handle 44 African languages, many of which are not covered by any other model. For these languages, we train sentence encoders and mine bitexts. Adding these mined bitexts yielded an improvement of 3.8 BLEU for NMT into English.

## introduction there is increasing interest in multilingual sentence representations since they promise an appealing approach to extend nlp tasks to a large number of languages, without the need to separately train a language-specific model. most of the current works on multilingual sentence representations have focused on training one model which handles all languages of interest, e.g. @xcite @xcite . the main motivation is that languages with limited resources will benefit from the fact that the same model has learned other (similar) languages. zero-shot performance is of particular interest: the model generalizes well to a new language although it has never seen training data in that language. training massively multilingual models faces several problems with increasing number of languages: how to make sure that all languages are learned, how to to account for the large imbalance of available training, or the high computational complexity. @xcite proposed a teacher-student approach to extend an existing (monolingual) sentence embedding space to new languages. we build on this generic idea and propose multiple improvements which significantly improve performance, namely different teacher and student architectures, several supervised and unsupervised training criteria, and language-specific encoders. we also investigate challenges when handling low-resources languages, showcased by training models for 50 african languages. to the best of our knowledge, many of these languages are not handled by any other sentence encoder or pretrained model. we dispose test data for 44 out of 50 languages, mine bitexts against 21.5 billion english sentences, and train smt models to translate into english. multilingual sentence embeddings have many applications which is reflected by several metrics to evaluate them, e.g. the xtreme bench mark @xcite . in this work, we focus on the use of multilingual sentence embeddings for similarity-based bitext mining, as proposed by @xcite , and on using these mined bitexts to improve nmt. consequently, our primary metric is nmt performance. however, mining and nmt training is computationally expensive and it is intractable to systematically perform this evaluation for many different sentence encoder variants. as an evaluation proxy, we use multilingual similarity search error rate. in contrast to previous work which used the tatoeba test set, e.g. @xcite @xcite , we switch to the flores evaluation benchmark, which contains high-quality human translated texts from wikipedia @xcite and covers many low-resource languages. the contributions of this work can be summarized as follows: 1) we move away from the popular one-for-all approach and train multiple, mutually compatible language (family) specific encoders; 2) we explore several variants and improvements of teacher-student training for multilingual sentence representations (section 3), and propose a new approach which combines supervised teacher-student with self-supervised mlm training to better handle very low-resource languages (subsection 5.3); 3) the new model substantially improves 12 languages which were badly handled by the original laser encoder (subsection 5.1); and 4) we train sentence encoders for 50 african languages, mine bitexts, and train nmt systems (section 6). to the best of our knowledge, many of these languages are not handled by any other nmt system. ## related work multilingual sentence representation examples of approaches to learn multilingual representations are multilingual bert (m-bert) which covers 104 languages @xcite , xlm @xcite , and xlm-r which was trained on 100 languages using crawled web data @xcite . however, as these approaches do not take into account a sentence-level objective during training, they can result in poor performance when applied to tasks which use sentence representations such as bitext retrieval @xcite . in order to address this, methods such as sentencebert (sbert) make use of a siamese network to better model sentence representations @xcite . @xcite uses a dual-encoder approach with a transformer-based architecture and additive margin softmax loss @xcite . it covers 109 languages, and is pre-trained using a masked language modelling (mlm) and translation language modelling (tlm) objective @xcite . labse was used to mine bitexts in eleven indian languages @xcite . another popular multilingual sentence embedding model is laser @xcite . it is based on a lstm encoder/decoder architecture with a fixed-size embedding layer and no pre-training. laser covers 93 languages. when learning a multilingual embedding space, a limitation of many existing approaches is that they require training a new model from scratch each time a language is to be added. however, there have been various methods proposed to address this. @xcite provide one such technique which extends m-bert to low-resource languages by increasing the size of the existing vocabulary, and then continuing self-supervised training using monolingual data for a low-resource language. another example by @xcite uses multilingual distillation. in this supervised teacher-student approach, the teacher is a monolingual model pre-trained on english (sbert), and the student is a pre-trained multilingual model (xlm-r). using bitexts, the student then extends the embedding space to the desired language(s) by applying regression loss between the english sentence representation of the teacher, and the target language sentence representation of the student. scaling multilinguality several recent works have addressed the challenges faced when scaling multilingual models to a hundred languages and beyond, namely massively multilingual nmt systems @xcite @xcite . a recent study explored the extension to more than a thousand languages @xcite . training nmt models for a large number of languages faces many challenges and a large variety of architectures have been explored @xcite @xcite . to the best of our knowledge, similar modelling techniques were not yet considered to train (massively) multilingual sentence encoders. resources for african languages collecting resources, training nmt systems, or performing evaluations for african languages is the focus of several works @xcite @xcite @xcite @xcite . the masakhane project foot_0 aims at providing resources to both strengthen and spur nlp research in african languages. a workshop focused on the evaluation of african languages will be held at emnlp'22. 2 in the framework of the data track, several parallel corpora were made available. in general, the number of languages covered is well below the 44 languages we evaluate in this work. is your watch broken? saatin bozuk mu? this is a nice house. this is a nice house. cosine loss sentence embedding sentence embedding bitext monolingual saatin [mask] mu? cross entropy loss masked language modeling multilingual distillation teacher student student ## architecture the overall architecture of our approach is summarized in figure 1 . the teacher is an improved laser encoder. compared to the original training procedure described in @xcite , we use spm instead of bpe preprocessing, up-sampling of low-resource languages, and a new implementation in fairseq. all the other parameters are unchanged, namely a 5-layer bil-stm encoder, the 1024 dimensional sentence embeddings are obtained by max-pooling over the last layer, and training is performed for 93 languages with public resources obtained from opus. the reader is referred to @xcite for details on the original laser training procedure. we use this new multilingual sentence encoder as the teacher in all our experiments and in this work refer to our teacher as laser2, and student models as laser3. the code to train the teacher or student models is freely available in the fairseq github repository. 3 training of the students follows the general idea of a teacher-student approach as initially proposed by @xcite , but with several important differences. we want to scale encoder training and bitext mining well beyond the roughly 100 languages handled by current multilingual encoders. this may include languages which are not covered by existing pretrained models, and retraining them would be computationally very expensive. also, some languages may be written in a new script which is not covered. therefore, we made the following design choices: (1) we do not initialize the student with some pretrained model, e.g. xlm-r, but use a random initialization; (2) the student is trained on 2m sentences of english monolingual data, and we also add 2m sentence of english-spanish bitexts from ccmatrix @xcite to better align with the teacher's multilingual embedding space; (3) instead of one massively multilingual model, we train multiple students for a small subset of (similar) languages, or even a single language; (4) use of separate spm vocabularies for teacher and student, better accommodating scripts and tokens in the student languages which were unseen by the teacher (cf. subsection 5.2); (5) optimization of the cosine loss between the teacher and student embedding, since this is the relevant metric for bitext mining (cf. figure 1 above); (6) jointly train distillation alongside a mlm criterion to benefit additional learning from monolingual data in a foreign language (cf. figure 1 , and subsection 5.3); (7) addition of curriculum learning in the form of progressive distillation. in this strategy, instead of sending the entire sentence pairs all at once, we send incremental versions of the respective sentence pairs to both teacher and student, which we found to be helpful for some particularly challenging low-resource languages. our motivation of using a total of 4m english sentences is to "anchor" the student encoder to the embedding space, and hopefully be able to learn new languages with a limited amount of parallel texts. in initial experiments, we used a 6layer bilstm encoder architecture as in @xcite , but we saw consistent improvements by switching to a 12-layer transformer. we keep the same student architecture for all languages (@xmath0, @xmath1, @xmath2, 250m params). teacher-student training was performed on 16 gpus, adam optimizer, a learning rate of 0.0005 and with a batch size of 2,000. when we minimize the cosine distance only, max-pooling of the transformer outputs to achieve the fixed-size sentence representations worked best, compared to using a special token like @xcite . for curriculum learning using progressive distillation, we incrementally send a percentage of subwords from each sentence pair (e.g. 10%, 20%, ..., 100%). we experimented sending various incremental percentages of the sentence pairs to both teacher and student (e.g. 20%, 40%), but found 10% increments to perform best. ## training and evaluation resources evaluation data creating high-quality evaluation data for low-resource languages is challenging. in this work we evaluate our approach on two publicly available corpora: tatoeba and flo-res. the tatoeba corpus is a test set covering 112 languages @xcite , and contains up to 1000 sentences for each language pair. flores is a freely available n -way parallel test with 1012 sentences the devtest set @xcite . 4 it initially covered 101 languages, and was recently extended to 200 languages @xcite , including 44 african languages on which we report results in this paper. monolingual data comes mostly from common crawl and other public sources like paracrawl, 5 and some additional targeted crawling for several low-resource languages. we have extended and improved fasttext lid @xcite to include additional languages considered in this work. we trained this new lid model on publicly available monolingual data and evaluated it on flores. preprocessing includes: sentence splitting, filtering of sentences in the wrong script or with more than 20% of numbers or punctuation, lid and deduplication, as well as lm filtering on english. bitexts are obtained from opus foot_5 @xcite and used to train the sentence encoders and nmt systems. the amount of available resources is summarized in ## experimental evaluation: multilingual similarity search in this section, we provide some evaluations of our proposed multilingual distillation approach, based on multilingual similarity search. ## acknowledgements for their helpful contributions to this work, we would like to thank: bapi akula, pierre andrews, angela fan, cynthia gao, kenneth heafield, philipp koehn, janice lam, alex mourachko, christophe rogers and guillaume wenzek. ## conclusion multilingual sentence representations are key to extend nlp approaches to more languages, and they are the underlying engine for distance-based bitext mining, which turned out to be crucial to scale nmt to more languages. in this work, we attack the challenge to scale the laser encoder and cover 50 african languages. to the best of our knowledge, only 14 african languages are handled and evaluated by current multilingual encoders. we achieve this by moving away from a onefor-all approach to an improved teacher-student training of several encoders, each one trained on a small subset of languages. this enabled us to better adapt the encoders to language specificities, e.g. a particular writing script, while maintaining mutual compatibility. our new models significantly outperform the original laser model on the flores test set, and we are on par or better than all other publicly available multilingual sentence encoders, namely labse. we were also able to integrate monolingual data by jointly minimizing a cosine and mlm loss. we showcase the potential of this technique for the wolof language, reducing the xsim error rate from 21.1% down to 5.9%. we performed bitext mining for 44 african languages. adding this data yielded an average improvement of 3.8 point bleu for translation into english. training code, the models and the mined bitexts are freely available. 9