Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining (2022)

Abstract: This work presents methods for learning cross-lingual sentence representations using paired or unpaired bilingual texts. We hypothesize that the cross-lingual alignment strategy is transferable, and therefore a model trained to align only two languages can encode multilingually more aligned representations. We thus introduce dual-pivot transfer: training on one language pair and evaluating on other pairs. To study this theory, we design unsupervised models trained on unpaired sentences and single-pair supervised models trained on bitexts, both based on the unsupervised language model XLM-R with its parameters frozen. The experiments evaluate the models as universal sentence encoders on the task of unsupervised bitext mining on two datasets, where the unsupervised model reaches the state of the art of unsupervised retrieval, and the alternative single-pair supervised model approaches the performance of multilingually supervised models. The results suggest that bilingual training techniques as proposed can be applied to get sentence representations with multilingual alignment.

## introduction cross-lingual alignment as evaluated by retrieval tasks has been shown to be present in the representations of recent massive multilingual models which are not trained on bitexts @xcite . other studies further show that sentence representations with higher crosslingual comparability can be achieved by training a cross-lingual mapping @xcite or fine-tuning @xcite for every pair of languages. these two lines of research show that, on the one hand, multilingual alignment arises from training using monolingual corpora alone, and, on the other, bilingual alignment can be enhanced by training on bitexts of specific language pairs. combining these insights yields a question: can training with bilingual corpora help improve multilingual alignment? given a language model encoding texts in different languages with some shared structure already, we can expect that the model further trained to align a pair of languages will take advantage of the shared structure and will therefore generalize the alignment strategy to other language pairs. from a practical point of view, bitexts for some pairs of languages are more abundant than others, and it is therefore efficient to leverage data from resource-rich pairs for the alignment of resource-poor pairs in training multilingual language models. to better understand the cross-lingual structure from the unsupervised models, we also ask the following question: how can multilingual alignment information be extracted from the unsupervised language models? unsupervised multilingual models out-of-the-box as sentence encoders fall short of their supervised counterparts such as laser @xcite in the task of bitext mining @xcite . the discovery of cross-lingual structure in the hidden states in the unsupervised model @xcite , however, raises the possibility that with relatively light post-training for better extraction of deep features, the unsupervised models can generate much more multilingually aligned representations. in this paper, we address both questions with the design of dual-pivot transfer, where a model is trained for bilingual alignment but tested for multilingual alignment. and we hypothesize that training to encourage similarity between sentence representations from two languages, the dual pivots, can help generate more aligned representations not only for the pivot pair, but also for other pairs. in particular, we design and study a simple extraction module on top of the pretrained multilingual language model xlm-r @xcite . to harness different training signals, we propose two training architectures. in the case of training on unpaired sentences, the model is encouraged by adversarial training to encode sentences from the two languages with similar distributions. in the other case where bitexts of the two pivot languages are used, the model is encouraged to encode encountered parallel sentences similarly. both models are then transferred to language pairs other than the dual pivots. this enables our model to be used for unsupervised bitext mining, or bitext mining where the model is trained only on parallel sentences from a single language pair. the experiments show that both training strategies are effective, where the unsupervised model reaches the state of the art on completely unsupervised bitext mining, and the one-pair supervised model approaching the state-of-the-art multilingually-supervised language models in one bitext mining task. our contributions are fourfold: ## related work alignment with adversarial nets this work follows the line of previous studies which use adversarial networks (gans) @xcite to align cross-domain distributions of embeddings without supervision of paired samples, in some cases in tandem with cycle consistency @xcite , which encourages representations "translated" to another language then "translated" back to be similar to the starting representations. conneau et al. (2018)'s muse project trains a linear map from the word-embedding space of one language to that of another using gans, the method of which is later applied to an unsupervised machine translation model @xcite . cycle consistency in complement to adversarial training has been shown to be effective in helping to learn cross-lingual lexicon induction @xcite @xcite . our work is the first to our knowledge to apply such strategy of adversarial training and cycle consistency to the task of bitext mining. alignment with pretrained lms we adopt the training strategy aforementioned on top of pretrained multilingual language models, the extractability of multilingual information from which has been studied in several ways. pires et al. (2019) find multilingual alignment in the multilingual bert (mbert) model (devlin et al., 2019) pretrained on monolingual corpora only, while conneau et al. (2020b) identify shared multilingual structure in monolingual bert models. other work studies the pretrained models dynamically by either fine-tuning the pretrained model for crosslingual alignment (cao et al., 2020) or learning cross-lingual transformation (aldarmaki and diab, 2019) with supervision from aligned texts. recently, yang et al. (2020) use multitask training to train multilingual encoders focusing on the performance on retrieval, and reimers and gurevych (2020) use bitexts to tune multilingual language models and to distill knowledge from a teacher model which has been tuned on paraphrase pairs. also, chi et al. (2021a) pretrain an alternative xlm-r on a cross-lingual contrastive objective. our work falls in the line of exploring multilinguality of pretrained models with a distinct emphasis on investigating the multilingual structure induced by bilingual training without fine-tuning or alternative pretraining. xlm-r x i s linear combination linear map y i s xlm-r x j t linear combination linear map y j t f g cycle consistency loss l cycle discriminator adversarial loss l adv source sentence target sentence figure 1: schematic depiction of the unsupervised model with adversarial and cycle consistency losses. unsupervised parallel sentence mining the evaluation task of our work is bitext mining without supervision from any bitexts or from bitexts of the pair of languages of the mining task. such experiments have been explored previously. hangya et al. (2018) show that unsupervised bilingual word embeddings are effective on bitext mining, and hangya and fraser (2019) further improve the system with a word-alignment algorithm. kiros (2020) trains a lensing module over mbert for the task of natural language inference (nli) and transfers the model to bitext mining. keung et al. (2020)'s system uses bootstrapped bitexts to finetune mbert, while kvapilíková et al. (2020)'s system uses synthetic bitexts from an unsupervised machine translation system to fine-tune xlm (conneau and lample, 2019). results from the three aforementioned studies are included in section 5 for comparisons. methodologically, our approach differs from the above in that our system is based on another pretrained model xlm-r (conneau et al., 2020a) without fine-tuning, for one of the goals of the study is to understand the extractability of the alignment information from the pretrained model; and our model receives training signals from existing monolingual corpora or bitexts, instead of from nli, bootstrapped, or synthesized data. ## training the model is trained with the adam optimizer (kingma and ba, 2014) and learning rate 0.001 with the parameters in xlm-r frozen. our training program is built upon allennlp @xcite , huggingface transformers @xcite , and pytorch @xcite . the code for this study is released publicly. 1 for adversarial training, the discriminator is updated κ times for every step of backpropagation to the encoder. other hyperparameters include the dimension of the output representations d, number of negative samples n, margin value α, and weight of the cycle loss λ. the hyperparameters and the the values which are experimented with are summarized in the bilingual corpora used to train the encoder is taken from opus @xcite as produced for the training of xlm @xcite . 2 we experimented with two language pairs for training the model-arabic-english (ar-en) and german-english (de-en)-to explore potential effects of the choice of the dual-pivots. after being trained, the encoder is evaluated on two tasks of bitext mining between texts in english and in another language. additionally, we train the models with the pivot pair of arabic-german (ar-de), which does not include english, to be evaluated on the second task. ## evaluations four models, two unsupervised and two one-pair supervised trained on either of the two language pairs, are evaluated on two bitext mining or retrieval tasks of the bucc corpus @xcite and of the tatoeba corpus (artetxe and schwenk, 2019a). ## analysis to understand the factors affecting the performance of the model, we consider several variants. all models presented in this section are trained with the same hyperparameters presented in the evaluation section above using either de-en corpora or corpora of multiple language pairs (section 6.3). ## conclusion this work shows that training for bilingual alignment benefits multilingual alignment for unsupervised bitext mining. the unsupervised model shows the effectiveness of adversarial training with cycle consistency for building multilingual language models, and reaches the state of the art of unsupervised bitext mining. both unsupervised and one-pair supervised models show that significant multilingual alignment in an unsupervised language model can be recovered by a linear mapping, and that combining monolingual and bilingual training data can be a data-efficient method for promoting multilingual alignment. future work may combine both the supervised and the unsupervised techniques to attain sentence embeddings with stronger multilingual alignment through the transferability of bilingual alignment demonstrated in this work, and such work will benefit tasks involving languages of low resources in bitexts.