Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings (2021)

Abstract: Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes more complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during fine-tuning to further improve multilingual contextualized representations for downstream cross-lingual transfer. This approach does not require word-alignment pairs prior to fine-tuning that may lead to sub-optimal matching and instead learns the word alignments within context in an unsupervised manner. It also allows different types of mappings due to soft matching between source and target sentences. We benchmark our proposed method on two tasks (XNLI and XQuAD) and achieve improvements over baselines as well as competitive results compared to similar recent works.

## introduction contextualized word embeddings have advanced the state-of-the-art performance in different nlp tasks @xcite @xcite . similar advancements have been made for languages other than english using models that learn cross-lingual word representations leveraging monolingual and/or parallel data @xcite @xcite . such cross-lingual ability helps in mitigating the lack of abundant data (labelled or unlabelled) and computational resources for languages other than english, with lesser cost. yet, there exists a challenge for improving multilingual representations and cross-lingual transfer learning, especially for low resource languages. recent studies proposed different techniques to improve multilingual representations in contextualized settings with additional objectives such as translation language modeling @xcite , integrating language and task adapters @xcite , and applying alignment techniques in the embedding spaces @xcite . previous studies concerning alignment in the embedding space show promising directions to improve cross-lingual transfer abilities for low resource languages @xcite @xcite . the objective is to align source and target language representations into the same embedding space, for instance by encouraging similar words to be closer to each other (e.g. cat in english and katze in german) with least cost in terms of data and computational resources. such methods require some form of cross-lingual signal, such as alignment in non-contextualized embeddings, mainly utilize bilingual/multilingual lexicon that have been learned with unsupervised or supervised techniques @xcite @xcite . however, when it comes to contextualized embeddings, alignment becomes more complex as we additionally utilize context (e.g. "match" in "your shoes don't match clothes" is similar to word "passen" in "ihre schuhe passen nicht zu kleidung" but not to "match" in "hast du das cricket match gesehen?"). in this work, we use only parallel sentences as an informative crosslingual training signal. along these lines, previous studies mainly followed two approaches: (1) rotation based techniques with the procrustes objective where the source embedding space is rotated to match that of the target @xcite ; (2) fine-tuning the pre-trained language model (lm) with explicit alignment objectives such that similar words in parallel sentences are closer in representation space @xcite @xcite . finetuning with alignment objective function provides simple yet effective and promising solution to improve contextualized word representations, especially for low resource languages. as opposed to rotation based approaches which require generating a transformation matrix for each language pair of interest, the alignment objective allows simultaneous learning from multiple languages. majority of previous studies concerning finetuning with alignment objective start with precollected word pairs generated using unsupervised or supervised methods (e.g. fastalign @xcite ) which aligns words in source and target sentences based on semantics, and subsequently applies some heuristics to obtain one-to-one word alignments. however, this leads to losing other word relationships (e.g. many-to-one) which exist in some language pairs (figure 2 ) inspired by the limitations of previous works, we propose the use of optimal transport (ot henceforward) to transfer knowledge across languages and improve multilingual word representation for cross-lingual transfer in zero-shot setting. this method learns word alignments while fine-tuning the pre-trained representations in an end-to-end fashion. as opposed to previous studies, this eliminates the need for pre-collected word pairs and allows many-to-many mappings between source and target words. furthermore, our approach directly utilizes the continuous representation of con-textualized word embeddings for alignment which helped broaden the scope of alignments to include additional linguistic information embedded in the lm (e.g. semantic and syntactic structure). specifically, we optimize a regularized variant of ot, i.e. sinkhorn divergence @xcite , on parallel sentences and use that as a guidance to fine-tune the pre-trained lm. we learn several independent ot mappings per language pair, each guiding the model to further shift contextualized word embeddings in the source language towards the ones in the target language (refer to figure 1 ). compared to the baseline mbert, we obtain improvements of 1.9% and 1.3% f1 on average in xnli @xcite and xquad @xcite benchmarks, respectively. before we dive deep into our method (section 4), we briefly describe ot in section 2 and related work in section 3. we discuss the experimental setup, results, analysis and finally conclusion in sections 5, 6, 7 and 8 respectively. our contribution is mainly three-fold: ## optimal transport in nlp optimal transport (ot) provides powerful tools to compare between different probability distributions and learn the similarities/differences to move the mass from source to target distributions @xcite . it has a strict requirement where source distribution must be completely transferred to target distribution making it rigid for machine learning based models. @xcite relaxed this constraint by allowing masses to be partially transferred to more than one point in the target distribution resulting in development of different regularized ot variants that improve both computational and statistical properties @xcite @xcite . regularized ot variants led to the increased adoption of ot in a wide range of areas such as graph applications, computer vision, and natural language processing @xcite @xcite @xcite . downstream applications utilize ot properties to obtain either of the following outputs @xcite : (1) optimal mapping (or transport) that aligns between points in the two distributions for applications like graph matching @xcite or word translations (alvarez-melis and jaakkola, 2018); (2) the optimal values (wasserstein distance) which is computed based on the optimal mappings and subsequently guide the model learning for applications like document similarity @xcite and machine translation @xcite . similar to this line of work, we use optimal values for guiding model learning. optimal transport for alignment: ot has been used as a fully unsupervised algorithm to align points between two distributions making use of the linguistic and structural similarity for applications like bilingual lexical induction @xcite and word translations @xcite @xcite . for instance, zhang et al. (2017) use ot to induce translations for a source word from the target language in bilingual lexicon induction whereas xu et al. (2018) use ot and back-translation losses to align traditional monolingual word embeddings that do not leverage context (i.e. word type level). ## related work alignment as a post-processing technique on distributional embedding spaces provides an effective solution to improve cross-lingual downstream applications. for non-contextualized embeddings, alignment based techniques for word embeddings have been thoroughly surveyed in @xcite . for contextualized embeddings, one direction of efforts to improve cross-lingual word representations is to use the procrustes objective to project the monolingual embeddings from one language to the monolingual embedding space in another @xcite . however, this generates a transformation matrix for each language pair which can be inconvenient to apply in downstream tasks. another direction is to use explicit alignment objective at the sentence level, word level, or both which allows simultaneous learning from different languages, as opposed to rotation based approaches. studies that depend on sentence level alignment achieve significantly high performance on bitext sentence retrieval tasks @xcite , and by design they are not applicable to word based applications. for instance, laser (artetxe and schwenk, 2019) learns massively multi-lingual encoder using a huge parallel corpus whereas @xcite trains a bi-directional dual encoder with an additive softmax margin loss to perform translation ranking among in-batch examples. similar to this line of work, we rely on only parallel sentences as external sources to fine-tune the model, but we define word alignment objective instead. other studies use word alignment objective to align parallel word pairs and fine-tune the contextualized multi-lingual lm @xcite @xcite . @xcite use regularized l2 based alignment objective to align parallel word pairs. wu and dredze (2020) use contrastive learning to align parallel word pairs relative to negative pairs in the batch. these approaches rely on unsupervised word aligners which are often sub-optimal to generate the parallel word pairs (e.g. fastalign @xcite or optimal transport @xcite ) and use these pairs as weak form of supervision. our work is most similar to these methods in that we use word level alignment objective; however, we learn the aligned word pairs implicitly during optimization rather than obtaining them beforehand using external aligners and applying heuristics to keep only one-to-one mappings. more recently, @xcite developed an end-to-end model that first aligns both source and target words with ot and then use the alignments as self-labels to fine-tune the contextualized lm. they use three objective functions for fine-tuning: masked language modeling (mlm) @xcite , translation language modeling (tlm) @xcite , and the cross entropy between predicted masked words and their corresponding alignments obtained from ot. similar to their work, we use ot based signals to finetune the contextualized lm, but we instead use the average cost of ot alignments for fine-tuning. there are other studies that attempt to combine various objectives for learning cross-lingual super-vision. for example, @xcite ; @xcite incorporate the following objectives on cross-lingual data: mlm, tlm, sentence level alignment (e.g. parallel sentence identification objective), and word level alignment. in this paper, we do not investigate combined objective functions similar to these works. we believe that adding more objectives can further boost the performance and we leave it for future work. ## method figure 1 shows the overall fine-tuning process. as input, we require parallel sentences (i.e. pairs of aligned sentences in source and target languages) and contextualized multilingual lm. we use english as fixed target language and other non-english languages as source language (more details in section 5.1). for each model iteration, we first embed words in source and target sentences independently with the pre-trained contextualized lm (section 4.1). these representations are then used as input for ot optimization applied for each source-target language pair. we then fine-tune the contextualized lm with the accumulated regularized loss across all language pairs as a guidance (section 4.3). we formulate the task of ot as minimizing the cost of transferring knowledge within context, from a non-english source sentence to an english target sentence in an unsupervised fashion. foot_0 ## results and discussion compared to the baseline mbert, we obtain +1.9% and +1.3% f1 scores on average in xnli and xquad, respectively. compared to l2 @xcite , we obtain an average improvement of +0.8% for xnli and +6.7% for xquad in f1 scores. in xnli, we obtain comparable f1 score (-1.0%) to the more recent model -amber @xcite . this could be attributed to the tlm @xcite objective used in am-ber which provides additional cross-lingual signal and hence, further boosts the performance. in xquad, we obtain better f1 score (+4.5%) than the more recent work -mad-x @xcite -showing the effectiveness of our method. more languages during optimization: in our previous results, we fine-tuned mbert with parallel sentences drawn from a set of 5 languages (refer to section 5.1). we also investigate whether adding more languages during fine-tuning with ot (largewordot) would help improve the performance. we expanded the set of languages to all 15 languages as described in section 5.1 (also we also examine the impact of ot fine-tuning on unseen languages from the performance of wor-dot*. we notice similar or better performance compared to largewordot on average for all languages for both tasks, thereby showing that the performance on remaining languages on average is comparable. in addition, notes on ot efficiency: to examine the efficiency of our proposed method, we computed the time taken by one epoch of fine-tuning mbert with five language pairs (250k parallel sentences for each pair). on a single nvidia v100 gpu, it took approximately 8 hours to complete one epoch, which is relatively 30% higher compared to l2 based alignment method which took approximately 6 hours with the same settings. this increase is expected as our method considers every combination of words from source to target in order to find ot mapping with minimum cost for each step of fine-tuning. hence, it performs at least o(n * m) operations, where n and m are the number of words in source and target languages, respectively. on the other hand, l2 based alignment considers only precomputed one-to-one mapping which speeds up the process. this is a trade-off between time and accuracy where ot outperforms l2 in both tasks for seen and unseen languages in terms of accuracy. the time complexity only impacts the model during fine-tuning which is done once. word vs. subword alignments: as discussed in section 4.1, ot is flexible to align different textual units. we compare between fine-tuning at word level (wordot) and subword level (subot). ## conclusion in this paper, we investigated ot to align the space of contextualized embeddings of a source and a target sentence in order to improve contextualized word embeddings for cross-lingual settings. we trained an independent ot per language pair and used the resultant cost as a guidance to fine-tune the contextualized lm and encourage the alignment of the corresponding contextual embeddings. we obtain improvements in sentence level evaluation tasks: xnli and xquad. as an improvement for our proposed technique, we intend to use different variants of ot such as goromov-wasserstein which performs the same logic presented in this paper in addition to its ability to align embeddings of different spaces, mapping both geometry and points of different embedding spaces. we would also like to combine more cross-lingual objectives using additional signals and perform evaluation on more tasks and languages. yuxuan wang, wanxiang che, jiang guo, yijia liu, and ting liu. 2019. cross-lingual bert transformation for zero-shot dependency parsing. adina williams, nikita nangia, and samuel r. bowman. 2018. a broad-coverage challenge corpus for sentence understanding through inference. in naacl. shijie wu and mark dredze. 2020. do explicit alignments robustly improve multilingual encoders? l. xu, han sun, and y. liu. 2019. learning with batchwise optimal transport loss for 3d shape recognition. 2019 ieee/cvf conference on computer vision and pattern recognition (cvpr), pages 3328-3337. ruochen xu, yiming yang, naoki otani, and yuexin wu. 2018. unsupervised cross-lingual transfer of word embedding spaces. in emnlp. meng zhang, yang liu, huanbo luan, and maosong sun. 2017. earth mover's distance minimization for unsupervised bilingual lexicon induction. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 1934-1945, copenhagen, denmark. association for computational linguistics. pierre zweigenbaum, s. sharoff, and r. rapp. 2017. overview of the second bucc shared task: spotting parallel sentences in comparable corpora. in bucc@acl. a parallel corpus details data source lang #pairs data source lang #pairs europarl corpora bg-en 371k iit bombay hi-en 618k de-en 1m opensubtitles th-en 278k el-en 157k tr-en 9m es-en 990k vi-en 608k fr-en 1m tanzil ur-en 422k multiun ar-en 2m tanzil global-voices sw-en 128k ru-en 9m zh-en 3m