Improving Multi-lingual Alignment Through Soft Contrastive Learning (2024)

Abstract: Making decent multi-lingual sentence representations is critical to achieve high performances in cross-lingual downstream tasks. In this work, we propose a novel method to align multi-lingual embeddings based on the similarity of sentences measured by a pre-trained mono-lingual embedding model. Given translation sentence pairs, we train a multi-lingual model in a way that the similarity between cross-lingual embeddings follows the similarity of sentences measured at the mono-lingual teacher model. Our method can be considered as contrastive learning with soft labels defined as the similarity between sentences. Our experimental results on five languages show that our contrastive loss with soft labels far outperforms conventional constrastive loss with hard labels in various benchmarks for bitext mining tasks and STS tasks. In addition, our method outperforms existing multi-lingual embeddings including LaBSE, for Tatoeba dataset.

## introduction learning good representations (or embeddings) of sentences and passages is crucial for developing decent models adaptive to various downstream tasks in natural language processing. compared with the high quality mono-lingual sentence embeddings developed in recent years @xcite , multi-lingual sentence embeddings have a room for improvement, mostly due to the difficulty of gathering translation pair data compared to mono-lingual data. this motivated recent trials on improving the performance of multi-lingual embeddings. one of the prominent approaches trains the multi-lingual embeddings using contrastive learning @xcite . given * equal contribution † corresponding authors a translation pair for different languages, this approach trains the model in a way that the embeddings for translation pairs are brought closer together, while embeddings for non-translation pairs are pushed further apart @xcite . despite several benefits of this contrastive learning approach, @xcite pointed out that current training method ruins the mono-lingual embedding space. to be specific, this issue arises from the fact that existing contrastive loss treats sentences that are not exact translation pairs identically (as negative pairs), irrespective of the semantic similarity of those sentences. another prominent approach is distilling monolingual teacher embedding space to a multi-lingual student model. the basic idea is, letting the multilingual embeddings of student models follow the mono-lingual embeddings of teacher model. this approach is motivated by the assumption that english embeddings are well constructed enough to guide immature multi-lingual embeddings. for example, @xcite proposed a distillation method using mean-squared-error (mse) loss, which is shown to be effective in learning embeddings for low-resource languages. also, @xcite used a distillation method where the teacher is the english embedding of a multi-lingual model. unfortunately, existing distillation methods cannot fully utilize the translation pairs. since conventional methods choose the most reliable english embeddings as the teacher model, translation pairs from non-english language parallel corpus are not fully leveraged. in this paper, we propose a novel distillation method for improving multi-lingual embeddings, by using soft contrastive learning. see figure 1 . given n translation pairs {(s i , t i )} n @xmath0, our method first computes the mono-lingual sentence similarity matrix from the teacher model. each element of this similarity matrix is a continuous value. we distill such soft label to the cross-lingual sentence similarity matrix computed for the multi-lingual student model. in other words, the anchor similarity matrix computed from the teacher model is used as a pseudo-label for contrastive loss. our main contributions are as follows: ## related works constructing multi-lingual embedding has been actively studied for recent years @xcite @xcite . for example, labse @xcite shows remarkable bitext retrieval performances, which is first pretrained with masked language modeling (mlm) @xcite and translation language modeling (tlm) @xcite tasks, and then fine-tuned with translation pairs using contrastive loss, i.e. translation ranking task. also, muse @xcite uses translation based bridge tasks from @xcite to make a multilingual embedding space. in short, muse @xcite is trained for translation ranking task with hard negatives. reimers and gurevych (2020) introduced distilling the mono-lingual embedding of a teacher model (using sbert @xcite ) to the multi-lingual embedding of a student model (using xlm-r @xcite ) with mse loss, which enables a good bitext retrieval performance only with small amounts of parallel data. several follow-up papers @xcite achieved good performances by using this distillation approach. for example, heffernan et al. (2022) successfully improved the per- figure 1: overall framework of our method. given n sentence pairs from source/target languages, we train a multi-lingual student model f by using the similarity between sentences measured by a mono-lingual teacher model g. our contrastive loss function in eq. 2 uses soft-label w(i, j) defined in eq. 4 and 5. ## proposed method suppose we are given n translation pairs, denoted by (s 1 , t 1 ), (s 2 , t 2 ), • • • , (s n , t n ), where s i is the i-th sentence in the source language and t i is the corresponding sentence in the target language. we train a multi-lingual student model f by using the similarities between mono-lingual sentences measured by a teacher model g. here, g can be either a mono-lingual model or using only a single language from multi-lingual models. specifically, we first use the teacher network g to measure the similarity of sentences {s i } n @xmath1. the cosine similarity between sentences s i and s j measured by encoder g is denoted by in fact, this only works when the teacher model is multi-lingual, and the student encoder f trains in a following way, which is different from the eq. 1. the contrastive loss discussed above is unidirectional. following the common symmetric bidirectional contrastive loss, e.g., @xcite , the symmetric loss using our soft label is defined as training monolingual space (tms) note that our objective l cross given in eq. 6 is to learn only the cross-lingual similarity of the student model. in addition to that, we consider learning with additional mono-lingual loss l mono in eq. 7,the distillation loss measured by the similarity between each monolingual sentence pair. this approach of using l mono on top of l cross is dubbed as training monolingual space (tms). the combined loss term is shown in eq. 8, where the parameter controls the balance between the cross-lingual loss and the mono-lingual loss term. note that using tms is orthogonal to the choice of using the priority label or the average label. this section describes the details of our experimental setting, for both training and evaluation. ## results we first test the model trained with a single language pair, and then show the result when the model is trained with multiple language pairs. ## conclusion in this paper, we proposed a method of improving multi-lingual embeddings, with the aid of the sentence similarity information measured at the mono-lingual teacher models. our method can be considered as a variant of existing contrastive learning approach, where our method uses soft labels defined as the sentence similarity, while existing methods use hard labels. we tested our method on five different languages including en, ko, ja, fr, and ru. our method shows the best performance in the tatoeba dataset, and achieved high performance in other bitext mining tasks as well as sts tasks.