Semantic Alignment with Calibrated Similarity for Multilingual Sentence Embedding (2021)

Abstract: Measuring the similarity score between a pair of sentences in different languages is the essential requisite for multilingual sentence embedding methods. Predicting the similarity score consists of two sub-tasks, which are monolingual similarity evaluation and multilingual sentence retrieval. However, conventional methods have mainly tackled only one of the sub-tasks and therefore showed biased performances. In this paper, we suggest a novel and strong method for multilingual sentence embedding, which shows performance improvement on both sub-tasks, consequently resulting in robust predictions of multilingual similarity scores. The suggested method consists of two parts: to learn semantic similarity of sentences in the pivot language and then to extend the learned semantic structure to different languages. To align semantic structures across different languages, we introduce a teacher-student network. The teacher network distills the knowledge of the pivot language to different languages of the student network. During the distillation, the parameters of the teacher network are updated with the slow-moving average. Together with the distillation and the parameter updating, the semantic structure of the student network can be directly aligned across different languages while preserving the ability to measure the semantic similarity. Thus, the multilingual training method drives performance improvement on multilingual similarity evaluation. The suggested model achieves the state-of-the-art performance on extended STS 2017 multilingual similarity evaluation as well as two sub-tasks, which are extended STS 2017 monolingual similarity evaluation and Tatoeba multilingual retrieval in 14 languages.

## introduction representing semantics of sentences as embedding vectors on a vector space is crucial for various natural language processing (nlp) tasks. the funda-mental inductive bias of the sentence embedding methods is to place sentences having similar semantics close to each other in the vector space, which is advantageous to sentence-based tasks such as clustering and semantic retrieval. building upon well-known pre-trained english language models bert @xcite and roberta @xcite , @xcite introduced a fine-tuning method to learn the semantic similarity between two english sentences using siamese networks. however, it has focused only on monolingual settings. multilingual sentence embedding models should be able to measure the semantic similarity between a pair of sentences not only in the same language but also in different languages. there are two conditions to satisfy when measuring the similarity between a pair of sentences in different languages. firstly, monolingual sentences need to be closely placed as similar as they are. secondly, a translation pair, which have the same meaning in different languages, should be placed in close proximity to each other. for example, in figure 1 , the similarity score between a) and b) is 0.4, and a korean translation of b) is c). if two english sentences are positioned to express a similarity score of 0.4, and the translation pair are placed very close, we can measure the similarity score between a) and c) as around 0.4. therefore, satisfying the two conditions enables the model to calculate similarity scores of multilingual sentences. the first condition can be evaluated by the monolingual sentence similarity evaluation task, and the second condition can be assessed by a multilingual sentence retrieval task. even though several methods have been recently proposed for multilingual sentence embedding, they fail to achieve high performance on the multilingual sentence similarity evaluation because they have not succeeded in reaching both conditions. laser @xcite and @xcite . two kinds of the dataset have been widely used, which are semantic similarity scores for english sentences (0.4 for a and b, 0.2 for a and d) and translation dataset @xcite . the key idea of the suggested method is to learn the semantic structure of pivot language (english) with the first dataset and then extend the structure to different languages by aligning embeddings of translation pairs directly (left). as the left figure shows, the suggested method can correctly measure similarities between different languages (0.4 for a and c) because embeddings of b and c are closely aligned and the semantic relationship between a and b is trained. on the other hand, laser @xcite and labse @xcite suggested contrastive objectives to keep alignment between translation pairs @xcite , thus shows inferior performance on similarity evaluation tasks compared to their retrieval performance. recent work @xcite also suggested a method to align semantic structures across different languages. however, their alignment method is indirect (right), thus shows inferior performance on the retrieval. labse @xcite , which are trained with contrastive objectives using translation pairs, show inferior performance on the similarity evaluation tasks compared to their retrieval accuracy. in figure 1 , they focus on aligning b) and c), but not a) and b). a model proposed by @xcite is successful at measuring the similarity of monolingual sentences but shows a performance drop in the retrieval task because the model less directly aligns sentence vectors across the different languages. in other words, as shown in figure 1 , the model reimers and gurevych (2020) embeds a) and b) to correctly present the similarity score 0.4, but is unsuccessful to represent the relationship between b) and c). in this paper, we introduce a powerful multilingual sentence embedding model which is able to measure similarity between multilingual sentences. the main idea of the suggested model is to align semantics across different languages after training on a monolingual semantic similarity dataset. first of all, the model is trained to measure the similarity between two sentences in the pivot language using a semantic similarity dataset. to extend the monolingual model to multilingual settings, we suggest a teacher-student network architecture. the student network, which is the final model of our approach, should align sentence vectors across multiple languages while keeping the learned semantic structure. the teacher network, which captures the semantic similarity in the pivot language, distills the semantics of the pivot language to other languages of the student network using translation pairs. in the meantime, the teacher network is not fixed but slowly adapted to the student network. the distillation and the adaptation together enable the alignment between the pivot language and the other language of the student network. finally, the student network produces the multilingual sentence embedding that can measure the calibrated similarity which can not only determine whether a translation pair are close but also quantify the semantic similarity. we demonstrate that the suggested method achieves the state-of-the-art performance on multilingual sentence similarity evaluation of extended sts 2017 @xcite as well as sts 2017 monolingual similarity evaluation and tatoeba-14 languages (artetxe and schwenk, 2019) multilingual sentence retrieval. ## tasks we conduct three tasks to evaluate the semantic alignment of the multilingual sentence embedding model: multilingual similarity evaluation, monolingual similarity evaluation, and multilingual sentence retrieval tasks. ## parallel data similar to previous work m-use @xcite , we consider the following 14 languages as the multilingual setting: ar, de, es, fr, it, ja, ko, nl, pl, pt, ru, th, tr, zh foot_0 . ## experimental results in this section, we evaluate the multilingual sentence embedding model on the multilingual similarity evaluation task, the monolingual similarity evaluation task, and the multilingual sentence retrieval task. the first task is the goal of the suggested model, and the latter two tasks are precedent tasks to be successful on the multilingual similarity evaluation task. all tasks include only the test set, not the train/validation set. also, the proposed model is not trained with any task-specific finetuning process. we compare the proposed model to previous multilingual sentence embedding models, m-use, laser, labse, and @xcite . moreover, we did not train the models from the previous papers. the reported scores are from the papers or evaluation using the publicly available models. ## conclusion this paper introduces the multilingual sentence embedding model, which can compare the similarity between sentences in different languages. the proposed model shows the state-of-the-art performance on the multilingual similarity evaluation task as well as the monolingual similarity evaluation task and the multilingual sentence retrieval task for the languages it has learned. starting from a model learned to catch similarities between pivot language sentences, the proposed model is trained to align sentence embedding vectors between different languages. the teacher network which produces the pivot language sentence vectors is updated by a slow-moving average rather than fixed. because the teacher network moves slower than the student network, it conveys sentence embedding vectors which are preserved from the monolingual similarity training. while the teacher network gradually moves to the student network, the alignment between the pivot language sentence of the teacher network and other language sentences of the student network turns into the alignment between the sentence vectors of the student network.