Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation (2020)

Abstract: We present an easy and efficient method to extend existing sentence embedding models to new languages. This allows to create multilingual versions from previously monolingual models. The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence. We use the original (monolingual) model to generate sentence embeddings for the source language and then train a new system on translated sentences to mimic the original model. Compared to other methods for training multilingual sentence embeddings, this approach has several advantages: It is easy to extend existing models with relatively few samples to new languages, it is easier to ensure desired properties for the vector space, and the hardware requirements for training are lower. We demonstrate the effectiveness of our approach for 50+ languages from various language families. Code to extend sentence embeddings models to more than 400 languages is publicly available.

## introduction mapping sentences or short text paragraphs to a dense vector space, such that similar sentences are close, has wide applications in nlp. however, most existing sentence embeddings models are monolingual, usually only for english, as applicable training data for other languages is scarce. for multiand cross-lingual scenarios, only few sentence embeddings models exist. in this publication, we present a new method that allows us to extend existing sentence embeddings models to new languages. we require a teacher model m for source language s and a set of parallel (translated) sentences ((s 1 , t 1 ), ..., (s n , t n )) with t i the translation of s i . note, the t i can be in different languages. we train a new student model m such that m (s i ) ≈ m (s i ) and m (t i ) ≈ m (s i ) using mean squared loss. we call this approach multilingual knowledge distillation, as the student m distills the knowledge of the teacher m in a multilingual setup. we demonstrate that this type of training works for various language combinations as well as for multilingual setups. we observe an especially high improvement of up to 40 accuracy points for low resource languages compared to laser @xcite . the student model m learns a multilingual sentence embedding space with two important properties: 1) vector spaces are aligned across languages, i.e., identical sentences in different languages are close, 2) vector space properties in the original source language from the teacher model m are adopted and transferred to other languages. the presented approach has various advantages compared to other training approaches for multilingual sentence embeddings. laser @xcite ) trains an encoder-decoder lstm model using a translation task. the output of the encoder is used as sentence embedding. while laser works well for identifying exact translations in different languages, it works less well for assessing the similarity of sentences that are not exact translations. multilingual universal sentence encoder (muse) @xcite was trained in a multi-task setup on snli @xcite and on over a billion question-answer pairs from popular online forums and qa websites. in order to align the crosslingual vector spaces, muse used a translation ranking task. given a translation pair (s i , t i ) and various alternative (incorrect) translations, identify the correct translation. first, multi-task learning is difficult since it can suffer from catastrophic forgetting and balancing multiple tasks is not straight-forward. further, running the translation ranking task is complex and results in a huge computational overhead. selecting random alternative translations usually leads to mediocre results. instead, hard negatives @xcite are required, i.e., alternative incorrect translations that have a high similarity to the correct translation. to get these hard negatives, muse was first trained with random negatives samples, then, this preliminary sentence encoder was used to identify hard negative examples. they then re-trained the network. in this work, we use sentence-bert (sbert) @xcite , which achieves state-of-the-art performance for various sentence embeddings task. sbert is based on transformer models like bert @xcite and applies mean pooling on the output. in our experiments we use xlm-r @xcite , a pre-trained network on 100 languages, as student model. note, the described approach is not limited to be used with transformer models and should also work with other network architectures. ## training we require a teacher model m , that maps sentences in one or more source languages s to a dense vector space. further, we need parallel (translated) sentences ((s 1 , t 1 ), ..., (s n , t n )) with s i a sentence in one of the source languages and t i a sentence in one of the target languages. we train a student model m such that m (s i ) ≈ m (s i ) and m (t i ) ≈ m (s i ). for a given minibatch b, we minimize the mean-squared loss: m could have the structure and the weights of m , or it can be a different network architecture. this training procedure is illustrated in figure 1 . we denote trained models with m ← m , as the student model m learns the representation of the teacher model m . in our experiments, we mainly use an english sbert model as teacher model m and use xlm-roberta (xlm-r) as student model m . the english bert models have a wordpiece vocabulary size of 30k mainly consisting of english tokens. using the english sbert model as initialization for m would be suboptimal, as most words in other latin-based languages would be broken down to short character sequences, and words in nonlatin alphabets would be mapped to the unk token. in contrast, xlm-r uses sentencepiece foot_1 , which avoids language specific pre-processing. further, it uses a vocabulary with 250k entries from 100 different languages. this makes xlm-r much more suitable for the initialization of the multilingual student model. ## training data in this section, we evaluate the importance of training data for making the sentence embedding model multilingual. the opus website foot_2 @xcite provides parallel data for hundreds of language pairs. in our experiments, we use the following datasets: getting parallel sentence data can be challenging for some low-resource language pairs. hence, we also experiment with bilingual dictionaries: the data set sizes for english-german (en-de) and english-arabic (en-ar) are depicted in we trained xlm-r as our student model and used sbert fine-tuned on english nli and sts data foot_3 as our teacher model. we trained for a maximum of 20 epochs with batch size 64, 10,000 warm-up steps, and a learning rate of 2e-5. as development set, we measured the mse loss on hold-out parallel sentences. in @xcite , we showed that the random seed can have a large impact on the performances of trained models, especially for small datasets. in the following experiments, we have quite large datasets of up to several million parallel sentences and we observed rather minor differences (∼ 0.3 score points) between random seeds. ## experiments in this section, we conduct experiments on three tasks: multi-and cross-lingual semantic textual similarity (sts), bitext retrieval, and cross-lingual similarity search. sts assigns a score for a pair of sentences, while bitext retrieval identifies parallel (translated) sentences from two large monolingual corpora. note, evaluating the capability of different strategies to align vector spaces across languages is nontrivial. the performance for cross-lingual tasks depends on the ability to map sentences across languages to one vector space (usually the vector space for english) as well as on the properties this source vector space has. differences in performance can then be due to a better or worse alignment between the languages or due to different properties of the (source) vector space. we evaluate the following systems: sbert-nli-stsb: the output of the bert-base model is combined with mean pooling to create a fixed-sized sentence representation @xcite . it was fine-tuned on the english allnli (snli @xcite and multi-nli @xcite ) dataset and on the english training set of the sts benchmark @xcite using a siamese network structure. laser: laser (artetxe and schwenk, 2019b) uses max-pooling over the output of a stacked lstm-encoder. the encoder was trained in an encoder-decoder setup (machine translation setup) on parallel corpora over 93 languages. labse: language-agnostic bert sentence embedding (labse) @xcite was trained similar to muse with a dual-encoder transformer architecture based on bert with 6 billion translation pairs for 109 languages. ## evaluation of training datasets to evaluate the suitability of the different training sets, we trained bilingual xlm-r models for en-de and en-ar on the described training datasets. english and german are fairly similar languages and have a large overlap in their alphabets, while english and arabic are dissimilar languages with distinct alphabets. we evaluate the performance on the sts 2017 dataset. the results for training on the full datasets are shown in first, we observe that the bilingual models are slightly better than the model trained for 10 languages (section 4.1): 2.2 points improvement for en-de and 1.2 points improvement for en-ar. @xcite calls this curse of multilinguality, where adding more languages to a model can degrade the performance as the capacity of the model remains the same. for en-de we observe only minor differences between the datasets. it appears that the domain of the training data (news, subtitles, parliamentary debates, magazines) is of minor importance. further, only little training data is necessary. for the dissimilar languages english and arabic, the results are less conclusive. in this section we evaluate whether it is better to transfer an english model to a certain target language or if training from-scratch on suitable datasets in the target language yields better results. for this, we use the kornli and korsts datasets from @xcite . they translated the english snli @xcite , multinli @xcite , and stsbenchmark (stsb) @xcite datasets to korean with an internal machine translation system. the dev and tests were postedited by professional translators. ham et al. fine-tuned korean roberta and xlm-r on these datasets using the sbert framework. we use the translated sentences they provide and tuned xlm-r using multilingual knowledge distillation. we use sbert-nli-stsb as teacher model. results are shown in we observe a slight improvement of using multilingual knowledge distillation over training the models directly on the translated nli and sts data. this is great news: training on the korean datasets yields a model only for korean, while with multilingual knowledge distillation, we get a model for english and korean with aligned vector spaces. further, we do not necessarily have a performance drop if there is only training data for the sentence embedding method in english available. figure 2 shows the plot of the first two principle components for different multi-lingual sentence embeddings methods. in the plot, we encoded the english premise sentences from xnli @xcite with their russian translation. the plot shows for the labse model a drastic separation between the two languages, indicating that the language significantly impacts the resulting embedding vector. the experiments in section 4 used so far monolingual sentence pools, i.e., all sentences in the source / target pool were of the same language. hence, these benchmarks are not suited to measure a potential harmful effect from language bias. in order to measure a potential negative effect from language bias, we combine all sentence pairs from the multilingual sts dataset and compute similar-ity scores as described in section 4.1. note, in section 4.1, the models had only to score e.g. en-ar sentence pairs. now, there are 10 language combinations in one joined set. a model without language bias would achieve on this joined set a performance similar to the average of the performances over the individual subsets. however, if a model has a language bias, sentence pairs from specific language combinations will be ranked higher than others, lowering the spearman rank correlation for the joint set. in summary, muse and the proposed multilingual knowledge distillation approach can be used on multilingual sentence pools without a negative performance impact from language bias, while laser and labse prefer certain language combinations over other, impacting the overall result. ## related work sentence embeddings are a well studied area with dozens of proposed methods @xcite @xcite . most of the methods have in common that they were only trained on english. multilingual representations have attracted significant attention in recent times. most of it focuses on cross-lingual word embeddings @xcite . a common approach is to train word embeddings for each language separately and to learn a linear transformation that maps them to a shared space based on a bilingual dictionary @xcite . this mapping can also be learned without parallel data @xcite . a straightforward approach for creating crosslingual sentence embeddings is to use a bag-ofwords representation of cross-lingual word embeddings. however, @xcite showed that this approach works poorly in practical crosslingual transfer settings. laser (artetxe and 1 & 2). actual score is the correlation for one joined set of sentence pairs. models without language bias would score on the joined set similar to the average over the individual sets. the difference shows the negative impact from the language bias. schwenk, 2019b) uses a sequence-to-sequence encoder-decoder architecture @xcite based on lstm networks. it trains on parallel corpora from neural machine translation. to create a fixed sized sentence representation, they apply max-pooling over the output of the encoder. laser was trained for 93 languages on 16 nvidia v100 gpus for about 5 days. multilingual universal sentence encoder (muse) 10 (chidambaram et al., 2019; yang et al., 2019) uses a dual-encoder architecture. it was trained in a multi-task setup on snli @xcite and over 1 billion crawled question-answer pairs from various communities. a translation ranking task was applied: given a sentence in the source language and a set of sentences in the target languages, identify the correct translation pair. to work well, hard negative examples (similar, but incorrect translations) must be included in the ranking task. muse was trained for 16 languages with 30 million steps. labse @xcite is based on a bert architecture and used masked language model and 6 billion translation pairs for training. it was trained similar to muse with a translation ranking loss, however, without any other training data. in this publication, we extended sentence-bert (sbert) @xcite . sbert is based on transformer models like bert @xcite and fine-tunes those using a siamese network structure. by using the pre-trained weights from bert, suitable sentence embeddings methods can be trained efficiently. multilingual bert (mbert) was trained on 104 languages using wikipedia, while xlm-r @xcite was trained on 100 languages using commoncrawl. mbert and xlm-r were not trained on any parallel data, hence, their vector spaces are not aligned. ## conclusion we presented a method to make monolingual sentence embeddings multilingual with aligned vector spaces between the languages. this was achieved by using multilingual knowledge distillation. we demonstrated that this approach successfully transfers properties from the source language vector space (in our case english) to various target languages. models can be extended to multiple languages in the same training process. this stepwise training approach has the advantage that an embedding model with desired properties, for example for clustering, can first be created for a high-resource language. then, in an independent step, it can be extended to support further languages. this decoupling significantly simplifies the training procedure compared to previous approaches. further, it minimizes the potential language bias of the resulting model. we extensively tested the approach for various languages from different language families. we observe that laser and labse work well for retrieving exact translations, however, they work less well assessing the similarity of sentence pairs that are not exact translations. further, we noticed that laser and labse show a language bias, preferring some language combinations over other. figure 2 : first two principle components for parallel sentences in english and russian. for labse, we observe a strong separation between the vector spaces, i.e., it is more biased for same language pairs. muse and the proposed multilingual knowledge distillation approach show nearly no language bias.