Cross-lingual Sentence Embedding using Multi-Task Learning (2021)

Abstract: Multilingual sentence embeddings capture rich semantic information not only for measuring similarity between texts but also for catering to a broad range of downstream cross-lingual NLP tasks. State-of-the-art multilingual sentence embedding models require large parallel corpora to learn efficiently, which confines the scope of these models. In this paper, we propose a novel sentence embedding framework based on an unsupervised loss function for generating effective multilingual sentence embeddings, eliminating the need for parallel corpora. We capture semantic similarity and relatedness between sentences using a multi-task loss function for training a dual encoder model mapping different languages onto the same vector space. We demonstrate the efficacy of an unsupervised as well as a weakly supervised variant of our framework on STS, BUCC and Tatoeba benchmark tasks. The proposed unsupervised sentence embedding framework outperforms even supervised state-of-the-art methods for certain under-resourced languages on the Tatoeba dataset and on a monolingual benchmark. Further, we show enhanced zero-shot learning capabilities for more than 30 languages, with the model being trained on only 13 languages. Our model can be extended to a wide range of languages from any language family, as it overcomes the requirement of parallel corpora for training.

## introduction sentence embeddings provide an efficient way to encode semantic information of text by mapping texts onto a shared vector space, such that sentences with similar meaning are represented by similar representations. with the abundance of data in diverse languages, cross-lingual sentence embedding enable the mapping of multilingual texts into a single unified vector space for a wide range of natural language processing (nlp) tasks. current sentence embedding methods are predominantly monolingual systems, geared mainly towards english @xcite . while there exist multilingual sentence embedding frameworks, they are mostly supervised methods requiring a large parallel corpus for training. for underresourced languages, there is not sufficient training data to effectively learn a model and we show that our unsupervised approach can better exploit the available unsupervised data, and thus produce better results for under-resourced languages. this is achieved by using a dual-encoder architecture based on word-level semantic similarity score (via word mover's distance) and learning to embed this into a single vector for sentences. supervised sentence embedding approaches map parallel sentences from source and target languages into the same vector space by either maximising their cosine similarity or minimising the distance between the generated embeddings @xcite . for example, recent supervised methods using parallel corpus rely on a teacher-student model to minimize cross-lingual embedding distance @xcite or an additive margin softmax function based dual sentence encoder to maximally separate the sentences that are true translations from similar overlapping sentences @xcite . although such methods produced good results, the use of these loss functions in unsupervised settings fails to efficiently capture crosslingual semantic similarities across sentences. the state-of-the-art unsupervised approach relies on automated machine translation to generate a "pseudo parallel corpus" @xcite for training. this method is affected by presence of translation errors and fails to generalize to low-resource languages for which translations are not available. to alleviate the above challenges, in this paper, we propose dueam, a cross-lingual sentence embedding framework based on a novel dual encoder architecture with an unsupervised joint loss function using an anchor-learner approach, a variant of the teacher-student model. we also depict the performance of a weakly-supervised variant of our unsupervised dueam architecture (obtained by simply changing the training dataset). the weaklysupervised framework to learn semantic relationship between cross-lingual sentences is motivated by the existence of the multilingual natural language inference dataset (xnli) @xcite , and the possible creation of such a dataset from existing comparable corpora. the unsupervised dueam framework learns from randomly chosen sentence pairs from the xnli dataset (see sec. 5). thus, we overcome the need for parallel sentences for multilingual sentence embedding generation. to understand the degree of similarity between monolingual and multilingual sentence pairs during training, the anchor module uses the word mover's distance (wmd) @xcite (used as a scalar value during backpropagation), while the learner module is trained to generate sentence embeddings (refer to fig. 1 ). thus, we learn a low-dimensional embedding of the sentences from the more complex encoding generated by wmd. we show that our joint loss formulation effectively captures cross-lingual semantic similarity between sentences by preserving distances between points across languages. extensive experiments (in section 6) on multilingual sentence similarity and parallel sentence mining tasks have showcased the efficacy of our sentence embedding framework. for example, on the cross-lingual sts benchmark @xcite , our unsupervised approach achieves state-of-the-art average spearman rank correlation score of 62.1, comparable to the supervised sentence embedding approach of laser @xcite with an average of 65.8. in fact, for certain languages, our models are even seen to outperform laser (e.g., for en-de our unsupervised model achieves a spearman rank correlation score of 64.6 and weaklysupervised model achieves a spearman rank correlation score of 69.4 compared to 64.2 for laser). on the bucc task (bitext mining task) @xcite our model achieves a better f1 score compared to the existing unsupervised model of @xcite . interestingly, for certain under-resourced languages, we outperform both laser and multilingual s-bert @xcite by an average of 10% on the tatoeba benchmark. we also show better or comparable performance to laser even on monolingual classification benchmark tasks. thus, our model is robust across diverse language families for multilingual sentence embeddings. in a nutshell, our contributions are: (i) dueam, a novel dual encoder based on an anchor-learner architecture for unsupervised and weakly-supervised multilingual sentence embedding generation, (ii) a joint loss function coupling word mover's distance and cosine similarity to capture the degree of text similarity and relatedness between sentence pairs, (iii) experimental evaluations, on monolingual as well as several cross-lingual benchmark tasks, depict that our model efficiently captures semantic similarity across languages, and provides state-of-the-art unsupervised performance, comparable with supervised models, (iv) robustness in zero-shot transfer learning for low-resource languages across language families, outperforming state-of-the-art supervised approaches on sentence matching tasks in certain scenarios. ## related work paragraph vectors were first proposed as sentence embeddings for computing document similarity @xcite . the majority of the current multilingual sentence embedding methods are supervised approaches. there exist some unsupervised sentence embedding frameworks @xcite , but are mostly for english sentence embeddings. initial methods generated sentence embeddings based on neural machine translation system with a shared encoder @xcite @xcite . the use of cosine similarities between source and target language parallel sentences was studied by @xcite using a bidirectional dual encoder architecture. @xcite proposed multilingual universal sentence encoder (muse), a dual-encoder model trained on large web-mined translation parallel corpora, along with data from reddit, wikipedia, and stanford natural language inference (snli) @xcite to learn more context, supporting 16 languages. a translation ranking task was used to identify a correct translation pair, and the architecture assumes 5 hard negative pairs for each sample while training. subsequently, the laser @xcite framework considered a sequence-to-sequence architecture using lstm networks, and was trained on parallel corpora designed for neural machine translation across 93 languages. expanding beyond translation-based approaches, the multilingual sentence encoder model of @xcite was trained for semantic retrieval on three different tasks: multi-feature question-answer prediction, translation ranking, and natural language inference (nli). recently, @xcite proposed conditional masked language modeling (cmlm) to generate sentence embeddings, by co-training the system with bi-text retrieval and natural language inference (nli) tasks. to generate sentence embeddings beyond the naïve cls token and simple pooling strategies of language models, sentence transformer architectures were proposed @xcite . for multilingual s-bert models, reimers and gurevych (2020) utilized the teacher-student model where the student model is tuned with a parallel corpus from 50 languages based on knowledge transfer from a fine-tuned teacher model developed by @xcite . labse @xcite was designed on the bert architecture and trained on 6 billion sentence pairs by use of additive margin softmax loss with in-batch negative sampling. another thread of research for sentence embedding involves improving the alignment of contextual embeddings into a shared vector space using iterative self-supervised learning or tuning with synthetic parallel corpora. @xcite introduced the enhancing multilingual sentence embeddings (emu) framework which tries to semantically enhance pre-trained multilingual sentence embeddings. that is, instead of building sentence embeddings from scratch, emu fine-tunes pre-trained multilingual sentence embeddings with two major components: enhancement of semantic similarity and multilinguality of sentence embeddings using multlingual adverserial training. further, @xcite used a parallel corpus as an anchor to align representations in a multilingual language model whereas @xcite used iterative self-learning to perform the task. recently, @xcite proposed an unsupervised method for improving pre-trained cross-lingual context vectors using synthetic parallel sentences and extracted sentence embeddings via mean pooling. however, use of machine translation to generate synthetic parallel data for such methods fails to generalize to low-resourced languages for which translations might be erroneous. ## dueam model in this section, we describe the components and working of the proposed dual encoder with anchor model (dueam) architecture for multilingual sentence embeddings, trained using an unsupervised multi-task joint loss function. ## intuitions behind loss function in dueam, wmd is computed between the contextual token embeddings (of the pair of input sentences) obtained from anchor encoders, without any stopwords removal (as standard while using language models @xcite ). while multi-lingual language models capture different languages in a common space, wmd captures the contextualized semantic distance between the multi-lingual input sentences. our use of wmd is motivated by moverscore @xcite . specifically, the use of wmd and cosine in the loss function of dueam is based on the following intuitions: weakly supervised data. this training data contains both monolingual and cross-lingual sentence pairs, where the monolingual sentence pairs are same as those of xnli (without annotated labels). to create cross-lingual sentence pairs, we keep the premises from the source language and replace the hypothesis with target language hypothesis sentences, and vice-versa (example shown in validation (using accuracy of finding parallel sentences) is done on held out 1k parallel sentences (across languages pairs) from the ted2020 corpus. the dueam models trained on the above unsupervised and weakly supervised datasets are henceforth denoted as dueam unsupv and dueam wklysupv respectively. more details on dataset and training are given in the appendix. ## experimental evaluation we evaluate the performance of our proposed dueam framework on the following 3 benchmark tasks: (a) sts: monolingual and cross-lingual semantic textual similarity; (b) bucc: bitext mining to extract parallel sentences; and (c) tatoeba: cross-lingual parallel sentence matching. we compare our model with the following supervised and unsupervised state-of-the-art approaches: (i) mbert / xlm-r -language model with mean pooling, (ii) muse -dual-encoder transformer architecture @xcite , (iii) laser -encoder-decoder architecture using lstm @xcite , (iv) labse -dual-encoder model based on bert @xcite , (v) xlm-r ← sbert-nli-stsb / xlm-r ← sbert-paraphrases -sentence transformer models @xcite . further details on training setup and baseline methods can be found in section a. ## ablation study we now study the effects of different components of dueam on the quality of generated embeddings. necessity of multi-task learning. one of the important features of dueam is multi-task joint learning via the dual-encoder based anchor-learner architecture. to explore the necessity of the different factors for our learning loss function, we use the tatoeba dataset for de-en, fr-en, and hi-en language pairs. we train weakly-supervised dueam wklysupv in three variants: (i) with only the anchor module loss term l a (eq. 1), which considers semantic similarity between sentences, (ii) using only the translation mining loss term l b (eq. from figure 2 , we observe that the learning objective factors l a and l b individually perform quite poorly. however, the proposed multi-task training performs efficiently providing a high accuracy in the range of 92% to 96%. similar results are observed across the language pairs considered. training dataset size. in weight parameter value selection. weight parameter selection while training the multi-task model is very important. we experimented with the weight parameter over a range of values and set to the value for which the model has performed best while training. we have given a snapshot of the model performances over the different weight parameters in figure 3 . we have calculated the average spearman rank correlation (ρ) results for sts datasets for six language pairs. from the figure we can see that the best model performance achieved with weight parameter 0.003. higher weight parameter value decreased the performance. this helps us to understand the importance of the weight parameter while training the model in multi-task settings. ## discussion: semantic similarity in general, applications use cosine similarity between sentence embeddings to gauge the semantic textual similarity. we provide a performance analysis of dueam based on the raw cosine sim-ilarity score on tatoeba de-en data. for example, the german sentence "das ist der geburtstag von muiriel!" ("that is the birthday of muiriel") has the highest cosine similarity with its english translation "it is muiriel's birthday!", although the sentence "happy birthday, muiriel!" is very similar (refer ## conclusion this paper proposed an unsupervised loss function based dueam framework for multilingual sentence embeddings based on dual encoder with anchor-learner model via multi-task learning. experiments on monolingual and cross-lingual benchmarks showcase the efficacy of our sentence embeddings in capturing semantic similarities across languages. we demonstrate that dueam significantly outperforms existing unsupervised models for textual similarity understanding. we also depicts robustness in zero-shot learning and limited training, for catering to under-resourced languages, and achieve results better or comparable to existing supervised methods in certain cases.