E nglish Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings (2022)

Abstract: Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space. Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sentences. In this work, we propose mSimCSE, which extends SimCSE to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data.In unsupervised and weakly supervised settings, mSimCSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks. The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving low-resource languages and multilingual STS.The performance can be further enhanced when cross-lingual NLI data is available.

## introduction universal cross-lingual sentence embeddings map the sentences from multiple languages into a shared embedding space, where semantically similar sentences across languages are close to each other. these embeddings have a wide spectrum of applications such as multi-lingual document retrieval @xcite , multi-lingual question answering @xcite , unsupervised machine translation @xcite , and zeroshot transfer learning @xcite . as shown in figure 1 (a), without finetuning on downstream tasks, the embedding space of pretrained multilingual language models such as m-bert @xcite or xlm-r @xcite separate the embeddings of each language into different clusters. to align cross-lingual figure 1 : we visualize the sentence embeddings on xnli corpus, where blue dots and green dots denote the sentences from english and swahili respectively. here, red dots, black dots, and purple dots denote the parallel sentences from different languages. in (a), the sentence embeddings from different languages are clearly separated into two clusters. in (b), after english nli training, the embedding space becomes indistinguishable for different languages, and the parallel sentences are aligned to each other. pre-trained multilingual language models @xcite have shown impressive performance on cross-lingual zero-shot transfer @xcite ) that a model finetuned on a source language can generalize to target languages. this implies the representations finetuned on downstream tasks are universal across various languages. in this work, we explore various crosslingual transfer settings on sentence retrieval tasks, especially in the setting of using english data only. we propose multilingual-simcse (msimcse) which extends simcse @xcite , a famous sentence embedding method on english, to multilingual for cross-lingual transfer. simcse is a contrastive learning @xcite @xcite ) method that pulls closer semantically similar sentences (i.e. positive sentence pairs) in embeddings space. as done in simcse, we obtain positive training pairs by either natural language inference (nli) @xcite supervision or unsupervised data augmentation using dropout. we also investigate model performance when a small amount of parallel data or cross-lingual nli data are available. in our experiments, as shown in figure 1 (b), we are surprised to find that contrastive learning on pure english data seems to be able to align cross-language representations. sentences that are semantically similar across languages are clearly closer together. compared with previous unsupervised or weakly supervised methods, our unsupervised method significantly improves the performance on cross-lingual sts and sentence retrieval tasks. in retrieving low-resource languages and sts tasks, our method is even on par with fully supervised methods trained on billions of parallel data. our results show that using contrastive learning to learn sentence relationships is more efficient than using massively parallel data for learning universal sentence embeddings. to the best of our knowledge, we are the first to demonstrate that using only english data can effectively learn universal sentence embeddings. ## discussion our experimental results demonstrate that in both unsupervised and english nli supervised settings, using english data alone can surprisingly align cross-lingual sentence embeddings. by comparing unsupervised results with nli supervised results, we observe that learning more meaningful sentence relationships can further enhance the alignment. in our analysis, we find that infinitely increasing parallel training data is not the most efficient manner to learn universal sentence embeddings; instead, our results suggest that designing a more challenging contrastive task or more effective sentence embedding learning method on english data may be a more efficient direction. also, contrastive learning may be a promising direction for improving the zero-shot transfer of pre-trained multilingual language models. we attribute the alignment to language-invariant contrastive training. because multilingual language models have shown good performance on zero-shot transfer, we speculate that multilingual language models encode texts into two disentangled embeddings, a language-specific embedding and a general language agnostic embedding. because english contrastive task doesn't require mlms to capture language identity, it only pulls closer language-agnostic sentence embeddings while weakening language-specific embedding. this property can be verified in figure 1 and ## conclusion in this work, we demonstrate that using only english data can effectively learn universal sentence embeddings. we propose four different strategies to extend simcse to multilingual, including unsupervised, english nli supervised, cross-lingual nli supervised, and supervised strategies. we surprisingly find that the english nli supervised strategy can achieve performance on par with previous supervised methods that leverage large amounts of parallel data. our work provides a new perspective on learning universal sentence embeddings and cross-lingual transfer that using contrastive learning to learn sentence semantic relationships on monolingual corpora may be a promising direction.