Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding (2024)

Abstract: Learning multilingual sentence embeddings is a fundamental task in natural language processing. Recent trends in learning both monolingual and multilingual sentence embeddings are mainly based on contrastive learning (CL) among an anchor, one positive, and multiple negative instances. In this work, we argue that leveraging multiple positives should be considered for multilingual sentence embeddings because (1) positives in a diverse set of languages can benefit cross-lingual learning, and (2) transitive similarity across multiple positives can provide reliable structural information for learning.In order to investigate the impact of multiple positives in CL, we propose a novel approach, named MPCL, to effectively utilize multiple positive instances to improve the learning of multilingual sentence embeddings. Experimental results on various backbone models and downstream tasks demonstrate that MPCL leads to better retrieval, semantic similarity, and classification performance compared to conventional CL. We also observe that in unseen languages, sentence embedding models trained on multiple positives show better cross-lingual transfer performance than models trained on a single positive instance.

## introduction multilingual sentence embedding transforms sentences in different languages into a shared embedding space @xcite , where sentences with similar meanings are positioned close to each other. this is a fundamental and important task in natural language processing (nlp), with various applications including multilingual retrieval @xcite , cross-lingual classifications @xcite , and multilingual inference @xcite . as contrastive learning (cl) exhibits great strength on learning sentence representation, clbased methods have become the common practice for learning monolingual @xcite @xcite as well as multilingual sentence embeddings @xcite . typically, conventional cl is performed with an anchor, a positive and multiple negative examples. the learning objective of cl is to pull the anchor and the positive closer and push the anchor and the negatives apart @xcite . our work aims at improving cl for multilingual sentence embedding with multiple positives. while existing approaches in multilingual sentence embedding only takes the naive cl with a single positive example, we argue that leveraging multiple positives should be considered especially for multilingual sentence embeddings. in contrast to monolingual cl, richer and more complex relationships exist among multiple positives in multilingual cl: to show such properties, we calculate similarity scores among multiple positives to emphasize the unique effect of multiple positives especially in multilingual scenarios. in the monolingual setting shown in figure 1 (a) and 1(c), although multiple positive examples all share high similarity with the anchor, the transitive similarity does not always exist among positives, e.g., (s1, s2) and (s1, s3) have similar meanings but (s2, s3) do not. by contrast, in multilingual settings shown in figure 1(b ) and 1(d) where translations are used as positives, multiple positives can provide cross-lingual information from diverse languages. moreover, multilingual translation guarantees transitivity of similarity across positive examples, leading to effective cl with multiple positives. motivated by the aforementioned discussion, in this paper, we investigate the impact of cl with multiple positives especially for multilingual sentence embeddings and propose mpcl (multilingual positives in contrastive learning), a novel approach for sentence embedding to effectively leverage multiple positives to improve the quality of multilingual sentence embeddings. specifically, we construct multiple positive instances by collecting multilingual translations for the anchor sentence. besides, we propose to utilize a multipositive loss function to effectively learn from the multiple positives, in which the conventional contrastive correlation and structural information among multilingual translations are learned simultaneously. to the best of our knowledge, we are the first to explore the impact of multiple positives on multilingual sentence embeddings. extensive experiments on various models and downstream tasks are conducted to evaluate the proposed approach. experimental results confirm that leveraging multiple positives leads to better semantic similarity, retrieval, and classification performance on labse @xcite , with an improvement of 4.1 on the bucc task, 2.7 on the sts17 task, and 1.5 on the mtop domain classifi-cation task, respectively. the improvement holds for a diverse set of backbone models, including the continual training on well-trained sentence embedding models such as msimcse @xcite , as well as training from scratch on pretrained language models such as mbert @xcite and xlm-roberta @xcite . a variant of mpcl outperforms the stateof-the-art model, msimcse, in various evaluation tasks. we also observe better cross-lingual transfer performance on unseen languages with our proposed mpcl compared to conventional cl with a single positive instance. moreover, to investigate the effectiveness of mpcl, we evaluate variants of mpcl by incorporating different languages and adjusting the number of languages included in our training dataset. ## multiple positives in contrastive learning for multilingual sentence embeddings this section begins with a formal definition of learning sentence embeddings with cl, followed by the construction of training data with multiple positives and the utilization of multiple positives for sentence embeddings. ## conclusion in this work, we propose mpcl, which improves multilingual sentence embeddings by utilizing a set of positive examples, specifically multilingual translations, for each anchor sentence. through the incorporation of multiple positives, mpcl captures both linguistic diversity and transitive similarities, thereby enriching the embedding space. it updates pairwise similarity distributions into group-wise similarity ones by contrasting the anchor with multiple positives and employs a novel multi-positive loss function that simultaneously learns contrastive correlations and structural information among translations. by doing so, mpcl can improve performance in semantic similarity, retrieval, and classification tasks while exhibiting better robustness during training. moreover, it is also observed in the experiments that mpcl shows a better transfer ability on unseen languages.