Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast (2021)

Abstract: In this paper, we propose to align sentence representations from different languages into a unified embedding space, where semantic similarities (both cross-lingual and monolingual) can be computed with a simple dot product. Pre-trained language models are fine-tuned with the translation ranking task. Existing work (Feng et al., 2020) uses sentences within the same batch as negatives, which can suffer from the issue of easy negatives. We adapt MoCo (He et al., 2020) to further improve the quality of alignment. As the experimental results show, the sentence representations produced by our model achieve the new state-of-the-art on several tasks, including Tatoeba en-zh similarity search (Artetxe andSchwenk, 2019b), BUCC en-zh bitext mining, and semantic textual similarity on 7 datasets.

## introduction pre-trained language models like bert @xcite and gpt @xcite have achieved phenomenal successes on a wide range of nlp tasks. however, sentence representations for different languages are not very well aligned, even for pre-trained multilingual models such as mbert @xcite . this issue is more prominent for language pairs from different families (e.g., english versus chinese). also, previous work @xcite has shown that the out-of-box bert embeddings perform poorly on monolingual semantic textual similarity (sts) tasks. there are two general goals for sentence representation learning: cross-lingual representations should be aligned, which is a crucial step for tasks like bitext mining @xcite , unsupervised machine translation @xcite , and zero-shot cross-lingual transfer @xcite etc. another goal is to induce a metric space, where semantic similarities can be com-puted with simple functions (e.g., dot product on l 2 -normalized representations). translation ranking @xcite can serve as a surrogate task to align sentence representations. intuitively speaking, parallel sentences should have similar representations and are therefore ranked higher, while non-parallel sentences should have dissimilar representations. models are typically trained with in-batch negatives, which need a large batch size to alleviate the easy negatives issue @xcite . @xcite use cross-accelerator negative sampling to enlarge the batch size to 2048 with 32 tpu cores. such a solution is hardware-intensive and still struggles to scale. momentum contrast (moco) @xcite ) decouples the batch size and the number of negatives by maintaining a large memory queue and a momentum encoder. moco requires that queries and keys lie in a shared input space. in selfsupervised vision representation learning, both queries and keys are transformed image patches. however, for translation ranking task, the queries and keys come from different input spaces. in this paper, we present dual momentum contrast to solve this issue. dual momentum contrast maintains two memory queues and two momentum encoders for each language. it combines two contrastive losses by performing bidirectional matching. we conduct experiments on the english-chinese language pair. language models that are separately pre-trained for english and chinese are fine-tuned using translation ranking task with dual momentum contrast. to demonstrate the improved quality of the aligned sentence representations, we report state-of-the-art results on both cross-lingual and monolingual evaluation datasets: tatoeba similarity search dataset (accuracy 95.9% → 97.4%), bucc 2018 bitext mining dataset (f1 score 92.27% → 93.66%), and 7 english sts datasets (average spearman's correlation 77.07% → 78.95%). we also carry out several ablation studies to help understand the learning dynamics of our proposed model. dual momentum contrast is a variant of the moco proposed by @xcite . our method fits into the bigger picture of contrastive learning for self-supervised representation learning @xcite . given a collection of parallel sentences {x i , y i } n @xmath0, as illustrated in figure 1 , we first encode each sentence using languagespecific bert models (base encoder), then apply mean pooling on the last-layer outputs and l 2 normalization to get the representation vector h x i , h y @xmath1. each bert encoder has a momentum encoder, whose parameters θ are updated by exponential moving average of the base encoder as follows: where t is the iteration step. two memory queues are maintained for each language to store k vectors encoded by the corresponding momentum encoder from most recent batches. the oldest vectors are replaced with the vectors from the current batch upon each optimization step. the momentum coefficient @xmath2, 1] is usually very close to 1 (e.g., 0.999) to make sure the vectors in the memory queue are consistent across batches. k can be very large (>10 5 ) to provide enough negative samples for learning robust representations. to train the encoders, we use the infonce loss @xcite : symmetrically, we can get l(y, x). the final loss function is the sum: after the training is done, we throw away the momentum encoders and the memory queues, and only keep the base encoders to compute the sentence representations. in the following, our model is referred to as moco-bert. application given a sentence pair (x i , y j ) from different languages, we can compute cross-lingual semantic similarity by taking dot product of l 2normalized representations h x i • h y j . it is equivalent to cosine similarity, and closely related to the euclidean distance. our model can also be used to compute monolingual semantic similarity. given a sentence pair (x i , x j ) from the same language, assume y j is the translation of x j , if the model is well trained, the representations of x j and y j should be close to each other: h x j ≈ h y j . therefore, we have h ## related work multilingual representation learning aims to jointly model multiple languages. such representations are crucial for multilingual neural machine translation @xcite , zero-shot cross-lingual transfer @xcite , and cross-lingual semantic retrieval @xcite etc. multilingual bert @xcite simply pre-trains on the concatenation of monolingual corpora and shows good generalization for tasks like cross-lingual text classification @xcite . another line of work explicitly aligns representations from language-specific models, either unsupervised @xcite or supervised @xcite . contrastive learning works by pulling positive instances closer and pushing negatives far apart. it has achieved great successes in self-supervised vision representation learning, including simclr @xcite , moco @xcite , byol @xcite , clip @xcite etc. recent efforts introduced contrastive learning into various nlp tasks @xcite @xcite . concurrent to our work, simcse @xcite uses dropout and hard negatives from nli datasets for contrastive sentence similarity learning, sentence-t5 @xcite outperforms simcse by scaling to larger models, and xmoco @xcite adopts a similar variant of moco for open-domain question answering. semantic textual similarity is a long-standing nlp task. early approaches @xcite use lexical resources such as wordnet to measure the similarity of texts. a series of semeval shared tasks @xcite provide a suite of benchmark datasets that is now widely used for evaluation. since obtaining large amounts of high-quality sts training data is non-trivial, most sts models are based on weak supervision data, including conversations @xcite , nli @xcite , and qa pairs @xcite . ## conclusion this paper proposes a novel method that aims to solve the easy negatives issue to better align crosslingual sentence representations. extensive experiments on multiple cross-lingual and monolingual evaluation datasets show the superiority of the resulting representations. for future work, we would like to explore other contrastive learning methods @xcite , and experiment with more downstream tasks including paraphrase mining, text clustering, and bilingual lexicon induction etc.