Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularization (2023)

Abstract: Multilingual sentence representations are the foundation for similarity-based bitext mining, which is crucial for scaling multilingual neural machine translation (NMT) system to more languages. In this paper, we introduce MuSR: a one-for-all Multilingual Sentence Representation model that supports 223 languages. Leveraging billions of English-centric parallel corpora, we train a multilingual Transformer encoder, coupled with an auxiliary Transformer decoder, by adopting a multilingual NMT framework with CrossConST, a cross-lingual consistency regularization technique proposed in Gao et al. (2023). Experimental results on multilingual similarity search and bitext mining tasks show the effectiveness of our approach. Specifically, MuSR achieves superior performance over LASER3 (Heffernan et al., 2022) which consists of 148 independent multilingual sentence encoders.

## introduction multilingual sentence representation models @xcite @xcite @xcite align different languages in a shared representation space, facilitating similarity-based bitext mining that extracts parallel sentences for learning multilingual neural machine translation (nmt) systems @xcite . specifically, laser3 @xcite scales the original laser @xcite beyond the 93 widely used languages and achieves the stateof-the-art (sota) performance on the multilingual sentence alignment tasks over 200 languages. 1 in its original context, laser3 refers solely to the language-specific models presented in @xcite . for simplicity, we use laser3 as an umbrella term encompassing the multilingual model laser2 and the languagespecific models discussed in this paper. 2 previous presentations of this work are available at https: //arxiv.org/abs/2306.06919 . although laser3 exhibits remarkable performance, it is not a one-for-all multilingual sentence representation model. instead, it comprises of one multilingual model called laser2 and 147 language-specific models, which are learned through a teacher-student training mechanism. such model strategy, although effective, results in substantial storage overhead of 78gb and degraded transfer performance from high-resource to low-resource languages, which hinders its practical value in natural language processing (nlp). in this paper, our primary goal is to learn a unified multilingual sentence encoder, musr, to handle a wide range of languages such that semanticequivalent sentences in different languages are close to each other in the representation space. inspired by the cross-lingual consistency for multilingual nmt @xcite , we learn multilingual sentence embeddings by utilizing a many-to-one multilingual nmt training paradigm with crosslingual consistency regularization (figures 1 and 2 ). in order to support a wide range of languages, we collect about 5.5 billion english-centric parallel sentences covering 223 languages from both opensource and in-house datasets. to the best of our knowledge, musr is the first one-for-all multilingual sentence representation model that supports more than 220 languages. the contributions of this paper can be summarized as follows: method #models #parameters #languages task architecture monolingual pretrain laser2 1 45m 93 seq2seq bi-lstm laser3 1 + 147 n/a 205 dual encoder transformer ✓ labse 1 471m 109 dual encoder transformer ✓ ✓ musr 1 434m 223 seq2seq transformer ## methodology following the similar problem formulation of @xcite , our approach is based on a transformer encoder-decoder architecture trained with english-centric parallel corpora. we discuss the details of our model architecture and training strategy as follows. ## experimental evaluation following the evaluation setup of heffernan et al. ## conclusion in this paper, we propose musr: a one-for-all multilingual sentence representation model supporting 223 languages. experimental results show that musr could yield strong performance on various bitext retrieval and mining tasks compare with the sota models labse and laser3, while also providing increased language coverage in a single model. extensive analysis shows that crossconst and the sentence embedding dimension play the key roles in learning multilingual sentence representations. as for future work, we could explore the development of lightweight models by distilling knowledge from musr for multilingual sentence alignment, which would potentially lower the computational requirements and make the model more accessible for a variety of applications.