Language-agnostic BERT Sentence Embedding (2022)

Abstract: While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM), dual encoder translation ranking, and additive margin softmax. We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by LASER, while still performing competitively on monolingual transfer learning benchmarks. Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE .

## introduction in this paper, we systematically explore using pretraining language models in combination with the best of existing methods for learning cross-lingual sentence embeddings. such embeddings are useful for clustering, retrieval, and modular use of text representations for downstream tasks. while existing cross-lingual sentence embedding models incorporate large transformer models, using large pretrained language models is not well explored. rather in prior work, encoders are trained directly on translation pairs @xcite @xcite , or on translation pairs combined with monolingual inputresponse prediction @xcite . in our exploration, as illustrated in figure 1 , we make use of dual-encoder models, which have been demonstrated as an effective approach for learning bilingual sentence embeddings @xcite . however, diverging from prior work, rather than training encoders from scratch, we investigate using pre-trained encoders based on large language models. we contrast models with and without additive margin softmax @xcite foot_0 . figure 2 illustrates where our work stands (shaded) in the field of lm pre-training and sentence embedding learning. our massively multilingual models outperform the previous state-of-the-art on large bi-text retrieval tasks including the united nations (un) corpus @xcite and bucc @xcite . the contributions of this paper are: we release the pre-trained model at https:// tfhub.dev/google/labse . ## cross-lingual sentence embeddings dual encoder models are an effective approach for learning cross-lingual embeddings @xcite . such models consist of paired encoding models that feed a scoring function. the source and target sentences are encoded separately. sentence embeddings are extracted from each encoder. cross-lingual embeddings are trained using a translation ranking task with inbatch negative sampling: dual encoder models trained using a translation ranking loss directly maximize the similarity of translation pairs in a shared embedding space. ## results row 4-7 shows the performance of models that use the public mbert vocabulary. the baseline model shows reasonable performance on un ranging from 57%-71% p@1. it also perform well on tatoeba with 92.8% and 79.1% accuracy for the 36 language group and all languages, respectively. adding pre-training both helps models converge faster (see details in section 6.2) and improves performance on the un retrieval task using both vocabularies. pre-training also helps on taoeba, but only using the customized vocabulary. 17 additive margin softmax significantly improves the performance on all model variations. the last two rows contain models using the customized vocab. both of them are trained with additive margin softmax given the strong evidence from the experiments above. both models outperform the mbert vocabulary based models, and the pre-trained model performs best of all. the top model (base w/ customized vocab + ams + pt) achieves a new state-of-the-art on 3 of the 4 languages, with p@1 91.1, 88.3, 90.8 for en-es, en-fr, en-ru, respectively. it reaches 87.7 on zh-en, only 0.2 lower than the best bilingual en-zh model and nearly 9 points better than the previous best multilingual model. on tatoeba, the best model also outperform the baseline model by a large margin, with +10.6 accuracy on the 36 language group model un (@xmath0, 2019b) -----84.4 65.5 m-use @xcite 86.1 83.3 88.9 78.8 84.3 -- yang et al. (2019a from xtreme and +18.2 on all languages. it is worth noting that all our models perform similarly on tatoeba but not on un. this suggests it is necessary to evaluate on large scale bitext retrieval tasks to better discern differences between competing models. for the rest of the paper we refer to labse as the best performing model here, base w/ customized vocab + ams + pt, unless otherwise specified. ## analysis 6.1 additive margin softmax the above experiments show that additive margin softmax is a critical factor in learning good crosslingual embeddings, which is aligned with the findings from @xcite . we further investi- ## mining parallel text from commoncrawl we use the labse model to mine parallel text from commoncrawl, a large-scale multilingual web corpus, and then train nmt models on the mined data. we experiment with two language pairs: english-to-chinese (en-zh) and english-to-german (en-de). we mine translations from monolingual commoncrawl data processed as described above for self-supervised mlm pretraining. after processing, there are 1.17b, 0.6b, 7.73b sentences for chinese (zh), german (de), and english (en), respectively. labse embeddings are used to pair each non-english sentence with its nearest english neighbor, dropping pairs with a similarity @xmath1.6. foot_16 for en-de and en-zh, we train a model with transformer-big @xcite in the following way: first we train the model on the mined data as is for 120k steps with batch size 10k. then we select the best 20% using wang results in ## conclusion this paper presents a language-agnostic bert sentence embedding (labse) model supporting 109 languages. the model achieves state-of-the-art performance on various bi-text retrieval/mining tasks compare to the previous state-of-the-art, while also providing increased language coverage. we show the model performs strongly even on those languages where labse doesn't have any explicit training data, likely due to language similarity and the massively multilingual natural of the model. extensive experiments show additive margin softmax is a key factor for training the model, parallel data quantity matters, but the effect of increased amounts of parallel data diminishes when a pre-trained language model is used. the pretrained model is released at https://tfhub. dev/google/labse .