Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval (2023)

Abstract: Contrastive learning has been successfully used for retrieval of semantically aligned sentences, but it often requires large batch sizes or careful engineering to work well. In this paper, we instead propose a generative model for learning multilingual text embeddings which can be used to retrieve or score sentence pairs. Our model operates on parallel data in N languages and, through an approximation we introduce, efficiently encourages source separation in this multilingual setting, separating semantic information that is shared between translations from stylistic or language-specific variation. We show careful large-scale comparisons between contrastive and generation-based approaches for learning multilingual text embeddings, a comparison that has not been done to the best of our knowledge despite the popularity of these approaches. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval - the last of which we introduce in this paper. Overall, our model outperforms both a strong contrastive and generative baseline on these tasks.

## introduction contrastive learning is the dominant paradigm for learning text representations from parallel text @xcite @xcite @xcite . however, contrastive learning requires strong negative examples in the data and finding these negatives can be expensive in terms of compute or manual effort. in this paper, we propose a generative foot_1 model for learning multilingual text embeddings which encourages source separation, separating semantic information that is shared between translations from stylistic or language-specific variation. we find that by filtering this variation into separate variables, performance of the remaining representations, that encode shared semantic information, increases across all downstream tasks. through an approximation that greatly reduces the memory footprint of our model, we scale our model and train on 92 languages. we systematically compare our model, the variational multilingual source-separation transformer (vmsst) to strong contrastive and generative baselines on a suite of tasks including semantic similarity, bitext mining, and question retrieval, which we introduce for the cross-lingual setting, using the same training data and architecture. we show that our model outperforms these models and is also competitive with the state-of-the-art. we analyze vmsst with careful ablations, showing the contribution of each aspect of the model to performance. we also show that even at large batch sizes, the advantage over contrastive learning remains, especially for large models. furthermore, we also find the learned embedding space of our model to be smoother, making it less affected by the "hubness problem" @xcite in representation learning, and more suitable for large-scale retrieval than the baseline methods. to the best of our knowledge, this is the first work to systematically compare generative and contrastive models for learning multilingual embeddings on a large parallel corpus containing many languages in a carefully controlled experimental setup-despite the popularity of these approaches @xcite . we carry out these experiments with both pretrained and randomly initialized models. the comparison of objective functions is an important research question due to the large amounts of multi-lingual text available to train models and the many uses of these models in downstream tasks. to that end, another contribution of this paper is showing these comparisons and the surprising result that contrastive objectives do not provide the overall best accuracy on downstream tasks. moreover, our generative vmsst increasingly outperforms the contrastive model when more layers are added and when training with larger batches and more training data, suggesting that as models continue to scale in the future, this performance gap may continue to increase further motivating the use of generative approaches for learning multilingual text embeddings. ## related work there has been a number of approaches proposed for learning bilingual and multilingual text embeddings. one popular approach is contrastive learning @xcite @xcite @xcite where translation pairs are positive examples and text from other pairs are used as negative examples. an alternative approach is to use a neural machine translation objective, where the representation from the hidden states of the encoder is used as the sentence embedding @xcite @xcite . other approaches include multi-task learning approaches which often use some type of contrastive learning of parallel text to align representations among languages @xcite , cross-lingual pretraining @xcite , and model distillation from a large pretrained multilingual model @xcite ). an alternative approach that is more closely related to our work is generative models that separate the linguistic variation from the shared semantic information in translation pairs. @xcite considered this for bitext, with each language having its own encoder and decoder parameters. this approach however does not scale, since it is not feasible to have thousands of encoders and decoders if one wants to model all of the more than 7,000 languages in the world. ## model the generative process of our underlying probabilistic model and the computation graph of our training objective procedure are depicted in fig- gaussian prior language-specific latent vector 𝒩 (0 ,𝐼 ) gaussian prior 𝒩 (0 ,𝐼 ) common semantic latent vector x l i observed sentences n-way translations n z sem semantic inference net shared semantic variation linguistic variation specific to languages to l 1 l n z l i n language decoders n language inference nets figure 1 : the generative process of our model. latent variables, z li , modeling the variation x li specifically due to language l i , as well as a latent variable modeling the common semantics, z sem , are drawn from a multivariate gaussian prior. the observed translation in each language is then conditioned on its language-specific variable and z sem . in practice, we approximate this model to make learning and inference tractable. ure 1 and figure 2 respectively. in the generative story for vmsst, we first sample a semantic variable z sem for the sentence. then for each of the n languages, we sample a language-specific variable z l i . each latent variable z is sampled from a multivariate gaussian prior n(0, i k ). these variables are then fed into a decoder that samples each of the n sentences in the translation set. each observed translation x l i , is sampled conditioned on z sem and its language variable z l i . because z sem will be used to generate the sampled sentences in all languages, we expect that this variable will encode semantic, syntactic, or stylistic information that is shared in all of the translations. conversely, the language variables z l i will handle languagespecific peculiarities or specific style differences that are not central to the meaning of the translation and are therefore not contained in many of the sentences. concretely, the likelihood function of our model can be written for a single n -way tuple of translations @xmath0, ..., x n ): in the next section, we discuss how this separation of information is encouraged during learning. ## learning and inference we would like to train our model on a set of parallel sentences x consisting of m examples in n languages and a collection of latent variables z. however, n -way parallel corpora are not available at the scale of bilingual text, and so we therefore approximate an n -way parallel corpus by sampling translation pairs from a large pool of pairs containing text in n languages. therefore in our model, @xmath1 in contrast to variational autoencoders, which have only a single latent variable for each example, we have three in our model for each example. to encourage source separation, we make several independence assumptions for q and factor it into three terms: lastly, we note that the elbo contains a kl term that acts to regularize the latent variables. in our model, the kl term encourages z sem , z l i , and z l j to be close to a zero-centered gaussian prior. the kl term thus encourages source separation, as encoding information shared by the translation pair in the shared variable results in only a single penalty from the kl loss, while encoding the information separately in the language-specific variables unnecessarily doubles the overall cost. in effect, we can view these language-specific latent variables as collecting information that cannot be captured in a common semantic space, separating it out from the variables collecting shared semantic information that we use for downstream tasks. objective function. the overall objective function for vmsst consists of consists of two terms, the first being elbo as described earlier: the second term, which we found necessary for strong performance, is the sum of p(x l i |µ sem l j ) and p(x l j |µ sem l i ) which can be interpreted as samples from the mean of the posterior distribution using semantic variables generated from both input sentences. when training variational objectives, where the model ignores the latent variables and the learned posterior remains close to the prior. examples of other approaches to address these issues include: @xcite @xcite . we weight the elbo by λ giving the total objective as: therefore, our objective resembles translation with a weighted source-separation term. we show the effectiveness of this formulation compared to a pure translation objective in our experiments in section 6. ## architecture our architecture is an encoder-decoder model, where the encoder produces a single representation that is fed into the decoder. cross-attention between the encoder and decoder is not used, therefore the decoder has no full sequence visibility and more pressure is applied on the encoder to create a semantically meaningful representation. specifically, we follow the approach of @xcite which uses a transformer @xcite encoder-decoder model, where the sentence embeddings are used in two places: at each layer of the decoder in place of cross-attention and in the computation of the logits. enc. embs. language-sp. layers enc. embs. inference net l i enc. embs. language-sp. layers x l j x l i concat concat generative model reconstruction of input translation pair concatenate sampled latent vectors and decode decoder p(x l i | z sem , z l i ; θ) p(x l j | z sem , z l j ; θ) x j semantic inference net inference net l j latent space kl between prior and approximate posteriors inference networks merge input translation pair for languages and sampled from pool of translations pairs figure 2 : the computation graph for the variational lower bound used to train vmsst. the text for languages l i and l j , with their respective language embeddings, are fed into the encoder acting as their inference networks. the text is also fed into the semantic inference network which is a separate encoder. the output of these networks are the language variables z li and z lj and semantic variable z sem . each language-specific variable is then concatenated to z sem and used by a single shared decoder to reconstruct the input sentence pair. decoder architecture. the decoder models p(x l i |z sem , z l i ; θ) for each language i (see right side of figure 2 ). the inputs to the decoder are the language-specific variable z l i and the semantic variable z sem , which are concatenated and used to condition the decoder to generate the reconstruction of the observed text x l i . we use a single decoder for all languages. encoder architecture. the encoders play an important role in the source separation as well as inference as detailed below. in order to motivate the separation of the linguistic and semantic information we split the encoder into two parts, only sharing the embedding table. we use one of these encoders to be the semantic inference network, which produces the semantic variable. the other encoder represents the n language inference networks and produces the language variables for each language. these inference networks are shown on the left side of figure 2 . we mean-pool the hidden states followed by a linear projection to produce each variable from the encoders. the semantic inference network, which models q(z sem |x l i , x l j ; ϕ), is a multilingual encoder that encodes each language. for each translation pair, we alternate which of the two parallel sentences is fed into the semantic encoder within a batch for the elbo term in the objective. since the semantic encoder is meant to capture language agnostic semantic information, its outputs for a translation pair should be similar regardless of the language of the input sentence. we use the mean of the semantic encoder as the sentence representation for downstream tasks. ## analysis in this section, we analyze vmsst with additional experiments. we give a high-level overview in this section and put details and results in appendix a. we first investigate different ablations of the model. we analyzed aspects such as factorizing the projection layer, using weaker decoders, using 4 language-specific encoders instead of 1, removing the kl term, using a single encoder for the semantic and language embeddings, and removing the language embeddings from the encoder and from the decoder. secondly, we analyze the effect of the parameter sharing approximation in vmsst, where we train a full model with separate encoders and decoders for each language. this experiment uses data in 4 languages to make this experiment tractable. we found the performance to be similar enough that we can say the approximation holds, but there does remain a small gap. we hypothesize however, that this performance gap will shrink as the number of layers of the model increases. thirdly, we evaluate the performance of zeroshot bitext mining on languages that were unseen in the training data. we find significant improvement in this setting over the baseline bitranslation. since bitranslation can be seen as an ablation of vmsst, we see that the source-separation loss especially helps with generalization to new languages. lastly, we investigate the impact of batch size on performance, comparing vmsst with con-trastive. it is common knowledge that contrastive models learn better representations when given harder negative examples, and bigger batch sizes increases the chances of finding these harder negatives. we experiment with batch sizes of 4096 and 8192, for both the 6 layer and 24 layer randomly initialized versions of contrastive and vmsst. we find that both models improve when trained with larger batches, with the very best model being the 24 layer vmsst. ## conclusion we present vmsst, a generative massively multilingual text embedding model trained to separate semantic information from language-specific information. vmsst also outperforms strong contrastive and generative baselines on a variety of tasks. there are several avenues for future work including alternative pretraining objectives that better fit the use case of the decoder, explore incorporating monolingual data into the generative objective, investigate synergy between vmsst and contrastive methods as they seem to specialize in different tasks, and lastly scale up to bigger models, more data, and languages to further investigate vmsst versus contrastive methods.