A Bilingual Generative Transformer for Semantic Sentence Embedding (2020)

Abstract: Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model’s posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks – contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.

## introduction learning useful representations of language has been a source of recent success in natural language processing (nlp). much work has been done on learning representations for words @xcite and sentences @xcite . more recently, 1 code and data to replicate results available at https: //www.cs.cmu.edu/ ˜jwieting. there are three main ingredients when designing a sentence embedding model: the architecture, the training data, and the objective function. many architectures including lstms @xcite @xcite , transformers @xcite , and averaging models @xcite are capable of learning sentence embeddings. the choice of training data and objective are intimately intertwined, and there are a wide variety of options including next-sentence prediction @xcite , machine translation @xcite @xcite , natural language inference (nli) @xcite , and multi-task objectives which include some of the previously mentioned objectives @xcite potentially combined with additional tasks like constituency parsing @xcite . surprisingly, despite ample testing of more powerful architectures, the best performing models for many sentence embedding tasks related to semantic similarity often use simple architectures that are mostly agnostic to the interactions between words. for instance, some of the top performing techniques use word embedding averaging @xcite , character n-grams @xcite , and subword embedding averaging @xcite to create representations. these simple approaches are competitive with much more complicated architectures on in-domain data and generalize well to unseen domains, but are fundamentally limited by their inability to capture word order. training these approaches generally relies on discriminative objectives defined on paraphrase data @xcite or bilingual data @xcite @xcite . the inclusion of latent variables in these models has also been explored @xcite . intuitively, bilingual data in particular is promising because it potentially offers a useful signal for learning the underlying semantics of sentences. within a translation pair, properties shared by both sentences are more likely semantic, while those that are divergent are more likely stylistic or languagespecific. while previous work learning from bilingual data perhaps takes advantage of this fact implicitly, the focus of this paper is modelling this intuition explicitly, and to the best of our knowledge, this has not been explored in prior work. specifically, we propose a deep generative model that is encouraged to perform source separation on parallel sentences, isolating what they have in common in a latent semantic embedding and explaining what is left over with language-specific latent vectors. at test time, we use inference networks @xcite for approximating the model's posterior on the semantic and source-separated latent variables to encode monolingual sentences. finally, since our model and training objective are generative, our approach does not require knowledge of the distance metrics to be used during evaluation, and it has the additional property of being able to generate text. in experiments, we evaluate our probabilistic source-separation approach on a standard suite of sts evaluations. we demonstrate that the proposed approach is effective, most notably allowing the learning of high-capacity deep transformer architectures @xcite while still generalizing to new domains, significantly outperforming a variety of state-of-the-art baselines. further, we conduct a thorough analysis by identifying subsets of the sts evaluation where simple word overlap the observed text in each language is then conditioned on its languagespecific variable and z sem . finally, we analyze our model to uncover what information was captured by the source separation into the semantic and language-specific variables and the relationship between this encoded information and language distance to english. we find that the language-specific variables tend to explain more superficial or language-specific properties such as overall sentence length, amount and location of punctuation, and the gender of articles (if gender is present in the language), but semantic and syntactic information is more concentrated in the shared semantic variables, matching our intuition. language distance has an effect as well, where languages that share common structures with english put more information into the semantic variables, while more distant languages put more information into the language-specific variables. lastly, we show outputs generated from our model that exhibit its ability to do a type of style transfer. x fr x en semantic inference net english inference net french inference net latent space x fr x en concat concat generative model kl between prior and approximate posteriors reconstruction of input translation pair concatenate sampled latent vectors and decode inference networks merge english decoder french decoder input english and french translation pair french encoder english encoder semantic encoder semantic encoder q(z sem | x en , x fr ; ϕ) q(z en | x en ; ϕ) q(z fr | x fr ; ϕ) p(x en | z sem , z en ; θ) p(x fr | z sem , z fr ; θ) 𝒩(μ sem , σ sem ) 𝒩(μ en , σ en ) 𝒩(μ fr , σ fr ) μ sem , σ sem μ fr , σ fr μ en , σ en semantic encoding z sem merging operation (section 3) figure 2 : the computation graph for the variational lower bound used during training. the english and french text are fed into their respective inference networks and the semantic inference network to ultimately produce the language variables z f r and z en and semantic variable z sem . each language-specific variable is then concatenated to z sem and used by the decoder to reconstruct the input sentence pair. ## model our proposed training objective leverages a generative model of parallel text in two languages. our model is language agnostic, and applies to a wide variety of languages (see section 5) but we will use the running example of english (en) and french (fr) pairs consisting of an english sentence x en and a french sentence x f r . importantly, the generative process utilizes three underlying latent vectors: language-specific variation variables (language variables) z f r and z en for each side of the translation, as well as a shared semantic variation variable (semantic variable) z sem . in this section we will first describe the generative model for the text and latent variables. in the following section we will describe the inference procedure of z sem given an input sentence, which corresponds to our core task of obtaining sentence embeddings useful for downstream tasks such as semantic similarity. the generative process of our model, the bilingual generative transformer (bgt), is depicted in figure 1 and the training computation graph is shown in figure 2 . first, we sample latent variables z f r , z en , z sem , where z @xmath0, from a multivariate gaussian prior n(0, i k ). these variables are then fed into a decoder that samples sentences; x en is sampled conditioned on z sem and z en , while x f r is sampled conditioned on z sem and z f r . because sentences in both languages will use z sem in generation, we expect that in a well-trained model this variable will encode semantic, syntactic, or stylistic information shared across both sentences, while z f r and z en will handle any language-specific peculiarities or specific stylistic decisions that are less central to the sentence meaning and thus do not translate across sentences. in the following section, we further discuss how this is explicitly encouraged by the learning process. decoder architecture. many latent variable models for text use lstms @xcite as their decoders @xcite @xcite . however, state-of-the-art models in neural machine translation have seen increased performance and speed using deep transformer architectures. we also found in our experiments (see appendix c for details) that transformers led to increased performance in our setting, so they are used in our main model. we use two decoders in our model, one for modelling p(x f r |z sem , z f r ; θ) and one for modeling p(x en |z sem , z en ; θ) (see right side of figure 2 ). each decoder takes in a language variable and a semantic variable, which are concatenated and used by the decoder for reconstruction. we explore four ways of using this latent vector: (1) concatenate it to the word embeddings (word) (2) use it as the initial hidden state (hidden, lstm only) (3) use it as you would the attention context vector in the traditional sequence-to-sequence framework (attention) and ( 4 ) concatenate it to the hidden state immediately prior to computing the logits (logit). unlike attention, there is no additional feedforward layer in this setting. we experimented with these four approaches, as well as combinations thereof, and report this analysis in appendix a. from these experiments, we see that the closer the sentence embedding is to the final word predictor, the better the performance on downstream tasks evaluating its semantic content. we hypothesise that this is due to better gradient propagation because the sentence embedding is now closer to the error signal. since attention and logit performed best, we use these in our main experiments. ## learning and inference our model is trained on a set of parallel sentences x consisting of n examples, @xmath1, x 1 f r , . . . , x n en , x n f r }, and z is our collection of latent variables @xmath2, z 1 f r , z 1 sem , . . . , z n en , z n f r , z n sem ). we wish to maximize the likelihood of the parameters of the two decoders θ with respect to the observed x, marginalizing over the latent variables z. unfortunately, this integral is intractable due to the complex relationship between x and z. however, related latent variable models like variational autoencoders (vaes; @xcite ) learn by optimizing a variational lower bound on the log marginal likelihood. this surrogate objective is called the evidence lower bound (elbo) and introduces a variational approximation, q to the true posterior of the model p. the q distribution is parameterized by a neural network with parameters φ. elbo for our model is written as: this lower bound on the marginal can be optimized by gradient ascent by using the reparameterization trick @xcite . this trick allows for the expectation under q to be approximated through sampling in a way that preserves backpropagation. we make several independence assumptions for q(z sem , z en , z f r |x en , x f r ; φ) to match our goal of source separation: we factor q as q( the parameters of the encoders that make up the inference networks, defined in the next paragraph, are denoted as φ. lastly, we note that the kl term in our elbo equation explicitly encourages explaining variation that is shared by translations with the shared semantic variable, and explaining language-specific variation with the corresponding language-specific variables. encoding information shared by the two sentences in the shared variable results in only a single penalty from the kl loss, while encoding the information separately in both language specific variables will cause unnecessary replication, doubling the overall cost incurred by the kl term. encoder architecture. we use three inference networks as shown on the left side of figure 2 : an english inference network to produce the english language variable, a french inference network to produce the french language variable, and a semantic inference network to produce the semantic variable. just as in the decoder architecture, we use a transformer for the encoders. the semantic inference network is a bilingual encoder that encodes each language. for each translation pair, we alternate which of the two parallel sentences is fed into the semantic encoder within a batch. since the semantic encoder is meant to capture language agnostic semantic information, its outputs for a translation pair should be similar regardless of the language of the input sentence. we note that other operations are possible for combining the views each parallel sentence offers. for instance, we could feed both sentences into the semantic encoder and pool their representations. however, in practice we find that alternating works well and also can be used to obtain sentence embeddings for text that is not part of a translation pair. we leave further study of combining views to future work. ## analysis we next analyze our bgt model by examining what elements of syntax and semantics the language and semantic variables capture relative both to each-other and to the sentence embeddings from the bilingualtrans models. we also analyze how the choice of language and its lexical and syntactic distance from english affects the semantic and syntactic information captured by the semantic and language-specific encoders. finally, we also show that our model is capable of sentence generation in a type of style transfer, demonstrating its capabilities as a generative model. 3 : average sts performance for the 2012-2016 datasets, measured in pearson's r × 100, followed by probing results on predicting number of subjects, number of objects, constituent tree depth, top constituent, word content, length, number of punctuation marks, the first punctuation mark, and whether the articles in the sentence are the correct gender. all probing results are measured in accuracy ×100. ## conclusion we propose bilingual generative transformers, a model that uses parallel data to learn to perform source separation of common semantic information between two languages from language-specific information. we show that the model is able to accomplish this source separation through probing tasks and text generation in a style-transfer setting. we find that our model bests all baselines on unsupervised semantic similarity tasks, with the largest gains coming from a new challenge we propose as hard sts, designed to foil methods approximating semantic similarity as word overlap. we also find our model to be especially effective on unsupervised cross-lingual semantic similarity, due to its stripping away of language-specific information allowing for the underlying semantics to be more directly compared. in future work, we will explore generalizing this approach to the multilingual setting, or applying it to the pre-train and fine-tune paradigm used widely in other models such as bert.