Multi-way Variational NMT for UGC : Improving Robustness in Zero-shot Scenarios via Mixture Density Networks (2023)

Abstract: This work presents a novel Variational Neural Machine Translation (VNMT) architecture with enhanced robustness properties, which we investigate through a detailed case-study addressing noisy French user-generated content (UGC) translation to English. We show that the proposed model, with results comparable or superior to state-of-the-art VNMT, improves performance over UGC translation in a zero-shot evaluation scenario while keeping optimal translation scores on in-domain test sets. We elaborate on such results by visualizing and explaining how neural learning representations behave when processing UGC noise. In addition, we show that VNMT enforces robustness to the learned embeddings, which can be later used for robust transfer learning approaches.

## introduction the specificities of user-generated content (ugc) leads to a wide range of vocabulary and grammar variations @xcite @xcite . these variations result in a large increase of out-of-vocabulary words (oovs) in ugc corpora with respect to canonical parallel training data and raise many challenges for machine translation (mt), all the more since common language variations found in ugc are actually productive (there will always be new forms that will not have been seen during training). this fact limits the pertinence of "standard" domain adaptation methods such as fine-tuning foot_0 or normalization techniques @xcite and urges the development of robust machine translation models able to cope with out-of-distribution (ood) texts in a challenging zeroshot scenario in which the target distribution is unknown during training. to address the problem raised by ood texts, an increasing number of works @xcite @xcite @xcite explore the possibility to combine deep learning with latent variable (lv) models, which are indeed able to capture underlying structure information and to model unobserved phenomena. the combination of these models with neural networks was shown to increase performance in several nlp tasks @xcite . in this work, we focus on a specific latent variable model for mt, variational nmt (vnmt) @xcite which has been reported to have good performance and interesting adaptability properties @xcite . the goal of this work is twofold. first, we aim to evaluate the performance of vnmt when translating a special kind of ood texts: french socialmedia noisy ugc. to account for the challenges raised by the productive nature of ugc, we consider a highly challenging zero-shot scenario and assume that only canonical texts foot_1 are available for training the system. we hypothesize that, by leveraging on variational nmt, latent models can build more robust representations able to represent ood observations that are symptomatic of noisy ugc and automatically map them to in-distribution instances, which can be more easily translated. furthermore, to account for the diversity of ugc phenomena, we introduce a new extension of vnmt that relies on mixture density networks @xcite and normalizing flows @xcite . intuitively, each mixture component extracts an independent latent space to represent the source sentence and can potentially model different ugc specificities. interestingly, extracting embeddings from our zero-shot model that has never seen any ugc data and using them in a classic transformer-based nmt model leads to a stronger, more robust to ugc noise model. this is in line with the regularizing character of vnmt @xcite . our contributions can be summarized as follows: variational neural machine translation variational inference (vi) methods (kingma and ba, 2015) are generative architectures capable, from a distributional perspective, of modeling the hidden structures that can be found in a corpus. in a sequence-to-sequence mt task, where x and y are respectively the source and target sentences, vnmt @xcite architectures assume there exists an hidden variable z modeling the implicit structure (i.e. relations) between the bilingual sentence pairs. in the context of ugc translation, we believe that this latent variable can capture the variations between a source sentence and its canonical, normalized form, recovering its underlying meaning and ensuring that the representation of the former is close to the representation of the latter. to make computations tractable, in spite of the latent variable, vi combines a so-called variational posterior q ϕ (z|x, y) that is chosen to approximate the true posterior distribution, with prior p(z|x); and a neural decoder generative distribution, p θ (y|x, z), in charge of generating the translation hypothesis conditioned on the latent variable. once the family of densities q is chosen, the parameters of the two distributions are jointly estimated to model the output y by looking for the parameters (θ, ϕ) that maximizes the evidence lower bound objective function: normalizing flows one of the major caveats of variational methods is that choosing the prior q(z) is a complicated process that requires some a priori knowledge of the task. in practice, a normal distribution with fixed parameters (generally µ = 0.0 and σ = 1.0) is often chosen due to the simplicity of its re-parametrization for sampling. however, such an assumption can be restrictive when modeling more complex processes. regarding this issue, @xcite propose to enhance variational methods with normalizing flows (nf) @xcite . a chain of normalizing flows is a series of simple bijective functions automatically chosen to extract a more suitable representation for the task at hand from a random variable, by alleviating the restrictions of choosing a default fixed prior. concretely, a base distribution q 0 (z 0 ), that generates the initial latent codes, z 0 , undergoes a series of invertible and smooth transformations f : r @xmath0, called flows. then, the random latent variables z are transformed to the random variable z ′ = f (z) after each flow: finally, we can build an arbitrarily k-long chain of f k transformations to generate the final latent variables, z k , from the initial random variables, z 0 , which is drawn from the base distribution q 0 (z 0 ) (often chosen to be n (0, 1)): mixture density networks mixture density networks (mdn) are another interesting generalization of variational encoding for modeling ugc. by using mdn, the posterior distribution of the current decoding step p(z|x, y t ) is no longer approximated by a single variational distribution q ϕ (z|x, y 1:t-1 ) but by a linear combination of variational posteriors qm ϕ (z|x, y 1:t-1 ): p(z|x, y t ) = m @xmath1, y 1:t-1 )• qm (z|x, y 1:t-1 ) gumbel-softmax sampling regarding the mixing coefficients definition, we also explore the use of a categorical probability distribution, for which probabilities are calculated by the network, such as in @xcite . unlike theirs, our supervised end-to-end training requires backpropagating the error gradient through the variational network via reparametrized sampling (kingma and welling, 2014) which poses optimization challenges because of the discrete random variables used as latent vector for categorical distributions. for this reason, we use the reparametrization of this distribution via the gumbel-softmax sampling @xcite , such that, the arg max function is approximated by a softmax and generates the relaxed one-hot encoded samples, which correspond to the mixing coefficients: ## extending variational methods for robust mt our model adopts a variational encoder-decoder architecture inspired by sketchrnn ( §2) that uses an mdn on the decoder's variational network to model multiple and independent continuous generative variational distributions. however, unlike sketchrnn, we use a transformer backbone for the encoder and the decoder and train our model in a end-to-end manner on canonical parallel corpora. in the following, we will first describe the general architecture of our model, denoted multi-vnmt, and then detail the encoder and decoder parameters. general architecture figure 1a represents the architecture of our model. the input sentence is first processed by a standard transformer encoder, the output of which is used by a variational encoder enhanced with nf to predict a latent representation of the input sentence. the latent representation and the output of the last layer of the transformer encoder are combined using the gating mechanism of @xcite . this combined representation is then fed to the decoder that has a similar architecture: it is made of an "usual" transformer decoder and a variational mdn that is sampled to obtain a prediction that will be combined to the transformer output by (again) a gating mechanism. the model can be trained in an end-to-end fashion using the "reparametrization trick" of @xcite . in order to ensure that the estimated variances for the variational posteriors are positive, we used the softplus activation function @xcite , as done in van den berg et al. (2018)'s implementation. in addition, concerning the training of the decoder's mdn, we compare two different ways to compute the mixing coefficient: the first one consists in a vanilla non-latent softmax, the second on a relaxed categorical variational method that relies on a gumbel-softmax sampling ( §2). the model has been implemented in opennmt-py 3 @xcite . encoder our encoder backbone is the "standard" transformer of @xcite , made of 6-layered transformer layers each with 8 attention heads. the feed-forward layers have 2,048 parameters and the dimension of lexical embeddings is 512. the dimension of the encoder vari-3 https://github.com/josecar25/mdn-vnmt ational network is 128. the network is extented with 4-flows normalizing planar flows (rezende and mohamed, 2015). 4 following to @xcite , we combine the last transformer layer output to the latent vectors using a gating mechanism. we used a feed-forward network to transform the representation of dimension 128 predicted by the variational network into a representation of size 512 that matches the transformer representation dimension. in figure 1b , we show the transformer and variational encoding latent state z as being estimated (p θ (z|x)) approximating the posterior's mean and variance, both learned using the reparametrization trick. in the figure, we can also observe how our model's encoder comprises the transformer backbone and vi network. decoder as for the encoder, the first component of the decoder is the "standard" transformer decoder of @xcite and uses the same parameters as the transformer encoder. the transformer decoder's last layer output is passed to a 128-component mdn, with trainable parameters ϕ and π: ϕ encodes the mean and variance of each multivariate gaussian components; π contains the probabilities of the categorical distribution that generates the mixing coefficient for each component. concisely, we estimate a posterior as a series of m posteriors parameterized by ⟨ϕ, π⟩, i.e. qϕ;π m (z dec |x, y 1:t-1 ), conditioned via the decoder's transformer, on both the gated latent encoder's output and previous predicted tokens, y 1:t-1 . the mdn's mixing coefficient (α m (x, y 1:t-1 )) network also takes the same input and is computed separately, by either using a fully-forward layer with softmax activation or the relaxed categorical gumbel distribution. both networks computing qm and α m are jointly trained in an end-to-end fashion, such that translation loss is minimal for representations sampled from the resulting mixture, obtained according to equation 4. ## training models all systems are trained using a batch size of 4,096 tokens using the adam optimizer (kingma and ba, 2015) accumulating gradients every 2 steps, and the noam learning rate schedule @xcite with 8k warmup steps. throughout training, learning rate attains a maximum of 7e-4 and minimum of 1e-5. both encoder and decoder transformers are trained using 0.1 dropout and we employed 0.1 label smoothing @xcite . training for, at most, 300k training iterations on a single nvidia v100 took about 40 hours to converge for the multi-vnmt models, 34 hours for vnmt-baseline and 28 hours for the non-latent transformer baseline. in order to avoid posterior collapse, and as done in setiawan et al. (2020), we use β c -vae @xcite , with values β = 1 and @xmath2.1. additionally, we used a kullback-leibler (kl) annealing schedule of 100k iterations for training. we set a 10% probability of dropping the target word @xcite . we have chosen, as initial experimental configuration, τ = 1.0 for the gumbel-softmax sampling temperature, which was selected mainly aiming to avoid artificial gradient scaling during backpropagation (c.f. equation ( 5 )). a beam of width 5 has been used for evaluation. ## experiments datasets we train our different mt models on two different french to english canonical parallel corpora: the first one is a subset of the wmt corpus, i.e. europarl (v7) and newscommentary(v10) @xcite and the second one is theopensubtitles'18 corpus @xcite . we used bpe tokenization @xcite with 16k merge operations. detailed statistics on our corpora can be found in ugc test sets to evaluate the different nmt models, we consider two data sets of manually translated ugc: mtnt @xcite and the parallel french social media bank corpus (pfsmb) @xcite foot_3 which extends the french social media bank @xcite with english translations. these two data sets raise many challenges for mt systems: they notably contain characters that have not been seen in the training data (e.g. emojis), rare character sequences (e.g. inconsistent casing or usernames) as well as many oovs denoting url, mentions, hashtags or more generally named entities (ne). most of the time, soovs are exactly the same in the source and target sentences. we also consider the 4square corpus (berard et al., 2019) as a blind test to validate our conclusions. to analyze our neural representations ( §7), we use a subset of the pfsmb, called pmumt, which contains 400 annotated and normalized french to english ugc sentences @xcite . protocols translation quality was evaluated using bleu @xcite and chrf2 @xcite both computed by sacrebleu @xcite with the 'intl' tokenization, after detokenizing the systems outputs. in all the experiments we used the hyperparameters values reported by @xcite and only choose the number of components of the mdn and the dimension of the latent representation on the validation set. 6 regarding the latent dimension, we conducted the same study with 128, 256 and 512 dimensions, with 128 being the best value. a beam of size 5 has been used for evaluation. ## results in this section we present the main mt results to study mt performance of our methods. ## analyzing latent representations in this section, we describe several experiments aiming at understanding how multi-vnmt uncovers more robust representations than the vnmt baseline. impact of noise in the source first, to evaluate the perturbations that the model suffers when noise is present in the source, we measure the cosine similarity between the representations of the french noisy sentences and their normalized version, taking advantage of the pmumt corpus ( §5). more precisely, we compare the source-side embeddings of the 400 original noisy ugc sentences and their corresponding 400 fullynormalized versions built by vnmt-baseline and multi-vnmt. we observe that the average cosine similarity between the noisy and normalized learning representations of multi-vnmt is 0.36 compared to an average similarity of 0.26 for the representations of vnmt-baseline, suggesting that the former provides more robust representations of ugc than the latter, a conclusion supported by the distribution of similarities shown in figure 2 . noisy vs normalized data to complete the previous analysis, we have reported, in figure 3 , the projection of the representations of noisy and normalized sentences computed by t-sne. we can notice how both vnmt systems have a tendency to separate noisy and normalized sentences compared to transformer, while both having higher cosine similarity than the latter. transferring learning representations as discussed above, in figure 3 we noticed that vnmt seems to enforce noisy morphology modeling to the transformer's embeddings in an implicit way. this motivated us to study whether the information in such learning representations can be used by the transformer backbone model and benefit from improved robustness while removing the direct latent space contribution, and notably, with the same number of parameters and architecture as transformer. thus, in results in ## blind test sets scores we evaluated our best performing model (multi-vnmt trained on opensubtitles) on the blind test sets described in § 5, translating another set of tests to assess whether our approach proves useful for generalization over different types of ugc. we have also included the 4square corpus @xcite to validate our vnmt system on other domain of ugc (restaurant reviews). we also display the results when using the vnmt-baseline baseline and the transformer model to assess improvement of our proposed architecture. we report such results in ## discussion how mdn behaves under noise in appendix a, we discuss how mdn components are activated when translating canonical in-domain and ood texts, as well as ugc and normalized ugc. in figure 4 and however, an open question arising from this work, it is currently unclear if the performance gain we observed is due to a better generalisation to distributional shift or if it corresponds to a better adaptation to noise in the input. future works will be devoted to this question, which can be abstracted away to study whether ugc idiosyncrasies are a form of noise, some parts being learnable, or are rather points to a new domain.