Target Conditioning for One-to-Many Generation (2020)

Abstract: Neural Machine Translation (NMT) models often lack diversity in their generated translations, even when paired with search algorithm, like beam search. A challenge is that the diversity in translations are caused by the variability in the target language, and cannot be inferred from the source sentence alone. In this paper, we propose to explicitly model this one-to-many mapping by conditioning the decoder of a NMT model on a latent variable that represents the domain of target sentences. The domain is a discrete variable generated by a target encoder that is jointly trained with the NMT model. The predicted domain of target sentences are given as input to the decoder during training. At inference, we can generate diverse translations by decoding with different domains. Unlike our strongest baseline (Shen et al., 2019), our method can scale to any number of domains without affecting the performance or the training time. We assess the quality and diversity of translations generated by our model with several metrics, on three different datasets.

## introduction neural machine translation (nmt) models are trained to translate a sentence from a source language into a target language. there are many translations of the same sentence that are both grammatically correct and faithful to the source, but these translations may differ greatly in their vocabulary, style or grammar. inferring the best translation among them requires to explore a vast output space to cover this variability. this is typically handle as a post-processing step using a search algorithm, like beam search. this procedure is known to produce translations that lack in diversity, often differing only by a punctuation or a word @xcite . while the search algorithm can certainly be improved, part of the problem resides also in the training of the nmt models; they are trained on 1-to-1 translation datasets without any objective to encourage diverse translations. there are many ways to model the diversity of translations from data that contain only one translation, such as mixture of experts @xcite or variational autoencoders @xcite . a particularity of machine translation is that it is a one-to-many mapping problem. this means that the variability should be encoded by the target sentence and the question is how to combine a nmt system with a target sentence encoder with no posterior collapse. in this work, we propose to combine the encoder of the nmt with a discrete target encoder. similar to other discrete autoencoders @xcite , each target sentence is assigned to a discrete variable, or domain, and each domain is associated with an embedding. the embeddings from both encoders are then fed to the decoder of the nmt to form a translation. the discrete latent representation follows a categorical distribution that is constrained to be uniform over the dataset to avoid a mode collapse. since each domain has its own embedding, changing the domain embedding changes the translation. at test time, we can thus condition the generation on each domain embedding to produce multiple translations with high diversity. our approach is general and can be applied on top of any model with little computational overhead. an advantage of our approach is that the number of domains can be arbitrarily large without affecting the performance or the running time. our approach can replace or work with beam search during inference. we assess the quality and diversity of translations generated by our model with several metrics, on three different datasets. at training time, a target sentence is encoded with the target transformer encoder to get a latent representation z y . the latent representation is linearly mapped to a vector of size k on which apply a softmax to obtain domain probabilities. each domain is associated with an embedding. the decoder is fed with both the source encoding, and the sum of the domain embeddings reweighted by their probabilities. during inference, we can generate k different hypotheses by switching the domain embedding that is fed to the decoder. to prevent a train-test discrepancy, during training we apply an argmax operator on domain probabilities, with probability p hard . ## related work several studies have proposed to sample diverse sequences by changing the value of a latent variable. for example, one possibility is to add noise to the latent space of a variational auto-encoder @xcite to diversify samples in machine translation @xcite , language modeling @xcite or question generation @xcite . in particular, @xcite also condition the decoder of a nmt model on a target encoder. as opposed to our work, the output of their encoder is continuous and sampling diverse generation requires to inject random noise, while we obtain diversity by switching between discrete domains. similar noise injection mechanisms have been investigated to improve the diversity of responses in dialogue @xcite @xcite , and image captioning @xcite . closer to our work, @xcite and @xcite use domain embeddings to condition their generations. unlike us, they do not condition the domain on the target, but select the domain which minimizes the reconstruction loss, which becomes expensive as the number of domains increases. another relevant work is the fast decoder of @xcite where they also combine a discrete encoder applied on the target sentence with the nmt encoder. their goal is to accelerate the decoding process of a machine translation system, while we are interested in efficiently sampling diverse translations. another line of work focuses on improving the generation by changing the decoding scheme during inference @xcite or by matching the training of the model to the decoding scheme @xcite . this is done by either training through a beam search decoder @xcite or by reframing generation as a reinforcement learning problem @xcite . these works focus on the decoding scheme to improve generation, but do not address the problem of diversifying the outputs generated from the same input. ## model in this section, we describe our target encoder and how to train it along with a translation model. the target encoder learns to map target sentences to discrete domains, and we show how to use these domains to efficiently sample diverse translations. ## experiments in this section, we describe an evaluation protocol similar to @xcite , and compare our approach to several baselines on 3 mt datasets. then, we show the importance of different components in our model in an ablation study. to obtain e tgt (y), we linearly map z y to a k dimensional vector and perform a "soft-disctretization" by applying either a softmax or an argmax operator. we then compute the target domain vector e as the sum of the domain embeddings e reweighted by their probabilities contained in e tgt (y). the vector e is fed to the decoder as the embedding of the first token, along with the source encoding z @xmath0. ## conclusion in this paper, we presented an efficient way to sample diverse translations by adding a discrete target encoder to a nmt model. the discrete representation allows to change the domain of the translation and can be trained without supervision. the advantages of using a discrete encoder is that it is both general and scales with the number of domains with no additional computational time. in the future, we plan to test our discrete target encoder to diversify generations in other domains, such as language modeling, image captioning or image inpainting.