Principled Paraphrase Generation with Parallel Corpora (2022)

Abstract: Round-trip Machine Translation (MT) is a popular choice for paraphrase generation, which leverages readily available parallel corpora for supervision. In this paper, we formalize the implicit similarity function induced by this approach, and show that it is susceptible to non-paraphrase pairs sharing a single ambiguous translation. Based on these insights, we design an alternative similarity metric that mitigates this issue by requiring the entire translation distribution to match, and implement a relaxation of it through the Information Bottleneck method. Our approach incorporates an adversarial term into MT training in order to learn representations that encode as much information about the reference translation as possible, while keeping as little information about the input as possible. Paraphrases can be generated by decoding back to the source from this representation, without having to generate pivot translations. In addition to being more principled and efficient than round-trip MT, our approach offers an adjustable parameter to control the fidelity-diversity trade-off, and obtains better results in our experiments.

## introduction paraphrase generation aims to generate alternative surface forms expressing the same semantic content as the original text @xcite , with applications in language understanding and data augmentation @xcite . one popular approach is to use an mt system to translate the input text into a pivot language and back @xcite @xcite . while it intuitively makes sense that translating to another language and back should keep the meaning of a sentence intact while changing its surface form, it is not clear what exactly would be considered a paraphrase by such a system. in this work, we show that the probability of a paraphrase x p given a source sentence x s under a given the input x s , we aim to learn a representation t (x s ) that encodes as much information as possible about it's reference translation y (ensuring that the meaning is preserved), and as little information as possible about x s itself (ensuring that surface information is removed). we achieve this through adversarial learning, where the encoder minimizes λl m t -(1 -λ)l adv and the two decoders minimize l m t and l adv . at inference time, we couple the english encoder and decoder to generate a paraphrase x p which, being conditioned on t (x), will preserve the meaning of x s but use a different surface form. so as to address this issue, we design an alternative similarity function that requires the entire translation distribution to match, and develop a relaxation of it through the information bottleneck (ib) method. we implement this approach using an adversarial learning system depicted in figure 1 . he will go to school él irá a la escuela x s y j y i he will go to school x s our model combines an encoder that, for a given sentence, removes the information that is not relevant to predict its translation, and a decoder that reconstructs a paraphrase from this encoding. in addition to being more principled, our approach is more efficient than round-trip mt at inference, can be tuned to favor fidelity or diversity, and achieves a better trade-off between the two. our code is freely available foot_0 . ## related work we next review the paraphrase generation literature ( §2.1), and describe the information bottleneck method ( §2.2), which is the basis of our proposal. ## characterizing round-trip mt let x be a random variable representing a sequence in the source language, and y be a random variable representing its translation into a pivot language. 3 given an input sequence x @xmath0, we can use round-trip mt to generate a paraphrase x @xmath1, according to the forward and backward translation models p (y|x s ) and p (x p |y). as such, we can formulate the probability of round-trip mt generating a particular paraphrase x p by marginalizing over the set of possible pivot translations: in what follows, we will characterize the paraphrases produced by this approach, i.e. the properties that x p needs to meet in relation to x s for p (x p |x s ) to be high. 4 by applying bayes' rule, we can rewrite eq. 1 as follows: the sum on the right hand side can be interpreted as a symmetric similarity function, s m t (x p , x s ) = s m t (x s , x p ) = y p (y|xs)p (y|xp) p (y) but when are x s and x p considered similar according to the above definition of s m t (x s , x p )? intuitively, s m t is a measure of the overlap between the conditional distributions that x s and x p induce over y . this will be highest when p (y|x s )p (y|x p ) is as large as possible for as many y as possible. at the same time, p (y|x s )p (y|x p ) will be high when both p (y|x s ) and p (y|x p ) are high, that is, when y is a probable translation of both x s and x p . this captures the intuition that two sentences are similar when they can be translated into the same text in the pivot language. but what if x s and x p have one particular highprobability translation y j in common, but differ in the rest? as illustrated in figure 2 , this can happen when y j is ambiguous in the target language and can mean both x s and x p , even if x s and x p are not equivalent (e.g., when x s uses the masculine form, x p the feminine form, and y j does not mark the gender). in this case, the sum y p (y|xs)p (y|xp) p (y) will be dominated by p (y j |xs)p (y j |xp) p (y j ) we can thus conclude that the implicit similarity function underlying round-trip mt is flawed, as it assigns a high score to a pair of sequences (x s , x p ) that have an ambiguous translation in common. as a consequence, round-trip mt will generate x p as a paraphrase of x s with a high probability, even if the two sequences have a different meaning. ## principled paraphrasing as shown in the previous section, the implicit similarity function induced by round-trip mt is not adequate in the general case, as it assigns a high score to pairs of sequences that share a single translation, despite differing in the rest. so as to address this, we can define an alternative similarity function that requires the entire translation distribution to match: s(x p , x s ) = 1 p (y|x p ) = p (y|x s )∀@xmath2. 2 so that p (x p |x s ) ∝ p (x p )s(x p , x s ). however, this definition is too strict, as it is virtually impossible that p (y|x p ) and p (y|x s ) are exactly the same for all @xmath3. 5 in 4.1, we define a relaxation of it through the ib method, which introduces an adjustable parameter β to control how much we deviate from it. in 4.2, we characterize the paraphrases generated by this approach, showing that they are less susceptible to the problem of confounding translations described in the previous section. ## proposed system in this section, we describe a practical implementation of the ib-based paraphrasing approach defined theoretically in §4. as illustrated in figure 1 , our system can be seen as an extension of a regular encoder-decoder mt architecture with an additional adversarial decoder, which is trained with an auto-encoding objective to reconstruct the original sentence x s from the encoder representation t (x s ). the encoder is trained to minimize the cross-entropy loss of the mt decoder, while maximizing the loss of the adversarial decoder. this way, the encoder is encouraged to remove as much information about x s as possible, while retaining the information that is necessary to predict its reference translation y. thanks to this, t (x s ) should capture the semantic content of x s (which is relevant to predict y), without storing additional surface information (which is not relevant to predict y). once the model is trained, the adversarial decoder can be used to generate paraphrases of x s from this representation t (x s ). this adversarial architecture can be interpreted as an implementation of the ib method as follows. following @xcite , we start by adding a decoder q(y|t) on top of the encoder t (x), and rewrite the i(t, y ) term as: in practice, we parametrize both the encoder t and the decoder q with transformer neural networks, and learn them from a parallel corpus. since log q(y|t (x)) is a lower bound of i(t, y ) -h(y ), maximizing this term is theoretically sound. mini- during training, the expectation in eq. 11 is approximated by sampling batches from the training data. care must be taken when optimizing the loss, as we do not want to propagate gradients of the adversarial loss to the adversarial decoder. if we did, a trivial way to minimize (1 -λ) log q(x|t (x)) would be to make the decoder bad at recovering x, which would not encourage t (x) to encode as little information as possible. to prevent this, we use a percentage k of the batches to learn the adversarial decoder log q(x|t (x)), where the encoder is kept frozen. the rest of the batches are used to optimize the full term -λ log q(y|t (x)) + (1 -λ) log q(x|t (x)), but the gradients for the second term are only propagated to the encoder. ## experimental design we experiment with the following systems: we use mbart @xcite to initialize both our proposed system and round-trip mt, and train them using the same hyper-parameters as in the original work. 8 in both cases, we use the english-french wmt14 dataset @xcite as our parallel corpus for training. 9 we report results for two decoding strategies: beam search with a beam size of 5, and top-10 sampling with a temperature of 0.9 (optimized in the development set). 10 for development, we extracted 156 paraphrase pairs from the sts benchmark dataset @xcite , taking sentence pairs with a similarity score above 4.5. for our final evaluation, we used the multiple translations chinese (mtc) corpus @xcite , which comprises three sources of chinese journalistic text translated into english by multiple translation agencies. we extract the translations of the first two agencies to obtain an test set of 993 paraphrase pairs, where one is the source and the other the reference paraphrase. the third sentence if kept as an additional paraphrase for estimating human performance. ## results we next report our main results ( §7.1), followed by a qualitative analysis ( §7.2). ## conclusions in this work, we have shown that the implicit similarity function present in round-trip mt is not appropriate in the general case, as it considers sentence pairs that share a single ambiguous translation to be paraphrases. we address this issue by designing an alternative similarity function that requires the entire translation distribution to match, and develop a relaxation of it through the ib method, which we prove to be less susceptible to the problem of confounding translations. we implement this approach through adversarial learning, training an encoder to preserve as much information as possible about the reference translation, while encoding as little as possible about the source. not only is our approach more principled than round-trip mt, but it is also more efficient at inference, as it does not need to generate an intermediate translation. in addition, it offers a knob to adjust the fidelity-diversity trade-off through the λ parameter, and obtains strong results in our experiments, outperforming round-trip mt.