Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity (2020)

Abstract: Recent work has shown that a multilingual neural machine translation (NMT) model can be used to judge how well a sentence paraphrases another sentence in the same language (Thompson and Post, 2020); however, attempting to generate paraphrases from such a model using standard beam search produces trivial copies or near copies. We introduce a simple paraphrase generation algorithm which discourages the production of n-grams that are present in the input. Our approach enables paraphrase generation in many languages from a single multilingual NMT model. Furthermore, the amount of lexical diversity between the input and output can be controlled at generation time. We conduct a human evaluation to compare our method to a paraphraser trained on the large English synthetic paraphrase database ParaBank 2 (Hu et al., 2019c) and find that our method produces paraphrases that better preserve meaning and are more gramatical, for the same level of lexical diversity. Additional smaller human assessments demonstrate our approach also works in two non-English languages.

## introduction paraphrase generation is the task of producing a fluent output sentence which is semantically similar to the input sentence while being syntactically and/or lexically different from it @xcite . paraphrasing has been of longstanding interest in the nlp community @xcite and has been used for data augmentation in question answering @xcite , machine translation (mt) @xcite , task oriented dialog @xcite , and mt metrics @xcite @xcite . @xcite recently released the prism mt metric, which uses a multilingual neural mt (nmt) model as a paraphraser to score paraphrastic pairs; they treat paraphrasing as a zero-shot translation task (e.g., "translation" from english to english) and force-decode and score mt system outputs conditioned on their respective human translations. they denote their paraphraser as lexically/syntactically unbiased as it does not prefer output that differs lexically or syntactically from the input; this is advantageous for an mt metric as it assigns the highest score to an mt output which matches or nearly matches a human reference, but generating from the prism model using standard beam search produces trivial copies or near copies. we introduce a simple method to enable paraphrase generation from a multilingual nmt model. 1 our method discourages the model from producing n-grams that match n-grams in the input sentence. this serves to lexically bias the output away from the input sentence, resulting in nontrivial paraphrases. when considered together with prism model of @xcite , our paraphrase generation approach offers several potential advantages over the common technique of training a paraphrase model on synthetic paraphrases generated by translating one side of bitext into the language of the other side @xcite @xcite : we conduct human evaluations to compare our proposed method to a strong english baseline paraphraser trained on the parabank 2 dataset @xcite , which consists of 50 million synthetic examples generated by translating the czech side of czech-english bitext into english and pairing it with the original english. we find that our method outperforms this baseline-both in terms of semantic similarity and grammaticality-when our system is adjusted to match the lexical diversity of the baseline. we also present small scale evaluations that suggest our method is effective in other languages. ## related work paraphrase generation machine translation techniques can be used to train paraphrase models @xcite . another method to generate a paraphrase is to translate a text to a different language and then back again @xcite . multiple pivot languages can be used to lessen the effect of inherent ambiguities (aziz and specia, 2013), at the expense of complication. several works have focused on training on paraphrase data, including synthetic data created by starting with bitext and translating one side into the language of the other side to create synthetic paraphrases @xcite @xcite . ideas such as adversarial training @xcite , reinforcement learning @xcite , and variational autoencoders @xcite have also been explored in the context of paraphrase generation. ## method let x and y be sentences, let m(x) represent the meaning of x, and let s(x, y) measure the lexical and/or syntactic similarity between the two sentences. formally, we can state the problem of para-algorithm 1 before paraphrasing a sentence, buildpenalties() is called to construct a mapping of word prefixes to subwords that require penalties. then, penalize() is called to modify the model prediction targetlogprobs at every decoder timestep. def buildpenalties(source): @xmath0, 2, 3, 4]: for ngram of size n in subwords2words(source): prefix, @xmath1:-1], ngram[-1] for subword in targetvocab: if word.lower().startswith(subword.lower()): penalties[prefix].append(subword) return penalties def penalize(history, penalties, targetlogprobs): for n in [1, 2, 3, 4]: @xmath2:] for subword in penalties[prefix]: targetlogprobs[id(subword)] -= alpha * (n ** beta) phrase generation as finding ŷ: ## discussion we hypothesize that our method outperforms the baseline because it does not suffer from a fundamental shortcoming in creating synthetic paraphrase data from bitext: namely that inherent ambiguities present in one language (but not the other) can cause erroneous synthetic paraphrases in the other language (aziz and specia, 2013). for the sake of discussion we consider gender 8 as an ambiguity. suppose we create synthetic english paraphrases from turkish-english data, and our bitext contains the following (valid) sentence pair: ("o magazaya gitti.", "she went to the store.") turkish is a gender-neutral language, so when we translate the turkish side to english it is perfectly valid to translate the sentence to "he went to the store." pairing the original english translation with the translation results in the synthetic paraphrase example ("she went to the store.", "he went to the store."). since english is gendered, this results in an invalid synthetic paraphrase. in contrast, consider what happens if "she went to the store." is paraphrased by our method. first, the sentence is converted to an intermediate representation by the encoder. if the encoder were from an @xmath3, it is plausible that the encoder would discard gender information, as it is not needed in the target language. however, our encoder comes from a multilingual system which can produce output in many different languages. thus, as long as the model has seen a sufficient number of training examples between english and at least one other gendered language, we can reasonably expect that the intermediate representation will preserve gender. thus, when this representation is passed to the decoder and english is requested as the target language, the model should put low probability on any output for which the subject is male. an alternative way to address pivot language ambiguities is to use multiple pivot languages, as proposed by @xcite . however, it is not clear how best to extend this idea to neural sequence-to-sequence models, or to a multilingual paraphraser. combining synthetic paraphrases for training using several different pivot languages would mitigate the errors due to ambiguities from any one pivot language, at the expense of errors due to ambiguities in other pivot languages. to really address such errors would require combining models of different language pairs; see @xcite for one such solution. ## conclusions we treat paraphrasing as a zero-shot translation task and present a method to control the lexical 8 czech is, of course, gendered, so we would not expect the parabank 2 dataset (which was created from czech-english bitext) to have gender errors. but the logic presented here should generalize to other ambiguities. diversity of paraphrases generated from a multilingual nmt model, enabling paraphrase generation in many languages. our approach gives a user finegrained control over the amount of lexical diversity at generation time, and also allows models and generation algorithms to be developed and evaluated with less interdependencies. there are likely many other ways that the output could be controlled to vary other aspects, such as syntactic diversity @xcite ; we would like to explore such methods in future work. our work outperforms an english baseline trained on a large synthetic paraphrase dataset @xcite . this improvement in performance may be because our method does not suffer from the issue that ambiguities in the pivot language used to create synthetic paraphrase data can cause errors in synthetic data. small experiments indicate our method also performs well in other languages. multilingual nmt is an active research area and we are optimistic that this approach will pave the way for even stronger paraphrase generation in the future, as multilingual nmt methods continue to improve and models are publicly released.