Paraphrase Generation as Unsupervised Machine Translation (2022)

Abstract: In this paper, we propose a new paradigm for paraphrase generation by treating the task as unsupervised machine translation (UMT) based on the assumption that there must be pairs of sentences expressing the same meaning in a large-scale unlabeled monolingual corpus. The proposed paradigm first splits a large unlabeled corpus into multiple clusters, and trains multiple UMT models using pairs of these clusters. Then based on the paraphrase pairs produced by these UMT models, a unified surrogate model can be trained to serve as the final model to generate paraphrases, which can be directly used for test in the unsupervised setup, or be finetuned on labeled datasets in the supervised setup. The proposed method offers merits over machine-translation-based paraphrase generation methods, as it avoids reliance on bilingual sentence pairs. It also allows human intervene with the model so that more diverse paraphrases can be generated using different filtering criteria. Extensive experiments on existing paraphrase dataset for both the supervised and unsupervised setups demonstrate the effectiveness the proposed paradigm.

## introduction the goal of paraphrase generation @xcite @xcite is to generate a sentence semantically identical to a given input sentence but with variations in lexicon or syntax. it has been applied to various downstream nlp tasks such as parsing @xcite , question answering @xcite , summarization @xcite and machine translation @xcite . building a strong paraphrase generation system usually requires massive amounts of high-quality annotated paraphrase pairs, but existing labeled datasets @xcite @xcite are either of small sizes or restricted in narrow domains. to avoid such a heavy reliance on labeled datasets, recent works have explored unsupervised methods @xcite @xcite to generate paraphrase without annotated training data, among which the backtranslation based model is an archetype @xcite . it borrows the idea of back-translation (bt) in machine translation @xcite where the model first translates a sentence s 1 into another sentence s 2 in a different language (e.g., @xmath0, and then translates s 2 back to s 1 . in this way, the model is able to generate paraphrases by harnessing bilingual datasets, removing the need for label paraphrase data. however, bt-based models for paraphrase generation have the following severe issues: firstly, btbased systems rely on external resources, i.e., bilingual datasets, making them hard to be applied to languages whose bilingual datasets are hard to obtain. secondly, translation errors, such as duplicate words @xcite , missing words @xcite and polysemous words @xcite , will accumulate during the forward and backward translations, resulting in inferior performances. thirdly, machine translation models work like blackboxs, making it hard for humans to intervene with the model and control the generation process. in this work, we propose to address these problems based on the assumption that there must be pairs of sentences expressing the same meaning in a large-scale unlabeled corpus. inspired by unsupervised machine translation (umt) models, which align semantic spaces of two languages using monolingual data, we propose a pipeline system to generate paraphrases following two stages: (1) splitting a large-scale monolingual corpus into multiple clusters/sub-datasets, on which umt models are trained based on pairs of these sub-datasets; and the proposed framework provides the following merits over existing bt-based methods: (1) it is purely based on a large-scale monolingual corpus, which removes the reliance on bilingual datasets; we conduct extensive experiments on a wide range of paraphrase datasets to evaluate the effectiveness of the proposed framework, and we are able to observe performance boosts against strong baselines in both supervised and unsupervised setups. @xcite proposed to use different syntactic structure templates. a line of work @xcite formalized paraphrase generation as machine translation. unsupervised paraphrase generation is primarily based on reinforcement learning (rl) generative models @xcite . rl optimizes certain criteria, e.g. bleu, to reward paraphrases with higher quality @xcite . bowman et al. (2016); yang et al. (2019) trained a variational auto-encoder (vae) (kingma and welling, 2013) to generate paraphrases. other unsupervised methods for paraphrase generation include vae (vq-vae) @xcite , latent bagof-words alignment @xcite and simulated annealing @xcite . adapting large-scale pretraining @xcite @xcite @xcite to paraphrase generation has been recently investigated @xcite @xcite and has shown promising potentials to improve generation quality. our work is distantly related to unsupervised text style transfer @xcite @xcite @xcite , where the model alters a specific text attribute of an input sentence (such as sentiment) while preserving other text attributes. regarding soliciting large-scale paraphrase datasets, @xcite used statistical machine translation methods obtain paraphrases in parallel text, the technique of which is scaled up by @xcite to produce the paraphrase database (ppdb). @xcite translate the non-english side of parallel text to obtain paraphrase pairs. @xcite collected paraphrase dataset with million of pairs via machine translation. @xcite produced paraphrases from a bilingual corpus based on the techniques of negative constraints and inference sampling. unsupervised machine translation unsupervised machine translation(umt) has been an active research direction in nlp @xcite . pioneering work for unsupervised neural machine translation used denoising auto-encoders and back-translation @xcite to iteratively refine the generated translation. @xcite proposed to use a shared encoder to encode source input sentences from different languages. @xcite additionally used adversarial and cross-domain training objectives to better identify different language domains. @xcite relaxed the strategy of sharing the entire encoder in @xcite by building independent encoders to maintain unique characteristics of each language. another line of work for umt is to combine statistical machine translation (smt) and nmt. artetxe et al. (2018); lample et al. (2018) built a phrase-level mapping table from the source language to the target language. following works improved umt by combining smt and nmt in different ways, such as warming up an nmt model with a trained smt model @xcite and using smt as posterior regularization @xcite . other works involve initializing the model using retrieved semantically similar sentence pairs @xcite @xcite , using auxiliary par-allel data @xcite and pretraining on large-scale multi-lingual data @xcite @xcite . ## background for unsupervised machine translation we use the unsupervised machine translation (umt) framework proposed by @xcite as the backbone. we briefly go though the model structure in this section. let c src and c tgt respectively denote the monolingual dataset for the source and target language, on which a translation model m is trained to to generate target sequences y based on source sequences x, @xmath1. the model is first initialized by training in a word-byword translation manner using a parallel dictionary. the initial parallel dictionary is thus a word being translated to itself. next, the model is iteratively trained based on a denoising auto-encoding (dae), back-training (bt) and adversarial learning (al). dae allows the model to reconstruct the translation from a noisy input sentence by dropping and swapping words in the original sentence. the training objective of dae is given by: the encoder is trained to fool the discriminator so that the encoder and the discriminator perform together in an adversarial style @xcite : the final training objective is given by: ## model the core idea of the proposed strategy is to use two subdatasets from a large monolingual corpus c and train unsupervised nmt models based on the two subdatasets. the path towards this goal naturally constitutes two modules: (1) constructing two subdatasets c src and c tgt from c; and (2) training the umt model based on c src and c tgt . ## ablation study in this section, we perform comprehensive ablation studies on wikianswers dataset for understanding behaviors of the proposed model. and we report ibleu score for comparison. size of c for umt training first, we explore how the size of c, the commoncrawl corpus used for dataset split and umt training, affects downstream performances. the number of lda topics pairing c src and c tgt in our main experiments, we randomly select c tgt given c src . it would be interesting to see the effects of different cluster selection strategies. we consider four strategies: largest (select c tgt with the largest distance to c src ), medium (select c tgt with the medium distance to c src ), smallest (select c tgt with the smallest distance to c src ) and for referring purposes, input bt umt time is the most accurate test of how puberty is going to progress. time is the most accurate test of how puberty is going to progress. (en-fr) how adolescence develops is most accurately test by time. gm told truck owners about the defect in the first half of october. gm owners told truck about the defect in the first half of october . (en-fr) gm informed truck owners of the defect in the first half of october . to keep him alive , well , there 's no reason to keep him alive . to keep him alive , well , there 's no reason to keep him alive . (en-fr) to let him live , well , there 's no reason to let him live . washington did not have the formal education his elder brothers received at appleby grammar school in england, but he did learn mathematics, trigonometry, and land surveying . washington did not pursue the same formal education as his older brothers at england's appleby grammar school, but he did study geometry, trigonometry, and land surveying . unlike his older brothers, who studied at england's appleby grammar school, washington did not receive formal education but studied mathematics, trigonometry, and land surveying . ## conclusion in this paper, we propose a new framework for paraphrase generation by treating the task as unsupervised machine translation (umt). the proposed framework first splits a large unlabeled corpus into multiple sub-datasets and then trains one or multiple umt models based on one or more pairs of these sub-datasets. experiments and ablation studies under supervised and unsupervised setups demonstrate the effectiveness of the proposed framework.