P ara M ac: A General Unsupervised Paraphrase Generation Framework Leveraging Semantic Constraints and Diversifying Mechanisms (2022)

Abstract: Paraphrase generation reflects the ability to understand the meaning from the language surface form and rephrase it to other expressions. Recent paraphrase generation works have paid attention to unsupervised approaches based on Pre-trained Language Models (PLMs) to avoid heavy reliance on parallel data by utilizing PLMs’ generation ability. However, the generated pairs of existing unsupervised methods are usually weak either in semantic equivalence or expression diversity. In this paper, we present a novel unsupervised paraphrase generation framework called Paraphrase Machine. By employing multi-aspect equivalence constraints and multi-granularity diversifying mechanisms, Paraphrase Machine is able to achieve good semantic equivalence and expressive diversity, producing a high-quality unsupervised paraphrase dataset. Based on this dataset, we train a general paraphrase model, which can be directly applied to rewrite the input sentence of various domains without any fine-tuning, and achieves substantial gains of 9.1% and 3.3% absolutely in BLEU score over previous SOTA on Quora and MSCOCO. By further fine-tuning our model with domain-specific training sets, the improvement can be increased to even 18.0% and 4.6%. Most importantly, by applying it to language understanding and generation tasks under the low-resource setting, we demonstrate that our model can serve as a universal data augmentor to boost the few-shot performance (e.g., average 2.0% gain on GLUE).

## introduction paraphrases are sentences that convey the same meaning with different forms of expressions. automatic generation of paraphrases has been an essential task in natural language processing (nlp) ## paramac in this section, we will introduce our unsupervised paraphrase generation framework paramac. our idea starts from two basic assumptions: the first is the paraphrase must contain some key information of the original sentence, which we use keywords to represent, and they can be rendered in expressions; the second is there exists a different order for keywords to reform a fluency sentence, as long as proper connecting parts are filled between them. based on these two assumptions, our unsupervised paraphrase generation framework can be divided into three parts: 1) keywords processing: we extract the keywords of input, rephrase and reorder them; 2) plm-based generation: with the help of context regulation and the linguistic ability of plms, we connect the rephrased and reordered keywords into a fluency sentence as a paraphrase candidate; 3) candidates ranking: we use a series of metrics to select the best candidate. to ensure semantic consistency during the process and the expression diversity of the outputs, we design multi-aspect equivalence constraints and multi-granularity diversifying mechanisms. specifically, the equivalence constraints are: the diversifying mechanisms are embedded at three levels: figure 2 provides an overview of the whole generation picture and where exactly these constraints and mechanisms are applied. ## paranet and paramod with the above framework, we can now choose a proper corpus to generate our paraphrase dataset paranet. as the generation process needs the context of inputs, we choose the long-form bookcorpus @xcite as our generation input corpus. since t5 uses bookcorpus as one of its pretraining corpora and we don't want the plm of paramac to have seen the input sentences before, we use a newly crawled version foot_1 (sep. 2020 by shawn presser) excluding the original bookcorpus, which finally leaves us 3551 books. the genres of these books include fiction, nonfiction, essay, poetry, plays, and screenplays, ranging from up to 100 topics such as romance, science fiction, fantasy, thriller, and suspense. from this subset of bookcorpus, we randomly sample 10k examples. each input examples contains 1) a complete sentence s with a length between 60 and 100 characters; 2) the context before and behind s, both with an average length of 250 characters. given the 10k examples, we generate the paranet in an unsupervised way using para-mac. then, based on paranet, we are able to train a seq2seq language model paramod, which can generate paraphrased sentences given any sentence. the implementation details can be found in appendix a. ## experiments in this section, we evaluate our proposed paraphrase generation model paramod in both unsupervised and supervised settings. furthermore, we use our paramod as a data augmentation and generation tool to validate its effectiveness in downstream nlp tasks. ## ablation studies in this section, we study various factors that can affect the performance of our paramod. ## conclusions in this paper, we introduce a novel unsupervised paraphrase generation framework paramac, which utilizes plms' linguistic ability, combined with multi-aspect equivalence constraints and multigranularity diversifying mechanisms, to improve the generation quality in terms of semantic equivalence and expression diversity. moreover, we demonstrate the generality and value of our general paraphrase model in several downstream tasks.