Unifying Discrete and Continuous Representations for Unsupervised Paraphrase Generation (2023)

Abstract: Unsupervised paraphrase generation is a challenging task that benefits a variety of downstream NLP applications. Current unsupervised methods for paraphrase generation typically employ round-trip translation or denoising, which require translation corpus and result in paraphrases overly similar to the original sentences in surface structure. Most of these methods lack explicit control over the similarity between the original and generated sentences, and the entities are also less correctly kept. To obviate the reliance on translation data and prompt greater variations in surface structure, we propose a self-supervised pseudo-data construction method that generates diverse pseudo-paraphrases in distinct surface structures for a given sentence. To control the similarity and generate accurate entities, we propose an unsupervised paraphrasing model that encodes the sentence meaning and the entities with discrete and continuous variables, respectively. The similarity can be controlled by sampling discrete variables and the entities are kept substantially accurate due to the specific modeling of entities using continuous variables. Experimental results on two benchmark datasets demonstrate the advantages of our pseudo-data construction method compared to round-trip translation, and the superiority of our paraphrasing model over the state-of-the-art unsupervised methods.

## introduction the task of paraphrase generation aims to reconstruct a sentence in a different surface structure (grammar and words) while preserving its underlying semantics. it plays an essential role in many downstream tasks like text summarization @xcite , machine translation @xcite , question answering @xcite ., * corresponding author. existing unsupervised paraphrase generation methods primarily employ round-trip translation @xcite @xcite or denoising @xcite @xcite , which leads to four flaws. firstly, round-trip translation requires extra translation corpus. secondly, the generated paraphrases are similar to the original sentences in surface structure, as faithful round-trip translation and denoising tend to maintain the syntax, sentence pattern, and tense of the original sentence. thirdly, most existing methods cannot explicitly control the similarity of the generated paraphrases to the original sentences. explicitly controlling the similarity allows for greater variations in surface structure and the generation of diverse paraphrases meets the requirements of different downstream tasks. lastly, the existing methods all ignore the importance of generating accurate entities, while entity errors usually lead to serious factual errors (e.g. paraphrasing "jobs created apple" to "gates created apple"). to produce paraphrases with great variation in surface structure without translation data, we propose a self-supervised pseudo-data construction method. also, to enable explicit control of similarity and accurate generation of entities, we propose a paraphrasing model which unifies discrete and continuous variables to encode sentences. 1 the pseudo-data constructor is trained on nonparallel sentences in a self-supervised manner, eliminating the need for translation data. from tom wants to buy a mercedes benz. ## related works unsupervised paraphrase generation methods often employ translation or denoising. @xcite and @xcite employ a straightforward round-trip translation approach. @xcite propose a hybrid decoder taking the hidden states of a set-to-sequence model @xcite and a round-trip translation model. @xcite and @xcite expand translation datasets into paraphrase datasets with lexically constrained decoding @xcite . principled paraphrasing introduces an adversarial reconstruction decoder to encourage the removal of structural information from the original sentence, followed by a translation task to ensure the retention of semantics @xcite . these methods rely on translation datasets and often produce paraphrases that resemble the original sentences. as for denoising, @xcite model paraphrase generation as an optimization problem and design a sophisticated objective function to enhance semantic preservation, expression diversity, and fluency of paraphrases, failing to explicitly control the similarity to generate diverse paraphrases. @xcite propose dynamic blocking to enforce a surface structure dissimilar from the input, which easily leads to key phrases being blocked. @xcite propose to learn paraphrasing models from unlabeled monolingual corpus with vq-vae @xcite . yet discrete variables are expressively less powerful than continuous variables @xcite , and it is difficult to encode large quantities of entities with a limited number of discrete variables in a codebook. we present a self-supervised pseudo-data constructor that generates paraphrase-like data for training the paraphrasing model, which eliminates the need for the translation corpus. and we propose a paraphrasing model to generate diverse paraphrases with controlled similarity through sampling discrete variables and generate accurate entities through specific modeling entities with continuous variables. bert-score self-bleu 0.55 0.58 ✘ 1.00 1.00 ✘ 0.87 0.43 ✔ he wants tom to buy a mercedes benz. tom, wants, buy, mercedes benz. tom wants to buy a mercedes benz. wants, tom, buy, mercedes benz. [per] tom [pro] mercedes benz a [pro] is what [per] wants to buy. keywords tom wants to buy a mercedes benz. original sentence original sentence pseudo-data constructor 1) train constructor 2) construct training data for paraphrasing model pivot sentence original entities [s] a [pro] is what [per] wants to buy. [sep] [per] tom [pro] mercedes benz. concatenated input tom wants to buy a mercedes benz. original sentence 3) train paraphrasing model tom, wants, buy, mercedes benz. mercedes benz, tom, wants, buy. shuffled keywords tom wants to buy a mercedes benz. a mercedes benz is what tom wants to buy. lee wants to buy a switch. a switch is what lee wants to buy. input 4) inference [s] [per] wants to buy a [pro]. [sep] [per] lee [pro] switch . generated paraphrase paraphrasing model paraphrasing model ## approach we describe the proposed approach in this section. besides, there are some efforts that show similarities to our work, we give detailed comparisons between them and our method in appendix a. ## conclusion in this paper, we propose a self-supervised pseudo-data construction method that generates paraphrases with more variation in surface structure than round-trip translation without the need for translation data. we propose a paraphrasing model that employs discrete and continuous variables to represent sentence meaning and entity information respectively for unsupervised paraphrase generation. our approach achieves stateof-the-art performance on two benchmark paraphrase datasets and far outperforms other methods in terms of accuracy in generating entities. in addition, we proposed an evaluation metric called entity-score for evaluating the precision and recall of the generated entities. entity-score correlates better with the human evaluation of entity than existing metrics. to the best of our knowledge, this is the first work concerned with ensuring the quality of the generated entities in the paraphrase generation task. the framework encoding text with discrete and continuous variables is promising on many nlp tasks, e.g. summarization and translation. future work will focus on extending our model to a more general seq2seq framework.