Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs (2021)

Abstract: Paraphrase generation plays an essential role in natural language process (NLP), and it has many downstream applications. However, training supervised paraphrase models requires many annotated paraphrase pairs, which are usually costly to obtain. On the other hand, the paraphrases generated by existing unsupervised approaches are usually syntactically similar to the source sentences and are limited in diversity. In this paper, we demonstrate that it is possible to generate syntactically various paraphrases without the need for annotated paraphrase pairs. We propose Syntactically controlled Paraphrase Generator (SynPG), an encoder-decoder based model that learns to disentangle the semantics and the syntax of a sentence from a collection of unannotated texts. The disentanglement enables SynPG to control the syntax of output paraphrases by manipulating the embedding in the syntactic space. Extensive experiments using automatic metrics and human evaluation show that SynPG performs better syntactic control than unsupervised baselines, while the quality of the generated paraphrases is competitive. We also demonstrate that the performance of SynPG is competitive or even better than supervised models when the unannotated data is large. Finally, we show that the syntactically controlled paraphrases generated by SynPG can be utilized for data augmentation to improve the robustness of NLP models.

## introduction paraphrase generation @xcite ) is a longlasting task in natural language processing (nlp) and has been greatly improved by recently developed machine learning approaches and large data collections. paraphrase generation demonstrates the potential of machines in semantic abstraction and sentence reorganization and has already been applied to many nlp downstream applications, given a source sentence and a target syntactic specification (either a full parse tree or top levels of a parse tree), the model is expected to generate a paraphrase with the syntax following the given specification. in recent years, various approaches have been proposed to train sequence-to-sequence (seq2seq) models on a large number of annotated paraphrase pairs @xcite @xcite . some of them control the syntax of output sentences to improve the diversity of paraphrase generation @xcite @xcite . however, collecting annotated pairs is expensive and induces challenges for some languages and domains. on the contrary, unsupervised approaches build paraphrase models without using parallel corpora @xcite @xcite . most of them are based on the variational autoencoder @xcite or back-translation @xcite @xcite . nevertheless, without the consideration of controlling syntax, their generated paraphrases are often similar to the source sentences and are not diverse in syntax. this paper presents a pioneering study on syntactically controlled paraphrase generation based on disentangling semantics and syntax. we aim to disentangle one sentence into two parts: 1) the semantic part and 2) the syntactic part. the semantic aspect focuses on the meaning of the sentence, while the syntactic part represents the grammatical structure. when two sentences are paraphrased, their semantic aspects are supposed to be similar, while their syntactic parts should be different. to generate a syntactically different paraphrase of one sentence, we can keep its semantic part unchanged and modify its syntactic part. based on this idea, we propose syntactically controlled paraphrase generator (synpg) foot_0 , a transformer-based model @xcite that can generate syntactically different paraphrases of one source sentence based on some target syntactic parses. synpg consists of a semantic encoder, a syntactic encoder, and a decoder. the semantic encoder considers the source sentence as a bag of words without ordering and learns a contextualized embedding containing only the semantic information. the syntactic encoder embeds the target parse into a contextualized embedding including only the syntactic information. then, the decoder combines the two representations and generates a paraphrase sentence. the design of disentangling semantics and syntax enables synpg to learn the association between words and parses and be trained by reconstructing the source sentence given its unordered words and its parse. therefore, we do not require any annotated paraphrase pairs but only unannotated texts to train synpg. we verify synpg on four paraphrase datasets: paranmt-50m @xcite , quora @xcite , pan @xcite , and mrpc @xcite . the experimental results reveal that when being provided with the syntactic structures of the target sentences, synpg can generate paraphrases with the syntax more similar to the ground truth than the unsupervised baselines. the human evaluation results indicate that synpg achieves competitive paraphrase quality to other baselines while its generated paraphrases are more accurate in following the syntactic specifications. in addition, we show that when the training data is large enough, the performance of synpg is competitive or even better than supervised approaches. finally, we demonstrate that the syntactically controlled paraphrases generated by synpg can be used for data augmentation to defense syntactically adversarial attack @xcite and improve the robustness of nlp models. ## unsupervised paraphrase generation we aim to train a paraphrase model without using annotated paraphrase pairs. given a source sentence @xmath0, x 2 , ..., x n ), our goal is to generate a paraphrase sentence @xmath1, y 2 , ..., y m ) that is expected to maintain the same meaning of x but has a different syntactic structure from x. syntactic control. motivated by previous work @xcite @xcite , we allow our model to access additional syntactic specifications as the control signals to guide the paraphrase generation. more specifically, in addition to the source sentence x, we give the model a target constituency parse p as another input. given the input (x, p), the model is expected to generate a paraphrase y that is semantically similar to the source sentence x and syntactically follows the target parse p. in the following discussions, we assume the target parse p to be a full constituency parse tree. later on, in section 2.3, we will relax the syntax guidance to be a template, which is defined as the top two levels of a full parse tree. we expect that a successful model can control the syntax of output sentences and generate syntactically different paraphrases based on different target parses, as illustrated in figure 1 . similar to previous work @xcite , we linearize the constituency parse tree to a sequence. for example, the linearized parse of the sentence "he eats apples." is (s(np(prp))(vp(vbz)(np(nns)))(.)). accordingly, a parse tree can be considered as a sentence @xmath2, p 2 , ..., p k ), where the tokens in p are non-terminal symbols and parentheses. ## experimental settings we conduct extensive experiments to demonstrate that synpg performs better syntactic control than other unsupervised paraphrase models, while the quality of generated paraphrases by synpg is comparable to others. in addition, we show that the performance of synpg is competitive or even better than supervised models when the training data is large enough. ## improving robustness of models recently, a lot of work show that nlp models can be fooled by different types of adversarial attacks @xcite @xcite @xcite . those attacks generate adversarial examples by slightly modifying the original sentences without changing the meanings, while the nlp models change the predictions on those examples. however, a robust model is expected to output the same labels. therefore, how to make nlp models not affected by the adversarial examples becomes an important task. since synpg is able to generate syntactically different paraphrases, we can improve the robustness of nlp models by data augmentation. the models trained with data augmentation are thus more robust to the syntactically adversarial examples @xcite , which are the adversarial sentences that are paraphrases to the original sentences but with syntactic difference. we conduct experiments on three classification tasks covered by glue benchmark @xcite : sst-2, mrpc, and rte. for each training example, we use synpg to generate four syntactically different paraphrases and add them to the training set. we consider the setting to generate syntactically adversarial examples by scpn @xcite . for each testing example, we generate five candidates of adversarial examples. if the classifier gives at least one wrong prediction on the candidates, we treat the attack to be successful. we compare the model without data augmentation (base) and with data augmentation (synpg) in recently, neural models make success on paraphrase generation @xcite @xcite @xcite . these approaches treat paraphrase generation as a translation task and design seq2seq models based on a large amount of parallel data. to reduce the effort to collect parallel data, unsupervised paraphrase generation has attracted attention in recent years. @xcite ; @xcite use translation models to generate paraphrases via back-translation. @xcite ; @xcite generate paraphrases based on variational autoencoders. reinforcement learning techniques are also considered for paraphrase generation @xcite . ## conclusion we present syntactically controlled paraphrase generator (synpg), an paraphrase model that can control the syntax of generated paraphrases based on the given syntactic specifications. synpg is designed to disentangle the semantics and the syntax of sentences. the disentanglement enables synpg to be trained without the need for annotated paraphrase pairs. extensive experiments show that synpg performs better syntactic control than unsupervised baselines, while the quality of the generated paraphrases is competitive to supervised approaches. finally, we demonstrate that synpg can improve the robustness of nlp models by generating additional training examples. synpg is especially helpful for the domain where annotated paraphrases are hard to obtain but a large amount of unannotated text is available. one limitation of synpg is the need for mannually providing target syntactic templates at inference time. we leave the automatic template generation as our future work.