Syntactically-Informed Unsupervised Paraphrasing with Non-Parallel Data (2021)

Abstract: Previous works on syntactically controlled paraphrase generation heavily rely on large-scale parallel paraphrase data that is not easily available for many languages and domains. In this paper, we take this research direction to the extreme and investigate whether it is possible to learn syntactically controlled paraphrase generation with nonparallel data. We propose a syntactically-informed unsupervised paraphrasing model based on conditional variational auto-encoder (VAE) which can generate texts in a specified syntactic structure. Particularly, we design a two-stage learning method to effectively train the model using non-parallel data. The conditional VAE is trained to reconstruct the input sentence according to the given input and its syntactic structure. Furthermore, to improve the syntactic controllability and semantic consistency of the pre-trained conditional VAE, we fine-tune it using syntax controlling and cycle reconstruction learning objectives, and employ Gumbel-Softmax to combine these new learning objectives. Experiment results demonstrate that the proposed model trained only on non-parallel data is capable of generating diverse paraphrases with specified syntactic structure. Additionally, we validate the effectiveness of our method for generating syntactically adversarial examples on the sentiment analysis task.

## introduction paraphrases are texts or passages conveying the same meaning but with different surface realization. paraphrase generation (pg) is a key technology of automatically generating a restatement for a given text, which has the potential use in many downstream tasks, such as question answering @xcite , machine translation @xcite ) and text summarization @xcite . recent years have witnessed that learning controllable paraphrase generation (cpg) with specified styles is attracting intense research interests, e.g., satisfying particular syntactic templates @xcite or exemplars @xcite . as cpg can produce diverse paraphrases by exposing syntactic control, it can be also employed for adversarial example generation @xcite . existing syntactically controlled paraphrase networks @xcite rely on large paraphrase parallel data for training. unfortunately, paraphrase parallel corpora are not easily available for many languages, and are expensive to build. conversely, non-parallel data is much easier to find, and many languages with limited parallel data still possess a huge amount of non-parallel data. in this paper, we propose a syntacticallyinformed unsupervised paraphrasing (sup) framework based on conditional variational auto-encoder (vae) to generate syntactic paraphrases with specified syntactic skeletons, which does not require any parallel paraphrase data. the basic assumption behind sup is that, given a sentence, there may exist many valid paraphrases with different syntactic structures. specifically, as shown in figure 1 , sup runs in two stages. at stage 1, we train a conditional vae to reconstruct a given input sentence according to the sentence itself and its syntactic parse tree. the model trained at this stage is endowed with basic ability to generate texts of desired syntax structures (similar to a warmup procedure). at stage 2, to improve the syntactic controllability and semantic consistency of generated sentences, we fine-tune the model trained at stage 1 using carefully-designed objective functions involving syntax controlling and cycle reconstruction. after the conditional vae model is fine-tuned, given an input sentence and a different syntactic structure, the model can generate a paraphrase according to the given structure. we evaluate sup on both syntactic paraphrase generation and adversarial example generation tasks. experiments show that sup outperforms previous unsupervised paraphrasing method sivae @xcite . it is also capable of generating syntactically adversarial examples that have a significant impact on the performance of attacked neural models. we further show that augmenting training data with such examples can improve the robustness of target neural models. in summary, the major contributions of this paper are as follows: ## approach we use the constituency parse tree to provide syntactic information. given a set of training instances @xmath0, s i )} |d| @xmath1, where s i is the syntactic parse tree of the sentence x i , we aim to train a syntactic paraphrasing model which can produce more diverse paraphrases given arbitrary syntax. however, using a full parse tree (the whole tree without leaf nodes) is too specific and poses the challenge of selecting such a tree for a given input as syntactic structures of two different sentences are not easily compatible to each other. therefore, we mainly use a general template (the top 3 layers of a parse tree), as shown in figure 2 , as the controlling signal which is beneficial to generate meaningful paraphrases. specifically, we employ the conditional variational autoencoder (vae) framework which have proven to be able to generate diverse texts conditioned on certain attributes. in this work, we view syntax as the conditional attribute. the training process consists of two stages. in the first stage, we train the model in an auto-encoding manner, while in the second stage, we use new objective functions to fine-tune it, as shown in figure 1 . the two stages will be described in detail below. ## experiments in this section, we will answer the following questions: ## conclusions we have presented an unsupervised syntacticallyinformed paraphrasing model based on conditional this film seems thirsty for taking on adolescent qualities. ( s ( cc ) ( np ) ( vp ) ) but this film seems thirsty for taking on adolescent qualities. vae and two-stage training process. we first train the conditional vae model to generate sentences in desired syntactic structures. to further improve the syntactic controllability and semantic consistency of generated sentences, we introduce syntax controlling and cycle reconstruction objective functions to fine-tune the pre-trained model. experiments show that our model achieves strong improvements over baselines on unsupervised setting and can generate syntactically controlled paraphrases. furthermore, adversarial example generation experiments also validate that our model is able to generate syntactically adversarial examples for sentiment analysis, which can be used to improve the robustness of the sentiment classifier model via adversarial training.