AESOP : Paraphrase Generation with Adaptive Syntactic Control (2021)

Abstract: We propose to control paraphrase generation through carefully chosen target syntactic structures to generate more proper and higher quality paraphrases. Our model, AESOP, leverages a pretrained language model and adds deliberately chosen syntactical control via a retrieval-based selection module to generate fluent paraphrases. Experiments show that AESOP achieves state-of-the-art performances on semantic preservation and syntactic conformation on two benchmark datasets with ground-truth syntactic control from human-annotated exemplars. Moreover, with the retrieval-based target syntax selection module, AESOP generates paraphrases with even better qualities than the current best model using human-annotated target syntactic parses according to human evaluation. We further demonstrate the effectiveness of AESOP to improve classification models’ robustness to syntactic perturbation by data augmentation on two GLUE tasks.

## introduction syntactically-controlled paraphrase generation, which aims to generate paraphrases that conform with given syntactic structures, has drawn increasing attention in the community. on the one hand, paraphrase generation has benefited a wide range of nlp applications, such as neural machine translation @xcite , dialogue generation @xcite , as well as improving model robustness @xcite and interpretability @xcite . on the other hand, syntacticallycontrolled paraphrasing has been used for diverse question generation @xcite , diversifying creative generation @xcite and improving model robustness @xcite . however, selecting suitable target syntactic structures to control paraphrase generation for diverse and high-quality results is a lesser explored direction. prior works usually use a fixed set of syntactic aesop what to say? how to guide paraphrasing? structures for all input sentences @xcite . a challenge with this method is that not all sentences can be paraphrased into the same set of syntactic structures. for example, it is impossible to turn a long sentence with multiple clauses into a noun phrase. thus, @xcite proposed to use crowd-sourcing to collect exemplars that can provide compatible syntax with the source sentence to guide generation. disadvantages with this method are that the crowdsourcing process is costly, and one exemplar sentence can only provide a specific syntactic guidance, while there are many syntactic parses that can properly guide the paraphrase generation (as shown in figure 1 ). in contrast, we propose to automatically select multiple syntactic parse structures to control paraphrase generation for more diverse and higher quality generation. our first contribution is the proposal of aesop (adaptive syntactically-controlled paraphrasing), a model that integrates pretrained language models (lms) with a novel retrieval-based target syntactic parse selection module to control paraphrase generation. by leveraging the expressiveness of pretrained lms and the adaptive selection module, aesop is capable of generating fluent and syntactically-diverse paraphrases. with ground-truth target syntactic parses from human-annotated exemplars, aesop achieves the state-of-the-art performance on both semantic preservation and syntactic conformation metrics. by human evaluation, we show that ae-sop can generate paraphrases with even better quality than the current best model using human annotated exemplars, which points out the importance of studying the adaptive target parse selection for future works on controlled paraphrase generation. our second contribution is the construction of two datasets containing adversarial examples with syntactic perturbation generated by aesop that are further validated and labeled by crowd workers. experiments show that the two datasets are challenging to current classification models, and using aesop to augment the training data can effectively improve classification models' robustness to syntactic attacks. foot_0 ## task formulation we formulate the task of adaptive syntacticallycontrolled paraphrase generation as: given an input sentence x, find a set of proper syntactic controls y to generate paraphrases z, such that z's syntax conforms to y while retaining the semantics of x. we use the term target syntactic parses to refer to the syntactic structure that guides the generation, which could be from exemplar sentences, a set of fixed templates, or our adaptive selection module. ## paraphrase generation with syntactic control we train and evaluate aesop on paranmtsmall @xcite and qqp-pos @xcite . our train/dev/test split follows previous work @xcite . during our experiments, we aim to answer three research questions: baselines. for supervised models that utilize exemplar sentences to get target parses, we compare with cgen @xcite and two versions of sgcp @xcite : sgcp-r and sgcp-f. sgcp prunes constituency parse trees of exemplar sentences from height 3 up to 10. during the evaluation, sgcp-r chooses the best paraphrase out of many, and sgcp-f uses the full parse tree. to the best of our knowledge, sgcp-r is the current state-of-the-art model under this setting. for models that utilize a fixed set of target syntactic parses, we compare with scpn @xcite that proposes 10 syntactic parses at height 2 to guide the generation. ## model analysis and interpretation ablation studies. we take out each part of sequence in both encoder and decoder and conduct ## improve robustness recent works show that powerful lms (e.g., bert @xcite ) are capturing the superficial lexical features @xcite and are vulnerable to simple perturbations @xcite . motivated by this, we first test if bert is robust to syntactic perturbations by paraphrasing. we fine-tune bert models on two glue @xcite tasks (sst-2 and rte). then, we generate 10 paraphrases using aesop-h2 for each test instance in the dev set and choose top-5 to get 2 larger dev sets. 9 we run trained bert models on new dev sets again. human annotation. we collect the paraphrases where models fail but succeeded at their original 9 as we do not have the label for test sets on glue. augmentation. we augment each training instance with 5 best paraphrases from aesop-h2. for synpg and scpn, as the pre-designed templates for synpg is a subset of scpn's. we generate 5 paraphrases using selected templates in synpg. then, we retrain bert models with augmented training data from each model. then, we retrain bert models after augmentation and get their test accuracies. we define paragap as the accuracy difference for after-and before-augmentation using paraphrase generation models. paragap indicates how efficient the augmentation is to improve the model robustness to syntactic perturbations. experiment result. as shown in recent advances have been using neural models for syntactically controlled paraphrase generation. from the modeling perspective, there are roughly two categories: unsupervised and supervised methods. unsupervised models do not use parallel paraphrases during training. wieting and gimpel (2018); wieting et al. (2017) use back-translation to generate paraphrases. huang and chang (2021) propose a transformer-based model synpg for paraphrase generation. aesop is a supervised paraphrase generation model, which means that we require parallel paraphrases during training. previous supervised paraphrase models are mostly rnnbased models, including scpn @xcite , cgen @xcite and sgcp @xcite . such models suffer from generating long sentences and do not utilize the power of recent pretrained language models. goyal and durrett (2020a) is a concurrent work with ours that also builds on bart to generate paraphrases but has a different model design. for syntactic control, goyal and durrett (2020b) use target syntactic parses to reorder source sentences to guide the generation, while other works, including aesop, directly use target syntactic parses to guide the generation. cgen @xcite and sgcp @xcite use target syntactic parses from crowd-sourced exemplars, scpn @xcite and synpg @xcite use pre-designed templates, while aesop retrieves target syntactic parses automatically. ## conclusion and future works in this work, we propose aesop for paraphrase generation with adaptive syntactic control. one interesting and surprising finding of this paper is that using automatically retrieved parses to control paraphrase generation can result in better qualities than the current best model using human-annotated exemplars. such finding manifests the benefits of adaptive target parse selection for controlled paraphrase generation -it does not only generate diverse paraphrases, but also higher quality paraphrases. this suggests future works on syntactically controlled paraphrase generation to pay more attention to target parse selection, and we hope ae-sop can serve as a strong baseline for this direction. in our work, we use generated paraphrases to reflect the quality of automatically-selected target parses; future works can design specific metrics to evaluate the quality of retrieved syntactic parses. in addition, we find that having the control signal in the decoder can lead to better controllability of aesop. future works can test the generalizability of this modeling strategy in other controlled generation tasks. in addition, we show that aesop can effectively attack classification models and contribute two datasets to test models' robustness to syntactic perturbation. we find that using aesop to augment training data can effectively improve classification models' robustness to syntactic perturbations.