Syntax-Guided Controlled Generation of Paraphrases (2020)

Abstract: Given a sentence (e.g., “I like mangoes”) and a constraint (e.g., sentiment flip), the goal of controlled text generation is to produce a sentence that adapts the input sentence to meet the requirements of the constraint (e.g., “I hate mangoes”). Going beyond such simple constraints, recent work has started exploring the incorporation of complex syntactic-guidance as constraints in the task of controlled paraphrase generation. In these methods, syntactic-guidance is sourced from a separate exemplar sentence. However, this prior work has only utilized limited syntactic information available in the parse tree of the exemplar sentence. We address this limitation in the paper and propose Syntax Guided Controlled Paraphraser (SGCP), an end-to-end framework for syntactic paraphrase generation. We find that Sgcp can generate syntax-conforming sentences while not compromising on relevance. We perform extensive automated and human evaluations over multiple real-world English language datasets to demonstrate the efficacy of Sgcp over state-of-the-art baselines. To drive future research, we have made Sgcp’s source code available.1

## introduction controlled text generation is the task of producing a sequence of coherent words based on given constraints. these constraints can range from simple attributes like tense, sentiment polarity, and word-reordering @xcite @xcite to more complex syntactic information. for example, given a sentence ''the movie is awful!'' and a simple constraint like flip sentiment to positive, a controlled text generator is expected to produce the sentence ''the movie is fantastic!''. these constraints are important in not only providing information about what to say but also how to say it. without any constraint, the ubiquitous sequence-to-sequence neural models often tend to produce degenerate outputs and favor generic utterances @xcite . although simple attributes are helpful in addressing what to say, they provide very little information about how to say it. syntactic control over generation helps in filling this gap by providing that missing information. incorporating complex syntactic information has shown promising results in neural machine translation @xcite @xcite , data-to-text generation @xcite , abstractive textsummarization @xcite , and adversarial text generation @xcite . additionally, recent work @xcite has shown that augmenting lexical and syntactical variations in the training set can help in building better performing and more robust models. in this paper, we focus on the task of syntactically controlled paraphrase generation, that is, given an input sentence and a syntactic exemplar, produce a sentence that conforms to the syntax of the exemplar while retaining the meaning of the original input sentence. while syntactically controlled generation of paraphrases finds applications in multiple domains like dataaugmentation and text passivization, we highlight its importance in the particular task of text simplification. as pointed out in @xcite , depending on the literacy skill of an individual, certain syntactical forms of english because it is raining today, you should carry an umbrella. s2 you should carry an umbrella today, because it is raining. connectives that permit pre-posed adverbial clauses have been found to be difficult for third to fifth grade readers, even when the order of mention coincides with the causal (and temporal) order @xcite . hence, they prefer sentence s2. however, various other studies @xcite @xcite have suggested that for older school children, college students, and adults, comprehension is better for the cause-effect presentation, hence sentence s1. thus, modifying a sentence, syntactically, would help in better comprehension based on literacy skills. prior work in syntactically controlled paraphrase generation addressed this task by conditioning the semantic input on either the features learned from a linearized constituency-based parse tree @xcite , or the latent syntactic information @xcite learned from exemplars through variational auto-encoders. linearizing parse trees typically results in loss of essential dependency information. on the other hand, as noted in @xcite , an autoencoderbased approach might not offer rich enough syntactic information as guaranteed by actual constituency parse trees. moreover, as noted in @xcite , scpn @xcite , and cgen @xcite tend to generate sentences of the same length as the exemplar. this is an undesirable characteristic because it often results in producing sentences that end abruptly, thereby compromising on grammaticality and semantics. please see to address these gaps, we propose syntax guided controlled paraphraser (sgcp) which uses full exemplar syntactic tree information. additionally, our model provides an easy mechanism to incorporate different levels of syntactic control (granularity) based on the height of the tree being considered. the decoder in our framework is augmented with rich enough syntactical information to be able to produce source how do i predict the stock market ? exemplar can a brain transplant be done ? scpn how can the stock and start ? cgen can the stock market actually happen ? sgcp (ours) can i predict the stock market ? source what are some of the mobile apps you ca n't live without and why ? exemplar which is the best resume you have come across ? scpn what are the best ways to lose weight ? cgen which is the best mobile app you ca n't ? sgcp (ours) the main contributions of this work are as follows: paraphrase generation. while generation of paraphrases has been addressed in the past using traditional methods @xcite @xcite @xcite @xcite , they have recently been superseded by deep learning-based approaches @xcite @xcite @xcite . the primary task of all these methods @xcite @xcite is to generate the most semantically similar sentence and they typically rely on beam search to obtain any kind of lexical diversity. @xcite try to tackle the problem of achieving lexical, and limited syntactical diversity using submodular optimization but do not provide any syntactic control over the type of utterance that might be desired. these methods are therefore restrictive in terms of the syntactical diversity that they can offer. controlled paraphrase generation. our task is similar in spirit to @xcite and @xcite , which also deals with the task of syntactic paraphrase generation. however, the approach taken by them is different from ours in at least two aspects. firstly, scpn @xcite uses an attention-based @xcite pointer-generator network @xcite to encode input sentences and a linearized constituency tree to produce paraphrases. because of the linearization of syntactic tree, considerable dependency-based information is generally lost. our model, instead, directly encodes the tree structure to produce a paraphrase. secondly, the inference (or generation) process in scpn is computationally very expensive, because it involves a two-stage generation process. in the first stage, they generate full parse trees from incomplete templates, and then from full parse trees to final generations. in contrast, the inference in our method involves a single-stage process, wherein our model takes as input a semantic source, a syntactic tree and the level of syntactic style that needs to be transferred, to obtain the generations. additionally, we also observed that the model does not perform well in low resource settings. this, again, can be attributed to the compounding implicit noise in the training due to linearized trees and generation of full linearized trees before obtaining the final paraphrases. @xcite propose a syntactic exemplar-based method for controlled paraphrase generation using an approach based on latent variable probabilistic modeling, neural variational inference, and multi-task learning. this, in principle, is very similar to @xcite . as opposed to our model, which provides different levels of syntactic control of the exemplarbased generation, this approach is restrictive in terms of the flexibility it can offer. also, as noted in @xcite , an autoencoder-based approach might not offer rich enough syntactic information as offered by actual constituency parse trees. additionally, vaes (kingma and welling, 2014) are generally unstable and harder to train @xcite than seq2seq-based approaches. ## sgcp: proposed method in this section, we describe the inputs and various architectural components essential for building sgcp, an end-to-end trainable model. our model, as shown in figure 1 , comprises a sentence encoder (3.2), syntactic tree encoder (3.3), and a syntactic-paraphrase-decoder (3.4). ## experiments our experiments are geared towards answering the following questions: q1. is sgcp able to generate syntax conforming sentences without losing out on meaning? (section 5.1, 5.4) q2. what level of syntactic control does sgcp offer? (section 5.2, 5.3, 5.2) q3. how does sgcp compare against prior models, qualitatively? (section 5.4) ## conclusion in this paper we proposed sgcp, an end-toend framework for the task of syntactically controlled paraphrase generation. sgcp generates paraphrase of an input sentence while conforming to the syntax of an exemplar sentence provided along with the input. sgcp comprises a grubased sentence encoder, a modified rnn-based tree encoder, and a pointer-generator-based novel decoder. in contrast to previous work that focuses on a limited amount of syntactic control, our model can generate paraphrases at different levels of granularity of syntactic control without compromising on relevance. through extensive evaluations on real-world datasets, we demonstrate sgcp's efficacy over state-of-the-art baselines. we believe that the above approach can be useful for a variety of text generation tasks including syntactic exemplar-based abstractive summarization, text simplification and data-totext generation.