Learning Structural Information for Syntax-Controlled Paraphrase Generation (2022)

Abstract: Syntax-controlled paraphrase generation aims to produce paraphrase conform to given syntactic patterns. To address this task, recent works have started to use parse trees (or syntactic templates) to guide generation.A constituency parse tree contains abundant structural information, such as parent-child relation, sibling relation, and the alignment relation between words and nodes. Previous works have only utilized parent-child and alignment relations, which may affect the generation quality. To address this limitation, we propose a Structural Information-augmented Syntax-Controlled Paraphrasing (SI-SCP) model. Particularly, we design a syntax encoder based on tree-transformer to capture parent-child and sibling relations. To model the alignment relation between words and nodes, we propose an attention regularization objective, which makes the decoder accurately select corresponding syntax nodes to guide the generation of words. Experiments show that SI-SCP achieves state-of-the-art performances in terms of semantic and syntactic quality on two popular benchmark datasets. Additionally, we propose a Syntactic Template Retriever (STR) to retrieve compatible syntactic structures. We validate that STR is capable of retrieving compatible syntactic structures. We further demonstrate the effectiveness of SI-SCP to generate diverse paraphrases with retrieved syntactic structures.

## introduction paraphrases are texts or passages conveying the same meaning but with different surface realization. paraphrase generation (pg) is a key technology of automatically generating a restatement for a given text, which has the potential use in many downstream tasks, such as question answering (gan and ng, 2019), machine translation @xcite , and sentence simplification (zhao et al., 2018). however, a natural sentence can be paraphrased into various surface forms. to obtain diverse paraphrases, controllable paraphrase generation (cpg) with specified styles has recently attracted growing interests e.g., satisfying particular sentiment @xcite @xcite or syntactic structure @xcite @xcite @xcite . as cpg can produce diverse paraphrases by exposing syntactic control, it can be used in a wide range of application scenarios, such as dialogue generation @xcite , data augmentation @xcite @xcite , and diverse question generation @xcite , etc. generally, syntax-controlled paraphrase generation needs to tackle two major challenges. the first challenge is how syntax-controlled generating is achieved? for this challenge, @xcite use two encoders to encode input sentences and linearized parse trees to produce paraphrases. a constituency parse tree contains abundant structural information, such as parent-child relation, sibling relation, and the alignment relation between words and nodes, as shown in figure 1 . linearizing parse trees, typically, result in loss of structural informa-tion. @xcite encode parse trees in a top-down manner, and then a queue-based decoding mechanism is proposed to incorporate syntax information. @xcite employ a path attention mechanism to capture the tree structure of the syntax. however, these methods still have some limitations. the top-down encoding manner and path attention only consider parent-child relation. additionally, due to the complexity of the queuebased decoding mechanism @xcite , the predicted pop sequences cannot be guaranteed to be perfect, which may cause error propagation. the second challenge is how compatible syntactic structures can be retrieved to guide generation in practice? previous works @xcite use common syntactic templates that appear most frequently from the corpus, which means that these works generate syntactically diverse paraphrases using a fixed set of syntactic structures for all input sentences. the diversity in syntax is hence limited. moreover, not all sentences can be paraphrased into the same set of syntactic structures. in order to address these problems, we propose a structural information-augmented syntax-controlled paraphrasing (si-scp) model based on attention network. particularly, we design a tree transformer to capture parent-child and sibling relation. to learn the alignment relation between words and nodes, we propose an attention regularization objective. the basic motivation is that a syntactic template contains several syntax nodes which guide the generation of different words, respectively. as shown in figure 1 , the np node guides the generation of noun phrase. learning alignment relation makes the decoder accurately select corresponding syntax nodes to guide the generation of words. additionally, to enhance diversity in syntax-controlled generation, we propose a syntactic template retriever (str) to retrieve compatible syntactic structures for any input sentence. we evaluate our model on two popular benchmark datasets. experiment results show that si-scp achieves the state-of-the-art performence in syntactic and semantic quality. the visualization results show the attention regularization makes the decoder accurately attend to corresponding syntactic nodes during decoding. human evaluation also demonstrates that our method is able to generate semantically and syntactically better sentences than previous methods. we further show that str can retrieve more compatible structures compared with the common syntactic templates method. si-scp can generate more syntactically diverse paraphrases with retrieved syntactic structures. in summary, the major contributions of this paper are as follows: ## related work we focus primarily on the task of syntactically controlled paraphrase generation, which has recently attracted increasing attention @xcite @xcite @xcite . according to the control element, previous works can be divided into two categories. the first strand of research uses sentential exemplar as syntactic control. @xcite and @xcite design a latent variable to learn syntax by encoding the exemplar itself, and then use a syntactic latent variable to guide the generation of paraphrases. however, a latent based approach might not offer enough explicit syntactic information as guaranteed by actual constituency parse trees @xcite . the second strand of research takes a parse tree as a syntactic input, controlling the syntax of generated text with the structure specified by the parse tree. @xcite use two encoders to encode input sentences and linearized parse trees to produce paraphrases. due to the linearization of syntactic tree, a lot of structural information is generally lost. to capture the tree structure of the parse tree, @xcite encode parse trees in a top-down manner, and then a queue-based decoding mechanism is proposed to incorporate syntactic information. li et al. (2020) design a syntax encoder based on a path attention mechanism. however, these methods only consider parent-child relation in parse trees. differing from previous approaches, we propose a novel tree-transformer to model parent-child and sibling relation for better capturing the tree structure of the parse tree. to learn the alignment relation between words and nodes, we also propose an attention regularization objective, which makes the decoder accurately select corresponding syntax nodes to guide the generation of words. additionally, we propose a syntactic template retriever that can help to retrieve compatible syntactic structures for any input sentences. ## syntactic template retriever a sentence cannot be converted to any syntactic structures. incompatible syntax will lead to imperfect paraphrase conversion and nonsensical sentences. in this work, we propose a syntactic template retriever (str) to find compatible syntax for a sentence in practice. the basic assumption is that syntactic templates expressing the same meaning are relatively close in semantic space. specifically, given an input sentence and its parse tree, the retriever retrieves compatible syntactic templates based on similarity from a pre-collected template library. ## experiments in this section, we conducted experiments to answer the following questions: ## conclusion we have presented a structural informationaugmented syntax-controlled paraphrasing (si-scp) model which can directly generate syntactic paraphrases with syntactic templates. particularly, we propose a tree-transformer and an attention regularization. the tree transformer can model parentchild and sibling relation of the syntactic template. the attention regularization method makes the decoder accurately select corresponding syntax nodes to guide the generation of words. to retrieve compatible syntactic templates in practice, we further propose a syntactic template retriever (str). experiments show that si-scp achieves substantial improvements over previous strong baselines. furthermore, we also validate that str is capable of retrieving compatible syntactic templates. si-scp can produce more syntactically paraphrases with retrieved syntactic templates.