Neural Syntactic Preordering for Controlled Paraphrase Generation (2020)

Abstract: Paraphrasing natural language sentences is a multifaceted process: it might involve replacing individual words or short phrases, local rearrangement of content, or high-level restructuring like topicalization or passivization. Past approaches struggle to cover this space of paraphrase possibilities in an interpretable manner. Our work, inspired by pre-ordering literature in machine translation, uses syntactic transformations to softly “reorder” the source sentence and guide our neural paraphrasing model. First, given an input sentence, we derive a set of feasible syntactic rearrangements using an encoder-decoder model. This model operates over a partially lexical, partially syntactic view of the sentence and can reorder big chunks. Next, we use each proposed rearrangement to produce a sequence of position embeddings, which encourages our final encoder-decoder paraphrase model to attend to the source words in a particular order. Our evaluation, both automatic and human, shows that the proposed system retains the quality of the baseline approaches while giving a substantial increase in the diversity of the generated paraphrases.

## introduction paraphrase generation @xcite has seen a recent surge of interest, both with large-scale dataset collection and curation @xcite and with modeling advances such as deep generative models @xcite . paraphrasing models have proven to be especially useful if they expose control mechanisms that can be manipulated to produce diverse paraphrases @xcite @xcite , which allows these models to be employed for data augmentation @xcite and rearrangement aware paraphrasing source order rewriting s clippers won the game np vp np vbd x np won y np y np won by x np 4 3 1 2 the game was won by the clippers. ## method given an input sentence @xmath0, x 2 , . . . , x n }, our goal is to generate a set of structurally distinct paraphrases @xmath1, y 2 , . . . , y k }. we achieve this by first producing k diverse reorderings for the input sentence, @xmath2, r 2 , . . . , r k }, that guide the generation order of each corresponding y. each reordering is represented as a permutation of the source sentence indices. our method centers around a sequence-tosequence model which can generate a paraphrase roughly respecting a particular ordering of the input tokens. formally, this is a model p (y | x, r). first, we assume access to the set of target reorderings r and describe this rearrangement aware paraphrasing model (reap) in section 2.2. then, in section 2.3, we outline our reordering approach, including the source order rewriting (sow) model, which produces the set of reorderings appropriate for a given input sentence x during inference (@xmath3. ## data and training we train and evaluate our model on the paranmt-50m paraphrase dataset @xcite constructed by backtranslating the czech sentences of the czeng @xcite corpus. we filter this dataset to remove shorter sentences (less than 8 tokens), low quality paraphrase pairs (quantified by a translation score included with the dataset) and examples that exhibit low reordering (quantified by a reordering score based on the position of each word in the source and its aligned word in the target sentence). this leaves us with over 350k paired paraphrase pairs. ## conclusion in this work, we propose a two-step framework for paraphrase generation: construction of diverse syntactic guides in the form of target reorderings followed by actual paraphrase generation that respects these reorderings. our experiments show that this approach can be used to produce paraphrases that achieve a better quality-diversity trade-off compared to previous methods and strong baselines. sow input sow output removing the nn from np excluding this nn from np they might consider vp if np were imposed in the case of imposition of np , they would consider vp np lingered in the deserted nns . in the abandoned nns , there was np . pp was a black nn archway . was a black nn passage pp . there is already a ring nn pp . pp circular nn exist . this scoring procedure is exactly same as the one proposed by @xcite to evaluate sentence and phrase similarities. 4. finally, the phrases @xmath4: these aligned set of phrase pairs (p, p) are used to construct tuples (t a , b, c) and (t a , b , c ), as outlined in section 3.2.