Paraphrase Generation by Learning How to Edit from Samples (2020)

Abstract: Neural sequence to sequence text generation has been proved to be a viable approach to paraphrase generation. Despite promising results, paraphrases generated by these models mostly suffer from lack of quality and diversity. To address these problems, we propose a novel retrieval-based method for paraphrase generation. Our model first retrieves a paraphrase pair similar to the input sentence from a pre-defined index. With its novel editor module, the model then paraphrases the input sequence by editing it using the extracted relations between the retrieved pair of sentences. In order to have fine-grained control over the editing process, our model uses the newly introduced concept of Micro Edit Vectors. It both extracts and exploits these vectors using the attention mechanism in the Transformer architecture. Experimental results show the superiority of our paraphrase generation method in terms of both automatic metrics, and human evaluation of relevance, grammaticality, and diversity of generated paraphrases.

## introduction paraphrases are texts conveying the same meaning while using different words @xcite . paraphrase generation is an important task in natural language processing (nlp) that has many applications in other down-stream tasks, such as text summarization, question answering, semantic parsing, and information retrieval @xcite @xcite . early works on paraphrasing mostly investigated rule-based or statistical machine translation approaches to this task @xcite . with the recent advances of neural sequence-to-sequence (seq2seq) framework in different nlp tasks, especially in machine translation, an increasing amount of literature have also applied what is the best way to overcome absence of mind ? figure 1 : an overview of the proposed model. this model retrieves the most similar paraphrase pair to the input x from the training corpus (retriever), computes a set of edit vectors [m , z] based on the retrieved pair (edit provider), and applies these edits to the input sequence x to generate its paraphrase (edit performer). seq2seq models to the task of paraphrase generation @xcite @xcite . although the proposed seq2seq methods for paraphrase generation have shown promising results, they are not yet as dominant as their counterparts used in neural machine translation. the main reason is that the available training data for paraphrasing is scarce and domain-specific @xcite . in fact, the necessity to generate sequences from scratch, which is a major drawback of traditional seq2seq models @xcite , magnifies itself when dealing with scarce training data. thus, one can expect that the model would not be trained well and consequently, would not be able to generate diverse outputs. although retrieval-based text generation has been evaluated recently in @xcite ; @xcite ; @xcite as a remedy for this problem, to the best of our knowledge, there is no previous study exploring the usage of this approach in paraphrase generation. moreover, none of the existing works in the realm of retrieval text generation, such as @xcite ; @xcite ; @xcite , focuses on learning how to extract edits from the retrieved sentences. indeed, @xcite ; @xcite computes a single edit vector heuristically through concatenating the weighted sum of the inserted word embeddings and the weighted sum of deleted word embeddings. moreover, hashimoto et al. (2018) only focuses on improving the retrieving stage and uses a standard seq2seq model to edit the retrieved sentence. in this paper, we present an effective retrievalbased approach to paraphrase generation by proposing a novel editor module. our method can be summarized as follows: given an input sentence x, the model first retrieves a similar sentence p and its associated paraphrase q from the training data. then, by getting x and (p, q), the editor both learns how to extract the fine-grained relations between p and q as a set of edits, and also when and how to use these extracted edits to paraphrase x. by incorporating the retrieved pairs into the editing process, we invigorate our model with a non-parametric memory, which enables it to produce non-generic and more diverse outputs. both the retriever and editor components of our method are modeled by deep neural networks. we employ the transformer architecture @xcite as the backbone of our model, and use its attention mechanism as an effective tool to apply edits in a selective manner. our main contributions are: ## proposed approach let @xmath0, y n } n @xmath1, and y n is its target paraphrase. in the paraphrasing task, our goal is to find the set of parameters of the model that maximizes n @xmath2. figure 1 illustrates the overview of our proposed model which is composed of a retriever and an editor. given an input sequence x, the retriever first finds a paraphrase pair (p, q) from the training corpus based on similarity of x and p. then, the editor utilizes the retrieved pair (p, q) to paraphrase x. we discuss the details in the following subsections. ## experiments in this section, we empirically evaluate the performance of our proposed method in the task of paraphrase generation, and compare it with various other methods, including previous state-of-the-art paraphrasing models. ## conclusion in this paper, we proposed a retrieval-based paraphrase generation model which includes a novel fully-attentional editor. this editor learns how to extract edits from a paraphrase pair and also when and how to apply these edits to a new input sentence. we also introduced the new idea of micro edit vectors, where each one of these vectors represents a small edit that should be applied to the source sentence to get its paraphrase. we incorporated transformer modules in our editor and augmented them with attention over micro edit vectors. the proposed model outperforms the previous state-of-the-art paraphrase generation models in terms of both automatic metrics and human evaluation. moreover, the outputs show that our model is able to produce paraphrases by editing sentences in a fine-grained manner using the idea of mevs. in future work, we intend to adapt our editor module for other learning tasks with both the structured input and structured output.