Towards Document-Level Paraphrase Generation with Sentence Rewriting and Reordering (2021)

Abstract: Paraphrase generation is an important task in natural language processing. Previous works focus on sentence-level paraphrase generation, while ignoring document-level paraphrase generation, which is a more challenging and valuable task. In this paper, we explore the task of document-level paraphrase generation for the first time and focus on the inter-sentence diversity by considering sentence rewriting and reordering. We propose CoRPG (Coherence Relationship guided Paraphrase Generation), which leverages graph GRU to encode the coherence relationship graph and get the coherence-aware representation for each sentence, which can be used for re-arranging the multiple (possibly modified) input sentences. We create a pseudo document-level paraphrase dataset for training CoRPG. Automatic evaluation results show CoRPG outperforms several strong baseline models on the BERTScore and diversity scores. Human evaluation also shows our model can generate document paraphrase with more diversity and semantic preservation.

## introduction paraphrase generation @xcite is an important task in natural language processing, and it aims to rewrite a text in other forms while preserving original semantics. paraphrase generation has many applications in other down-stream tasks, such as text summarization @xcite , dialogue system, question answering @xcite , semantic parsing @xcite and so on. inspired by the success of deep learning, most paraphrase systems leverage existing paraphrase corpora to train a seq2seq model, such as variational auto-encoder @xcite , syntactic pre-ordering @xcite ) and so on. all these works focus on sentence-level paraphrase generation. document-level paraphrase generation, which aims to rewrite a passage or a document without original: 1sustainability has become the foundation for almost all economic thinking nowadays. 2it is essential not only to economic recovery today, but to ensuring peace and security tomorrow. 3factoring sustainability into all our thinking is necessary because, as a global society, we are living on the edge. 4the last two years have brought a series of crises: energy, food, climate change, and global recession. 5i fear that worse may be in store. paraphrase: 1today, sustainability has been the basement for almost all economic mind. 3it is necessary to reflect on sustainable development in our planning, 2because it is indispensability not only for current economic recovery, but for the peace and security tomorrow. 3,4the global society has experienced a series of crises in the past two years, such as energy, food, climate change and global economic recession. 3we have on the edge of collapse, 5but i worry that the worse things are yet to come. in this work, we conduct a pilot study of this challenging task and focus only on rewriting and reordering the sentences in original document while still maintaining the original semantics and inter-sentence coherence. due to the lack of parallel document-level paraphrase pairs, it is not possible to straightforwardly train a sequence-tosequence paraphrasing model to address this task. we thus propose corpg (coherence relationship guided paraphrase generation), which is based on an automatically constructed pseudo document paraphrase dataset. though the paraphrases in the pseudo dataset do not involve inter-sentence diversity, our model can learn the coherence relations between sentences via a coherence relationship graph generated by albert @xcite , and make use of the learned coherence-aware representations of sentences to reorder them, while keeping good coherence of the generated document. our model consists of three parts: sentence encoder, graph gru and decoder. sentence encoder only encodes each sentence in the document individually. we propose graph gru, which combines graph attention @xcite and gru, to catch the coherence relationship information. finally, the outputs of graph gru and sentence encoder are concatenated and used as input to decoder to generate the paraphrase. extensive evaluations are performed and our model gets the best scores on most metrics in both automatic evaluation and human evaluation. the contributions of our work are summarized as below: ## related work with the development of deep learning, most paraphrasing models are based on seq2seq model. @xcite leveraged stacked residual lstm networks to generate paraphrases. @xcite found deep generative model such as variational auto-encoder can improve the quality of paraphrase significantly. @xcite proposed dnpg to decompose a sentence into sentence-level pattern and phrase-level pattern to make neural paraphrase generation more controllable. goyal and durrett (2020) used syntactic transformations to softly "reorder" the source sentence and guide neural model to generate more diverse paraphrase. @xcite explored to generate paraphrase by editing the original sentence. beside, there is another way to generate paraphrase, called "pivoting", which leverages backtranslation to introduce diversity. recently, @xcite revisited this method with neural machine translation to improve the paraphrase quality. wieting and gimpel (2018) leveraged bidirectional translation model to construct paranmt, which is a very large sentence-level paraphrase dataset. all works above focus on sentence-level paraphrase generation, and to the best of our knowledge, there is no research on document-level paraphrase generation. ## conclusion in this paper, we explore a challenging documentlevel paraphrase generation task and propose a novel model called corpg to generate document paraphrases with good relevancy, coherence and inter-sentence diversity. both automatic and human evaluation show the efficacy of our model. in the future, we will try to incorporate the operations of sentence splitting and merging, which is not well addressed by our model, to further improve the quality of document paraphrase.