Paraphrase Generation: A Survey of the State of the Art (2021)

Abstract: This paper focuses on paraphrase generation,which is a widely studied natural language generation task in NLP. With the development of neural models, paraphrase generation research has exhibited a gradual shift to neural methods in the recent years. This has provided architectures for contextualized representation of an input text and generating fluent, diverseand human-like paraphrases. This paper surveys various approaches to paraphrase generation with a main focus on neural methods.

## introduction paraphrases are texts that convey the same meaning while using different words or sentence structures. the generation of paraphrases is a longstanding problem for natural language learning. for example, the question how do i improve my english could be equivalently phrased as what is the best way to learn english. paraphrasing can be play an important role in language understanding tasks, such as question answering @xcite , machine translation @xcite , and semantic parsing @xcite . and it is also a good way for data augmentation @xcite . given a sentence, paraphrase generation aims to create its paraphrases that can have a different wording or different structure from the original sentence, while preserving the original meaning. the focus of paraphrase generation has exhibited a gradual shift from classical approaches to more advanced neural approaches in the recent years with the rapid development of various neural models. neural models have changed the traditional way paraphrase generation is performed and also provided new directions and architectures for the nlp community. while several surveys on the traditional methods and limited neural methods for paraphrase generation have been published @xcite , there is no thorough and comprehensive survey on neural methods for paraphrase generation. to our best knowledge, this is the first survey on neural methods for paraphrase generation. therefore, our goal in this paper is to provide a timely survey on paraphrase generation, with a main focus on neural methods. in the following section, we will first introduce the most frequently used datasets for paraphrase generation (section 2). then we list the traditional evaluation metrics in section 3. in section 4, we present some of the traditional approaches that were used before the neural methods. neural models, the main focus of this paper, will be discussed in section 5. after introducing all the methods, we compare the performance of the different models for paraphrase generation in section 6. finally, we identify some research gaps in paraphrase generation. ## datasets in this section, we describe several datasets that have been extensively used for paraphrase generation. wikianswer this dataset @xcite ) contains approximately 18 million word-aligned question pairs that are paraphrases. the word alignments provided by this dataset also relate the synonyms in the paraphrase sentences. however, all the sentences provided in this dataset are questions, which restricts the paraphrases to only questions. mscoco mscoco @xcite was originally described as a large-scale object detection dataset. it contains human-annotated captions of over 120k images, and each image is associated with five captions from five different annotators. there are about 500k pairs of paraphrases in this dataset. in most cases, annotators describe the most prominent object/action in an image, which makes this dataset suitable for paraphrase-related tasks. quora quora 1 released a dataset in 2017, which consists of over 400k lines of potential question duplicate pairs. among these potential question duplicate pairs, there are 150k question pairs annotated as paraphrases. for the paraphrase generation task only use these valid paraphrase question pairs are used for training and testing. like wikianswer, this dataset is restricted to questions. twitter url twitter url @xcite is constructed by collecting large-scale sentential paraphrases from twitter by linking tweets through paranmt paranmt @xcite is a dataset of more than 50 million english-english sentential paraphrase pairs. the pairs were generated automatically by using back-translation to translate the non-english side of a large czech-english parallel corpus. owing to its recency, it has not been used widely. ## evaluation methods two general types of evaluation metrics are commonly used to evaluate paraphrase generation: automatic evaluation and human evaluation. automatic evaluation several automatic evaluation metrics are used for the evaluation of paraphrase generation. the widely-used metrics include (1) bleu @xcite , which was originally developed to evaluate machine translation systems; (2) meteor @xcite , which aims to address bleu's weakness of being unable to measure semantic equivalents when applied to low-resource languages and has a better correlation with human judgment at the sentence/segment level than bleu; (3) rouge @xcite , a recall-based evaluation metric originally developed for text summarization, has also been used to evaluate paraphrase generation. its versions, rouge-n (computing the n-gram recall) and rouge-l (focusing on the longest common subsequence) are mostly used. (4) ter @xcite , which was also developed to evaluate machine translation. it measures the number of edits that a human translator would have to perform to change a translation so it exactly matches a reference translation. a ter score is a value in the range of 0-1, but is frequently presented as a percentage, where lower is better. human evaluation due to the fact that automatic evaluation metrics mainly focus on the ngram overlaps instead of meaning, human evaluation is used to provide a more accurate and qualitative evaluation of the generated output. in human evaluation, human annotators are asked to score generated paraphrases along multiple dimensions of quality such as similarity, clarity, and fluency. owing to the manual annotation efforts, human evaluation is naturally more costly compared to automatic evaluation, but more representative of the quality of the generated output. ## traditional approaches in this section, some traditional approaches without neural models will be introduced. ## neural approaches early works on paraphrasing mainly focused on template-based or statistical machine translation approaches. however, the matching of templates and modeling of a statistical translation model are both challenging tasks. with the recent advances of neural networks, especially the sequence-to-sequence framework, seq2seq models were first use for paraphrase generation by @xcite . their work inspired the wide use of neural models for paraphrase generation. below we introduce the main approaches based on neural models that are used for paraphrase generation. ## state-of-the-art performance second, a model that employs rl (row 7) has a great advantage for generating better paraphrases because of the reward provided. therefore, a well designed optimization goal plays an important role in the task of paraphrase generation. third, a novel decoding algorithm based on large pretrained language models helps to generate better paraphrases at the word level (row 8) because of the strength of large pretrained language models and the synonyms learned by decoding algorithm. fourth, the attempts to improve paraphrase generation with a special focus on combining multiple granularity levels also yield good performance (row 9,10). when learning to generate paraphrase in word level, phrase level and sentence level at the same time, their models improve the performance on multiple metrics compared with their backbone transformer model (row 5). finally, incorporating syntax control into paraphrase generation will also yield better results at word level and sentence level @xcite . compared with implicit control (row 11), explicit control has a much better performance (row 12) based on quora. it should be noted that most of the works utilize two datasets for experiments (as shown in for evaluation metrics, bleu is the most frequently used one. however, as proposed by @xcite , current automatic evaluation metrics are limited for evaluating paraphrase generation bleu quality target generated paraphrase high high a picture of someone taking a picture of herself a woman taking a picture with a cell phone. high low a batter swinging a bat at a baseball a batter swinging a baseball at a bat low high a man in sunglasses laying on a green kayak. the man lying on a boat in the water. low low people on a gold course enjoy a few games a group of peaple walking ## conclusion although recent neural models have shown great advances, state-of-the-art results are still not satisfactory enough. therefore, more advanced paraphrasing models still need to be explored. below we discuss several potential directions of research that we believe are worth studying. pretrained language models virtually all recent work related to the application of pretrained language models on paraphrase generation is quite naive. therefore, we could combine the large pretrained language models with other mechanisms, for example reinforcement learning, vae and gan.