Revisiting Pivot-Based Paraphrase Generation: Language Is Not the Only Optional Pivot (2021)

Abstract: Paraphrases refer to texts that convey the same meaning with different expression forms. Pivot-based methods, also known as the round-trip translation, have shown promising results in generating high-quality paraphrases. However, existing pivot-based methods all rely on language as the pivot, where large-scale, high-quality parallel bilingual texts are required. In this paper, we explore the feasibility of using semantic and syntactic representations as the pivot for paraphrase generation. Concretely, we transform a sentence into a variety of different semantic or syntactic representations (including AMR, UD, and latent semantic representation), and then decode the sentence back from the semantic representations. We further explore a pretraining-based approach to compress the pipeline process into an end-to-end framework. We conduct experiments comparing different approaches with different kinds of pivots. Experimental results show that taking AMR as pivot can obtain paraphrases with better quality than taking language as the pivot. The end-to-end framework can reduce semantic shift when language is used as the pivot. Besides, several unsupervised pivot-based methods can generate paraphrases with similar quality as the supervised sequence-to-sequence model, which indicates that parallel data of paraphrases may not be necessary for paraphrase generation.

## introduction paraphrase generation is an important and challenging task in the field of natural language processing (nlp), which can be applied in a variety of applications such as information retrieval @xcite , question answering @xcite , machine translation @xcite , and so on. traditionally, paraphrase generation is usually implemented using ruled-based models @xcite , lexicon-based methods @xcite , grammar-based methods @xcite , statistical machine translation-based methods @xcite . with the rapid development of deep learning techniques, neural methods have shown great power in paraphrase generation and achieve state-of-the-art results @xcite . neural paraphrase generation models usually follow the encoder-decoder paradigm. given a sentence x, these models generate the paraphrase y by directly modeling p (y |x) through a deep neural network. however, deep neural networks are sensitive to domains in general @xcite , while existing mainstream paraphrase corpora only cover a few specific domains, such as image caption @xcite and questions @xcite . highquality paraphrases for general domains are difficult to obtain in practice, which greatly restricts the application of these seq2seq models. benefiting from the rapid development of machine translation technologies, pivot-based methods @xcite @xcite have been proposed for paraphrase generation. formally speaking, pivot-based methods generate the paraphrase by following p (y |x) = p (z|x)p (y |z), where z denotes the pivot of x. existing pivot-based methods all choose z as representations in a different language, therefore the quality of the generated paraphrases largely depends on the pre-existing machine translation system. choosing language as pivot has some disadvantages, for example: (1) the pipeline translations may incur semantic shift @xcite , and (2) machine translation systems are sensitive to domain, and the quality of translating out-of-domain sentences can not be guaranteed. in this paper, we explore the feasibility of using different pivots for pivot-based paraphrasing models, including syntactic representation (universal dependencies @xcite , ud), semantic representation @xcite , amr), and latent semantic representation (lsr). compared with choosing other languages as pivot, choosing syntactic or semantic as pivot is a more direct way, and is less likely to incur semantic shift. apart from pipeline pivot-based generation, we also investigate how much an end-to-end pivot-based model, which can produce paraphrases in a single step with the help of pivot, affects the quality of paraphrases. in the end-to-end framework, the model directly learns the paraphrasing probability p (y |x) from text distribution p (x) and p (y ), pivot distribution p (z), as well as parallel text-pivot distribution p (z|x) and p (y |z). we conduct experiments on two benchmarks of paraphrasing tasks: parabank and quora datasets. we compared in detail the pros and cons of models using different pivots in terms of fidelity, fluency, diversity and so on in the experiments. the results show that using the amr as the pivot can also produce high-quality paraphrases. besides, the endto-end framework can reduce the semantic shift when language is the pivot. in sum, the prime contributions of this paper are as follows: the end-to-end framework can also reduce the semantic changes when language is used as the pivot. besides, several unsupervised pivot-based methods can generate paraphrases as good as the supervised encoder-decoder method, indicating that parallel samples may not be essential to generate high-quality paraphrases. ## pipeline pivot-based paraphrase generation in the pipeline process, we first translate the input texts to pivots (language, amr or ud) foot_1 , followed by generating paraphrases from pivots. this process is shown in figure 1 (a) . ## towards end-to-end paraphrase generation the above pivot-based methods are simple and straightforward, but have two disadvantages: (1) it is difficult to control and optimize the pipeline system, and the quality of the generated paraphrases is totally determined by the text-to-pivot and pivotto-text systems used. in this paper, we also investigate the feasibility of end2end methods. different from the supervised paraphrasing models, our model does not involve any explicit paraphrase sentences, so it needs to generate paraphrases in a "zero-shot" way. inspired by recent work on cross-lingual transfer @xcite , we propose a pretraining framework to endow the model with the ability of zero-shot paraphrasing. besides, we also experiment using auto-encoder to generate paraphrases. in the auto-encoder model, the encoded latent semantic representation (lsr) can be considered as a kind of semantic pivot. ## results and analysis the automatic evaluation results are shown in ## conclusions and future work in this work, we focus on pivot-based paraphrase generation. previous works leverage language as the pivot, which may introduce semantic shift. in this work, we explore whether we can use amr or ud as pivot. we also explore an end-to-end framework in a zero-shot way, using only parallel text-pivot data. results of the automatic metrics and human evaluations show that amr is a good choice of pivot, as amr graphs preserve important words as concepts and thus preserve semantics. moreover, replacing two-step pipeline process with the end-to-end framework is beneficial when language is the pivot, reducing the semantic change. besides, some unsupervised pivot-based methods can perform as well as supervised paraphrase models. in the future, we will focus on zero-shot paraphrase generation task and explore more semantic representations as pivots for pivot-based paraphrase generation.