A Learning-Exploring Method to Generate Diverse Paraphrases with Multi-Objective Deep Reinforcement Learning (2020)

Abstract: Paraphrase generation (PG) is of great importance to many downstream tasks in natural language processing. Diversity is an essential nature to PG for enhancing generalization capability and robustness of downstream applications. Recently, neural sequence-to-sequence (Seq2Seq) models have shown promising results in PG. However, traditional model training for PG focuses on optimizing model prediction against single reference and employs cross-entropy loss, which objective is unable to encourage model to generate diverse paraphrases. In this work, we present a novel approach with multi-objective learning to PG. We propose a learning-exploring method to generate sentences as learning objectives from the learned data distribution, and employ reinforcement learning to combine these new learning objectives for model training. We first design a sample-based algorithm to explore diverse sentences. Then we introduce several reward functions to evaluate the sampled sentences as learning signals in terms of expressive diversity and semantic fidelity, aiming to generate diverse and high-quality paraphrases. To effectively optimize model performance satisfying different evaluating aspects, we use a GradNorm-based algorithm that automatically balances these training objectives. Experiments and analyses on Quora and Twitter datasets demonstrate that our proposed method not only gains a significant increase in diversity but also improves generation quality over several state-of-the-art baselines.

## introduction paraphrase generation (pg) creates different expressions that share the same meaning (e.g., "how far is earth from sun" and "what is the distance between sun and earth"). it is a crucial technology in many downstream natural language processing (nlp) applications such as question answering @xcite , machine translation @xcite , and text summarization @xcite . diversity is an essential characteristic of human language, as the meaning of a text can often have multiple different expressions. a good paraphrase generation system is often required to conform to two desired properties @xcite . the first is diversity, capturing a wide range of linguistic variations. the second is fidelity, preserving semantic meanings while paraphrasing. therefore, we hope to generate diverse paraphrases while ensuring same meaning, which is important for enhancing generalization capability and robustness of downstream applications @xcite . as shown in most recent state-of-the-art approaches to pg @xcite @xcite employ neural sequence-to-sequence (seq2seq) models, which mainly uses one given reference for model learning, while the nature of paraphrasing indicates that we can paraphrase one sentence into several different sentences. meanwhile these methods usually adopt the cross-entropy loss which requires a strict pairwise matching at the word level between the predicted sentence and the ground truth sentence. it lacks flexibility and may penalize the generation model for a diverse paraphrase even if the sentence retains the meaning. for example, considering one model prediction "i watched a movie last original sentence how can i improve my english language? paraphrase-a how can i improve my english pronunciation? paraphrase-b how can i increase my knowledge in english language? paraphrase-c there is a possible way to improve my english language? paraphrase-d what is the best way to increase my english knowledge? in particular, we use the variations between the generated paraphrase and the original input as learning signal of diversity. to ensure expressive diversity and semantic fidelity simultaneously, we design two rewards and one for each aspect. one reward is to distinguish whether the explored sentence has diverse expression, the other one is to examine whether the explored sentence conveys the same meaning with original input. the higher the confidence of one sentence is to be judged as a good paraphrase explored by the sample algorithm, that will be used for better model training. we also use a reward evaluated with reference to train model. finally, we combine these rewards evaluated from different aspects as guiding signals to train the generation model via reinforcement learning. furthermore, in order to effectively combine these rewards that optimize different aspects of a generated paraphrase, we additionally use a gradnorm-based algorithm @xcite that automatically balances these training objectives. in summary, our contributions are as follows: input sentence: x explored sentences: 𝑌 paraphrase generator p(θ) how do you build a webite? what is the best way to build a website? how do you create a webite? how can i find a way to have a webite? … … evaluator rewards multi-objective deep reinforcement learning with rewards gradients 𝜃 reference sentence: y figure 1: the proposed learning-exploring paraphrase generation framework with multi-objective deep reinforcement learning. ## the proposed model in this section, we elaborate our proposed model, including its essential components and their working mechanisms. ## multi-objective learning this section explains how to learn the generator using multi-objective deep reinforcement learning. our training algorithm allows model to explore the space of possible paraphrases, making our model to generate more diverse paraphrases and also enhance model performance. as a result, multi-objective reinforcement learning helps paraphrase generation in two ways: (a) it directly optimizes the evaluation metric instead of maximizing the likelihood of the ground-truth reference and (b) it makes our model has the ability to explore unseen paraphrases beyond one single reference. ## experiment in this section, we described the datasets, experimental setup, evaluation metrics and the results of our experiments. ## related work neural paraphrase generation is often formalized as a sequence-to-sequence (seq2seq) learning formalism. @xcite employ a stacked residual lstm network in the seq2seq model to enlarge the model capacity. @xcite incorporate the attention mechanism @xcite to generate paraphrases. @xcite integrate transformer model @xcite and recurrent neural network gru @xcite to learn long-range dependencies in the input sequence. @xcite propose a generator-evaluator architecture to reinforce the paraphrase generator by a reward function. @xcite suppose a sentence-level paraphrase can be decomposed to word/phrase-level paraphrase and learn to generate paraphrases at different levels of granularity. more recent works also focus on generating diverse paraphrases, which is important for improving model generalization capability and robustness of downstream applications. @xcite use a variational autoencoder framework to generate diverse paraphrases by introducing random noise as input. iyyer et al. (2018) similar to these works, we also adopt seq2seq model for paraphrase generation. however, significantly different from them, our work extend the seq2seq model to use explored paraphrases for model training via deep reinforcement learning. we further introduce evaluation metrics in terms of expressive diversity and semantic similarity for model learning. finally, our model can effectively generate paraphrases by exploring unseen paraphrases beyond one single reference in a learning-exploring fashion. ## conclusion in this work, we have presented a novel method to paraphrase generation in a learning-exploring fashion via multi-objective reinforcement learning. we designed sample-based exploring algorithm to acquire diverse paraphrases for model training, and used reinforcement learning with expressive diversity and semantic similarity rewards. experiments and analyses on both quora and twitter datasets show that the proposed method can effectively learn to generate high-quality paraphrases and achieves better performance over several strong baselines. these results disclose that we can improve paraphrase generation by using explored sentences, breaking the restriction of single reference in supervised learning.