A Semantically Consistent and Syntactically Variational Encoder-Decoder Framework for Paraphrase Generation (2020)

Abstract: Paraphrase generation aims to generate semantically consistent sentences with different syntactic realizations. Most of the recent studies rely on the typical encoder-decoder framework where the generation process is deterministic. However, in practice, the ability to generate multiple syntactically different paraphrases is important. Recent work proposed to cooperate variational inference on a target-related latent variable to introduce the diversity. But the latent variable may be contaminated by the semantic information of other unrelated sentences, and in turn, change the conveyed meaning of generated paraphrases. In this paper, we propose a semantically consistent and syntactically variational encoder-decoder framework, which uses adversarial learning to ensure the syntactic latent variable be semantic-free. Moreover, we adopt another discriminator to improve the word-level and sentence-level semantic consistency. So the proposed framework can generate multiple semantically consistent and syntactically different paraphrases. The experiments show that our model outperforms the baseline models on the metrics based on both n-gram matching and semantic similarity, and our model can generate multiple different paraphrases by assembling different syntactic variables.

## introduction paraphrase generation is a longstanding problem in natural language processing (nlp) @xcite , which aims to generate semantically consistent sentences for a given sentence with different syntactic realizations. the task is not only an important building block for many text generation systems such as question answering @xcite , machine translation @xcite , but also beneficial to some nlp tasks such as semantic parsing @xcite , sentence-level representation learning @xcite , data augmentation @xcite . neural network-based methods @xcite @xcite have shown great progress on paraphrase generation. the models mainly rely on the sequence-tosequence (seq2seq) learning framework @xcite with typical encoder-decoders, which are relatively deterministic during the testing stage. generally, the models will select the best result through the beam search but are not able to produce multiple paraphrases in a principled way @xcite . due to the nature of beam-search, the quality of k-th variant will be worse than the first variant. in practice, the ability to generate multiple high-quality and diverse paraphrases is an important characteristic of text generation systems. a target-oriented seq2seq model is applaudable to achieve this goal. for example, @xcite applied variational inference (kingma and welling, 2014) on a target-related latent variable z. during testing, the model can sample multiple latent variables z from a prior distribution to generate multiple different paraphrases. but the remained problem is that z may be contaminated by the semantic information of other unrelated sentences in the training set, leading to an unexpected semantic change of the generated sentences. in this paper, we propose to constrain the target-related latent variable z to contain merely the syntactic information. to achieve this goal, we introduce a syntactic encoder to extract z syn from the target y, and develop a discriminator with adversarial learning to ensure z syn is semantic-free. the idea is inspired by @xcite , which disentangled the latent space of variational autoencoder (vae) into semantic and syntactic spaces. but they considered the bag of words (bows) as the semantic information for adversarial training. this is not optimal because human-generated paraphrases can use quite different words but still express the same meaning. instead, our model is data-driven. we do not constrain the semantic variables to be syntax-free, as the syntactic information entangled in the semantic variables will be overwritten by the target-oriented syntactic variables. when considering semantic consistency, there exists another problem in many text generation models that maximum likelihood estimation (mle) which is implemented by the cross-entropy function will penalize all the non-target words. an example is shown in the experiments on two datasets show that our model yields competitive results over other baseline models, and can generate multiple syntactically different and semantically consistent paraphrases. the main contributions of this work are as follows: ## related work recently, many neural network-based models are proposed for paraphrase generation and can be categorized into three groups: reconstruction-based learning, typical seq2seq learning, and target-oriented seq2seq learning. reconstruction-based learning. the first category of studies mainly deals with paraphrase generation in an unsupervised manner, which adds constraints on language models (lms) including rnn-lm @xcite or @xcite . @xcite introduced a similaritybased reconstruction loss to vae which considered similarities between words in the embedding space. @xcite introduces three kinds of constraints on rnn-lm including keywords matching, word embedding similarity, and skip-thoughts similarity. however, the similarity-based losses could not guarantee the semantic consistency between two words. for example, the words "good", "great" and "bad" are all close in the embedding space because they appear in similar contexts. recently, an intuitive approach was proposed which disentangled the latent space of vae into syntactic and semantic spaces @xcite . in their model, the constituency parse tree was used to supervise the syntactic latent variable, and the bows were used to supervise the semantic latent variable. although the proposal of disentanglement is promising, supervision with bows is not optimal because paraphrases are possible to use quite different words and still convey the same meaning. typical seq2seq learning. the second category of studies considered paraphrase generation as a typical seq2seq task with parallel data. @xcite proposed to use a seq2seq model for paraphrase generation with residual stack lstm, and still performs as a strong baseline @xcite . recent studies improved seq2seq models by involving some efficient mechanisms such as copy and constrained decoding @xcite , inverse reinforcement learning @xcite , decomposition of phrase-level and sentence-level patterns @xcite , and content planning with latent bag of words @xcite . when a sentence has multiple paraphrases in training data, these models will convert them into multiple pairwise sentences. from the perspective of probability modeling, these studies maximize the log conditional @xmath0. target-oriented seq2seq learning. compared with the second category of studies, the third included the target information which substantially maximized the log @xmath1, z y i ) where z y i conveyed the information of target y i . apparently, there was a train-test discrepancy because z y i was not available during testing. @xcite tackled the issue by a combination of the seq2seq architecture with vae which allowed z y i to sample from a prior distribution. the remained problem is that z y i may contain semantic information of other unrelated sentences, which is possible to mislead the model. ideally, for paraphrase generation, z y i is expected to only convey the syntactic information. @xcite implicitly tackled this problem by focusing on a slightly different task, the syntacticguided controlled paraphrase generation, which inputted an exemplar to tell the syntactic information. as a result, the train-test discrepancy does not exist in the controlled task. however, for the traditional paraphrase generation task, constraining z y i is still a problem.