Factorising Meaning and Form for Intent-Preserving Paraphrasing (2021)

Abstract: We propose a method for generating paraphrases of English questions that retain the original intent but use a different surface form. Our model combines a careful choice of training objective with a principled information bottleneck, to induce a latent encoding space that disentangles meaning and form. We train an encoder-decoder model to reconstruct a question from a paraphrase with the same meaning and an exemplar with the same surface form, leading to separated encoding spaces. We use a Vector-Quantized Variational Autoencoder to represent the surface form as a set of discrete latent variables, allowing us to use a classifier to select a different surface form at test time. Crucially, our method does not require access to an external source of target exemplars. Extensive experiments and a human evaluation show that we are able to generate paraphrases with a better tradeoff between semantic preservation and syntactic novelty compared to previous methods.

## introduction a paraphrase of an utterance is "an alternative surface form in the same language expressing the same semantic content as the original form" @xcite . for questions, a paraphrase should have the same intent, and should lead to the same answer as the original, as in the examples in recent approaches to paraphrasing use information bottlenecks with vaes @xcite or pivot languages @xcite to try to extract the semantics of an input utterance, before projecting back to a (hopefully different) surface form. however, these methods have lit-how is a dialect different from a language? the differences between language and dialect? what is the difference between language and dialect? what is the weight of an average moose? average weight of the moose? how much do moose weigh? how heavy is a moose? what country do parrots live in? in what country do parrots live? where do parrots naturally live? what part of the world do parrots live in? in this paper, we propose separator, a method for generating paraphrases that exhibit high variation in surface form while still retaining the original intent. our key innovations are: (a) to train a model to reconstruct a target question from an input paraphrase with the same meaning, and an exemplar with the same surface form, and (b) to separately encode the form and meaning of questions as discrete and continuous latent variables respectively, enabling us to modify the output surface form while preserving the original question intent. crucially, unlike prior work on syntax controlled paraphrasing, we show that we can generate diverse paraphrases of an input question at test time by inferring a different discrete syntactic encoding, without needing access to reference exemplars. we limit our work to english questions for three reasons: (a) the concept of a paraphrase is more what is the weight of an average moose? ## problem formulation the task is to learn a mapping from an input question, represented as a sequence of tokens x, to paraphrase(s) y which have different surface form to x, but convey the same intent. our proposed approach, which we call separator, uses an encoder-decoder model to transform an input question into a latent encoding space, and then back to an output paraphrase. we hypothesize that a principled information bottleneck (section 2.1) and a careful choice of training scheme (section 2.2) lead to an encoding space that separately represents the intent and surface form. this separation enables us to paraphrase the input question, varying the surface form of the output by directly manipulating the syntactic encoding of the input and keeping the semantic encoding constant (section 2.3). we assume access to reference paraphrase clusters during training (e.g., our model is a variant of the standard encoderdecoder framework @xcite , and consists of: (a) a vanilla transformer sentence encoder @xcite , that maps an input question x to a multi-head sequence of encodings, e h,@xmath0; (b) a principled choice of information bottleneck, with a continuous variational path and a discrete vector-quantized path, that maps the encoding sequence to a pair of latent vectors, z sem , z @xmath1,t ), represented in more detail in figure 1 ; (c) a vanilla transformer decoder, that attends over the latent vectors to generate a sequence of output tokens, ŷ = decoder(z sem , z syn ). the separation between z sem and z syn is induced by our proposed training scheme, shown in figure 1 and described in detail in section 2.2. ## experimental setup datasets we evaluate our approach on two datasets: paralex @xcite , a dataset of question paraphrase clusters scraped from wikianswers; and quora question pairs (qqp) 2 sourced from the community question answering forum quora. we observed that a significant fraction of the questions in paralex included typos or were ungrammatical. we therefore filter out any questions marked as non-english by a language detection script @xcite , then pass the questions through a simple spellchecker. while this destructively edited some named entities in the questions, it did so in a consistent way across the whole dataset. there is no canonical split for paralex, so we group the questions into clusters of paraphrases, and split these clusters into train/dev/test partitions with weighting 80/10/10. similarly, qqp does not have a public test set. we therefore partitioned the clusters in the validation set randomly in two, to give us our dev/test splits. summary statistics of the resulting datasets are given in appendix b. all scores reported are on our test split. model configuration following previous work @xcite , our quantizer uses multiple heads (@xmath2, with z syn given by the concatenation of their codebook embeddings c h (q h ). we use a relatively small codebook size of @xmath3, relying on the combinatoric power of the multiple heads to maintain the expressivity of the model. we argue that, assuming each head learns to capture a particular property of a template (see section 4.3), the number of variations in each property is small, and it is only through combination that the space of possible templates becomes large. we include a detailed list of hyperparameters in appendix a. our code is available at http:// github.com/tomhosking/separator. ## results our experiments were designed to answer three questions: (a) does separator effectively factorize meaning and form? (b) does separator manage to generate diverse paraphrases (while preserving the intent of the input)? (c) what does the underlying quantized space encode (i.e., can we identify any meaningful syntactic properties)? we address each of these questions in the following sections. ## related work paraphrasing prior work on generating paraphrases has looked at extracting sentences with similar meaning from large corpora @xcite @xcite , or identifying paraphrases from sources that are weakly aligned @xcite . more recently, neural approaches to paraphrasing have shown promise. several models have used an information bottleneck to try to encode the semantics of the input, including vaes @xcite , @xcite , and a latent bag-of-words model @xcite . other work has relied on the strength of neural machine translation models, translating an input into a pivot language and then back into english @xcite @xcite . @xcite use submodular function maximisation to improve the diversity of paraphrases generated by an encoder-decoder model. @xcite use an automatic paraphrasing system to rewrite inputs to a question answering system at inference time, reducing the sensitivity of the system to the specific phrasing of a query. ## conclusion we present separator, a method for generating paraphrases that balances high variation in surface form with strong intent preservation. our approach consists of: (a) a training scheme that causes an encoder-decoder model to learn separated latent encodings, (b) a vector-quantized bottleneck that results in discrete variables for the syntactic encoding, and (c) a simple model to predict different yet valid surface forms for the output. extensive experiments and a human evaluation show that our approach leads to separated encoding spaces with negligible loss of expressivity, and is able to generate paraphrases with a better balance of variation and semantic fidelity than prior methods. in future, we would like to investigate the properties of the syntactic encoding space, and improve on the code prediction model. it would also be interesting to reduce the levels of supervision required to train the model, and induce the separation without an external syntactic model or reference paraphrases.