Hierarchical Sketch Induction for Paraphrase Generation (2022)

Abstract: We propose a generative model of paraphrase generation, that encourages syntactic diversity by conditioning on an explicit syntactic sketch. We introduce Hierarchical Refinement Quantized Variational Autoencoders (HRQ-VAE), a method for learning decompositions of dense encodings as a sequence of discrete latent variables that make iterative refinements of increasing granularity. This hierarchy of codes is learned through end-to-end training, and represents fine-to-coarse grained information about the input. We use HRQ-VAE to encode the syntactic form of an input sentence as a path through the hierarchy, allowing us to more easily predict syntactic sketches at test time. Extensive experiments, including a human evaluation, confirm that HRQ-VAE learns a hierarchical representation of the input space, and generates paraphrases of higher quality than previous systems.

## introduction humans use natural language to convey information, mapping an abstract idea to a sentence with a specific surface form. a paraphrase is an alternative surface form of the same underlying semantic content. the ability to automatically identify and generate paraphrases is of significant interest, with applications in data augmentation @xcite , query rewriting, @xcite and duplicate question detection @xcite . while autoregressive models of language (including paraphrasing systems) predict one token at a time, there is evidence that in humans some degree of planning occurs at a higher level than individual words @xcite . prior work on paraphrase generation has attempted to include this inductive bias by specifying an alternative surface form as additional model input, either in the form of target parse trees @xcite @xcite , exemplars @xcite , or syntactic codes figure 1 : the generative models underlying our approach. given some semantic content z sem , we predict a hierarchical set of syntactic codes q d that describe the output syntactic form at increasing levels granularity. these are combined to give a syntactic embedding z syn , which is fed to the decoder along with the original semantic content to generate the output sentence y. during training, the encoder is driven by a paraphrase x sem and a syntactic exemplar x syn . @xcite . most of these approaches suffer from an 'all or nothing' problem: the target surface form must be fully specified during inference. however, predicting the complete syntactic structure is almost as difficult as predicting the sentence itself, negating the benefit of the additional planning step. in this paper, we propose a generative model for paraphrase generation, that combines the diversity introduced by an explicit syntactic target with the tractability of models trained end-to-end. shown in figure 1 , the model begins by assuming the existence of some semantic content z sem . conditioned on this semantic information, the model predicts a syntactic 'sketch' in the form of a hierarchical set of discrete codes q 1:d , that describe the target syntactic structure with increasing granularity. the sketch is combined into an embedding z syn , and fed along with the original meaning z sem to a de-coder that generates the final output utterance y. choosing a discrete representation for the sketch means it can be predicted from the meaning as a simple classification task, and the hierarchical nature means that the joint probability over the codes admits an autoregressive factorisation, making prediction more tractable. the separation between z sem and z syn is induced by a training scheme introduced in earlier work @xcite and inspired by prior work on separated latent spaces @xcite , whereby the model must reconstruct a target output from one input with the correct meaning, and another input with the correct syntactic form. to learn the discretized sketches, we propose a variant of vector-quantized variational autoencoders (vq-vae, or vq) that learns a hierarchy of embeddings within a shared vector space, and represents an input encoding as a path through this hierarchy. our approach, which we call hierarchical refinement quantized variational autoencoders or hrq-vae, leads to a decomposition of a dense vector into embeddings of increasing granularity, representing high-level information at the top level before gradually refining the encoding over subsequent levels. our contributions are summarized as follows: ## neural parameterisation we assume a gaussian distribution for z sem , with prior p(z sem ) ∼ n (0, 1). the encoders φ(z sem |x sem ) and φ(z syn |x syn ) are transformers @xcite , and we use an autoregressive transformer decoder for p(y|z sem , z syn ). the mapping f @xmath0:d to z syn and the posterior network φ(q d |@xmath1, z syn ) are more complex, and form a significant part of our contribution. our choice of parameterization is learned endto-end, and ensures that the sketches learned are hierarchical both in the shared embedding space and in the information they represent. ## experimental setup datasets a paraphrase is 'an alternative surface form in the same language expressing the same semantic content as the original form' @xcite , but it is not always clear what counts as the 'same semantic content'. our approach requires access to reference paraphrases; we evaluate on three english paraphrasing datasets which have clear grounding for the meaning of each sentence: paralex @xcite , a dataset of question paraphrase clusters scraped from wikianswers; quora question pairs (qqp) foot_0 sourced from the community question answering forum quora; and mscoco 2017 @xcite , a set of images that have been captioned by multiple annotators. for the question datasets, each paraphrase is grounded to the (hypothetical) answer they share. we use the splits released by @xcite . for mscoco, each caption is grounded by the image that it describes. we evaluate on the public validation set, randomly selecting one caption for each image to use as input and using the remaining four as references. model configuration hyperparameters were tuned on the paralex development set, and reused for the other evaluations. we set the depth of the hierarchy @xmath2, and the codebook size @xmath3. the transformer encoder and decoder consist of 5 layers each, and we use the vocabulary and token embeddings from bert-base @xcite . we use an initialisation decay factor of α @xmath4.5, and a depth dropout probability p @xmath5.3. a full set of hyperparameters is given in appendix a, and our code is available at https://github.com/tomhosking/hrq-vae . comparison systems as baselines, we consider three popular architectures: a vanilla autoencoder (ae) that learns a single dense vector representation of an input sentence; a gaussian variational autoencoder @xcite , which learns a distribution over dense vectors; and a vector-quantized variational autoencoder (vq-vae, @xcite , that represents the full input sentence as a set of discrete codes. all three models are trained to generate a sentence from one of its paraphrases in the training data, and are not trained with an autoencoder objective. we implement a simple tf-idf baseline @xcite , retrieving the question from the training set with the highest cosine similarity to the input. finally, we include a basic copy baseline as a lower bound, that simply uses the input sentences as the output. we also compare to a range of recent paraphrasing systems. latent bag-of-words (bow, @xcite uses an encoder-decoder model with a discrete bag-of-words as the latent encoding. sow/reap (goyal and durrett, 2020) uses a two stage approach, deriving a set of feasible syntactic rearrangements that is used to guide a second encoder-decoder model. btmpg @xcite uses multi-round generation to improve diversity and a reverse paraphrasing model to preserve semantic fidelity. we use the results after 10 rounds of paraphrasing. separator @xcite uses separated, non-hierarchical encoding spaces for the meaning and form of an input, and an additional inference model to predict the target syntactic form at test time. all comparison systems were trained and evaluated on our splits of the datasets. as an upper bound, we select a sentence from the evaluation set to use as an oracle syntactic exemplar, conditioning generation on a sketch that is known to represent a valid surface form. ## results our experiments were designed to test two primary hypotheses: (1) does hrq-vae learn hierarchical decompositions of an encoding space? and (2) does our choice of generative model enable us to generate high quality and diverse paraphrases? ## conclusion we present a generative model of paraphrasing, that uses a hierarchy of discrete latent variables as a rough syntactic sketch. we introduce hrq-vae, a method for mapping these hierarchical sketches to a continuous encoding space, and demonstrate that it can indeed learn a hierarchy, with lower levels representing more fine-grained information. we apply hrq-vae to the task of paraphrase generation, representing the syntactic form of sentences as paths through a learned hierarchy, that can be predicted during testing. extensive experiments across multiple datasets and a human evaluation show that our method leads to high quality paraphrases. the generative model we introduce has potential application for any natural language generation task; z sem could be sourced from a sentence in a different language, from a different modality (e.g., images or tabular data) or from a task-specific model (e.g., summarization or machine translation). furthermore, hrq-vae makes no assumptions about the type of space being represented, and could in principle be applied to a semantic space, learning a hierarchy over words or concepts. to this value, and our model outperforms all comparison systems on all datasets for 0.7 ≤ α ≤ 0.9. models were trained on a single gpu, with training taking between one and three days depending on the dataset. we use sacrebleu @xcite to calculate bleu scores.