MCPG : A Flexible Multi-Level Controllable Framework for Unsupervised Paraphrase Generation (2022)

Abstract: We present MCPG: a simple and effectiveapproach for controllable unsupervised paraphrase generation, which is also flexible toadapt to specific domains without extra training. MCPG is controllable in different levels: local lexicons, global semantics, and universal styles. The unsupervised paradigm ofMCPG combines factual keywords and diversified semantic embeddings as local lexical andglobal semantic constraints. The semantic embeddings are diversified by standard dropout,which is exploited for the first time to increaseinference diversity by us. Moreover, MCPGis qualified with good domain adaptability byadding a transfer vector as a universal style constraint, which is refined from the exemplars retrieved from the corpus of the target domain in atraining-free way. Extensive experiments showthat MCPG outperforms state-of-the-art unsupervised baselines by a margin. Meanwhile,our domain-adapted MCPG also achieves competitive performance with strong supervisedbaselines even without training.

## introduction paraphrase generation aims to restate a given sentence in a way that conveys the same semantic meaning but uses a different expression form. paraphrasing results benefit lots of downstream tasks, such as text classification @xcite , question answering @xcite , and semantic matching @xcite . traditional approaches @xcite require supervised training on large parallel corpora. however, it is expensive for manual annotation. thus, unsupervised methods are welcomed in the absence of annotated datasets @xcite @xcite @xcite . in this paper, we focus on controllable unsupervised paraphrase generation, which is a promising direction and some progress has been achieved. however, most existing works are either limited by the supervised settings @xcite @xcite @xcite or mainly explore to control generation from a single perspective (e.g., syntatic diversity) in the unsupervised condition @xcite . multi-level controllable approaches for unsupervised generation are still not well explored. to this end, we propose a simple and effective framework for unsupervised paraphrase generation called multi-level controllable paraphrase generator (mcpg), as shown in figure1. overall, mcpg is controllable in three levels, i.e., global semantics, local lexicons, and universal styles. this approach is motivated by a commonly observed phenomenon that the human paraphrasing process is decomposable at different levels. for example, given a sentence, we first read it through to capture the general semantic meaning (global semantics). then we read over to identify factual keywords like names of persons and locations, which should be protected from tampering (local lexicons). furthermore, if there are any exemplars for reference, we may optionally get inspirations from them regarding word editing or sentence restructuring (universal styles). finally, we rewrite the remaining parts as much as possible while preserving the original meaning. figure1(a) shows the basic paradigm of mcpg tailored for the unsupervised scenario. we employ t5 @xcite as the generator backbone to integrate both global semantic and local lexical constraints. the whole model is trained by reconstructing input text, which is compatible with the denoising objective for pretraining t5. the global semantics is controlled by the semantic embedding. we show that enabling dropout in- semantic encoder keyword selector what is it like to study in mcgill university? mcgill university semantic embed generator • (@xmath0.10) what is it like to study at mcgill university ? • (@xmath1.12) what is it like to study , to be at mcgill university ? • (@xmath2.16) what does it mean to study like this inside mcgill university ? • (@xmath3.22) what is it like to be a student in mcgill university ? (a) mcpg-basic semantic encoder keyword selector what is it like to study in mcgill university? mcgill university generator how does it feel to study at mcgill university ? retriever p: what is it like to live in canada ? q: how does it feel to live in canada ? semantic embed transfer vector (b) mcpg-adapt side the semantic encoder during inference works well in creating different semantic embeddings for the same sentence while keeping the semantic meaning. this provides a helpful guidance for mcpg to generate paraphrases with various expression forms. besides, we can further control the diversity degree by tuning the dropout probability and filter bad semantic embeddings according to the cosine similarity between them and the dropoutdisabled standard embedding of the same input. the local lexicons are controlled by the keywords. previous works @xcite @xcite pre-specify keywords using either rule-based or model-based methods, but neglect the importance of factual entities, which are actually the unchangeable words we must copy during paraphrasing. towards this, we adopt named entities recognition tools to identify factual entities like names of persons and locations as keywords. fixing this kind of keywords turns out to be an effective way for improving paraphrase quality. in figure 1 (b), we offer an alternative option to constrain the universal style of outputs, which helps mcpg adapt to the target domain efficiently without training. the universal styles are controlled by a transfer vector, which implicitly encodes the lexical and structural mapping mode between a pair of exemplar paraphrases retrieved from the training set of target domain. this mode is similar to style transfer @xcite . we derive the transfer vector via subtracting the semantic embeddings of two parallel paraphrases, which requires no extra training. by adding the transfer vector to the semantic embedding of the input sentence, our model is inspired to paraphrase in a tone more consistent with the target domain. our contributions are to: (1) present a multilevel controllable framework for unsupervised paraphrase generation which is flexible for domain adaptation; unsupervised approaches there are two main lines for unsupervised paraphrase generation: optimization-oriented and pivot-based methods. the goal of optimization-oriented methods @xcite @xcite is to find the optimal paraphrase by optimizing an objective function that considers semantic fidelity, expression diversity, language fluency, etc. nonetheless, designing reliable metrics to automatically evaluate the quality of paraphrases has been a long-standing problem. the pivot-based approaches aim to first represent the input sentence by some kind of pivot and then reconstruct the input from it @xcite . a popular pivot is the latent semantic representation learned by a variational autoencoder (vae) @xcite , which is mathematically interpretable but hard to train intrinsically. another method is to use other languages as a pivot @xcite @xcite , which relies on external machine translation systems. the works most similar to us explore to take advantages from large-scale language models and adopt corrupted sequences as paraphrasing pivot @xcite @xcite . however, these methods lack a global semantic guidance due to the absence of the complete input sentence. in addition to the above methods, huang and chang (2021) take the first step towards syntactically controllable unsupervised paraphrasing by manipulating the embedding of the potential constituency parse tree. whereas, it is time-consuming for constituent parsing, which may increase time complexity for paraphrase generation. ## experimental setup 4.1 datasets there are two variants of our proposed model, i.e., the unsupervised mcpg-basic and the domainadapted mcpg-adapt. we evaluate mcpg-basic on three widely-used datasets: quora foot_0 , mscoco @xcite and twitter @xcite under the unsupervised setting following in meng et al. @xcite . for mcpg-adapt, we compare it with supervised baselines on both quora and mscoco datasets follow the supervised evaluation protocol in @xcite . the details are as follows. quora is grounded from the duplicated question pairs sharing the same answers in the quora forum. we use 30k instances for testing in the unsupervised experiments. to validate the domain-adapted performance, the size of training, validation and test sets are 100k, 4k and 20k. mscoco was originally constructed for image caption, which contains 5 different captions for each image. we randomly pick one of them as source sentence and the rest as targets. we use 20k test instances under the unsupervised setting. for supervised experiments, we split the dataset into 93k, 4k and 20k for training, validation and testing. twitter was built from linked tweets sharing urls. there are both automatically and human annotated sentence pairs. we only use the pairs that are manually labeled as "paraphrases", which results in 566 instances for unsupervised testing. ## conclusion in this paper, we propose a simple and effective model mcpg for more controllable unsupervised paraphrase generation which is also flexible for domain adaptation. mcpg decomposes the generation constraints into multiple levels: the global semantics, the local lexicons, and the universal styles in a human-like manner. particularly, we manipulate a transfer vector derived from the semantic embeddings of exemplars to control output styles, which provides a new possibility for domain adaptation with annotated samples in a trainingfree way. in future works, we will explore to ( 1 ) control the model from more perspectives such as the syntactic structure, (2) generate paraphrases with higher quality than the input, i.e., to polish the input texts.