Vector-Quantized Prompt Learning for Paraphrase Generation (2023)

Abstract: Deep generative modeling of natural languages has achieved many successes, such as producing fluent sentences and translating from one language into another. However, the development of generative modeling techniques for paraphrase generation still lags behind largely due to the challenges in addressing the complex conflicts between expression diversity and semantic preservation. This paper proposes to generate diverse and high-quality paraphrases by exploiting the pre-trained models with instance-dependent prompts. To learn generalizable prompts, we assume that the number of abstract transforming patterns of paraphrase generation (governed by prompts) is finite and usually not large. Therefore, we present vector-quantized prompts as the cues to control the generation of pre-trained models. Extensive experiments demonstrate that the proposed method achieves new state-of-art results on three benchmark datasets, including Quora, Wikianswers, and MSCOCO. We will release all the code upon acceptance.

## introduction paraphrase generation aims to produce sentences that have different expressions but convey the same semantic meaning given a particular sentence. paraphrasing is a common phenomenon that reflects the diversity of human languages, serving as an important research topic in natural language processing. it has broad applications such as in question answering @xcite and information retrieval @xcite . however, automatically generating accurate and differentappearing paraphrases is still very challenging, since it requires the abilities of both understanding and generation. conventional methods draw on rule-based systems @xcite @xcite and statistical machine translation @xcite to generate paraphrases. these methods are easy to interpret and analyze, but struggle to yield fluent and diverse sentences. recently the accumulation of the paraphrase data provides an unprecedented opportunity to directly learn the paraphrasing transformations in an end-to-end manner @xcite . for instance, @xcite formulate paraphrasing as a supervised encoding-decoding problem and use stacked residual lstm networks to generate paraphrases. a good paraphrase is a sentence that shares similar semantics but has noticeable syntactical or lexical differences from the original one @xcite . to improve the diversity of generated sentences, @xcite introduce the variational auto-encoder (vae) to perform paraphrase generation. @xcite propose multiple generators with different granularity levels to learn the mapping relationship between input and output respectively, and then combine them to complete the paraphrase generation task. but those generated paraphrases tend to only make trivial changes to original sentences, such as modifications of synonyms. further, @xcite leverage autoencoder to encode the structure and semantics of the sentence separately, and generate paraphrases by perturbing the structure encoding. liu et al. integrate the word editing and rule-based transformation operations into deep learning and achieve the previous sota performance in paraphrase generation @xcite . however, due to the limitation of scales of the paraphrasing datasets, neural networks tend to generate the paraphrases with local changes to the inputs rather than global modifications on sentence structures. in this work, we aim to exploit the knowledge of the pre-trained language model to balance expression diversity and semantic preservation. there- pretrained language model what is the main reason of global warming? prompt a what is the reason of global warming? prompt b why does the earth's temperature rise? why does the global warming happen? fore, inspired by @xcite we propose a vector-quantized prompt learning framework, called vqprompt, to generate diverse and high-quality paraphrases. in particular, vqprompt comprises a prompt encoder and a pre-trained generative language model. the prompt encoder produces discrete prompts and the generative language model accepts both the prompts and the input sentence to generate the corresponding paraphrases. to make the vector-quantization work, we also introduce a k-means training strategy to dynamically update the codebook in the prompt encoder. we evaluate the effectiveness of our model on four paraphrasing datasets, namely, quora, wikianswers, and mscoco. experimental results show that vqprompt achieves a new state-of-the-art paraphrasing performance in terms of both automatic metrics and human evaluation. in summary, our contributions are as follows: ## related work one of the characteristics of the paraphrase generation task is that there exist several general transformation rules. therefore, rule-based methods have been used for paraphrase generation tasks as early as the last century. representative methods include dictionary-based and template-based methods. dictionary-based methods look up synonyms in the dictionaries such as hownet @xcite or wordnet @xcite to replace words in the original sentence, thereby generating corresponding paraphrases @xcite . the advantage of such a rulebased approach is that it is interpretable and controllable. its shortcomings lie in the heavy workload of manually writing rules, and the generated sentences are not smooth enough. with the accumulation of paraphrase corpus, researchers then start to model the paraphrase generation task as a single-language statistical translation process, thereby improving the fluency of generating paraphrase sentences @xcite . the statistical translation model learns the transition probability from the original sentence to the paraphrases from a large amount of training data. for example, dolan et al. collected a large number of news texts from the internet and built a paraphrase generation corpus, and then used statistical machine translation methods to generate paraphrase sentences @xcite . however, statistical paraphrasing methods still require heavy feature engineering. in recent years, deep neural network has become the mainstream paraphrase generation method due to its powerful fitting ability @xcite . similar to the statistical paraphrasing methods, the neuralbased paraphrase generation method formulates the paraphrase generation as a single-language translation process, but adopts an encoding-decoding network structure and an end-to-end training method. the first deep paraphrasing method takes the long short-term memory network lstm @xcite as the encoder and decoder. in order to solve the long-distance dependency problem in the encoding process, wang et al. used the multi-head attention network transformer @xcite as the encoder and decoder, and achieved further performance improvement @xcite ). an ideal paraphrase not only needs to have the same semantics but also should have a significant change in expression from the input sentence (i.e., expression difference) @xcite . aiming at the problem of expression differences in generated sentences, researchers have made a lot of attempts in different dimensions @xcite @xcite . for example, li et al. proposed multiple generators with different granularity levels to learn the mapping relationship between input and output respectively, and then combine them to complete the paraphrase generation task @xcite . lin and wan utilized back-translation and multiple rounds of iterative generation methods to produce paraphrased sentences with significant variance @xcite . hosking and lapata @xcite continued to use the idea of variational autoencoder to encode the structure and semantics of the sentence separately, and generate paraphrases by perturbing the structure encoding. different from these methods, this work learns to generate syntax-based prompts, which could induce the pre-trained model to generate diverse paraphrases. apart from the traditional methods working with language models (lms) that have parameters less than 1b, modern llms like chatgpt can also generate paraphrases with high quality. however, it costs much more than traditional methods since they require a huge training corpus and learnable parameters. ## experiments in this section, we first test the proposed model on the benchmarking datasets on both automatic evaluation and human evaluation. then we provide several detailed analyses to elucidate its effectiveness in generating paraphrases. ## conclusion paraphrasing aims to restate one sentence as another with the same meaning, but different wordings. in this paper, we establish a prompt learning framework, coined vqprompt, for paraphrase generation. vqprompt leverages vector quantization to learn finite prompt components and thus possess modest interpretability. we introduce a k-means training strategy to avoid index collapse in vq. experiments show vqprompt achieves impressive generation performance on multiple datasets.