C on RPG : Paraphrase Generation using Contexts as Regularizer (2021)

Abstract: A long-standing issue with paraphrase generation is the lack of reliable supervision signals. In this paper, we propose a new unsupervised paradigm for paraphrase generation based on the assumption that the probabilities of generating two sentences with the same meaning given the same context should be the same. Inspired by this fundamental idea, we propose a pipelined system which consists of paraphrase candidate generation based on contextual language models, candidate filtering using scoring functions, and paraphrase model training based on the selected candidates. The proposed paradigm offers merits over existing paraphrase generation methods: (1) using the context regularizer on meanings, the model is able to generate massive amounts of high-quality paraphrase pairs; (2) the combination of the huge amount of paraphrase candidates and further diversity-promoting filtering yields paraphrases with more lexical and syntactic diversity; and (3) using human-interpretable scoring functions to select paraphrase pairs from candidates, the proposed framework provides a channel for developers to intervene with the data generation process, leading to a more controllable model. Experimental results across different tasks and datasets demonstrate that the proposed paradigm significantly outperforms existing paraphrase approaches in both supervised and unsupervised setups.

## introduction paraphrase generation @xcite @xcite is the task of generating an output sentence which is semantically identical to a given input sentence but with variations in lexicon or syntax. it is a long-standing problem in the field of natural language processing (nlp) @xcite @xcite @xcite and has fundamental applications on end tasks such as semantic parsing @xcite , lan-guage model pretraining @xcite and question answering @xcite . a long-standing challenge with paraphrase generation is to obtain reliable supervision signals. one way to resolve this issue is to manually annotate paraphrase pairs, which is both labor-intensive and expensive. existing labeled paraphrase datasets @xcite @xcite are either of small sizes or restricted in narrow domains. for example, the quora dataset foot_0 contains 140k paraphrase pairs, the size of which is insufficient to build a large neural model. as another example, paraphrases in the larger mscoco @xcite dataset are originally collected as image captions for object recognition, and repurposed for paraphrase generation. the domain for the mscoco dataset is thus restricted to captions depicting visual scenes. unsupervised methods, such as reinforcement learning @xcite and auto-encoders @xcite , on the other hand, have exhibited their ability for paraphrase generation in the absence of annotated datasets. the core problem with existing unsupervised methods for paraphrase is the lack of an objective (or reward function in rl) that reliably measures the semantic relatedness between two diverse expressions in an unsupervised manner, with which the model can be trained to promote pairs with the same meaning but diverse expressions. for example, hegde and patil (2020) crafted unsupervised pseudo training examples by corrupting a sentence and then fed the corrupted one to a pretrained model as the input with the original sentence as the output. since the model is restricted to learning to reconstruct corrupted sentences, the generated paraphrases tend to be highly similar to the input sentences in terms of both wording and word orders. the issue in hegde and patil (2020) can be viewed as a microcosm of problems in existing unsupervised methods for paraphrase: we wish sentences to be diverse in expressions, but do not have a reliable measurement to avoid meaning change when expressions change. additionally, the action of sentence corrupting can be less controllable. in this work, we propose to address this issue by a new paradigm based on the assumption that the probabilities of generating two sentences with the same meaning based on the same context should be the same. with this core idea in mind, we propose a pipelined system which consists of the following steps: (1) paraphrase candidate generation by decoding sentences given its context using a language generation model; (2) candidate filtering based on scoring functions; and (3) paraphrase model training by training a seq2seq paraphrase generation model, which can be latter used for supervised finetuning on labeled datasets or directly used for unsupervised paraphrase generation. the proposed paradigm offers the following merits over existing methods: (1) using the context regularizer on meanings, the model is able to generate massive amounts of high-quality paraphrase pairs; and (2) using human-interpretable ranking scores to select paraphrase pairs from candidates, the proposed framework provides a channel for developers to intervene with the data generation process, leading to a more controllable paraphrase model. extensive experiments across different datasets under both supervised and unsupervised setups demonstrate the effectiveness of the proposed model. 2 related work supervised methods for paraphrase generation rely on annotated paraphrase pairs to train the model. iyyer et al. (2018); li et al. (2019); chen et al. (2019); goyal and durrett (2020) leveraged syntactic structures to generate diverse paraphrases with different syntax. xu et al. (2018); qian et al. (2019) used different semantic embeddings or generators to produce more diverse paraphrases. kazemnejad et al. ( 2020 ) proposed a retrievalbased approach to retrieve paraphrase from a large corpus. @xcite ; @xcite casted paraphrase generation as the task of machine translation. @xcite ; @xcite extended the idea of bilingual pivoting for paraphrase generation where the input sentence is first translated into a foreign language, and then translated back as the paraphrase. @xcite trained a mt model using multilingual parallel data and then finetuned the model using parallel paraphrase data. li et al. (2018 ); siddique et al. (2020) proposed to generate paraphrases using reinforcement learning, where certain rewarding criteria such as bleu and rouge are optimized. bowman et al. (2016); yang et al. (2019) used the generative framework for paraphrase generation by training a variational autoencoder (vae) (kingma and welling, 2013) to optimize the lower bound of the reconstruction likelihood for an input sentence. sentences sampled through the vae's decoder can be regarded as paraphrases for an input sentence due to the reconstruction optimization target. fu et al. (2019) similarly adopted a generative method but worked at the bagof-words level. other works explored paraphrase generation in an unsupervised manner by using vector quantised vae (vq-vae) (roy and grangier, 2019), simulated annealing (liu et al., 2019) or disentangled syntactic and semantic spaces @xcite . more recently, large-scale language model pretraining has also been proven to benefit paraphrase generation in both supervised learning @xcite and unsupervised learning @xcite . @xcite proposed diverse paraphrasing by warping the input's meaning through attribute transfer. ## model the key point of the proposed paradigm is to generate paraphrases based on the same context. this can be done in the following pipelined system: (1) we first train a contextual language generation model (context-lm) that predicts sentences given left and right contexts; (2) the pretrained contextual generation model decodes multiple sentences given the same context, and decoded sentences are treated as paraphrase candidates; (3) due to the fact that decoded sentences can be extremely noisy, further filtering is needed; (4) given the selected paraphrase, a seq2seq model @xcite ) is trained using one sentence of the paraphrase pair as the source and the other as the target; the seq2seq model can be directly taken for the use of paraphrase in the unsupervised learning setup, or used as initialization to be further finetuned on labeled paraphrase datasets in the supervised learning setup. an overview of the proposed framework in depicted in figure 1 , the constituent unit of which will be detailed in order below. ## conclusion in this paper, we propose conrpg, a paradigm for paraphrase generation using context regularizer. conrpg is based on the assumption that the probabilities of generating two sentences with the same meaning based on the same context should be the same. we acknowledge that the current system is rather complicated, which requires multiple pipelines and modules to build. we will simplify the system in future work.