Generative Pretraining for Paraphrase Evaluation (2022)

Abstract: We introduce ParaBLEU, a paraphrase representation learning model and evaluation metric for text generation. Unlike previous approaches, ParaBLEU learns to understand paraphrasis using generative conditioning as a pretraining objective. ParaBLEU correlates more strongly with human judgements than existing metrics, obtaining new state-of-the-art results on the 2017 WMT Metrics Shared Task. We show that our model is robust to data scarcity, exceeding previous state-of-the-art performance using only 50% of the available training data and surpassing BLEU, ROUGE and METEOR with only 40 labelled examples. Finally, we demonstrate that ParaBLEU can be used to conditionally generate novel paraphrases from a single demonstration, which we use to confirm our hypothesis that it learns abstract, generalized paraphrase representations.

## introduction representing the relationship between two pieces of text, be it through a simple algorithm or a deep neural network, has a long history and diverse usecases that include the evaluation of text generation models @xcite and the clinical evaluation of human speech @xcite . one of the earliest examples of such a representation is the levenshtein distance @xcite , which describes the number of character-level edits required to transform one piece of text into another. this metric now forms part of a wider family of edit-distance-based metrics that includes the word error rate (wer) and the translation error rate (ter) @xcite . other algorithms, such as rouge @xcite , meteor @xcite and the widely used bleu metric @xcite , perform exact or approximate n-gram matching between the two texts. these low-level approaches bear little resemblance to the human process of comparing two texts, which benefits from a deep prior understanding of the semantic and syntactic symmetries of language @xcite . for example, pairs like "she was no ordinary burglar" and "she was an ordinary burglar" are close in edit-distancespace but semantically disparate. the goal of an automatic text evaluation metric is typically to be a good proxy for human judgements, which is clearly task-dependent. more recently, neural approaches have begun to close the gap between automatic and human judgements of semantic text similarity using transformer-based language models such as bert @xcite . they aim to leverage the transferable knowledge gained by the model during pretraining on large text corpora. the relationship between two texts is similarly modelled, albeit implicitly, by sequenceto-sequence models such as bart @xcite and t5 @xcite . we consider paraphrase evaluation and paraphrase generation to be two instances of paraphrase representation learning. linguistically, a paraphrase is a restatement that preserves essential meaning, with arbitrary levels of literality, fidelity and completeness. in practice, what qualifies as a good paraphrase is contextspecific. one motivation for considering paraphrase evaluation as a representation learning problem is the varied nature of paraphrase evaluation tasks, which may have an emphasis on semantic equivalence (e.g. paws @xcite and @xcite ), logical entailment versus contradiction (e.g. multinli @xcite and @xcite ), and the acceptability of the generated text (e.g. the wmt metrics shared task @xcite ). considering even broader applications such as clinical speech analysis further motivates learning generalized paraphrase representations. in this paper, we introduce parableu, a paraphrase representation learning model that predicts a conditioning factor for sequence-to-sequence paraphrase generation as one of its pretraining objectives, inspired by style transfer in text-to-speech @xcite ) and text generation systems @xcite . para-bleu addresses the primary issue with neural paraphrase evaluation models to date: the selection of a sufficiently generalized pretraining objective that primes the model for strong performance on downstream paraphrase evaluation tasks when data is scarce. previous state-of-the-art neural models have either used a broad multi-task learning approach or eschewed additional pretraining altogether. the former case may encourage the model to learn the biases of inferior or inappropriate metrics, while the latter leaves room for optimization. non-neural models, such as bleu, ter, rouge and bertscore @xcite , benefit from requiring no training data and thereby avoid domain shift issues. they cannot, however, learn to exploit task-specific nuances of what defines 'good' paraphrasing. we evaluate parableu's ability to predict human judgements of paraphrases using the english subset of the 2017 wmt metrics shared task. a useful neural text similarity metric should be robust to data scarcity, so we assess performance as a function of the fine-tuning dataset size. finally, using the parableu pretraining model as a paraphrase generation system, we explore our hypothesis that the model reasons in high-level paraphrastic concepts rather than low-level edits through an explainability study, and demonstrate that parableu can operate as a conditional paraphrase generation model. ## approach in this section, we describe and justify the set of inductive biases we build into parableu, along with a description of the model architecture and pretraining/fine-tuning strategy. we consider a reference text x and a candidate text x. we wish to learn a function f : f (x, x) → y, where @xmath0, which could be a scalar score. ## data in this section, we describe the pretraining and fine-tuning datasets we use in our studies. ## experiments in this section, we present results on wmt17, benchmarked against the current state-of-the-art approach, along with widely used neural, n-gram and edit-distance-based metrics. we study parableu performance as a function of number of pretraining steps and the size of the fine-tuning dataset. finally, we perform ablations to test the impact of the inductive biases and resultant architectural decisions described in section 2. we report results for both parableu base , based on roberta base (12 layers, 768 hidden units, 12 heads), and our default model parableu large , based on roberta large (24 layers, 1, 024 hidden units, 16 heads). both models are trained nearidentically for 4 epochs on paracorpus. further pretraining details can be found in appendix a. for fine-tuning, we use a batch size of 32, a learning rate of 1e-5 and train for 40k steps, with a validation set size of 10% (unless otherwise stated). no reference texts are shared between the train and validation sets, following @xcite . pretraining parableu large takes ∼ 10h on a 16 a100 gpu machine. fine-tuning takes ∼ 8h on a single a100 gpu machine. ## one-shot paraphrase generation as our final study, we exploit the generative nature of the pretraining architecture to test our claim that the edit encoder reasons in high-level paraphrastic concepts rather than low-level edits. to do this, we diverge from the pretraining setup, in which the same reference text is passed to both the edit encoder and the sequence-to-sequence model, by passing a different, unseen reference to the sequence-to-sequence model. akin to @xcite , the hope is that the 'demonstration paraphrase' acts as a conditioning factor for paraphrasing the unseen sentence in a similar way. if the model is reasoning in low-level edits or otherwise 'cheating', we expect to see: if the model is reasoning in high-level paraphrastic concepts, we expect to see: we generate text using beam-search @xcite . we sample references at random from the mrpc dataset. the demonstration candidate is a hand-crafted paraphrase of the demonstration reference that embodies a pre-specified paraphrase type. we report the predicted entailment score of the demonstration reference and candidate, along with the candidate generated by the model. a summarized, random subset of generation results is shown in appendix c. we include two sets of results for each paraphrastic type (e.g. 'negative'): one where the demonstration reference/candidate differ in this concept, and one where both embody the concept. since we wish to encode the difference between the demonstration reference/candidate texts, the desired behaviour when the demonstration pair is identical is no change. if this is not the case, it is likely that the edit encoder is just autoencoding the candidate using high-level linguistic concepts, similar to linguistic style transfer. further randomly chosen examples are given in appendix f. the results present a strong case that the encoder is representing high-level paraphrastic concepts. it is able to successfully identify changes in mood, style and tense between the demonstration reference and candidate, and transpose them onto the unseen reference to make a largely grammatical and appropriately paraphrased sentence. we do not see significant leakage of concepts, words or styles between the demonstration candidate and the generated candidate, instead the expected transfer of paraphrase style. ## limitations limitations of this work include the relatively small set of baselines used; there is an ever-increasing number of text similarity metrics and so only a subset is presented here. as demonstrated in section 4.1, it seems likely that the performance is currently limited by pretraining time and so we have not yet probed the ceiling performance of this method. swapping out the edit encoder and encoder-decoder for current state-of-the-art models like deberta @xcite ) may offer further performance boosts. expanding this work to predicting on datasets beyond the 2017 wmt metrics shared task will probe the generalizability of the techniques in this paper. the application of para-bleu for paraphrase generation has not been quantitatively explored. although the results presented in section 5 are chosen at random, the analysis is qualitative. more rigorous methods for evaluating the quality of the paraphrase generation are left for future work. ## conclusions in this paper, we introduced parableu, a paraphrase representation learning model and associated paraphrase evaluation metric. we demonstrated that the metric yields state-of-the-art correlation with human paraphrase judgements and is robust to data scarcity. we motivated its pretraining strategy through a set of inductive biases, which we tested through ablation studies. finally, we reframed the pretraining as a one-shot paraphrase generation model and gathered evidence that parableu represents meaningful paraphrastic information.