Towards Better Characterization of Paraphrases (2022)

Abstract: To effectively characterize the nature of paraphrase pairs without expert human annotation, we proposes two new metrics: word position deviation (WPD) and lexical deviation (LD). WPD measures the degree of structural alteration, while LD measures the difference in vocabulary used. We apply these metrics to better understand the commonly-used MRPC dataset and study how it differs from PAWS, another paraphrase identification dataset. We also perform a detailed study on MRPC and propose improvements to the dataset, showing that it improves generalizability of models trained on the dataset. Lastly, we apply our metrics to filter the output of a paraphrase generation model and show how it can be used to generate specific forms of paraphrases for data augmentation or robustness testing of NLP models.

## introduction a robust understanding of semantic meaning, despite variances in sentence expression, is an integral part of natural language processing (nlp) tasks. however, many existing nlp models exhibit shortcomings in understanding real-world variations in natural language. these models are often overreliant on learned spurious correlations resulting in poor generalization @xcite . this problem is challenging to address since it is difficult to distinguish spurious correlations from useful features @xcite . one way of improving the performance and robustness of nlp model is to increase the size of the dataset @xcite . it is possible to do so in an efficient manner through data augmentation, or the process of generating new data out of existing examples, thus creating more training data or test cases @xcite . this would also enhance the capability to detecting error in a wide range of nlp systems. we can also condition language models to generate paraphrases of input sentences @xcite through the use of large language models such as gpt @xcite . however, commonly used paraphrase datasets and paraphrase generation techniques that rely on such datasets suffer from several shortfalls, such as being noisy due to loose labelling in these datasets and lack of accurate, controllable generation. in this paper, we make three key contributions to address this issue. firstly, we propose two new metrics for better understanding of paraphrase pairs: word position deviation and lexical deviation. we show, with examples, how these metrics are more effective at quantitatively capturing the linguistic characteristics of paraphrase pair than existing methods such as rogue-l, self-bleu and edit distance. secondly, we apply the proposed metrics to better understand the commonly used microsoft research paraphrase corpus (mrpc) @xcite dataset. we also study how mrpc differs from paraphrase adversaries from word scrambling (paws) @xcite , another paraphrase identification dataset. in the process, we perform a detailed study on mrpc and propose some revisions to the dataset. we demonstrate that this improves the quality of paraphrase identification models trained on mrpc, with higher transferability to other paraphrase identification datasets. lastly, we demonstrate the applicability of our proposed metrics. by applying our metrics to filter the output of a paraphrase generation model, we show how it can be used to generate specific forms of paraphrases, which can be used as training data for data augmentation purposes and to generate test cases for robustness testing of nlp models. ## related work there have been several survey papers done to better understand the task of paraphrase identification and generation. a survey of paraphrasing and textual entailment methods @xcite presented a comprehensive survey and review on the the aforementioned tasks. in this paper, the authors helped to properly define the tasks and identified some methods and their associated challenges. this was followed up by a more recent survey specifically on the task of paraphrase identification, a survey on paraphrase recognition @xcite , where the focus of the survey was the performance of various statistical and non-deep learning approaches on paraphrase identification on the mrpc dataset. additionally, in on paraphrase identification corpora @xcite , the authors performed a survey of various paraphrase datasets, also highlighting several issues with paraphrase datasets, including mrpc, and providing some recommendations for improving the curation of paraphrase datasets. there have also been previous work on the task of better quantifying various characteristics of paraphrase pairs. in texygen: a benchmarking platform for text generation models @xcite , self-bleu was proposed to measure the diversity in text generation. however, it suffers from limitations inherent to bleu-style metrics: it captures the differences in presence of n-grams, but not their sequence, and is thus mostly limited to capturing the differences of vocabulary, but not the overall structure of a sentence. in paraphrasing with large language models @xcite , rouge-l is used as a measurement of paraphrase diversity, where lower rouge-l scores correspond to greater diversity in paraphrasing generation. however, rouge-l mainly measures degree of similarity in sub-sequences, but not the order in which the sub-sequences occur, and thus cannot accurately capture the possible structural differences present in paraphrase pairs. in our paper, we take a deeper look at some of the issues related to mrpc, proposing some useful improvements. we also build upon previous attempts to characterise paraphrases through the use of quantitative metrics, demonstrating how our proposed metrics can capture various different paraphrasing techniques better than previously proposed metrics. ## application of metrics to demonstrate the applicability of our proposed metrics of wpd and ld, we compare them against other metrics with similar purposes: rogue-l @xcite , self-bleu @xcite and damerau-levenshtein edit distance @xcite . in the examples below, we show that with wpd and ld, we can effectively distinguish between different types of paraphrases that have similar scores via various other metrics. in example pair 1 (figure 3 ), we show that two paraphrases can have very similar rogue-l scores of 0.76 and 0.75, where rouge-l primarily measures the degree of sub-string similarity (longest common sub-strings). however, with wpd, we are able to additionally distinguish the degree in which the similar sub-strings have been shuffled in position, which is a structural alteration to the sentence. in example pair 2 (figure 4 ), we again show that two paraphrases can have very similar self-bleu scores of 0.60 and 0.59, where self-bleu primarily measures the degree n-gram overlap. however, similar to example pair 1, in one of the paraphrases, the two "halves" of the sentence has been swapped in position, and this structural alteration is captured by the wpd score. lastly, in example pair 3 (figure 5), we show that two paraphrases can have very similar damerau-levenshtein edit distance, but feature two completely different types of paraphrasing method. 5.1 comparing mrpc and paws 5.1.1 degree of structural paraphrasing using wpd, we are able to obtain an aggregate view of both the mrpc and paws datasets. we see that both datasets feature similar distributions of structural paraphrasing, where the average amount of structural paraphrasing is fairly low and mrpc features more structural paraphrasing compared to paws. a visualization is provided in figure 6 below. hence, we would expect the mrpc dataset to be somewhat more diverse in structural paraphrases as compared to paws. ## ethical considerations to the best of our knowledge, we do not introduce any ethical concerns in this work. our work is based on the existing mrpc and paws datasets, which are sampled from online news articles as well as wikipedia. hence we expect our findings to generalize well to other english datasets in the general domain. generalization of our work to domains where usage of language is markedly different (for example, in some forms of technical writing) is not certain. when our proposed metrics are used in conjunction with other technology (such as large generative language models), it does not affect the existing ethical considerations of using those technology. ## conclusions and future work in our paper, we have proposed two new metrics to better understand paraphrase pairs: word position deviation (wpd) and lexical deviation (ld). we have applied these metrics to better understand the mrpc and paws datasets, and also to filter the output of a paraphrase generation model to obtain specific forms of paraphrases. however, our metrics still have some limitations, which can be address in future work. although we are able to measure the extent of structural and lexical alterations, we cannot determine the fine-grained type of alterations that is being made, for example, a specific form of structural alteration or word substitution. we anticipate that improvements in this area would be valuable to improve our ability to effectively characterize various properties of paraphrases, leading to better data augmentation and robustness testing approaches that eventually resulting in better performing nlp systems.