Improving Paraphrase Detection with the Adversarial Paraphrasing Task (2021)

Abstract: If two sentences have the same meaning, it should follow that they are equivalent in their inferential properties, i.e., each sentence should textually entail the other. However, many paraphrase datasets currently in widespread use rely on a sense of paraphrase based on word overlap and syntax. Can we teach them instead to identify paraphrases in a way that draws on the inferential properties of the sentences, and is not over-reliant on lexical and syntactic similarities of a sentence pair? We apply the adversarial paradigm to this question, and introduce a new adversarial method of dataset creation for paraphrase identification: the Adversarial Paraphrasing Task (APT), which asks participants to generate semantically equivalent (in the sense of mutually implicative) but lexically and syntactically disparate paraphrases. These sentence pairs can then be used both to test paraphrase identification models (which get barely random accuracy) and then improve their performance. To accelerate dataset generation, we explore automation of APT using T5, and show that the resulting dataset also improves accuracy. We discuss implications for paraphrase detection and release our dataset in the hope of making paraphrase detection models better able to detect sentence-level meaning equivalence.

## introduction although there are many definitions of 'paraphrase' in the nlp literature, most maintain that two sentences that are paraphrases have the same meaning or contain the same information. @xcite define paraphrasing as "expressing the same information in multiple ways" and @xcite call paraphrases "alternative ways of conveying the same information." @xcite write that "paraphrases are differing textual realizations of the same meaning." a definition that seems to sufficiently encompass the others is given by @xcite : "paraphrases are sentences or phrases that use different wording to convey the same meaning." however, even that definition is somewhat imprecise, as it lacks clarity on what it assumes 'meaning' means. if paraphrasing is a property that can hold between sentence pairs, foot_0 then it is reasonable to assume that sentences that are paraphrases must have equivalent meanings at the sentence level (rather than exclusively at the levels of individual word meanings or syntactic structures). here a useful test is that recommended by inferential role semantics or inferentialism @xcite , which suggests that the meaning of a statement s is grounded in its inferential properties: what one can infer from s and from what s can be inferred. building on this concept from inferentialism, we assert that if two sentences have the same inferential properties, then they should also be mutually implicative. mutual implication (mi) is a binary relationship between two sentences that holds when each sentence textually entails the other (i.e., bidirectional entailment). mi is an attractive way of operationalizing the notion of two sentences having "the same meaning," as it focuses on inferential relationships between sentences (properties of the sentences as wholes) instead of just syntactic or lexical similarities (properties of parts of the sentences). as such, we will assume in this paper that two sentences are paraphrases if and only if they are m i. 2 in nlp, modeling inferential relationships between sentences is the goal of the textual entailment, or natural language inference (nli) tasks @xcite . we test mi using the version of roberta large released by @xcite trained on a combination of snli @xcite , multinli @xcite , fever-nli @xcite , and anli @xcite . owing to expeditious progress in nlp research, performance of models on benchmark datasets is 'plateauing' -with near-human performance often achieved within a year or two of their releaseand newer versions, using a different approach, are constantly having to be created, for instance, glue @xcite and superglue @xcite . the adversarial paradigm of dataset creation @xcite @xcite has been widely used to address this 'plateauing,' and the ideas presented in this paper draw inspiration from it. in the remainder of this paper, we apply the adversarial paradigm to the problem of paraphrase detection, and demonstrate the following novel contributions: we do this by collecting paraphrases that are mi but are as lexically and syntactically disparate as possible (as measured by low bleurt scores). we call this the adversarial paraphrasing task (apt). ## related work paraphrase detection (given two sentences, predict whether they are paraphrases) (zhang and patrick, 2005; @xcite @xcite is an important task in the field of nlp, finding downstream applications in machine translation @xcite @xcite ), text summarization, plagiarism detection @xcite , question answering, and sentence simplification @xcite . paraphrases have proven to be a crucial part of nlp and language education, with research showing that paraphrasing helps improve reading comprehension skills @xcite . question paraphrasing is an important step in knowledgebased question answering systems for matching questions asked by users with knowledge-based assertions @xcite . paraphrase generation (given a sentence, generate its paraphrase) @xcite is an area of research benefiting paraphrase detection as well. lately, many paraphrasing datasets have been introduced to be used for training and testing ml models for both paraphrase detection and generation. msrp @xcite ) contains 5801 sentence pairs, each labeled with a binary human judgment of paraphrase, created using heuristic extraction techniques along with an svm-based classifier. these pairs were annotated by humans, who found 67% of them to be semantically equivalent. the english portion of ppdb @xcite contains over 220m paraphrase pairs generated by meaning-preserving syntactic transformations. paraphrase pairs in ppdb 2.0 @xcite include fine-grained entailment relations, word embedding similarities, and style annotations. twitterppdb @xcite consists of 51,524 sentence pairs captured from twitter by linking tweets through shared urls. this ap-proach's merit is its simplicity as it involves neither a classifier nor a human-in-the-loop to generate paraphrases. humans annotate the pairs, giving them a similarity score ranging from 1 to 6. paranmt @xcite was created by using neural machine translation to translate the english side of a czech-english parallel corpus (czeng 1.6 @xcite ), generating more than 50m english-english paraphrases. however, paranmt's use of machine translation models that are a few years old harms its utility @xcite , considering the rapid improvement in machine translation in the past few years. to rectify this, we use the google-translate library to translate the czech side of roughly 300k czeng2.0 @xcite sentence pairs ourselves. we call this dataset paraparanmt (pp-nmt for short, where the extra paraprefix reflects its similarity to, and conceptual derivation from, paranmt). some work has been done in improving the quality of paraphrase detectors by training them on a dataset with more lexical and syntactic diversity. @xcite propose a paraphrase generation algorithm that penalizes the production of n-grams present in the source sentence. our approach to doing this is with the apt, but this is something worth exploring. @xcite use a machine translation model to generate paraphrases much like paranmt. an interesting application of paraphrasing has been discussed by @xcite who, given a sentence in one language, generate a diverse set of correct translations (paraphrases) that humans are likely to produce. in comparison, our work is focused on generating adversarial paraphrases that are likely to deceive a paraphrase detector, and models trained on the adversarial datasets we produce can be applied to mayhew et al.'s work too. anli @xcite , a dataset designed for natural language inference (nli) @xcite , was collected via an adversarial human-andmodel-in-the-loop procedure where humans are given the task of duping the model into making a wrong prediction. the model then tries to learn how not to make the same mistakes. aflite (bras et al., 2020) adversarially filters dataset biases making sure that the models are not learning those biases. they show that model performance on snli @xcite drops from 92% to 62% when biases were filtered out. however, their approach is to filter the dataset, which reduces its size, making model training more difficult. our present work tries instead to generate adversarial examples to increase dataset size. other examples of adversarial datasets in nlp include work done by @xcite ; @xcite . perhaps the closest to our work is paws @xcite , short for paraphrase adversaries from word scrambling. the idea behind paws is to create a dataset that has a high lexical overlap between sentence pairs without them being 'paraphrases.' it has 108k paraphrase and non-paraphrase pairs with high lexical overlap pairs generated by controlled word swapping and back-translation, and human raters have judged whether or not they are paraphrases. including paws in the training data has shown the state-of-the-art models' performance to jump from 40% to 85% on paws's test split. in comparison to the present work, paws does not explicitly incorporate inferential properties, and we seek paraphrases minimizing lexical overlap. ## adversarial paraphrasing task (apt) semantic textual similarity (sts) measures the degree of semantic similarity between two sentences. popular approaches to calculating sts include bleu @xcite , bertscore @xcite , and bleurt @xcite . bleurt is a text generation metric building on bert's @xcite contextual word representations. bleurt is warmed-up using synthetic sentence pairs and then fine-tuned on human ratings to generalize better than bertscore @xcite . given any two sentences, bleurt assigns them a similarity score (usually between -2.2 to 1.1). however, high sts scores do not necessarily predict whether two sentences have equivalent meanings. consider the sentence pairs in ## experiments to quantify our datasets' contributions, we designed experiment setups wherein we trained roberta base @xcite for paraphrase detection on a combination of twitterppdb and our datasets as training data. roberta was chosen for its generality, as it is a commonly used model in current nlp work and benchmarking, and currently achieves sota or near-sota results on a majority of nlp benchmark tasks @xcite ; ## discussions and conclusions this paper introduced apt (adversarial paraphrasing task), a task that uses the adversarial paradigm to generate paraphrases consisting of sentences with equivalent (sentence-level) meanings, but differing lexical (word-level) and syntactical similarity. we used apt to create a human-generated dataset / benchmark (ap h ) and two machinegenerated datasets (ap m t 5 and ap t w t 5 ). our goal was to effectively augment how paraphrase detectors are trained, in order to make them less reliant on word-level similarity. in this respect, the present work succeeded: we showed that roberta base trained on twitterppdb performed poorly on apt benchmarks, but this performance was increased significantly when further trained on either our human-or machine-generated datasets. the code used in this paper along with the dataset has been released in a publicly-available repository. 4 paraphrase detection and generation have broad applicability, but most of their potential lies in areas in which they still have not been substantially applied. these areas range from healthcare (improving accessibility to medical communications or concepts by automatically generating simpler language), writing (changing the writing style of an article to match phrasing a reader is better able to understand), and education (simplifying the language of a scientific paper or educational lesson to make it easier for students to understand). thus, future research into improving their performance can be very valuable. but approaches to paraphrase that treat it as no more than a matter of detecting word similarity overlap will not suffice for these applications. rather, the meanings of sentences are properties of the sentences as a whole, and are inseparably tied to their inferential properties. thus, our approaches to paraphrase detection and generation must follow suit. the adversarial paradigm can be used to dive deeper into comparing how humans and sota language models understand sentence meaning, as we did with apt. furthermore, automatic generation of adversarial datasets has much unrealized potential; e.g., different datasets, paraphrase generators, and training approaches can be used to generate future versions of ap t 5 in order to produce ap t passing sentence pairs with lower lexical and syntactic similarities (as measured not only by bleurt, but also by future state-of-the-art sts metrics). the idea of more efficient automated adversarial task performance is particularly exciting, as it points to a way language models can improve themselves while avoiding prohibitively expensive human participant fees. finally, the most significant contribution of this paper, apt, presents a dataset creation method for paraphrases that will not saturate because as the models get better at identifying paraphrases, we will improve paraphrase generation. as models get better at generating paraphrases, we can make apt harder (e.g., by reducing the bleurt threshold @xmath0.5). one might think of this as students in a class who come up with new ways of copying their assignments from sources as plagiarism detectors improved. that brings us to one of the many applications of paraphrases: plagiarism generation and detection, which inherently is an adversarial activity. until plagiarism detectors are trained on adversarial datasets themselves, we cannot expect them to capture human levels of adversarial paraphrasing.