Paraphrase Types for Generation and Detection (2023)

Abstract: Current approaches in paraphrase generation and detection heavily rely on a single general similarity score, ignoring the intricate linguistic properties of language. This paper introduces two new tasks to address this shortcoming by considering paraphrase types - specific linguistic perturbations at particular text positions. We name these tasks Paraphrase Type Generation and Paraphrase Type Detection. Our results suggest that while current techniques perform well in a binary classification scenario, i.e., paraphrased or not, the inclusion of fine-grained paraphrase types poses a significant challenge. While most approaches are good at generating and detecting general semantic similar content, they fail to understand the intrinsic linguistic variables they manipulate. Models trained in generating and identifying paraphrase types also show improvements in tasks without them. In addition, scaling these models further improves their ability to understand paraphrase types. We believe paraphrase types can unlock a new paradigm for developing paraphrase models and solving tasks in the future.

## introduction paraphrases are texts expressing identical meanings that use different words or structures @xcite @xcite . paraphrases exhibit humans' complex language's nature and diversity, as there are infinite ways to transform one text into another without altering its meaning. for example, one can change a text's morphology: "who they could might be?", syntax: "he drew a go was drawn by him.", lexicon: "she liked enjoyed it.". nonetheless, current paraphrase generation and detection systems are yet unaware of the lexical variables they manipulate @xcite . generative models cannot be asked to perform certain types of perturbations, and detection models are unable to understand which paraphrase types they detect @xcite @xcite , or they learn limited language aspects (e.g., primarily syntax @xcite @xcite ). the shallow notion of what composes paraphrases used by these systems limits their understanding of the task and makes it challenging to interpret detection decisions in practice. for example, although high structural and grammatical similarities can indicate plagiarism, detection systems are often not concerned with the aspects that make two texts segments alike @xcite . given the limitations of current paraphrase generation and detection tasks, their proposed solutions are also constrained @xcite . figure 1 gives an example of the difference between current tasks and their true linguistic diversity. therefore, we propose two new tasks to explore the role of paraphrase types, namely paraphrase type generation and paraphrase type detection. in the generation task, a model has to generate a paraphrased text for specific segments considering multiple paraphrase types ( §3.1). for the detection task, paraphrased segments must be classified into one or more paraphrase types (e.g., lexico-syntactic-based changes) ( §3.2). these tasks complement existing ones (without paraphrase types), enabling more granular assessments of paraphrased content. the shift from traditional paraphrase tasks towards a more specific scenario, i.e., including paraphrase types, has many benefits. a direct consequence of incorporating paraphrase types, and maybe the most impactful, lies in plagiarism detection. plagiarism detection systems based on machine learning often support their decision results using shallow high-level similarity metrics that only indicate how much a given text is potentially plagiarized, thus limiting their analysis @xcite . incorporating paraphrase types allows for more interpretable plagiarism detection systems, as more informative and precise results can be derived. additionally, automated writing assistants can be improved beyond simple probability distributions when suggesting alterations to a text. on top of that, second-language learners can correct their texts by considering specific paraphrase types in their daily lives (e.g., learning about contractions "does @xmath0, helping them to learn new languages faster. our proposed tasks indicate that language models struggle to generate or detect paraphrase types with acceptable performance, underlining the challenging aspect of finding the linguistic aspects within paraphrases. however, learning paraphrase types is beneficial in generation and detection as the performance of trained models consistently increases for both tasks. in addition, scaling models also suggest improvements in their ability to understand and differentiate paraphrase types when transferring to unseen paraphrasing tasks. in summary, we: first attempts to categorize the lexical variables manipulated in paraphrases into a taxonomy have been performed by @xcite , followed by their first typology annotated corpus @xcite . gold et al. (2019) categorize paraphrases on a higher level as meaning relations and present three additional categories: textual entailment, specificity, and semantic similarity. kovatchev et al. (2020) extend vila et al. (2015, 2014)'s typology and re-annotate the mrpc-a (dolan and brockett, 2005) corpus with fine-grained annotations with more than 26 lexical categories, such as negation switching and spelling changes in the etpc dataset. recent works model objective functions instead of taxonomies (e.g., word position deviation, lexical deviation) to automatically categorize paraphrases (liu and soh, 2022a). this approach is similar to bandel et al. (2022); liu et al. (2020a)'s proposed metrics for paraphrase quality (e.g., semantic similarity, expression diversity). recent work requires texts to satisfy certain stylistic, semantic, or structural requirements, such as using formal language or expressing thoughts using a particular template @xcite . in paraphrase generation, methods require texts to meet certain quality criteria, such as semantic preservation and lexical diversity @xcite or require syntactic criteria, such as word ordering @xcite @xcite . the development of the multi-topic paraphrase in twitter (multipit) corpus addresses quality issues in existing paraphrase datasets and facilitates the acquisition and generation of high-quality paraphrases @xcite . parse-instructed prefix (pip) tunes large pre-trained language models for generating paraphrases according to specified syntactic structures in a low-data setting, significantly reducing training costs compared to traditional finetuning methods @xcite . although current contributions to generate and detect different paraphrase forms, they do not use them to generate or detect paraphrase types directly. instead, they rely on shallow similarity measures and binary labels for identifying paraphrases. in this work, we propose two new tasks. one for generating specific perturbations when creating new paraphrases and one for detecting the lexical differences between paraphrases. we use the etpc dataset to evaluate these tasks and show that learning paraphrase types is more challenging than considering the binary notion of paraphrase. our results suggest that learning paraphrase types is beneficial for traditional paraphrase generation and detection. we further demonstrate these findings in our experiments ( §4). ## task formulation most paraphrase-related tasks focus on generating or classifying paraphrases at a general level @xcite @xcite . this goal is limited, as it provides little details on what composes a paraphrase or which aspects make original and paraphrase alike @xcite . we believe incorporating paraphrase types in generation-and detection-related tasks can help understand paraphrasing better. we propose specific tasks for paraphrase generation and detection to include paraphrase types. the goal of paraphrase type generation is to generate a paraphrased text that preserves the semantics of the source text but differs in certain linguistic aspects. these linguistic aspects are specific paraphrase types (e.g., lexicon change). in the paraphrase type detection task, the goal is to locate and identify the paraphrase types in which two pieces of the text differ. both tasks aim to include a fine-grained understanding of paraphrase types over current existing tasks. each specific task can also be formulated as a simple paraphrase generation or detection task (i.e., without paraphrase types) with a small error ϵ. thus, our tasks complement existing ones in paraphrase generation and detection. section 4 provides more information on the composition of the proposed datasets, their splits, and their structure. figure 2 illustrate our proposed tasks. ## experiments in our experiments, we first investigate the distinct differences in paraphrase types by evaluating their correlations. next, we measure how much language models already know about paraphrase types and how that changes when scaling them. finally, we empirically study how existing models perform in our proposed tasks. to test whether paraphrase types are a valuable extension to traditional paraphrasing tasks, we evaluate selected models after incorporating paraphrase types in their training to quantify the transfer from paraphrase types to traditional paraphrase tasks.