Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap (2023)

Abstract: Paraphrase detection is useful in many natural language understanding applications. Current works typically formulate this problem as a sentence pair binary classification task. However, this setup is not a good fit for many of the intended applications of paraphrase models. In particular, such applications often involve finding the closest paraphrases of the target sentence from a group of candidate sentences where they exhibit different degrees of semantic overlap with the target sentence. To apply models to this paraphrase retrieval scenario, the model must be sensitive to the degree to which two sentences are paraphrases of one another. However, many existing datasets ignore and fail to test models in this setup. In response, we propose adversarial paradigms to create evaluation datasets, which could examine the sensitivity to different degrees of semantic overlap. Empirical results show that, while paraphrase models and different sentence encoders appear successful on standard evaluations, measuring the degree of semantic overlap still remains a big challenge for them.

## introduction detecting paraphrases is useful in many natural language understanding applications, such as question answering @xcite , fact checking @xcite , and text summarisation @xcite . researchers have constructed paraphrase identification benchmarks, typically formulating the problem as a sentence pair classification task @xcite @xcite . sentence pairs that have the same or largely equivalent semantics are considered as paraphrases of each other @xcite . for example: a) more than half of the songs were purchased as albums, apple said. b) apple noted that half the songs were purchased as part of albums. not only is it unclear what the criteria is for determining when a sentence pair has sufficiently similar semantics to be considered paraphrases, but as @xcite point out, the standard paraphrase classification task is not a good fit for many of the intended applications of paraphrase models. in particular, such applications are often retrieval tasks that involve finding the closest paraphrases of some target sentences within a set of documents, where candidate sentences exhibit different degrees of semantic overlap with the target sentence. to apply models to a paraphrase retrieval scenario, a paraphrase model must be sensitive to the degree to which two sentences are paraphrases of one another. we use the term partial paraphrase to refer to situations where a sentence pair has some overlap in meaning, but this can range from nearly exact paraphrases to pairs that share very little meaning. an example of an intermediate case is given below: a) more than half of the songs were purchased as albums, apple said yesterday in a meeting with sony. b) apple noted that half the songs were purchased as part of albums. the setup used for standard paraphrase classification can be adapted to the partial paraphrase task, where the softmax confidence score is used as an estimate of the degree to which two sentences are paraphrases of one another. indeed, this has been used in ranking tasks across different domains @xcite @xcite . however, while pre-trained language models have shown good performance on the standard classification task @xcite , as we will show, these models are often fooled by partial paraphrases where there is significant, but far from complete, semantic over-lap. current paraphrase identification datasets do not test models in a partial paraphrase ranking setup. though the semantic textual similarity (sts) tasks @xcite exhibit similarities to this setup as they also try to measure gradations of meaning overlap, there are some significant differences. firstly, the ranking setup in sts concerns comparing completely different sentence pairs (e.g., (a, b) > (c, d)), while most paraphrase applications aim to compare different sentences with the same pivot sentence (e.g., (a, b) > (a, c)). secondly, as @xcite point out that the definition of similarity in sts is rather vague and various complicated relations between sentence pairs all contribute to the similarity score. the difference in the similarity score cannot guarantee the different degree of semantic overlap. our aim, in this paper, is to rectify this deficiency. we draw inspiration from previous adversarial testing works utilising word swapping and number replacement @xcite to produce negative examples. in this work, we propose adversarial paradigms (multiple word swap) to create evaluation datasets that consist of high-quality partial paraphrase pairs with graded semantic overlap. we aim to test whether the paraphrase score produced by existing paraphrase models and sentence encoders is a good reflection of the degree of semantic overlap. in contrast to their strong performance on standard paraphrase classification tasks, our analysis reveals that measuring the degree of semantic overlap still remains a challenge. our main contributions are as follow. first, in section 3, we follow the standard fine-tuning strategy to produce two paraphrase models and then demonstrate their good performance on standard evaluation tasks and insensitivity to partial paraphrases. we then present (in section 4) evaluation datasets which consist of high-quality partial paraphrase pairs with graded semantic overlap, constructed by multiple word swapping. we further show (in section 5) that the distinction between partial paraphrase and exact paraphrase is a challenge for paraphrase models, and that their paraphrase scores are not a good reflection of the degree of semantic overlap. finally, our work demonstrates that similarity scores produced by sentence encoders, though being widely used as a measure of similarity in meaning, are dominated by the degree of lexical overlap, and are poor estimators of the degree to which sentences are partial paraphrases. ## related work the definition of paraphrase has been long debated, as have the characteristics of paraphrase pairs @xcite @xcite . a widely accepted definition is that two sentences should exhibit the same or largely equivalent semantics, which suggests a bi-directional entailment relation. as @xcite pointed out, paraphrases may occur at different levels, such as word-level, phrase-level, and sentence-level. although there has been some work that concerned the identification of lexical and phrasal paraphrases @xcite , most recent work on paraphrase identification has been performed at the sentence level, and has involved determining whether a given sentence pair is a paraphrase or not in a classification setup @xcite @xcite @xcite . however, paraphrase detection has been utilised in other nlp tasks. in question answering, @xcite utilised paraphrase detection to discover most probable paraphrases of a given question from a group of potential paraphrases by comparing their paraphrase scores. similarly, @xcite integrated paraphrase detection in a information retrieval system to select the best paraphrased queries which are used to expand the original query list. accordingly, @xcite argued that the standard binary classification setup of paraphrase identification is ill-suited to many real-world applications which involve paraphrase retrieval. to apply paraphrase models to such a retrieval scenario, the model must be sensitive to the degree to which two sentences share semantic content. though pre-trained language models show good performance when fine-tuned on paraphrase identification datasets @xcite @xcite , a performance drop is often observed when being tested for robustness under different adversarial scenarios. zhang et al. (2019b) utilised word swapping and backtranslation to produce adversarial examples. yang et al. (2019) adopted the same approach to produce adversarial pairs in a multilingual scenario. shi and huang (2020) modified shared words to produce both positive and negative pairs. wang et al. (2021) additionally proposed a robustness evaluation platform which can perform different transformations to sentence pairs, including word swapping, template-based generation and number replacement. nighojkar and licato (2021) employed paraphrase generators to produce sentence pairs that are both lexically and syntactically disparate. such transformations can create partial paraphrases in different types. however, these partial paraphrases do not exhibit decreasing semantic overlap. to measure the sensitivity of models to different degrees of semantic overlap, we draw inspiration from them and create a list of partial paraphrases with decreasing semantic overlap for each paraphrase pair. ## background and preliminaries the classification setup for the evaluation of paraphrase identification involves identifying whether the given sentence pair is a paraphrase or not. in this section, we follow previous work and first finetune bert @xcite and roberta @xcite on widely used paraphrase datasets to produce standard paraphrase models and check whether their success on standard evaluation benchmarks could transfer to the recognition of partial paraphrases with different degrees of semantic overlap. ## partial paraphrase construction to create partial paraphrases at decreasing degrees of semantic overlap, while maintaining lexical overlap and sentence length, we draw inspiration from @xcite and @xcite who create negative examples by swapping words and entities. we take positive sentence pairs from 5 our initial experiments show that sentence encoders can achieve extremely high performance on ranking these sentence pairs by capturing such clues. paws test sets and create corresponding partial paraphrases with graded semantic overlap by making multiple word swaps. since the semantics are equivalent for positive sentence pairs, we always make modifications to sentence b to produce partial paraphrase variants and compare them with the original sentence a. this can increase the task difficulty as the lexical overlap will be high for negative pairs. models that produce high scores based on high lexical overlap are likely to fail in this scenario. figure 2 illustrates the multi-swap procedure. given a paraphrase pair (a, b), we first perform part-of-speech tagging foot_3 (pos) on b to obtain tags for each word. we further detect named entities like locations, person names, organisations, and dates using a named entity tagger, and replace pos tags with entity tags when there is overlap. words and phrases that have the same tag foot_4 are then grouped together. we deduplicate each group to avoid swapping the position between two identical words/phrases. given that a swap requires at least two words/phrases, we discard tag groups that have less than two words/phrases. in order to produce enough candidates for ranking, we filter out sentences that have less than 3 tag groups. for each sentence with at least three tag groups, we randomly sample three groups, and from each group we randomly sample two words/phrases. in the end, we swap the position of sampled words/phrases to produce swapped sentences. we perform each swap based on previous swaps, with a maximum of three swaps. in summary, given a positive sentence pair (a, b), we apply our multi-swap strategy on b, and produce a group of sentence pairs @xcite , ( b1swap , b), ( b2swap , b), ( b3swap , b)], where they exhibit decreasing semantic overlap. the statistics of the resulting evaluation datasets are given in ## experiments we compare previous fine-tuned paraphrase models (bert and roberta in section 3) with sentence encoders. sentence encoders, such as sbert @xcite and sim-cse @xcite , are widely used in various ranking scenarios which aim to measure the similarity in meaning between two sentences. they use a contrastive learning objective, intended to derive high-quality sentence representations by pulling sentences with similar semantics closer together and pushing dissimilar ones apart. although they have achieved relatively good performance on sts tasks, it is unclear whether the similarity score they produce can be used to measure the extent to which sentence pairs are paraphrases. in this experiment, we evaluate simcse foot_5 , two variants of simcse, namely, simcse+pas @xcite and simcse+bertscore foot_6 @xcite , and two sbert models 10 (reimers and gurevych, 2020) which are specifically trained on paraphrase datasets. we denote one as sbert v1 11 and the other as sbert v2 12 . for paraphrase models, we use its softmax confidence of being positive as the paraphrase score to rank sentence pairs. for sentence encoders, we rank sentence pairs using their default strategy to produce a paraphrase score. specifically, sbert and simcse utilise the cosine similarity between two sentences; simcse+pas increases the interaction between two sentences by considering the aggregated score over predicate-argument alignments; and simcse+bertscore considers the idf-weighted f1 measure in terms of word matching. ## conclusion in this paper, we explore whether paraphrase scores produced by paraphrase models and sentence encoders are reliable indicators of the degree to which two sentences share semantic content. accordingly, we propose an adversarial paradigms (multiple word swap) to create evaluation datasets that consist of high-quality partial paraphrases with graded semantic overlap in a ranking setup. our experimental results show that the similarity score produced by sentence encoders is not a good indicator of how close two sentences are to being exact paraphrases, and is heavily affected by lexical overlap. whilst paraphrase models show generally better performance, the confidence scores they produce are still far from acting as a reliable indicator to different degrees of semantic overlap. measuring the degree of semantic overlap between two sentences remains a significant challenge. our future work includes producing larger ranking datasets and extending this paradigm to other relevant datasets.