Pointwise Paraphrase Appraisal is Potentially Problematic (2020)

Abstract: The prevailing approach for training and evaluating paraphrase identification models is constructed as a binary classification problem: the model is given a pair of sentences, and is judged by how accurately it classifies pairs as either paraphrases or non-paraphrases. This pointwise-based evaluation method does not match well the objective of most real world applications, so the goal of our work is to understand how models which perform well under pointwise evaluation may fail in practice and find better methods for evaluating paraphrase identification models. As a first step towards that goal, we show that although the standard way of fine-tuning BERT for paraphrase identification by pairing two sentences as one sequence results in a model with state-of-the-art performance, that model may perform poorly on simple tasks like identifying pairs with two identical sentences. Moreover, we show that these models may even predict a pair of randomly-selected sentences with higher paraphrase score than a pair of identical ones.

## introduction paraphrase identification is a well-studied sentence pair modeling task that refers to the problem of determining whether two sentences are semantically equivalent. detecting paraphrases can be very useful for many nlp applications such as machine translation (mt), question answering (qa), and information retrieval (ir). in a qa system, we would like to find the most probable question paraphrases from a database of question answer pairs for a given input question @xcite . in a mt model, we would like to obtain the best translation by comparing the target sentence to a list of translated sentences. even though pretrained language models have reached state-of-theart performance on paraphrase identification tasks, the current problem setup is insufficient to produce models with consistent and robust performance on unseen samples and real world problems. the typical current problem setup for paraphrase identification is different from intended uses in real world applications. they often involve finding best paraphrases from a group of documents given a particular query, rather than just determining whether two sentences are paraphrases of each other. besides, getting the order and identifying the most relevant documents is usually more important than getting the binary decision of a pair of sentences @xcite . however, to make the task simpler, current methods and existing datasets such as quora question pairs (qqp) @xcite and microsoft research paraphrase corpus (mrpc) @xcite are all framed as a binary classification problem at the sentence pair level. contributions as a first step to improve the way paraphrase identification is evaluated for ranking tasks, we analyze some of the anomalies found in the current pointwise task setting. we first demonstrate the standard way of fine-tuning bert for pointwise paraphrase evaluation makes the model sometimes fails on simple problems including recognizing two identical sentences and reversing the order of two sentences in a pair (section 3). we find that it performs worse than a bag-of-words model due to its asymmetrical model architecture. lastly, we show that the model may fail to capture the correct relative order of two sentence pairs using the pointwise approach, sometimes even predicting a pair of random sentences with a higher paraphrase score than a pair of identical ones (section 4). ## background this section provides background on the paraphrase identification task, evaluation methods, and the datasets and models we use in our experiments. ## asymmetry for semantic matching tasks, the bert paraphrase identification model considers two sentences (s 1 , s 2 ) as a single sequence by concatenating them with a separator token. however, due to this asymmetrical approach, the sequence representations before the final classification layer would be entirely different if we permute the order of the two input sentences. we explore two implications of this method for identifying paraphrases: sensitivity to input order (section 3.1) and possibility of considering identical sentences non-paraphrases (section 3.2). ## problems with pointwise evaluation for a given query sentence, we assume that a well-generalized paraphrase identification model should output a higher paraphrase score to the query sentence itself than a randomly-selected sentence. however, models trained with pointwise evaluation cannot learn the relative order based on the degree of semantic equivalence. we test this by considering how often models recognize a random sentence as more similar than the query sentence itself, and looking at the distribution of paraphrase scores across a dataset. ## conclusion although the state-of-art paraphrase identification models can achieve impressive performance under the pointwise evaluation method, they cannot handle real-world problems and unseen data well and even have worse results than a bow model on simple tasks. we show that the asymmetry in bert can produce inconsistent prediction results when reversing the order of the two sentences. we examined the relation of semantic equivalence learned by models trained with pointwise approach, and found that they may consider a random sentence as more similar to the query sentence itself. this suggests future work to reconsider how to match the training and evaluation to the actual objective of downstream applications, and thus create more reliable evaluation metrics and benchmarks.