On the Evaluation Metrics for Paraphrase Generation (2022)

Abstract: In this paper we revisit automatic metrics for paraphrase evaluation and obtain two findings that disobey conventional wisdom: (1) Reference-free metrics achieve better performance than their reference-based counterparts. (2) Most commonly used metrics do not align well with human annotation.Underlying reasons behind the above findings are explored through additional experiments and in-depth analyses.Based on the experiments and analyses, we propose ParaScore, a new evaluation metric for paraphrase generation. It possesses the merits of reference-based and reference-free metrics and explicitly models lexical divergence. Based on our analysis and improvements, our proposed reference-based outperforms than reference-free metrics.Experimental results demonstrate that ParaScore significantly outperforms existing metrics.

## introduction paraphrase generation is a fundamental problem in natural language processing (nlp), which has been widely applied in versatile tasks, such as question answering @xcite @xcite , machine translation @xcite @xcite , and semantic parsing @xcite @xcite . in particular, paraphrase generation is also an important module in many writing assistant systems, such as quillbot foot_0 , wordtune foot_1 and effidit foot_2 @xcite . recent years have witnessed rapid development in paraphrase generation algorithms @xcite @xcite . however, little progress has been made in the automatic evaluation of this task. it is even unclear which metric is more reliable among many widely used metrics. most evaluation metrics used in previous paraphrase generation research are not designed for the task itself, but adopted from other evaluation tasks, such as machine translation (mt) and summarization. however, paraphrase evaluation is inherently different from the evaluation of most other tasks, because a good paraphrase typically obeys two criteria @xcite @xcite : semantic similarity (sim) and lexical divergence @xcite . sim means that the paraphrase maintains similar semantics to the input sentence, whereas div requires that the paraphrase possesses lexical or syntactic differences from the input. in contrast, tasks like machine translation have no requirement for div. it is therefore uncertain whether the metrics borrowed from other tasks perform well in paraphrase evaluation. in this paper, we revisit automatic metrics for paraphrase evaluation. we collect a list of popular metrics used in recent researches @xcite @xcite @xcite , and computed their correlation with human annotation. generally, these metrics fall into two categories, i.e., reference-based and reference-free metrics. the former is utilized much more frequently than the latter. we first empirically quantify the matching degree between metric scores and human annotation, on two datasets of different languages. upon both benchmarks, we make comprehensive experiments to validate the reliability of existing metrics. surprisingly, we obtain two important findings: (1) reference-free metrics better align with human judgments than reference-based metrics on our benchmarks, which is counter-intuitive in related evaluation tasks. (2) most of these metrics (especially the commonly-used bleu and rouge) do not agree well with human evaluation. then we explore the potential reasons behind the above findings through additional experiments. for the first finding, we demonstrate that the performance comparison between reference-free and reference-based metrics is largely affected by the input-candidate and reference-candidate distance distribution. specifically, reference-free metrics are better because most paraphrase candidates in the testset have larger lexical distances to the reference than to the input, but reference-based metrics may be better for the minority candidates. to study the second finding, we design an approach based on attribution analysis @xcite to decouple the factors of semantic similarity and lexical divergence. our experiments and analysis show that existing metrics measure semantic similarity well, but tend to ignore lexical divergence. based on our analyses, we propose a new family of metrics named parascore for paraphrase evaluation, which takes into account the merits from both reference-based and reference-free metrics and explicitly models lexical divergence. extensive experiments show that our proposed metrics significantly outperform the ones employed in previous research. in summary, our main contributions are: foot_3 ## reference-free vs. reference-based the results in the previous section indicate that reference-free metrics typically have better performance than their reference-based counterpart. in this section, we investigate this finding by answering the following question: when and why do reference-free metrics outperform their referencebased variants? ## decoupling semantic similarity and lexical divergence in this section, we investigate why most metrics do not align well with human annotation. ## conclusion this paper first reviews the reliability of existing metrics for paraphrasing evaluation by investigating how well they correlate with human judgment. then, we find two interesting findings and further ask two questions behind them that are overlooked by the community: (1) why do reference-free metrics outperform reference-based ones? ( 2 ) what is the limitation of existing metrics? we deliver detailed analyses of such two questions and present the explanation by disentangling paraphrase quality. based on our analyses, finally, we propose parascore (with both reference-based and reference-free implementations) for paraphrase evaluation, and its effectiveness is validated through comprehensive experiments. in addition, we call for building better benchmarks which can faithfully reflect the importance of lexical divergence in paraphrase evaluation; we hope it will shed light on the future direction.