Evaluating the Examiner: The Perils of P earson Correlation for Validating Text Similarity Metrics (2022)

Abstract: In recent years, researchers have developed question-answering based approaches to automatically evaluate system summaries, reporting improved validity compared to word overlap-based metrics like ROUGE, in terms of correlation with human ratings of criteria including fluency and hallucination. In this paper, we take a closer look at one particular metric, QuestEval, and ask whether: (1) it can serve as a more general metric for long document similarity assessment; and (2) a single correlation score between metric scores and human ratings, as the currently standard approach, is sufficient for metric validation. We find that correlation scores can be misleading, and that score distributions and outliers should be taken into account. With these caveats in mind, QuestEval can be a promising candidate for long document similarity assessment.

## introduction methods which can provide accurate estimates of document content similarity are critical to tasks such as news analysis and fact-checking @xcite . researchers have proposed a broad range of metrics to estimate document similarity @xcite , from n-gram overlap metrics such as bleu @xcite and meteor @xcite for machine translation, and rouge @xcite for automatic summarisation, to embedding-based metrics such as bertscore @xcite and moverscore @xcite . however, these metrics have been shown to rely heavily on superficial features, correlate poorly with human annotations, and perform poorly over longer document pairs @xcite @xcite . a more radical recent proposal has been to use question-answering (qa) based models @xcite , to automatically generate question-answer pairs from a source document, and estimate similarity by the proportion of questions that can be successfully answered on the basis of the target document. while such approaches were designed to evaluate automatic summarisation in a reference-free manner, i.e., compare a full (long) document with its (short) summary, they can in principle be applied to arbitrary document pairs. in this paper we ask whether the questeval method @xcite scales to varying-length document pairs, and in particular, can be used to calculate the similarity between same length documents reliably. in other words, we are comparing two evaluation settings: longshort document pairs vs. documents of the same length. in our contributions are as follows: (1) we eval-uate questeval on three different datasets, and demonstrate that it is robust to increasing document lengths; ## experimental setup here, we describe the datasets and evaluation methods we use to test questeval's applicability to long documents, as well as reliability across datasets and reference annotations. ## results in analysing the results, we investigate: (1) whether questeval is document-length agnostic, i.e., scales from the original scenario of article-summary (long-short) similarity to estimating article-article (long-long) similarity in terms of raw pearson correlation scores; (2) whether questeval correlates with ratings of document similarity, departing from the dimensions of coherence, consistency, fluency, and relevance as originally assessed; and (3) how robust the observed pearson and spearman correlations are across all data conditions and ground-truth labels. questeval as a measure of long document similarity the correlation coefficients reported in ## analysis and discussion from our results we can observe that summarisation evaluation metrics and more specifically, questeval have utility for tasks beyond summarisation, especially where there is no access to gold human annotations. in our case, we showed that questeval scores do correlate with the overall news article similarity scores of semeval. however, this is not the case for every metric, as we were also able to show with dimensions like document sim-ilarity, consistency, and fluency. moreover, we showed that in isolation pearson correlation coefficients with human ratings are not a reliable signal for the quality of an evaluation metric, due to their sensitiveness to outliers. we recommend to visualise score distributions in tandem with calculating the correlation to ensure that it is not affected by a minority of outliers. this is consistent with the observations of @xcite in their analysis of wmt task results. we observed that questeval scores are distributed in the range of 0-1 for almost all datasets/measurements except for abc news, motivating us to look more closely at this dataset. in the appendix we present some examples with high document similarities but low questeval scores. while we are aware that questeval values are lower than expected for those examples, the similarity rating is also arguable. for both cases, almost none of the entities overlap in the depicted documents; this could be the reason why questeval scores are low. we also propose to take into consideration several correlation coefficients as we show in ## conclusion in this paper we investigated whether automatic qa-based metrics for summarisation evaluation can be adopted to compare long documents. we also conducted a more detailed evaluation of the robustness of pearson correlation for similarity metric evaluation, and found that correlation-based metrics need to be validated by plotting and understanding labels and score distributions. in future work, we plan to extend our work to different languages.