Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora (2022)

Abstract: Similarity metrics for text corpora are becoming critical due to the tremendous growth in the number of generative models. These similarity metrics measure the semantic gap between human and machine-generated text on the corpus level. However, standard methods for evaluating the characteristics of these metrics have yet to be established. We propose a set of automatic measures for evaluating the characteristics of semantic similarity metrics for text corpora. Our measures allow us to sensibly compare and identify the strengths and weaknesses of these metrics. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by comparing it to a collection of classical and state-of-the-art metrics. Our measures revealed that recent metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.

## introduction while there has been a long-standing interest in developing semantic similarity metrics foot_0 @xcite , measuring how close two text corpora are remains an open problem @xcite . specifically, the recent advances in generative language models have led to an increased interest in the study of content similarity between human and generated language, as a mean for comparing the quality of generative models @xcite . the goal of a text corpus' dissimilarity or distance metric is to provide a broad representation of distance across specific linguistic aspects, such as lexical, morphological, syntactic, and semantic @xcite . such metrics are essential for measuring how well corpus-based linguistic analysis generalizes from one data-set to another. this work focuses on semantic similarity metrics. while one can reasonably measure the semantic distance between two individual sentences (e.g., by calculating the cosine distance between the sentence embeddings), measuring the dissimilarity between two text corpora remains a challenge @xcite . corpus-level metrics seek to assess semantic similarity at the group level. for instance, assessing generated text fidelity, diversity, and coverage compared to the reference corpus @xcite . thus, one common approach for measuring the semantic dissimilarity between two corpora is to compare the densities of their sentences in the embedding space @xcite . however, there are no standard automatic procedures for evaluating the precision and robustness of such similarity metrics. the semi-manual standard approach is to correlate the results of these metrics for human judgement. however, leveraging manual human judgements to construct numeric metrics has significant weaknesses. as we explain in section 2, human judgements are expensive to obtain, are difficult to aggregate consistently from individual text instances into a corpus-level metric, and can be subjective and non-robust. to mitigate the dependence on human judgement, controllable synthetic distributions have been used in recent work to evaluate the metric quality @xcite . for instance, this was done by calculating the distance of synthetic high dimensional samples generated by sampling from gaussian or mixture-of-gaussian distributions representing the reference p and target q data, and then calculating the distance measure by shifting away the two distributions. in this paper, we adopt a middle ground between validating the metric against human judgement on real data and evaluating the metric with synthetic distributions by building "controllable-distance real data corpora" (section 3). by precisely controlling the content of test corpora, we devised a unified evaluation of desired metric characteristics on real data. this tech-nique allows aggregation of many small-difference judgements that should correspond to what a human would logically decide, to evaluate the distance metric overall in terms of desirable properties. the middle ground thus attempts to reflect human logical judgement in an inexpensive way, while avoiding some of the weaknesses described, such as lack of consistency. to summarize, our contributions are as follows. first, we present a text similarity evaluation measures that allows researchers to compare the robustness of their newly proposed metrics against existing metrics using the same standards. second, we evaluate classical and state-of-the-art similarity metrics and show that our benchmark performs well in capturing their known properties. finally, we provide a pip-installable python package to compute an extensive set of text dissimilarity metrics, using a unified and simple api foot_1 . ## literature review the most widely-used method to compute the quality of text similarity metrics investigates the correlation between the scores given by the metric and human judgements. however, human judgement, even on the sentence level, has several shortcomings, mainly that it is expensive and can be inconsistent and subjective @xcite @xcite . also, superficial aspects of the sentences, such as text length or syllables per sentence, may influence human judgements of the semantic similarity @xcite . furthermore, though humans may be able judge the relative similarity of a pair of sentences, they are usually limited in their ability to make large-scale assessments of a similar type when comparing two corpora (i.e., two distributions of sentences) consistently and reliably. in an attempt to standardize metric evaluation, several competitions and standard datasets containing compared data and human assessment were created for specific tasks, such as translation @xcite . however, there is currently a lack of benchmarks against which to assess the semantic similarity between corpora. text similarity metrics can be thought of as belonging to several broad and overlapping classes (see e.g., wang and dong 2020), which partially depend on the form of the text representation (e.g., token-based or vector embedding). here, we inves-tigate metrics from three of these classes, comparing corpora based on these aspects: lexicographical (statistical properties of words and tokens), distribution ( densities of sentences represented in the embedding space), and discriminatability (ability to classify sentences as belonging to one corpus or the other). the metrics we use are summarized in lexicographical statistics these methods have been developed to compare various distributional properties of target text q with respect to the reference samples p , based on some statistic measures t (p ) and t (q), operating on the surface text level, e.g., sentence, words, word-parts, tokens, etc. such commonly-used measures include resemblance in vocabulary distribution @xcite , likelihood of repetition @xcite , and n-gram matching @xcite . however, these metrics tend to be overly sensitive or easily misled by adversarial samples or text peculiarities. in general, χ 2 -based metrics calculate distance between observed and expected frequencies of categorical variables. the metric in @xcite , denoted here as chi, calculates e, the average (between p and q) frequencies of the n most common tokens in the combined vocabulary of p and q, then sums the χ 2 statistics comparing each of p and q to the expected e, across tokens. here, for both chi and zipf, below, we use the top @xmath0. in contrast, the zipf metric @xcite compares the use of vocabulary using zipf's law, which suggests that the frequency of a given word in human text is inversely-proportional to its frequency rank. the zipfian coefficient is fitted on a given corpus and the further it is from 1, the more the observed corpus differs from the 'ideal' theoretical distribution @xcite . we can thus use z pz q as a distance metric between corpora p and q. distributional metrics these metrics are based on quantifying the distributional relationship between the reference and target corpora in the embedded vector space, thereby capturing semantics beyond superficial token-level statistics. here p and q denote the reference and target corpora in the embedding space. given samples from these, we can use the sample density estimates p and q to approximate the true unknown corpus population distributions p and q. type metric measures lexicographical chi (χ 2 ) @xcite word/token count comparison. statistics zipf @xcite unigram rank-frequency statistics. fid @xcite wasserstein distance between densities. distributional pr @xcite assessing distributional precision & recall. dc @xcite estimating manifolds density and coverage. mauve @xcite quality & diversity via divergence frontiers. discriminative classifier (2016) classifiability between reference and target. irpr @xcite average distance between closest samples pairs. the fréchet inception distance @xcite is computed by fitting a continuous multivariate gaussian to the p and q, and then calculating the wasserstein-2 distance between them. however, fid is sensitive to both the addition of spurious modes as well as to mode dropping @xcite . also, while fid is able to detect distributional distances in the high-dimensional space, it cannot shed light upon the nature of this distance. due to these weaknesses of fid, we additionally consider a metric denoted pr proposed in computer vision @xcite , which is inspired by the notion of precision and recall in machine learning. intuitively, the precision captures the average resemblance of the individual target samples to the reference set (i.e., fidelity), while the recall measures how well the target samples "cover" the full variability of the reference samples (i.e., diversity). to obtain a single distance value using the method in (kynkäänniemi et al., 2019), we calculate the f 1 measure based on the returned precision and recall, denoted here by pr. naeem et al. ( 2020 ) proposed an improved estimation of these precision and recall notions (called, density and coverage) by mitigating the overestimation of manifolds caused by outliers and underestimating the similarity when the target and reference are taken from the same distribution. similarly to pr, we calculate the f 1 to obtain a similarity value using this method, denoted as dc foot_2 . mauve @xcite is a recentlydeveloped metric that estimates the gap between human and generated text by computing the area under the information divergence frontiers in a quantized embedding space using the kl-divergence foot_3 . discriminatability metrics similar to the distributional metrics, discriminative metrics calculate the distance using the embedding of the individual sentences in the two corpora. however, they do not aim to specifically capture the overlap between the distribution induced by the compared corpora. rather, they calculate the relationship in classification terms, i.e., to what extent can sentences in one corpus be distinguished from the sentences in the other corpus, using a discriminative model. classifier: following (lopez-paz and oquab, 2016), we measure the similarity between corpora using a binary classifier. we used svm @xcite trained on samples of both source corpora to predict corpus membership in a test set of unseen samples. a higher test accuracy indicates higher inter-corpora distance. while classifier is a model-based metric that uses the entire corpus distribution, irpr (information-retrieval precision and recall) is an example of an instance-(individual sentence) based corpus distance metric. inspired by @xcite , we calculate the dissimilarity between corpora as follows. for each embedded sentence in corpus a, we find its closest neighbor in b by cosine distance. the average of these distances is then computed to find the "precision" value. the same procedure in reverse, from b to a, gives the "recall" value. we calculate the f 1 score of the recall and precision to obtain a single value. note that the classifier metric is used to represent model-based discriminative approaches, while irpr is used to represent instance-based discriminative methods. the values calculated by chi, irpr, pr, dc and mauve capture the similarity rather than the distance between two corpora (for all metrics @xmath1, 1]). to make these metrics represent distances, we take 1v. our model selection was based on considering the trade-off between embedding quality and calculation time. the code as well as the scripts to reproduce the experiments are available online. 5 3 known similarity corpora most of the metric quality measures we propose are primarily based on the notion of knownsimilarity corpora (ksc) introduced by @xcite . the ksc set is created by mixing samples from two different source corpora a and b in gradually-changing proportions. the ksc set, denoted ksc(a, b), consists of k corpora {c 1 , c 2 , . . . , c k }, each of size @xmath2, where corpus c i , @xmath3, . . . , k is constructed by sampling n k-i k-1 observations from a, and the remaining n i-1 k-1 from b (see figure 1 ). the sampling resolution gradation between corpora is a fixed 1 k-1 . we now introduce some notation on the ksc set, which are used to define the measures in section 4. let [k] = {1, 2, . . . , k}. for given source corpora a and b, for each ℓ ∈ [k -1] we define the ℓ-distant corpora set as follows: some of the metrics d have a pre-defined range (e.g., chi, mauve, dc, pr only return values in the range [0,1]) while others have no preset scale or operation range. therefore, to allow sensible comparison of distance metrics with different operation ranges and across source corpora, we obtain z-scores by normalizing the metric values, pooled across all d ℓ (a, b, d). in the following analysis, if not specified otherwise, d ℓ will always be the normalized rather than raw distances. ## metric robustness measures we now describe our measurements of desirable properties for distance metrics, given the normalized d ℓ on the ksc sets. in the three following measures (monotonicity, separability, and linearity), we aim to capture three attributes of wellbehaved metrics that can be understood by considering the top line scatter plots of figure 2 ; these show the relation between the d ℓ sets and ℓ. in these scatterplots, a high angle of the regression line, low vertical variability around it, and linearity are all desirable properties for the distance metric, and are captured in these measures. ## increasingly fine-tuned corpora here, we qualitatively investigate the metrics' ability to discriminate between generated and human text using the following procedure: we generated a sequence of equal-size synthetic corpora if @xmath4, g 2 , . . . , g n ) by sampling from a gradually fine-tuned language model on a specific source corpus a. namely, in each iteration, a fine-tuning step is performed by training the language model on a single epoch containing 1000 sentences randomly drawn from a, followed by a generation process to synthesize a corpus g i containing around 1000 sentences. the name ifc, or "increasingly fine-tuned corpora", was chosen to parallel the name ksc ("known similarity corpora"). for each generated corpus g i , we estimated the distance from a, i..e, d @xmath5, g i ), ∀@xmath6. while the true distance between those synthetic corpora and a are unknown, an effective metric should capture the decreasing distance between a and g i with increasing i, namely d 1 ≺ d 2 ≺ ⋅ ⋅ ⋅ ≺ d n . due to our results, which show low imbalance robustness of some metrics, we maintained the same-size corpora when calculating corpora distance. the results presented in figure 5 show the gap between human and generated text captured by each metric in each iteration. to calculate the average self-distance of the reference corpus (a), we take the mean distance between two randomly sampled sub-corpora r 1 and r 2 from a, i.e. d(a, a) = e r 1 ,r 2 ∼a [d(r 1 , r 2 )]). in our experiments we used two datasets, the banking77 dataset, mentioned above and the news dataset 10 , representing different domains of text corpora. the ifc set for the banking77 dataset was generated in an iterative two-step procedure similar to the one described in lambada (anaby-tavor et al.). this procedure first generates sentences conditioned on the label, then filters out sentences that are out-of-domain or incorrectly labeled. however, the ifc set for the news dataset was generated by finetuning the pre-trained gpt-2 medium model @xcite . the results in figure 5 show that chi is less effective than the other metrics in capturing the gradual nature of the if c. also, they show that fid and irpr are sensitive in discriminating between the original and generated corpora, even after many fine-tuning iterations. interestingly, the zipf distance increases with the iteration. this indicates that the generated text, despite becoming 10 huffpost ( www.huffpost.com ) news headlines collected from 2012 to 2018 containing around 200k headlines.( www.kaggle.com/rmisra/news-category-dataset ( https://www.huffpost.com ) semantically closer to the original with the increasing iterations, becomes less 'natural' in that the token frequencies deviate from that of human text and the reference corpus. this can be explained, at least in part, by the ttr measure. ttr is a standard word diversity measure, calculated by dividing the number of unique words in a text by the total word count. a high ttr indicates significant lexical variation. indeed, in the ifc of banking77, g 1 's ttr is 0.295 which is closer to the original dataset's ttr of 0.299 than g 40 's ttr of 0.322. ## conclusions in this work, we propose a principled set of automatic measures for evaluating the quality of text dissimilarity metrics. by testing various metrics using our measures, we show that they do a good job of capturing their known characteristics, hence increasing our confidence in these measures; also, overall, recent metrics exhibit more favorable traits than their predecessors. the radar chart in figure 4 shows that our measure scores correlate well with the compared distributional metrics recency m au v e ≻ dc ≻ p r ∼ f id as well as their known relative strengths.