O N THE U SEFULNESS OF E MBEDDINGS , C LUSTERS AND S TRINGS FOR T EXT G ENERATOR E VALUATION
A BSTRACT
A good automatic evaluation metric for language generation ideally correlates highly with human judgements of text quality.
Yet, there is a dearth of such metrics, which inhibits the rapid and efﬁcient progress of language generators.
One exception is the recently proposed M AUVE .
In theory, M AUVE measures an informationtheoretic divergence between two probability distributions over strings: one representing the language generator under evaluation and the other representing the true natural language distribution.
M AUVE ’s
authors argue that its success comes from the qualitative properties of their proposed divergence.
Yet in practice, as this divergence is uncomputable, M AUVE approximates it by measuring the divergence between multinomial distributions over clusters instead, where cluster assignments are attained by grouping strings based on a pre-trained language model’s embed dings.
As we show, however, this is not a tight approximation—in either theory or practice.
This begs the question why does M AUVE work so well?
In this work we show that M AUVE was right for the wrong reasons, and that its newly proposed divergence is not necessary for its high performance.
In fact, classical divergences paired with its proposed cluster-based approximation may actually serve as better evaluation metrics.
We ﬁnish the paper with a probing analysis; this analysis leads us to conclude that—by encoding syntactic- and coherence-level features of text, while ignoring surface-level features—such cluster-based substitutes to string distributions may simply be better for evaluating state-of-the-art language generators.
I NTRODUCTION
Probabilistic text generators have improved greatly over the last years, with models producing increasingly human-like text
(Yang et al., 2019; Brown et al., 2020; Raffel et al., 2020; Rae et al., 2021; Hoffmann et al., 2022).
As the gap between human and model-generated text closes, the quality of our evaluation metrics becomes ever more important for determining generator quality, especially given the increasing number of user-facing systems employing these generators.
While human evaluations serve as the gold standard, they are costly (in both time and money), leading researchers to rely on automatic metrics—i.e., metrics that can be measured by a computer—for the bulk of their development process.
Many automatic language generator evaluation metrics share the same underlying mechanism: the quantitative comparison of two probability distributions.
Speciﬁcally, most metrics measure a difference between the distributions over strings deﬁned by: (1) a language generation model 2 and (2) the natural language itself.
This includes some of the most widely used language evaluation metrics:cross-entropy (Shannon, 1948), perplexity (Jelinek et al., 1977), and (more recently)
M AUVE (Pillutla et al., 2021).
As typically applied to evaluate language generators, however, these metrics have a number of computational and qualitative issues (discussed in §3).
Such issues manifest empirically: the most commonly used automatic metrics are known to correlate poorly with human judgements (Wiseman et al., 2017; Reiter, 2018;
Sellam et al., 2020; Gehrmann et al., 2021).
A newly proposed metric stands apart: M AUVE (Pillutla et al., 2021).
In theory, M AUVE measures the area under the curve formed by the divergence between two probability distributions, qualitatively mimicking a precision–recall quantiﬁcation (Djolonga et al., 2020; Kynkäänniemi et al., 2019).
The authors attribute the success of their metric to the qualitative properties of this new class of divergences.
Yet, due to this divergence being in practice uncomputable, Pillutla et al. propose an approximation to it.
Speciﬁcally, rather than directly comparing the original two distributions over strings, M AUVE ﬁrst clusters samples taken from these distributions based on the embeddings of a pre-trained language model; it then estimates the proposed divergence using the samples’ empiricallyobserved multinomial distributions over cluster assignments.
As we will show, this approximation is bad in both theory §4 and practice §5.1—to the point that the term “approximation” is arguably a misnomer.
Thus, the reasons why M AUVE works well—knowledge which is important for the continued progress of language generator evaluation metrics—are still unknown.
In this work, we aim to uncover these reasons.
To this end, we consider the axes on which M AUVE differs from other evaluation metrics: is M AUVE ’s success due to the new divergence metric, to its “approximation”, or both?
Empirically, we identify M AUVE ’s substitution of probability distributions over strings with probability distributions over embedding-based clusters as the main factor for its success.
We show that mathematically, this substitution leads to a quite biased estimator of the original string-based divergences.
Yet it also leads to metrics with lower variance and stronger correlations with human judgements.
In fact, all divergence measures analysed here correlate more strongly with human judgements when cluster-based distributions are used in place of string-based ones.
Finally, in order to understand the root of the effectiveness of these cluster-based metrics, we probe the clusters themselves.
We ﬁnd that sentence-level permutations within texts noticeably affect cluster assignments, suggesting that cluster-based metrics are susceptible to attributes such as coherence.
On the other hand, basic manipulations that render text unhuman-like, such as removing all articles from the input text, do not seem to affect these metrics signiﬁcantly.
Together, these results lead us to conjecture that embedding-based metrics may be favourable when estimating the quality of state-of-the-art (SOTA) language generators, as SOTA models are known to (at least typically) produce grammatical text.
That is, by ignoring surface-level features of text—while emphasising discourse- and coherence-level ones—clustered embeddings may simply be better suited for the evaluation of the top language generation systems.
Yet these ﬁndings also suggest routes through which such metrics can be gamed, bringing into question their robustness.
We believe these ﬁndings, along with the theoretical framework we provide for evaluation metrics’ comparison, are important for the further development of language generator evaluation metrics.
D IVERGENCE M ETRICS FOR L ANGUAGE G ENERATOR E VALUATION
When evaluating language generation systems, we will ﬁrst assume the existence of an unknown ground-truth distribution p w .
This distribution is deﬁned over strings w and its domain spans W ≡ Σ ∗ , where Σ is an alphabet of words and Σ ∗ is its Kleene closure.
Second, we are given a probabilistic text generator q w , which is also a distribution over W .
An evaluation metric for a language generator q w can now be deﬁned as a measure of its “distance” from p w : ∆( p w , q w ) .
In short, ∆( · , · ) should return high values if q w is a bad approximation to p w , and it should return low values if it is a good one.
Notably, it is not clear whether q w being a good approximation to p w in terms of an arbitrary ∆( · , · ) guarantees that it will be a good language generator. Indeed, models that perform well in terms of standard metrics, such as perplexity, often still produce poor-quality text (Holtzman et al., 2020). Thus, we are interested speciﬁcally in ∆( · , · ) that correlate highly with human quality judgements. More formally, we deﬁne human quality judgements as a (potentially noisy) mapping α ( q w ) from a language generator to a real-valued score.
For ﬁxed p w , a useful metric ∆( p w , · ) for evaluating the quality of a language generator q w is one whose scores correlate highly with humanscores ( · ) .
This notion can be operationalised as follows.
Assume we have N language generator models.
Let us deﬁne: [Equation]
We then quantify a metric’s usefulness on a speciﬁc natural language task (and its distribution p w ) as:
[Equation]
We now review common choices for ∆( · , · ) .
Given the probabilistic nature of most language generators, a number of divergence measures are among these choices, which quantify the difference between two probability distributions. The rest of this work focuses primarily on this class of metrics.
Forward Divergence.
Cross-entropy, ∆ H ( p w , q w ) def = H( p w , q w ) , which is equivalent (up to an additive constant) to the forward Kullback–Leibler (KL) divergence, is one such choice:
[Equation]
where we use (cid:47) to signify additive or multiplicative equivalence.
(1) is true since H( p w ) is constant with respect to q w .
Since Pearson and Spearman correlations—the metrics we use to evaluate ∆ ’s quality—are invariant to translational shifts, the cross-entropy and forward KL are equivalent as language generator metrics.
We will refer to them interchangeably during subsequent comparisons.
Backward Divergence.
Albeit much less common, another potential evaluation metric would be the backward (exclusive) KL divergence:
[Equation]
As opposed to the forward KL , when use as an evaluation metric, Eq. (5) is not effectively equivalent to the cross-entropy between q w and p w , as H( q w ) is not constant across language generators q w .
Exponentiated Divergence.
By far, the most common choice of ∆ to evaluate language models is the perplexity: ∆ perp ( p w , q w ) def = e H( p w ,q w ) .
Notably, perplexity is equivalent (up to a multiplicative constant) to an exponentiated Kullback–Leibler divergence between p w and q w , which follows from the same relationship as in Eq.
Given the property that both Pearson and Spearman correlations are invariant to a change in scale, the perplexity and exponentiated KL will thus be equivalent as language generator metrics.
For consistency, we will use solely the exponentiated KL in our analyses:
[Equation]
Jensen–Shannon Divergence.
Note that the KL divergence is non-symmetric and unbounded.
On the other hand, the Jensen–Shannon (JS) divergence—deﬁned as the average of two KL s—is symmetric with respect to its inputs and is guaranteed to produce bounded values:
[Equation]
Area Under the Curve (AUC) Divergence.
Finally, information divergence frontiers are a recently proposed class of metrics for generative models (Sajjadi et al., 2018; Kynkäänniemi et al., 2019).
The variant proposed by Pillutla et al.
(2021) computes the area under the curve formed by a series of Kullback-Leibler divergences as we change a mixing parameter λ :
[Equation]
where λ is varied across the interval [0 , 1] , and s ∈ R > 0 is a strictly positive real-valued scaling constant. Note that we deﬁne the AUC divergence as 1 − AUC( · , · ) so that a larger value indicates a greater discrepancy with the reference corpus p w .
I NFELICITIES AND A PPROXIMATIONS
There are several issues, both computational and qualitative, with using the divergences presented in §2 to evaluate language generators.
We now review these issues, along with both commonly-used and newly-proposed methods to address them via approximations.
3. N ECESSITY OF F ULL S UPPORT
A well-known property of the (forward) KL divergence between two distributions p w and q w is that it is inﬁnite for any q w that assigns 0 probability to an event in the support of p w (i.e., for which p w ( w ) > 0 ).
The above is often not an issue for ∆ exp and ∆ → : most neural language generators cannot assign 0 probability to any string due to the ﬁnal softmax operation typically used to project their outputs onto the probability simplex.
However, these same models are often used with decoding strategies that prune the space W : e.g., both top- k and nucleus sampling modify q w such that strings which do not meet a certain criterion are reassigned 0 probability.
While top- k and nucleus sampling typically lead to systems with qualitatively better text, they will likely be given an inﬁnitely bad score by both ∆ exp and ∆ → , which is perhaps too harsh a penalty for an otherwise good language generator.
3. p w IS U NKNOWN
In practice, we do not have access to the true distribution p w .
Rather, we are typically given a corpus { w p w n } Nn =1 , whose instances we assume to be sampled i.i.d.
from p w .
The common approach to address this issue is thus to derive a statistical estimator (cid:98) ∆ that uses this corpus to approximate ∆ .
There are two common strategies for building such estimators: Monte Carlo and plug-in estimation.
Monte Carlo Estimation.
Our i.i.d.
assumption w.r.t.
samples in { w p w n } Nn =1 allows us to derive a
Monte Carlo estimator for certain divergences.
We start with the forward KL divergence:
[Equation]
where const ∈ R is constant with respect to q w .
Eq. (9) is an unbiased estimator of KL divergence, which in turn allows us to build estimators (cid:98) ∆ → and (cid:98) ∆ exp .
Unfortunately, unbiased estimates of ∆ ← , ∆ JS and ∆ AUC are not as straightforward to compute, as they require explicit knowledge of p w rather than just samples (see App.
A).
This issue motivates the use of our next set of estimation techniques.
Plug-in Estimation.
Here we consider estimation via building an approximation of p w itself to use in the formulas given in §2.
Speciﬁcally, we construct a density estimator for p w (which we denote as (cid:98) p w ) and “plug it into” a given ∆ .
However, this is a bit circular: the task of building a language generator q w itself is often framed as density estimation of p w .
Thus, if we think q w is the “best” estimator for p w , we should logically use it in our plug-in estimator.
Yet, using q w would be nonsensical; by the deﬁnition of a (shifted) divergence, it would always lead to the lowest possible value of ∆ , e.g., ∆ → ( q w , q w ) = 0 .
To use plug-in estimation in this setting, we should therefore choose a different estimator for p w , e.g., from a family of density estimators that differs from those used to create q w .
More formally, we consider a function π which takes a corpus as input and produces a (queryable) distribution (cid:98) p w def =
π ( { w p w n } Nn =1 ) .
This function typically induces a secondary model, e.g., an n -gram model or neural network, trained on the corpus { w p w n } Nn =1 .
Our chosen π may introduce biases (e.g., from the inductive biases of the architecture parameterising π ) into our metrics’ estimation.
To balance out such biases, we may consider using the same method to create an approximation (cid:98) q w for use in our plug-in estimators, rather than directly querying q w :
[Equation]
Plug-in estimators for ∆ JS and ∆ AUC are deﬁned similarly.
Further, if (cid:98) q w is a smoothed approximation to the original q w , using it may also mitigate the issues discussed in §3.1.
We thus also compute estimators for the forward/exponentiated divergences using plug-in estimators, e.g.,:
[Equation]
Unfortunately, most functions π cannot produce a good estimate of p w using only a small corpus, which is the case we consider since we rely on evaluation sets for { w p w n } Nn =1 .
While the best available language models are a class of π typically trained on millions (if not billions) of sentences, a standard evaluation set is quite small—on the order of one to ten thousand sentences—and we cannot expect π to provide a good (cid:98) p w when
ﬁt using only such a small dataset.
Accordingly, depending on our choice of π , this class of metrics may be either high variance or high bias, both of which are problematic.
3. C LUSTERING - BASED A PPROXIMATIONS
For the (cid:98) ∆ above that require density estimators for p w and/or q w , our choice of π will have a large effect on its value.
We may thus wish to rethink our approximation technique altogether, and instead work with different distributions for which we can create lower variance density estimators.
This is the approach used by Pillutla et al.
(2021) when approximating ∆ AUC .
Speciﬁcally, instead of computing the above metrics on the original distributions p w and q w , they use the cluster-based distributions p c and q c .
Given a pre-trained language model, these cluster-based distributions are deﬁned as:
[Equation]
where PLM ( · ) takes as input an utterance w and outputs an embedding r , and φ ( · ) is a pretrained clustering function.
Note that the function φ ( · ) is trained jointly on samples from the two distributions under consideration as we will detail later; we defer the reader to Pillutla et al.
for a more detailed explanation of the procedure.
Given these distributions, we can evaluate cluster-based versions of all the divergences above, simply by substituting the original p w and q w with the new p c and q c .
A NALYSING C LUSTER -B ASED A PPROXIMATIONS
We now take a closer look at the biases introduced by the substitution of cluster-based distributions suggested in §3.3.
For simplicity, we focus on the bias introduced to KL( p w || q w ) —a computation involved in M AUVE ’s ∆ AUC .
This divergence can be decomposed as:
[Equation]
where (1) follows from the fact that p ( c, w ) = p ( w ) , which is true because the cluster assignment is deterministic, i.e.: p ( c | w ) = 1 { c = φ ( PLM ( w )) } .
See the full decomposition of this equation in App.
B. Notably, as KL divergences are always non-negative, the cluster-based version is negatively biased, lower-bounding the string-based one.
Further, the actual measurement is done on the distribution over cluster assignments p ( c ) ; the distribution p ( w | c ) is completely ignored.
Assuming a reasonable number of clusters is used when deﬁning p c , however, it should be easier to create good approximations of distributions over clusters than distributions over strings due to the sheer difference in the size of the supports alone.
Consequently, the variance of cluster-based metrics should be lower, at the cost of the bias introduced by this substitution.
Further, it is not clear whether this bias is inherently bad when evaluating the quality of language generators: the answer to this question must be determined empirically (by measuring the correlation in Eq.
To this end, we now provide an empirical comparison between string- and cluster-based language generation evaluation.
E XPERIMENTS
Setup.
We follow the setup of Pillutla et al.
throughout our experiments.
We compare systems for open-ended text generation q w with human-generated text p w .
As human-generated samples, we use 5 k strings taken from WebText’s test set.
As model-generated text, we sample 5 k strings from each of our evaluated systems, conditioning our models on the ﬁrst 10 words of human-generated strings before sampling (i.e., a text-completion task).
For our language generators, we comparemodel architectures (all variants of GPT-2), each under two decoding strategies, giving us a total ofsystems.
Explicitly, we compare the small , medium , large , and XL versions of GPT-2, decoding strings using either ancestral or nucleus sampling.
Following Pillutla et al.
(2021), we use a nucleus probability of 0 . 9 for small and medium , while 0 .
95 for large and XL GPT-2’s.
Importantly, we run our experiments only on English text, which is a notable limitation of our work; future work should verify that ﬁndings hold across languages.
String-based Approximations (cid:98) p w .
To compute our string-based divergences, we require a secondary language model (cid:98) p w to estimate p w .
Further, following the issues highlighted in §3, we will also rely on a secondary language model (cid:98) q w to estimate q w .
We will use n -gram models for these approximations.
Speciﬁcally, we use Kneser-Essen-Ney smoothed 5-gram models, as implemented in
KenLM (Ney et al., 1994; Heaﬁeld, 2011).
We choose n -gram models explicitly because—while they are by no means SOTA language models—they should have inductive biases which are different from the models we are trying to evaluate.
We present results using LSTM-based estimators in App.
E.
When computing ∆ AUC , we use a scaling constant s of 0.2.
Cluster-based Approximations (cid:98) p c .
Cluster-based distributions, as presented in Eq.
(12), are deﬁned by a choice of PLM ( · ) and pre-trained clustering function φ ( · ) .
We rely on GPT-2 XL as our PLM, and use K -means as our clustering function; results using other pre-trained language models are in App.
E. Speciﬁcally, we ﬁrst extract embeddings from the last word in each sentence using GPT-2 XL and then use PCA to reduce their dimensionality (keeping 90% of the original variance explained); results using mean of word embeddings can be found in App.
E. We then train K -means (with K = 500 ) on a joint set of GPT-2 embeddings extracted from: the 5 k human-generated strings, and 5 k model-generated sentences.
Finally, we approximate (cid:98) p c and (cid:98) q c by computing the frequency with which strings (among these 5 k used ones) are assigned to each cluster.
To avoid inﬁnite divergence measures, we estimate distributions using Laplace smoothing with α = 1 (which is equivalent to imposing a Dirichlet distributed prior with α = 1 over the cluster allocation).
When computing ∆ AUC , we use a scaling constant s of 5.
[Figure]
Our ﬁrst experiment tries to identify whether p c and q c provide faithful approximations of p w and q w .
To this end, we compare both (cid:98) q w and (cid:98) q c to the true q w , i.e., the language generator under evaluation.
Explicitly, we compute the Spearman correlations between the probabilities assigned by each model to the strings in { w q w n } Nn =1 .
Fig.
1 presents these correlations.
We see that—despite being estimated on very little data—probability estimates from our n -gram models correlate strongly with the ground-truth probabilities of q w ; this result holds for all four GPT-2 architectures.
On the other hand, our cluster-based probabilities consistently present negative correlations with q w .
This result has an important implication: if cluster distributions do not correlate with q w , then KL( (cid:98) p c
|| (cid:98) q c ) is likely a poor estimate of KL( p w || q w ) .
This further implies that the approximation used by Pillutla et al.
is not an accurate estimate of ∆ AUC ( p w , q w ) , which brings into question whether this new divergence is really responsible for M AUVE ’s success.
5. ∆ AS T EXT E VALUATION M ETRICS
We now compare how various string- and cluster-based divergence measures correlate with human judgement scores.
In short, Fig. 2 shows that all divergences do better when estimated with cluster distributions.
These results evince that M AUVE ’s
(Pillutla et al., 2021) high correlations with human judgements (represented here as (cid:98) ∆ AUC ( p c , q c ) ) are mainly due to their use of clusterbased approximations ( p c , q c ), rather than to their proposed divergence ∆ AUC .
In fact, we see
( p c , q c ) ( p w , q w )
C o rr e l a ti on ( % ) (a) Interestingness ( p c , q c ) ( p w , q w )
C o rr e l a ti on ( % ) (b) Sensibility ( p c , q c ) ( p w , q w )
C o rr e l a ti on ( % ) (c) Human likeness slight improvements over (cid:98) ∆ AUC when using the divergences (cid:98) ∆ ← and (cid:98) ∆ JS instead. Furthermore, cluster-based divergences appear to be more stable, exhibiting smaller variances across random seeds.
Collectively, our results suggest that cluster-based divergences may produce better metrics of text quality than string-based divergences.
This motivates the question: Which components of natural language are captured by cluster-based distributions p c , and which are overlooked by ignoring p ( w | c ) when computing the cluster-based divergences?
Our next set of experiments aim to answer this.
P ROBING C LUSTERS
To better understand the aspects of natural language that our cluster distributions encode, we must ﬁrst understand how φ ( PLM ( · )) partitions the string space W .
In other words, we must understand what components of natural language—e.g., semantics, syntactic attributes, or surface features—lead to strings being assigned to similar or different clusters.
Such an analysis should provide a deeper insight into the actual similarity being measured by cluster-based divergences (while also revealing how such a metric might be gamed).
To this end, we probe (Alain & Bengio, 2016)
the clusters learned by the
M AUVE algorithm for a number of linguistic attributes—including subject matter , sentiment , prose style , word order , basic grammaticality and document length —looking at how they affect both cluster assignment and the divergence scores between texts that differ in these characteristics.
Notably, we probe cluster assignments directly—without relying on any diagnostic classiﬁers (Adi et al., 2017).
Our probing analyses are thus exempt from most recent criticism against probing methodologies (Hewitt & Liang, 2019; Pimentel et al., 2020a;b; Ravichander et al., 2021; Elazar et al., 2021).
6. F INDING F EATURES p c E NCODES
Setup.
We look at texts annotated with different attribute categories in order to explore correlations between the presence of these attributes and cluster membership.
Speciﬁcally, we analyse texts’: sentiment , authorship , and topic (using the Yelp Polarity, News Category, and 20 NewsGroup datasets, respectively).
Further details on datasets are provided in App.
D. For each of these classiﬁcation datasets, we compute the cluster–category distributions that result from the M AUVE algorithm using the standard training split for the respective datasets; all evaluations are then performed on test splits.
Explicitly, we ﬁrst learn a partitioning φ ( · ) of the embedding space (w.r.t a language model PLM ( · ) ).
Each cluster is then labelled with the majority category represented in that cluster by training examples; text categories in the test set are then predicted using this labelling, depending on which of the clusters the example falls into.
For comparison’s sake, we use four language models as PLM ( · ) :
GPT-2 with small , medium , large , and XL architectures.
Results using embeddings from BERT (Devlin et al., 2019) can be found in App.
E. Further, we use two methods for learning clusters:
• φ ( · ) Learned on WebText.
Using the same procedure as in §5.2, we train PCA and K -means functions to partition the embedding space (again relying on WebText’s test set for our data).
This mimics the setting under which our partitions would be learned in practice.• φ ( · ) Learned on Training Set.
We instead train the PCA and K - means clustering functions on the analysed dataset’s training set.
This setting studies the partitioning that our clustering functions have the capacity to learn in an ideal setting, i.e., where the attribute in question is one of the main differentiating factors between texts.
[Figure]
Results.
In Fig.
3a, we see that, at least for large numbers of clusters, cluster assignment is indeed indicative of a text’s sentiment.
Interestingly, this is the case even when clusters are trained on data that is not particularly polar in sentiment (i.e., on WebText).
On the other hand, we are only able to predict author and topic (with reasonable accuracy) when clusters are learned on text data with authorship and topic as distinguishing factors.
These results indicate that, while writing style and subject matter are captured by the text embeddings, they likely were not being used as distinguishing features between corpora in our cluster-based divergences.
We further see that, in all classiﬁcation settings, the capacity to encode these analysed attributes appears to increase with model size, perhaps suggesting that the embedding spaces of larger models decompose along higher-level features of text.
6. H OW T EXT F EATURES I MPACT ∆
We next assess how changing different features of our evaluated text impacts divergence scores.
Speciﬁcally, we look at the impact of: text truncation; article removal; stopword removal; sentencelevel permutations; and word-level permutations.
Setup.
We follow a similar setup to §5.
In order to create a more controlled setting, though, we primarily consider human-generated text in these experiments (i.e., the 5 k human-written articles in
WebText’s test set).
We take the ﬁrst 2500 articles of this dataset as our reference corpus { w p w n } Nn =1 .
We then use the remaining 2500 reference strings as the comparison corpus, i.e., in place of the model-generated text that we would typically evaluate { w q w n } Nn =1 .
In order to explore how changing speciﬁc features of text affects ∆ w.r.t.
the reference corpus, we then compute scores when making the following modiﬁcations to the comparison corpus:
• No modiﬁcation ( p ).
This is a baseline experiment where we keep the original strings unchanged.
• Text Truncation ( p short ).
We truncate texts to 1 / 3 of their original length.
This allows us to understand whether the divergence metrics pick up on differences in dataset length statistics.
• Article Removal ( p no art ).
We remove all articles (‘a’, ‘an’ and ‘the’) in the text.
This allows us to understand whether the divergence metrics can distinguish between texts with or without basic levels of ﬂuency and grammaticality.
• Stopwords Removal ( p no stop ).
We remove all stopwords (e.g., ‘that’ or ‘so’) in the text.
This allows us to understand whether the divergence metrics can detect differing levels of syntactic coherence, rather than just focusing on content words.• Sentence-level Permutation ( p swap ).
We permute the ﬁrst halves of texts (as delineated by sentences) across the entire corpus (i
e. randomly reassigning the strings’ ﬁrst halves).
This allows us to understand whether the divergence metrics detect coherence.
• Word-level Permutation ( p rand ).
We randomly permute all words in a text.
This allows us to understand whether the divergence metrics can only distinguish between bag-of-word level features.
• GPT-2 Baseline ( q ).
As an extra baseline, we also use the ﬁrst 2500 generations from GPTXL on the WebText text-completion task, as in §5.
Results.
Fig.
4 shows that certain alterations to the evaluated text—such as completely removing articles—have almost no impact on its divergences from the reference corpora for various ∆ .
In fact, text without any articles is judged as better than GPT-2 XL ’s by all of the cluster-based divergences (see Fig.
10 for a zoomed-in version).
Further, while this perturbation undoubtedly affects the text’s ﬂuency, it has less of an effect on ∆ than, e.g., truncating texts.
This is arguably undesirable: A metric of text quality should place more emphasis on ﬂuency than surface statistics, such as length.
On the other hand, our metrics deem text with stopwords removed as utterly different from the reference.
Permuting words within texts has a similar effect, demonstrating that, at least to some extent, the embedding space captures notions of syntax and grammaticality, rather than pure unigram statistics.
The increase in ∆ shown when performing sentence-level permutations likewise suggests that the clusters delineate different levels of coherence to some extent.
In Fig.
13 (in App.
E), we perform an additional experiment where we again probe the clusters (as in §6.1), but for surface features of text this time, such as the percentage of stopwords and punctuation symbols in a text.
There we see evidence that such features of text are not strongly encoded in the clustering scheme.
Perhaps surprisingly, when applied to cluster-based distributions, all of the studied ∆ metrics rank the distance of the perturbed texts from the reference texts in the exact same order (this is clear in Fig.
10, a zoomed-in version of Fig. 4).
For these perturbations, the ∆ differ only in the magnitude of their outputs, further suggesting that the ∆ AUC metric itself is likely not critical for the effectiveness of M AUVE .
One potential avenue for future research would be investigating whether different algorithms for discretisation of the embedding space create clusters that align with speciﬁc linguistic attributes; this could be a useful diagnostic for language generators’ improvement.
This section’s results—along with those of §6.1—suggest that divergences based on PLM embeddings are more sensitive to syntax- and coherence-related properties of the target text than to its superﬁcial features.
The opposite, however, might be said of our string-based distributions.
These ﬁndings offer a potential explanation for the effectiveness of metrics that make use of PLM embeddings, such as M AUVE or BERTS CORE (Zhang et al., 2020).
As current SOTA language generators already typically produce grammatical text, being invariant to surface statistics may perhaps be a feature—as opposed to a bug—when trying to assess the quality of the text they produce.
Yet, this may also reveal potential ways in which such metrics can be gamed, bringing this paradigm’s robustness into question.
C ONCLUSION
In this paper, we analyse M AUVE , a recently-proposed automatic metric for language generator evaluation.
While M AUVE correlates quite well with human quality judgements, it is unclear which of the metric’s design choices are in fact responsible for its success—a shortcoming that impedes the further development of language generator evaluation metrics.
We attempt to rectify this shortcoming.
We ﬁrst provide a general theoretical framework for the comparison of language generator evaluation metrics.
Then through a series of empirical studies, we identify M AUVE ’s substitution of probability distributions over embedding-based clusters—in place of the traditional distributions over strings—as the attribute largely responsible for the metric’s success.
In order to better understand the nature of this improvement, we probe the clusters used by the density estimators, analysing what they ignore and what they emphasise about the input text.
We ﬁnd that, while distributions over clusters are sensitive to syntax- or coherence-level perturbations to the text, this is not the case for several surfacelevel perturbations.
We thus conjecture that, by focusing on higher-level text features, cluster-based evaluation metrics may simply be better suited to rank high-performing models, and that this is a general paradigm worth further exploration.
We would like to thank the authors of M AUVE for sharing their human evaluation data with us. We would also like to thank Gábor Melis for his insightful feedback on an initial version of this paper, our anonymous reviewers for their help in improving our ﬁnal draft and Luca Malagutti for detailed feedback on clarity and presentation
A A M ONTE C ARLO E STIMATOR FOR B ACKWARD , JS, AND AUC D IVERGENCES
Consider a Monte Carlo estimator for ∆ ← :
[Equation]
where w q w n ∼ q w
We can easily sample from q w .
However, computing Eq.
(14) requires knowledge of log p w , which we do not have.
Thus we resort to other techniques for estimating these divergences, as discussed in §3.
B S TRING VS . C LUSTER - BASED K ULLBACK –L EIBLER D ECOMPOSITION
The decomposition in Eq.
(13) can be shown as follows:
[Equation] [Equation]
where again (1) follows from the fact that p ( c, w ) = p ( w ) , which is true because the cluster assignment is deterministic, i.e.:
[Equation]
Thus, KL( p ( c ) || q ( c )) provides a biased estimate of KL( p w || q w ) .
C R ELATED W ORK
Over the years, a number of evaluation metrics have been proposed for language generation tasks (such as translation and summarization); the most well-established and commonly-used include BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and METEOR (Banerjee & Lavie, 2005).
However, these metrics—which rely on n -gram statistics—have been shown to correlate poorly with human judgement (Reiter, 2018).
Recent metrics have improved upon these correlations with more advanced techniques, e.g., BEER (Stanojevi ´ c & Sima’an, 2014),
MoverScore (Zhao et al., 2019) and BLEURT (Sellam et al., 2020).
These metrics, though, are intended for language generation tasks with a strict set of reference texts.
While reasonably effective for directed generation tasks, they do not transfer well to the open-ended domain.
For tasks in which there is not a clear reference, e.g., story generation, basic statistics are typically employed to provide a preliminary evaluation of generated text.
Such statistics include n -gram repetitions (Welleck et al., 2020), Zipﬁan coefﬁcient (Holtzman et al., 2020), or the perplexity of generated text (Fan et al., 2018).
Final assessments of language generation systems are still often performed using human evaluations, as automatic metrics on their own have not proven sufﬁcient for differentiation between top-end language generation systems.
Automatic evaluation metrics for language models based on statistical divergences have been proposed by a number of different authors.
For instance, Meister & Cotterell (2021) assessed the quality of language models while using a number of divergences between distributions over surface statistics in text corpora.
Xiang et al.
(2021) propose the approximation of distributions over strings with distributions in the embedding space in standard divergence metrics.
Pillutla et al.
(2021) present
M AUVE —the object of study of this work—which is based on a new divergence metric inspired by the information frontier divergences Djolonga et al.
Besides proposing this AUC divergence metric, Pillutla et al.
(2021) also propose a new way to approximate it using clusters over word embeddings.
We provide an analysis of this paradigm, showing that in practice, we should expect this method to provide a poor estimation of the intended quantity.
We go on to perform empirical experiments to identify the proposed use of distributions over clusters itself is likely responsible for the metric’s success, rather than characteristics of the new divergence.
We see this work as complementary to Pillutla et al.
(2021), providing deeper and more comprehensive insights into the metric’s inner workings.
D E XPERIMENTAL S ETUP
Embedding Models.
To obtain text embeddings for our input strings, we use several different PLM .
Namely, we use the four sizes of GPT-2 (Radford et al., 2019), as well as BERT-base and BERT-large (both cased; Devlin et al., 2019).
For the former, we use the embedding of the ﬁnal token.
For the latter, we use the embedding associated with the special CLS token.
In both cases, we additionally present results using the average across token embeddings in App.
E. All texts are truncated totokens (for experiments in §6.2, this truncation is performed before any manipulations of the text), in order to ensure that all models can process the input text in their context.
For our probing analysis in §6, we employ the following datasets:
• Sentiment.
To analyse sentiment, we use Yelp Polarity (Zhang et al., 2015), a dataset extracted from the Yelp Dataset Challenge 2015, which contains binary sentiment classiﬁcations for highly polar Yelp reviews.
We use 10 k examples randomly sampled from the training set and 5 k examples randomly sampled from the test set.
• Authorship.
To analyse authorship, we use News Category (Misra & Grover, 2021; Misra, 2022), a dataset consisting of 200 k news headlines from the years 2012–2018 obtained from HuffPost.
We scrape entire articles from the URLs provided by the dataset.
We only use the subset of articles for which the article’s author has ≥ 400 articles within the dataset, giving us a training set of 32 k and a test set of 6 k with 46 unique authors.
• Topic.
To analyse topic, we use the 20 NewsGroup dataset, which contains 18 k newsgroups posts (partitioned into train and test sets using the original splits) on 20 topics, such as subcategories of science, politics and religion.
The distribution over text topics is relatively uniform.
E A DDITIONAL E XPERIMENTS
[Figure]
[Figure] [Figure]