The Vendi Score: A Diversity Evaluation Metric for Machine Learning
Anonymous authors Paper under double-blind review
Abstract
Diversity is an important criterion for many areas of machine learning (ML), including generative modeling and dataset curation.
Yet little work has gone into understanding, formalizing, and measuring diversity in ML.
In this paper we address the diversity evaluation problem by proposing the Vendi Score , which extends ideas from ecology to ML.
The Vendi Score is deﬁned as the exponential of the Shannon entropy of the eigenvalues of a similarity matrix.
This matrix is induced by a user-deﬁned similarity function applied to the sample to be evaluated for diversity.
In taking a similarity function as input, the Vendi Score enables its user to specify any desired form of diversity.
Importantly, unlike many existing metrics in ML, the Vendi Score does not require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be deﬁned.
We showcase the Vendi Score on molecular generative modeling where we found it addresses shortcomings of the current diversity metric of choice in that domain.
We also applied the Vendi Score to generative models of images and decoding algorithms of text where we found it conﬁrms known results about diversity in those domains.
Furthermore, we used the Vendi Score to measure mode collapse, a known shortcoming of generative adversarial networks (GANs).
In particular, the Vendi Score revealed that even GANs that capture all the modes of a labelled dataset can be less diverse than the original dataset.
Finally, the interpretability of the Vendi Score allowed us to diagnose several benchmark ML datasets for diversity, opening the door for diversity-informed data augmentation.
Introduction
Diversity is a criterion that is sought after in many areas of machine learning (ML), from dataset curation and generative modeling to reinforcement learning, active learning, and decoding algorithms.
A lack of diversity in datasets and models can hinder the usefulness of ML in many critical applications, e.g. scientiﬁc discovery.
It is therefore important to be able to measure diversity.
Many diversity metrics have been proposed in ML, but these metrics are often domain-speciﬁc and limited in ﬂexibility.
These include metrics that deﬁne diversity in terms of a reference dataset (Heusel et al., 2017; Sajjadi et al., 2018), a pre-trained classiﬁer (Salimans et al., 2016; Srivastava et al., 2017), or discrete features, like n-grams (Li et al., 2016).
In this paper, we propose a general, reference-free approach that deﬁnes diversity in terms of a user-speciﬁed similarity function.
Our approach is based on work in ecology, where biological diversity has been deﬁned as the exponential of the entropy of the distribution of species within a population (Hill, 1973; Jost, 2006;
Leinster, 2021). This value can be interpreted as the eﬀective number of species in the population. To adapt this approach to ML, we deﬁne the diversity of a collection of elements x 1 , . . . , x n as the exponential of the entropy of the eigenvalues of the n × n similarity matrix K , whose entries are equal to the similarity scores between each pair of elements.
This entropy can be seen as the von
Neumann entropy associated with K (Bach, 2022), so we call our metric the Vendi Score , for the von Neumann diversity.
[Figure]
does.
VS is highest when the items in the sample diﬀer in many attributes, and the attributes are not correlated with each other.
Contributions.
We summarize our contributions as follows:
• We extend ecological diversity to ML, and propose the Vendi Score, a metric for evaluating diversity in ML.
We study the properties of the Vendi Score, which provides us with a more formal understanding of desiderata for diversity.
• We showcase the ﬂexibility and wide applicability of the Vendi Score–characteristics that stem from its sole reliance on the sample to be evaluated for diversity and a user-deﬁned similarity function–and highlight the shortcomings of existing metrics used to measure diversity in diﬀerent domains.
Related Work
Are we measuring diversity correctly in ML?
Several existing metrics for diversity rely on a reference distribution/dataset, which hinders ﬂexibility.
These reference-based metrics deﬁne diversity in terms of coverage of a reference sample.
They assume access to an embedding function that maps samples to real-valued vectors, such as a pretrained Inception model (Szegedy et al., 2016).
The Fr´echet Inception Distance (FID; Heusel et al., 2017) measures the Wasserstein-2 distance between Gaussian distributions ﬁt to reference embeddings and sample embeddings.
FID was originally proposed for evaluating image GANs but has been applied to text (C´ıfka et al., 2018) and
molecules (Preuer et al., 2018) using domain-speciﬁc neural network encoders.
Sajjadi et al.
(2018) proposed a two-metric evaluation using precision and recall, with precision measuring quality and recall measuring diversity, in terms of coverage of the reference distribution, and a number of variations and improvements have been proposed (Kynk¨a¨anniemi et al., 2019;
Simon et al., 2019; Naeem et al., 2020).
Compared to these approaches, the Vendi Score can measure diversity without relying on a reference distribution/dataset.
Some other existing metrics evaluate diversity using a pre-trained classiﬁer.
Inception Score (IS;
Salimans et al., 2016) evaluates diversity using the entropy of the marginal distribution of class labels predicted by an ImageNet classiﬁer.
More generally, given a pretrained classiﬁer, mode coverage can be calculated directly by classifying test samples and calculating the number of unique modes or the entropy of the mode distribution (Srivastava et al., 2017).
These metrics deﬁne diversity in terms of predeﬁned categories, and require knowledge of the ground truth categories and a separate classiﬁer.
In some discrete domains, diversity is often evaluated in terms of the distribution of unique features.
For example in NLP, a standard metric is n-gram diversity, which is deﬁned as the number of distinct n-grams divided by the total number of n-grams (e.g. Li et al., 2016).
These metrics require an explicit, discrete feature representation.
There are proposed metrics that use similarity scores to deﬁne diversity.
The most widely used metric of this form is the average pairwise dissimilarity score.
In biology, average dissimilarity is known as Internal Diversity (IntDiv; Benhenda, 2017), with similarity deﬁned as the Jaccard (Tanimoto) similarity beteween molecular ﬁngerprints. Given samples x 1 , . . . , x n and a pairwise similarity function k taking values between 0 and 1, IntDiv is deﬁned: [Equation]
In text, variants of this metric include Pairwise BLEU (Shen et al., 2019) and D-Lex-Sim (Fomicheva et al., 2020), in which the similarity function is an n-gram overlap metric such as BLEU (Papineni et al., 2002).
As illustrated in Figure 1, IntDiv becomes less sensitive as diversity increases and does not account for correlations between features. Related to the metric we propose here is a similaritysensitive diversity metric proposed in ecology by Leinster and Cobbold (2012), and which was introduced in the context of ML by Posada et al.
This metric is based on a notion of entropy deﬁned in terms of a similarity proﬁle , a vector whose entries are equal to the expected similarity scores of each element. Like IntDiv, it does not account for correlations between features.
Some other diversity metrics in the ML literature fall outside of these categories.
The Birthday
Paradox Test (Arora and Zhang, 2018) aims to estimate the size of the support of a generative model, but requires some manual inspection of samples.
GILBO (Alemi and Fischer, 2018) is a referencefree metric but is only applicable to latent variable generative models.
Determinantal point processes.
The Vendi Score bears a relationship to determinantal point processes (DPPs), which have been used in machine learning for diverse subset selection (Kulesza et al., 2012). A DPP is a probability distribution over subsets of a ground set X parameterized by a positive semideﬁnite kernel matrix K . The likelihood of drawing any subset X ⊆
X is de- ﬁned as proportional to | K X | , the determinant of the similarity matrix restricted to elements in X : p ( X ) ∝ | K X | = (cid:81) i λ i , where λ i are the eigenvalues of K X .
The likelihood function has a geomet- ric interpretation, as the square of the volume spanned by the elements of X in an implicit feature space. However, the DPP likelihood is not commonly used for evaluating diversity, and has some limitations. For example, it is always equal to 0 if the sample contains any duplicates, and the geometric meaning is arguably less straightforward to interpret than the Vendi Score, which can be understood in terms of the eﬀective number of dissimilar elements. Spectral clustering.
The eigenvalues of the similarity matrix are also related to spectral clustering algorithms (Von Luxburg, 2007), which use a matrix known as the graph Laplacian, deﬁned L = D − K , where K is a symmetric, weighted adjacency matrix with non-negative entries, and D is a diagonal matrix with D i,i = (cid:80)
j K i,j . The eigenvalues of L can be used to characterize diﬀerent properties of the graph—for example, the multiplicity of the eigenvalue 0 is equal to the number of connected components. As a metric for diversity, the Vendi Score is somewhat more general than the number of connected components: it provides a meaningful measure even for fully connected graphs, and captures within-component diversity.
Measuring Diversity with the Vendi Score
We now deﬁne the Vendi Score, state its properties, and study its computational complexity.
(We relegate all proofs of lemmas and theorems to the appendix.)
3.1 Defining the Vendi Score
To deﬁne a diversity metric in ML we look to ecology, the ﬁeld that centers diversity in its work.
In ecology, one main way diversity is deﬁned is as the exponential of the entropy of the distribution of the species under study (Jost, 2006; Leinster, 2021).
This is a reasonable index for diversity.
Consider a population with a uniform distribution over n species, with entropy log( n ) .
This population has maximal ecological diversity n , the same diversity as a population with n members, each belonging to a diﬀerent species.
The ecological diversity decreases as the distribution over the species becomes less uniform, and is minimized and equal to one when all members of the population belong to the same species.
How can we extend this way of thinking about diversity to ML?
One naive approach is to deﬁne diversity as the exponential of the Shannon entropy of the probability distribution deﬁned by a machine learning model or dataset.
However, this approach is limiting in that it requires a probability distribution for which entropy is tractable, which is not possible in many ML settings.
We would like to deﬁne a diversity metric that only relies on the samples being evaluated for diversity.
And we would like for such a metric to achieve its maximum value when all samples are dissimilar and its minimum value when all samples are the same.
This implies the need to deﬁne a similarity function over the samples.
Endowed with such a similarity function, we can deﬁne a form of entropy that only relies on the samples to be evaluated for diversity.
This leads us to the Vendi Score:
Deﬁnition 3.1 (Vendi Score) .
Let x 1 , . . .
, x n ∈ X denote a collection of samples, let k : X ×X → R be a positive semideﬁnite similarity function, with k ( x, x ) = 1 for all x , and let K ∈ R n × n denote the kernel matrix with entry K i,j
= k
( x i , x j ) .
Denote by λ 1 , . .
, λ n the eigenvalues of K /n .
The
Vendi Score (VS) is deﬁned as the exponential of the Shannon entropy of the eigenvalues of K /n :
[Equation]
where we use the convention 0 log 0 = 0 .
To understand the validity of the Vendi Score as a mathematical object, note that the eigenvalues of K /n are nonnegative (because k is positive semideﬁnite) and sum to one (because the diagonal entries of K /n are equal to 1 /n ).
The Shannon entropy is therefore well-deﬁned and the Vendi Score is well-deﬁned.
In this form, the Vendi Score can also be seen as the eﬀective rank of the kernel matrix K /n .
Eﬀective rank was introduced by Roy and Vetterli (2007) in the context of signal processing; the eﬀective rank of a matrix is deﬁned as the exponential of the entropy of the normalized singular values.
The Vendi Score can be expressed directly as a function of the kernel similarity matrix K :
Lemma 3.1.
Consider the same setting as Deﬁnition 3.1.
Then
[Equation]
The lemma makes explicit the connection of the Vendi Score to quantum statistical mechanics:
the
Vendi Score is equal to the exponential of the von Neumann entropy associated with K /n (Bach, 2022).
Our formulation of the Vendi Score assumes that x 1 , . . .
, x n were sampled independently, and so p ( x i ) ≈ 1 n for all i .
This is the usual setting in ML and the setting we study in our experiments.
However, we can generalize the Vendi Score to a setting in which we have an explicit probability distribution over the sample space X (see Deﬁnition 3.1 in the appendix).
3.2 Understanding the Vendi Score
The Vendi Score has several desirable properties as a diversity metric.
We summarize them in the following theorem.
Theorem 3.1 (Properties of the Vendi Score) .
Consider the same deﬁnitions in Deﬁnition 3.1.
(a) Eﬀective number: If k ( x i , x j ) =
for all i (cid:54)
=
j , then VS k ( x 1 , . . .
, x n ) is maximized and equal to n .
If k ( x i , x j ) = 1 for all i, j , then VS k ( x 1 , . . . , x n ) is minimized and equal to 1.
(b) Partitioning: Suppose S 1 , . .
, S m are collections of samples such that, for any i (cid:54) =
j , for all x ∈ S i , x (cid:48)
∈
S j , k ( x, x (cid:48) )
= 0 .
Then the diversity of the combined samples depends only on the diversities of S 1 , . .
, S m and their relative sizes.
In particular, if p i = | S i | / (cid:80)
j | S j | is the relative size of S i and H ( p 1 , . .
, p m ) denotes the Shannon entropy, then VS is given by the geometric mean, VS [Equation]
S S S
Number of components S S S
S S Mixture proportions S
S S S S
Per-component variance
S S S 1 S 2 S 3 S 4 S
VS
S 1 S 2 S 3 S 4 S VS S 1 S 2 S 3 S 4 S
VS S 1 S 2 S 3 S 4 S0 .0 .1 .
IntDiv
S 1 S 2 S 3 S 4 S0 .0 .1 . IntDiv
S 1 S 2 S 3 S 4 S0 .0 .1 . IntDiv
[Equation]
Taking the exponential of entropy turns it into an eﬀective number .
The value of measuring diversity with eﬀective numbers has been argued in ecology (e g. Hill, 1973; Patil and Taillie, 1982; Jost, 2006) and economics (Adelman, 1969).
Eﬀective numbers provide a consistent basis for interpreting diversity scores and make it possible to compare diversity scores using ratios and percentages.
For example, suppose a population contains one million equally abundant species, and 50% go extinct.
The diversity score will decrease by 50%, while the entropy will decrease by only about 5%.
3.3 Calculating the Vendi Score
Calculating the Vendi Score for a sample of n elements requires ﬁnding the eigenvalues of an n × n matrix, which has a time complexity of O ( n 3 ) .
The Vendi Score can be approximated using column sampling methods (i.e. the Nystr¨om method; Williams and Seeger, 2000).
However, in many of the applications we consider, the similarity function we use is the cosine similarity between explicit feature vectors φ ( x ) ∈ R d , with d < n .
That is, K = X (cid:62) X , where X ∈ R n × d is the feature matrix with row
X i, : = φ ( x i ) / (cid:107) φ ( x i ) (cid:107) 2 .
The eigenvalues of K /n are the same as the eigenvalues of the matrix XX (cid:62) /n , therefore we can calculate the Vendi Score exactly in a time of O
( d 2 n + d 3 )
=
O ( d 2 n ) .
This is the same complexity as existing metrics such as FID (Heusel et al., 2017), which require calculating the covariance matrix of Inception embeddings.
Experiments
We illustrate the Vendi Score (VS) on synthetic data to illustrate that it captures intuitive notions of diversity, and then apply it to a variety of setting in ML.
We used VS to evaluate the diversity of generative models of molecules, an application where diversity plays an important role in enabling discovery, and found that VS identiﬁes some model weaknesses that are not detected by IntDiv, the standard metric in that domain.
We also applied VS to generative models of images, and decoding algorithms of text, where we found it conﬁrms what we know about diversity in those applications.
We also used VS to measure mode collapse in GANs and show that it reveals ﬁner-grained distinctions in diversity than current metrics for measuring mode collapse.
Finally, we used VS to analyze the [Figure]
il a r i t y diversity of several image, text, and molecule datasets, gaining insights into the least and most diverse elements of those datasets.
(Implementation details are provided in Appendix B.)
4.1 Synthetic experiments
To illustrate the behavior of the Vendi Score, we calculate the diversity of simple datasets drawn from a mixture of univariate normal distributions, varying either the number of components, the mixture proportions, or the per-component variance. We measure similarity using the RBF kernel: k ( x, x (cid:48) ) =
exp( (cid:107) x − x (cid:48) (cid:107) 2 / 2 σ 2 ) .
The results are illustrated in Figure 2.
VS behaves consistently and intuitively in all three settings: in each case, VS can be interpreted as the eﬀective number of modes, ranging between one and ﬁve in the ﬁrst two rows and increasing from ﬁve to seven in the third row as we increase within-mode variance.
On the other hand, the behavior of IntDiv is diﬀerent in each settings: for example, IntDiv is relatively insensitive to within-mode variance, and additional modes bring diminishing returns.
In Appendix C.1, we also validate that VS captures mode dropping in a simulated setting, using image and text classiﬁcation datasets, where we have information about the ground truth class distribution. In both cases, VS has a stronger correlation with the true number of modes compared to IntDiv. 4.2 Evaluating molecular generative models for diversity Next, we evaluate the diversity of samples from generative models of molecules.
For generative models to be useful for the discovery of novel molecules, they ought to be diverse.
The standard diversity metric in this setting is IntDiv.
We evaluate samples from generative models provided in the MOSES benchmark (Polykovskiy et al., 2020), using the ﬁrst 2,500 valid molecules in each sample.
Following prior work, our similarity function is the Morgan ﬁngerprint similarity (radius 2), implemented in RDKit.
In Figure 3, we highlight an instance where IntDiv and VS disagree: IntDiv ranks the HMM among the most diverse models, while VS ranks it as the least diverse (the complete results are in Appendix Table 4).
The HMM has a high IntDiv score because, on average, the HMM molecules have low pairwise similarity scores, but there are a number of clusters of identical or nearly identical molecules.
4.3 Assessing mode collapse in generative adversarial networks (gans)
Mode collapse is a failure mode of GANs that has received a lot of attention from the ML community (Metz et al., 2017; Dieng et al., 2019).
The main metric for measuring mode collapse, called number of modes , can only be used to assess mode collapse for GANs trained on a labelled dataset. [Table] [Table]
the diﬀusion model gets the best quality scores as measured by FID, but scores lower on VS.
No model matches the diversity score of the original dataset they were trained on.
Number of modes is computed by training a classiﬁer on the labeled training data and counting the number of unique classes that are predicted by the trained classiﬁer for the generated samples.
In
Table 1, we evaluate two models that were trained on the StackedMNIST dataset, a standard setting for evaluating mode collapse in GANs.
StackedMNIST is created by stacking three MNIST images along the color channel, creating 1000 classes corresponding to 1000 number of modes.
We calculate VS using the probability product kernel (Jebara et al., 2004): k ( x, x (cid:48) )
= (cid:80) y p
( y | x ) 12 p ( y | x (cid:48) ) 12 , where the class likelihoods are given by the classiﬁer.
We compare PresGAN (Dieng et al., 2019) and Self-conditioned GAN (Liu et al., 2020), two GANs that are known to capture all the modes.
Table 1 shows that PresGAN and Self-conditioned GAN have the same diversity according to number of modes, they capture all 1000 modes.
However, VS reveals a more ﬁne-grained notion of diversity, indicating that PresGAN is more diverse than Self-conditioned GAN and that both are less diverse than the original dataset.
One possibility is that VS is capturing imbalances in the mode distribution.
To see whether this is the case, we also calculate Mode Diversity, the exponential entropy of the predicted mode distribution: exp H (ˆ p ( y )) , where ˆ p ( y ) = 1 n (cid:80) ni =1 p ( y | x i ) .
The generative models score lower on VS than Mode Diversity, indicating that low scores cannot be entirely attributed to imbalances in the mode distribution.
Therefore VS captures more aspects of diversity, even when we are using the same representations as existing methods.
4.4 Evaluating image generative models for diversity
We now evaluate several recent models for unconditional image generation, comparing the diversity scores with standard evaluation metrics, Inception Score (IS; Salimans et al., 2016), Frechet
Inception Distance (FID; Heusel et al., 2017), Precision (Sajjadi et al., 2018), and Recall (Sajjadi et al., 2018).
The models we evaluate represent popular classes of generative models, including a variational autoencoder (VDVAE; Child, 2020), a ﬂow model (DenseFlow; Grci´c et al., 2021), diffusion models (IDDPM, Nichol and Dhariwal, 2021; ADM Dhariwal and Nichol, 2021), GAN-based models (Karras et al., 2019; 2020), and an auto-regressive model (RQ-VT; Lee et al., 2022).
The models are trained on CIFAR-10 (Krizhevsky, 2009), ImageNet (Russakovsky et al., 2015), or two categories from the LSUN dataset (Yu et al., 2015).
We either select models that provide precomputed samples, or download publicly available model checkpoints and sample new images using the default hyperparameters.
(More details are in Appendix B.)
The standard metrics in this setting use a pre-trained Inception ImageNet classiﬁer to map images to real vectors.
Therefore, we calculate VS using the cosine similarity between Inception embeddings, using the same 2048-dimensional representations used for evaluating FID and Precision/Recall.
As a result, the highest possible similarity score is 2048.
The baseline metrics are reference-based, with the exception of the IS.
FID and IS capture diversity implicitly.
Recall was introduced to capture diversity explicitly, with diversity deﬁned as coverage of the reference distribution.
The results of this comparison are in Table 2.
On the lower resolution datasets (left), VS generally agrees with the existing metrics.
On those datasets the diﬀusion model performs better on all of the metrics.
On the LSUN datasets (right), the diﬀusion model gets the best quality scores as measured by FID, but scores lower on VS.
No model matches the diversity score of the original dataset they were trained on.
In addition to comparing the diversity of the models, we can also compare the diversity scores between datasets: as a function of Inception similarity, the most diverse dataset is
ImageNet 64 × 64, followed by CIFAR-10, followed by LSUN Cat, and then LSUN Bedroom.
This agrees with the intuition that the Inception network will induce the highest-rank embedding space for datasets that most resemble the training distribution.
VS should be understood as the diversity with respect to a speciﬁc similarity function, in this case, the Inception ImageNet similarity.
We illustrate this point in the appendix (Figure 6) by comparing the top eigenvalues of the kernel matrices corresponding to the cosine similarity between Inception embeddings and pixel vectors.
Inception similarity captures a form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel provides a simple form of visual similarity, with components corresponding to broad diﬀerences in lightness, darkness, and color.
[Table]
We evaluate diversity on the MS COCO image-captioning dataset (Lin et al., 2014), following prior work on diverse text generation (Vijayakumar et al., 2018).
In this setting, the subjects of evaluation are diverse decoding algorithms rather than parametric models.
Given a ﬁxed model p ( x | c ) , where c is some conditioning context, the aim is to identify a “Diverse N-Best List”, a list of sentences that have high likelihood but are mutually distinct. a list of sentences that have high likelihood but are mutually distinct.
The baseline metric we compare to is n-gram diversity (Li et al., 2016), which is the proportion of unique n-grams divided by the total number of n-grams.
We deﬁne similarity using the n-gram overlap kernel: for a given n , the n-gram kernel k n is the cosine similarity between bag-of-n-gram feature vectors.
We use the average of k 1 , . . .
, k 4 .
This ensures that VS and n-gram diversity are calculated using the same feature representation.
Each image in the validation split has ﬁve captions written by diﬀerent human annotators, and we compare these with captions generated by a publicly available captioning model trained on this dataset.
For each image, we generate ﬁve captions using either beam search or diverse beam search (DBS Vijayakumar et al., 2018).
DBS takes a parameter, γ , called the diversity penalty, and we vary this between 0.2, 0. [Figure]
Table 3 shows that all diversity metrics increase as expected, ranking beam search the lowest, the human captions the highest, and DBS in between, increasing with the diversity penalty.
The human diversity score of 4.88 can be interpreted as meaning that, on average, all ﬁve human-written captions are almost completely dissimilar from each other, while beam search eﬀectively returns only three distinct responses for every ﬁve that it generates.
4.6 Diagnosing datasets for diversity
In Figure 4, we calculate VS for samples from diﬀerent categories in CIFAR-100, using the cosine similarity between either Inception embeddings or pixel vectors.
The pixel diversity is highest for categories like “aquarium ﬁsh”, which vary in color, brightness, and orientation, and lowest for categories like “cockroach” in which images have similar regions of high pixel intensity (like white backgrounds).
The Inception diversity is less straightforward to interpret, but might correspond to some form of semantic diversity—for example, the Inception diversity might be lower for classes like “castle,” that correspond to distinct ImageNet categories, and higher for categories like “clock” and “keyboard” that are more diﬃcult to classify.
In Appendix C.5, we show additional examples from text, molecules, and other image datasets.
Discussion
We introduced the Vendi Score, a metric for evaluating diversity in machine learning.
The Vendi
Score is deﬁned as a function of the pairwise similarity scores between elements of a sample and can be interpreted as the eﬀective number of unique elements in the sample.
The Vendi Score is interpretable, general, and applicable to any domain where similarity can be deﬁned.
It is unsupervised, in that it doesn’t require labels or a reference probability distribution.
Importantly, the Vendi Score allows its user to specify the form of diversity they want to measure via the similarity function.
We showed the Vendi Score can be computed eﬃciently exactly and showcased its usefulness in several
ML applications, diﬀerent datasets, and diﬀerent domains.
In future work, we’ll leverage the Vendi
Score to improve data augmentation, an important ML approach in settings with limited data.
A Proofs A.1 Probability-weighted Vendi Score
Deﬁnition A.1 (Probability-Weighted Vendi Score) . Let p ∈ ∆ n denote a probability distribution on a discrete space X = { x 1 , . . .
, x n } , where ∆ n denotes the ( n − 1) -dimensional simplex, let k : X × X
→ R be a positive semideﬁnite similarity function, with k ( x, x ) = 1 for all x , and let
K ∈ R n × n denote the kernel matrix with K i,j = k ( x i , x j ) .
Let ˜ K
p = diag( √ p )
K diag( √ p ) denote the probability-weighted kernel matrix.
Let λ 1 , . . .
λ n denote the eigenvalues of ˜ K p .
The
Vendi Score (VS) is deﬁned as the exponential of the Shannon entropy of the eigenvalues of ˜ K p :
[Equation]
When all elements in the sample are completely dissimilar, the probability-weighted Vendi Score deﬁned in Deﬁnition A.1 reduces to the exponential of the Shannon entropy of the weighting distribution:
Lemma A.1.
Let p ∈ ∆ n be a probability distribution over x 1 , . . .
{+, x n and suppose k ( x i , x j ) = 
+}
for all i (cid:54)
=
j .
Then VS k ( x 1 , . . .
, x n , p ) =
exp H ( p ) , the exponential of the Shannon entropy of p .
Rather than prove ?? directly, we prove a more general lemma for the probability-weighted kernel matrix ˜ K p deﬁned in Deﬁnition A.1. ?
? follows by setting p =
(1 /N, . .
, 1 /N ) to be the uniform distribution on
N elements.
Lemma A.2.
Let K ∈ R N × N denote a positive semi-deﬁnite kernel matrix with K ii = 1 for i ∈ { 1 , . .
. , N } .
Let ∆ N denote the ( N − 1) -dimensional simplex; its elements are vectors of dimension N whose entries are nonnegative and sum to one.
Let p =
( p 1 , . . .
, p n ) ∈ ∆ N denote a probability distribution on N elements, and let λ = ( λ 1 , . . .
, λ N ) denote the vector of eigenvalues of the probability-weighted kernel matrix
˜ K p = diag( √ p )
K diag( √ p ) .
Then λ ∈ ∆ N .
Proof.
λ is in the ( N − 1) -simplex if its entries are nonnegative and sum to one.
The entries of λ are nonnegative because the eigenvalues of a positive semi-deﬁnite matrix are nonnegative, and ˜ K p is the product of positive semi-deﬁnite matrices and so is positive semi-deﬁnite iteslf.
( diag( √ p ) is positive semi-deﬁnite because it is a diagonal matrix with nonnegative entries.)
The entries sum to one because the trace of a square matrix is equal to the sum of its eigenvalues, and tr( ˜ K p ) =
(cid:80) Ni =1 K ii p i = 1 .
A.3 Proof of Lemma 3.
Lemma.
Consider the same setting as Deﬁnition 3.1.
Then
[Equation]
Proof.
For any square matrix X ∈ R n × n , if X has an eigendecomposition X =
U Λ U − 1 , then log X
=
U (log Λ ) U − 1 , where log Λ = diag(log λ 1 , . .
, log λ n ) is a diagonal matrix whose diagonal entries are the logarithms of the eigenvalues of X .
Also, tr( X ) = tr (cid:0) U Λ U − 1 (cid:1) = tr( Λ ) , because the trace is similarity-invariant.
K /n is diagonalizable because it is positive semideﬁnite, so let K /n =
U Λ U − 1 denote the eigendecomposition.
Then
[Equation]
Therefore
VS
[Equation]
A.4 Proof of Lemma A.
Lemma.
Let p ∈ ∆ n be a probability distribution over x 1 , . . .
, x n and suppose k ( x i , x j ) =
for all i (cid:54)
=
j .
Then VS k ( x 1 , . . .
, x n , p ) =
exp H ( p ) , the exponential of the Shannon entropy of p .
Proof.
If all element in p are completely dissimilar, then ˜ K p is a diagonal matrix, and the eigenvalues λ 1 , . . .
, λ S are the diagonal entries, which are the entries of p .
So the von Neumann entropy of ˜ K p is identical to the Shannon entropy of p , and the exponential is the Vendi Score.
A.5 Proof of Theorem 3.
Proof.
(a) Eﬀective number: If p is the uniform distribution over N completely dissimilar elements, then ˜ K p is a diagonal matrix with each diagonal entry equal to 1 /N .
The eigenvalues of a diagonal matrix are the diagonal entries, so VS K ( p ) =
exp H (1 /N, . . .
, 1 /N )
=
exp log N = N .
On the other hand, if all elements are completely similar to each other, then ˜ K p has rank one
and so
the
Vendi Score is equal to one.
(b) Identical elements: The eigenvalues of ˜ K p are the same as the eigenvalues of the covariance matrix of the corresponding feature space:
[Equation]
Suppose elements i and j are identical, and let p (cid:48) denote the probability distribution created by combining i and j , i.e. p (cid:48)
i
= p i + p j and p (cid:48)
j = 0 .
Clearly, ˜ Σ p = ˜ Σ
p (cid:48) , and so VS k ( x 1 , . . . , x n , p ) =
VS k ( x 1 , . . . ,
x n , p (cid:48) ) .
(c) Partitioning: Suppose N samples are partitioned into M groups S 1 , . . .
, S M such that, for any i (cid:54) =
j , for all x ∈ S i , x (cid:48)
∈
S j , k ( x, x (cid:48) )
= 0 .
Let p i = | S i | / (cid:80) j | S j | denote the relative size of group i , and let K denote kernel matrix of ∪ i S i , sorted in order of group index, and let K S
i denote the restriction of
K to elements in S i .
Then K /N is a block diagonal matrix, with each block i equal to p
i K S
i .
The eigenvalues of a block diagonal matrix are the combined eigenvalues of each block, and the partitioning property then follows from the partitioning property of the Shannon entropy.
(e) Symmetry:
The eigenvalues of a matrix are unchanged by orthonormal transformation, and the
Shannon entropy is symmetric in its arguments, so the Vendi Score is symmetric.
B Implementation Details B.1 Images
Stacked MNIST We train GANs on Stacked MNIST using the publicly available code for PresGANs 4 and self-conditioned GANs 5 .
The models share the same DCGAN (Radford et al., 2015) architecture and are trained on the same dataset of 60,000 Stacked MNIST images, rescaled to 32 ×pixels, and other hyperparameters are set according to the descriptions in the papers.
The models are trained for 50 epochs and the diversity scores are evaluated every ﬁve epochs by taking 10,000 samples.
For both models, we report the scores from the epoch corresponding to the highest VS score.
As in prior work (Metz et al., 2017), we classify Stacked MNIST digits by applying a pretrained MNIST classiﬁer to each color channel independently.
The 1000-dimensional Stacked MNIST probability vector is then the tensor product of the three 10-dimensional probability vectors predicted for the three channels.
Obtaining Image Samples In Section 4.4, we calculate the diversity scores of several recent generative models of images.
We select models that represent a range of families of generative models and and provide publicly available samples or model checkpoints for common image datasets.
On the low-resolution datasets, we generate 50,000 samples from each model using the oﬃcial code for
VDVAE, 6 DenseFlow, 7 , and IDDPM, 8 , each of which provides a checkpoint for unconditional image generation models on CIFAR-10 and ImageNet-64.
For IDDPM, we sample using DDIM (Song et al., 2021) for 250 steps, and otherwise use the default sampling parameters.
For the higher-resolution datasets, we use the 50,000 precomputed samples provided by Dhariwal and Nichol (2021) 9 for ADM and StyleGAN models.
We obtain 50,000 samples from the RQ-VAE/Transformer model using the code and checkpoints provided by the authors, 10 with the default sampling parameters.
Calculating Image Metrics In Table 2, we calculate standard image quality and diversity metrics, which are based on Inception embeddings.
These Inception-based metrics are sensitive to a number of implementation details (Parmar et al., 2022) and in general cannot be compared directly between papers.
For a consistent comparison, we calculate all scores using the evaluation code provided by Dhariwal and Nichol (2021).
We also calculated FID and Precision/Recall using the provided reference images and statistics, with the exception of CIFAR-10, for which we use the training set as the reference.
(The diversity scores of the Original datasets in Table 2 are calculated using these reference images.)
As a result, the numbers in this table may not be directly comparable to results reported in prior work.
B.2 Text
Obtaining Image Captions In Section 4.5, we sample image captions from a pretrained imagecaptioning model, 11 which is publicly available in Hugging Face (Wolf et al., 2019), and we use the
Hugging Face implementation of beam search and diverse beam search.
For beam search we use a beam size of 5.
For diverse beam search, we use a beam size of 10, a beam group size of 10, and set the number of return sequences to 5.
Calculating Text Metrics
The text metrics we use are calculated in terms of word n-grams, and therefore depend on how sentences are tokenized into words.
We calculate all text metrics using the pre-trained wordpiece tokenizer used by the captioning models.
We use the implementation of the
BLEU score in NLTK (Bird, 2006).
C Additional Results C.1 Assessing Mode Dropping in Datasets
[Figure]
In Figure 5, we examine whether VS captures mode dropping in a controlled setting, where we have information about the ground truth class distribution.
We simulate mode dropping by sampling equalsized subsets of two classiﬁcation datasets, with each subset S i containing examples sampled uniformly from the ﬁrst i categories.
We perform this experiment one image dataset (MNIST) and one text dataset (MultiNLI; Williams et al., 2018), using simple similarity functions.
For MNIST, we use the cosine similarity between pixel vectors.
In MultiNLI, we use the premise sentences from the validation split (mismatched), which are drawn from one of ten genres.
We deﬁne similarity using the n-gram overlap kernel: for a given n , the n-gram kernel k n is the cosine similarity between bag-of-n-gram feature vectors, and we use the average of k 1 , . . .
, k 4 .
The results (Figure 5) show that VS generally increases with the number of classes.
In MNIST (left),
VS increases roughly linearly for the ﬁrst six digits (0-5) and then ﬂuctuates.
This could occur if the new modes are similar to the other modes in the sample, or have low internal diversity.
In MultiNLI (right), VS increases monotonically with the number of genres represented in the sample.
In both cases, VS has a stronger correlation with the number of modes compared to IntDiv.
C.2 Evaluating molecular generative models for diversity
We evaluate samples from generative models provided in the MOSES benchmark (Polykovskiy et al., 2020), using the ﬁrst 2,500 valid molecules in each sample.
Following prior work, our similarity function is the Morgan ﬁngerprint similarity (radius 2), implemented in RDKit.
12 IntDiv ranks the
HMM among the most diverse models, while VS ranks it as the least diverse (see Section 4.2).
C.3 Evaluating image generative models for diversity
In Table 5, we replicate the table described in Section 4.4 and add an additional column, which evaluates diversity using the cosine similarity between pixel vectors as the similarity function.
VS should be understood as the diversity with respect to a speciﬁc similarity function, in this case, the Inception ImageNet similarity.
We illustrate this point in Figure 6 by comparing the top eigenvalues of the kernel matrices corresponding to the Inception similarity and the pixel similarity, which we calculate by resizing the images to 32 × 32 pixels and taking the cosine similarity between pixel
[Table]
[Table]
vectors.
Inception similarity provides a form of semantic similarity, with components correspond
ing to particular cat breeds while the pixel kernel provides a
simple
form of visual similarity, with components corresponding to broad diﬀerences in lightness, darkness, and color. C.4 Evaluating decoding algorithms for text for diversity In Figure 7, we plot the relationship between VS and n-gram diversity using the MS-COCO captioning data and the n-gram overlap kernel described in Section 4.5.
The ﬁgure shows that VS is highly correlated with n-gram diversity, which is expected given that our similarity function is based on n-gram overlap.
Nonetheless, there are some data points that the metrics rank diﬀerently.
This is because n-gram diversity conﬂates two properties: the diversity of n-grams within a single sentences and the n-gram overlap between sentences.
We highlight two examples in Figure 8.
In general, the instances that n-gram diversity ranks lower compared to VS contain individual sentences that repeat phrases.
On the other hand, n-gram diversity can be inﬂated in cases when one sentence in the sample is much longer than the others, even if the other sentences are not diverse. Least positive Most positive
Inception kernel
K e r n e l p r i n c i p a l c o m p o n e n t s
Least positive
Most positive Pixel kernel K e r n e l p r i n c i p a l c o m p o n e n t s [Figure]
C.5 Diagnosing datasets for diversity
Molecules We evaluate the diversity scores of molecules in the GoodScents database of perfume materials, 13 which has been used in prior machine learning research on odor modeling (SanchezLengeling et al., 2019).
We use the standardized version of the data provided by the Pyrfume library.
14 Each molecule in the dataset is labeled with one or more odor descriptors (for example, “clean, oily, waxy” or “ﬂoral, fruity, green”).
We form groups of molecules corresponding to the seven most common odor descriptors, with each group consisting of 500 randomly sampled
High Vendi Score, low n-gram diversity:
• two men in bow ties standing next to steel rafter.
• several men in suits talking together in a room.
• an older man in a tuxedo standing next to a younger man in a tuxedo wearing glasses.
• two men wearing tuxedos glance at each other.
• older man in tuxedo sitting next to another younger man in tuxedo .
Low Vendi Score, high n-gram diversity:
• a man and woman cutting a slice of cake by trees.
• a couple of people standing cutting a cake.
• the dork with the earring stands next to the asian beauty who is way out of his league.
• a newly married couple cutting a cake in a park.
• a bride and groom are cutting a cake as they smile.
molecules.
We evaluate VS using two similarity functions: the Morgan ﬁngerprint similarity (radius 2), and the similarity between odor descriptors, deﬁned as the cosine similarity between descriptor indicator vectors φ ( x ) , where φ i ( x ) is equal to one if descriptor i is associated with molecule x and zero otherwise.
[Figure]
, and the odor-descriptor similarity is the cosine similarity between binary tag indicator vectors.
The diversity scores are plotted in Figure 9.
The molecular diversity score and the odor-descriptor diversity scores are correlated, meaning that words like “woody” and “green” are used to describe molecules that vary in molecular structure and also elicit diverse odor descriptions, while words like “waxy” and “fatty” are used for molecules that are similar to each other and elicit similar odor descriptions.
For example, the word “green” appears in tag sets such as “aldehydic, citrus, cortex, green, herbal, tart” and “ﬂoral, green, terpenic, tropical, vegetable, woody”, whereas the word “waxy” tends to co-occur with the same tags (“fresh, waxy”; “fresh, green, melon rind, mushroom, tropical, waxy”; “fruity, green, musty, waxy”).
Molecules from the categories with the highest and lowest scores are illustrated in Figure 10.
[Figure]
N-grams
S i m
C S E
Slate
Government
Telephone
Travel
Fiction
MultiNLI genres
N-grams
S i m
C S E 2 stars 4 stars 5 stars 1 stars 3 stars
Amazon reviews
Text In Figure 11, we evaluate the diversity scores of samples sentences with diﬀerent genres, from the MultiNLI dataset (Williams et al., 2018), and Amazon product reviews with diﬀerent star ratings (Keung et al., 2020), using either the n-gram overlap similarity or SimCSE (Gao et al., 2021).
SimCSE is a Transformer-based sentence encoder that achieves state-of-the-art scores on semantic similarity benchmarks.
The model we use initialized from the uncased BERT-base model (Devlin et al., 2019) and trained with a contrastive learning objective to assign high similarity scores to pairs of MultiNLI sentences that have a logical entailment relationship.
In MultiNLI, both models assign the highest score to Slate, which consists of sentences from articles published on slate.com.
SimCSE assigns a higher score to the “Fiction” category, possibly because it is less sensitive to common n-grams (e.g. “he said”), that appear in many sentences in this genre and contribute to the low N-gram diversity score.
In the Amazon review dataset, the 5-star reviews have the highest N-gram diversity but the lowest SimCSE diversity, perhaps because SimCSE assigns high similarity scores to sentences that have the same strong sentiment.
SimCSE assigns the highest diversity score to 3-star reviews, which can vary in sentiment.
Images Following the setting in Section 4.6, we evaluate two additional datasets, Fashion
MNIST (Xiao et al., 2017) and CelebA (Liu et al., 2015).
We use the same similarity scores as in Section 4.6.
Images in CelebA are associated with 40-dimensional binary attribute vectors.
We use these attributes as an additional similarity score, deﬁning the attribute similarity as the cosine similarity between attribute vectors.
These illustrations highlight the importance of the choice of similarity function in deﬁning a diversity metric.
[Figure]
I n c e p t i o n
Wearing_Necktie
Rosy_Cheeks
Least diverse
Blurry
Wearing_Hat
Most diverse
P i x e l s
Blond_Hair
Rosy_Cheeks
Least diverse
Wearing_Hat
Mustache
Most diverse
A tt r i b u t e s
Rosy_Cheeks
Heavy_Makeup
Least diverse
Big_Nose
Male
Most diverse