C OMPARING D ISTRIBUTIONS BY M EASURING D IFFERENCES THAT A FFECT D ECISION M AKING
A BSTRACT
Measuring the discrepancy between two probability distributions is a fundamental problem in machine learning and statistics.
We propose a new class of discrepancies based on the optimal loss for a decision task – two distributions are different if the optimal decision loss is higher on their mixture than on each individual distribution.
By suitably choosing the decision task, this generalizes the JensenShannon divergence and the maximum mean discrepancy family.
We apply our approach to two-sample tests, and on various benchmarks, we achieve superior test power compared to competing methods.
In addition, a modeler can directly specify their preferences when comparing distributions through the decision loss.
We apply this property to understanding the effects of climate change on different economic activities and selecting features targeting different decision tasks.
I NTRODUCTION
Quantifying the difference between two probability distributions is a fundamental problem in machine learning.
Modelers choose different types of discrepancies (or probability divergences) to encode their prior knowledge about which aspects are relevant to evaluate the difference.
Integral probability metrics (IPMs, M ¨ uller (1997)) and f -divergences (Csisz ´ ar, 1964) are widely used discrepancies in machine learning.
IPMs, such as the Wasserstein distance, maximum mean discrepancy (MMD) (Rao, 1982; Burbea & Rao, 1984; Gretton et al., 2012), are based on the idea that if two distributions are identical, any function should have the same expectation under both distributions.
IPMs are used to deﬁne training objectives for generative models (Arjovsky et al., 2017), perform independence tests (Doran et al., 2014), robust optimization (Esfahani & Kuhn, 2018) among many other applications.
f -divergences, such as the KL divergence and the Jensen Shannon divergence, are based on the idea that if two distributions are identical, they assign the same likelihood to every point.
One can then deﬁne a discrepancy based on how different the likelihood ratio is from one.
KL divergence underlies some of the most commonly used training objectives for both supervised and unsupervised machine learning algorithms, such as cross entropy loss.
We propose a third category of divergences called H-divergences that overlaps with but also extends the set of integral probability metrics or the set f -divergences.
Intuitively, H-divergence compares two distributions in terms of the optimal loss for a certain decision task.
This optimal loss corresponds to a generalized notion of entropy (DeGroot et al., 1962).
Instead of measuring the best average code length of any encoding scheme (Shannon entropy), the generalized entropy uses arbitrary loss function (rather than code length) and set of actions (rather than encoding schemes), and is deﬁned as the best expected loss among the set of actions.
In particular, given two distribution p and q , we compare the generalized entropy of the mixture distribution ( p + q ) / 2 and the generalized entropy of p and q individually.
Intuitively, if p and q are different, it is more difﬁcult to minimize expected loss under the mixture distribution ( p + q ) / 2 , and hence the mixture distribution should have higher generalized entropy; if p and q are identical, then the mixture distribution is identical to p or q , and hence should have the same generalized entropy.
Our divergence strictly generalizes the maximum mean discrepancy family and the Jensen Shannon divergence, which can be obtained with speciﬁc choices of the loss function.
We illustrate this via the Venn diagram in Figure 1.
Our formulation allows us to choose alternative losses to leverage inductive biases and machine learning models from different problem domains.
For example, if we choose the generalized entropy as the maximum log likelihood of deep generative models, we are able to leverage recent progress in modeling high dimensional images.
We demonstrate the effectiveness of H-divergence in two sample tests, i.e. to decide whether two sets of samples come from the same distribution or not.
A test based on a probability discrepancy declares two sets of samples different if their discrepancy exceeds some threshold.
We use Hdivergences based on generalized entropy deﬁned by the log likelihood of off-the-shelf generative models.
Compared to state-of-the-art tests based on MMD with deep kernels (Liu et al., 2020), tests based on the H-divergence achieve better test power (given identical type I error) on a large set of benchmarks.
More importantly, scientists and policy makers are often interested not only in if two distributions are different, but how two distributions are different and whether the differences affect decision making.
Typical divergence measures (such as KL) or two sample tests only quantify if two distributions are different, while we show that H-divergence is a useful tool for quantifying how distributions are different with three application examples: studying the effect of climate change, feature selection, and sample quality evaluation.
In each of these examples, we compare different aspects of the distributions by choosing speciﬁc decision loss functions.
For example, climate change (Figure 3) might impact agriculture in a region but not energy production, or vice versa.
By choosing suitable loss functions (related to agriculture, energy, etc) we can quantify and test if the change in climate distribution impact different economic activities.
B ACKGROUND 2. P ROBABILITY D IVERGENCES
Let X denote a ﬁnite set or a ﬁnite dimensional vector space, and P ( X ) denote the set of probability distributions on X that have a density.
We consider the problem of deﬁning a probability divergence between any two distributions in P ( X ) , where a probability divergence is any function D : P ( X ) × P ( X ) → R that satisﬁes D ( p (cid:107) q ) ≥ 0 , D ( p (cid:107) p ) = 0 , ∀ p, q ∈ P ( X ) .
We call the divergence D “strict” if D ( p (cid:107) q ) > 0 ∀ p (cid:54) =
q , and “non-strict“ otherwise.
In this paper we consider both types of divergences.
Integral Probability Metrics Let F denote a set of functions X
→ R .
An integral probability metric is deﬁned as IPM F ( p (cid:107) q ) =
sup f ∈F | E p [ f ( X )] − E q [ f ( X )] | .
Several important divergences belong to integral probability metrics.
Examples include the Wasserstein distance, where F is the set of 1 -Lipschitz functions; the total variation distance, where F is the set of functions X
→ [ − 1 , 1] .
The maximum mean discrepancy (MMD) (Rao, 1982; Burbea & Rao, 1984; Gretton et al., 2012) chooses a kernel function k : X × X
→ R + and is deﬁned by
[Equation]
MMD is an IPM where F is the unit norm functions in the reproducing kernel Hilbert space (RKHS) associated with the kernel k .
f -Divergences Given any convex continuous function f : R + → R such that f (1) = 0 , the f -Divergence is deﬁned as (assuming densities exist) D f ( p (cid:107) q ) =
E q [ f ( p ( X ) /q
( X ))] .
Examples include the KL divergence, where f : t (cid:55)→ t log t and the Jensen Shannon divergence, where f : t (cid:55)→ ( t + 1) log (cid:16) 2 t +1 (cid:17) + t log t .
2. H-E NTROPY
For any action space A and any loss function (cid:96) : X × A → R , the H-entropy (DeGroot et al., 1962; DeGroot, 2005; Gr¨unwald et al., 2004) is deﬁned as
[Equation]
In words, H -entropy is the Bayes optimal loss of a decision maker who must select some action a not for a particular x , but in expectation for a random x drawn from p ( x ) .
H-entropy generalizes several important notions of uncertainty.
Examples include: Shannon Entropy , where A as the set of probabilities P ( X ) , and (cid:96) ( x, a ) = − log a ( x ) ; Variance where A =
X , and (cid:96) ( x, a ) = (cid:107) x − a (cid:107) 22 ; Predictive V -entropy , where A ⊂ P ( X ) is some subset of distributions, and (cid:96) ( x, a ) =
− log a ( x ) (Xu et al., 2020).
A key property we will use is that H -entropy is concave (DeGroot et al., 1962).
Lemma 1.
For any choice of (cid:96) :
X × A → R , H (cid:96) is a concave function.
This Lemma can be proved by observing that inf is a concave function: it is always better to pick an optimal action for p and q separately rather than a single one for both.
[Equation]
This Lemma reﬂects why H (cid:96) can be thought of as a measurement of entropy or uncertainty.
If the distribution is more uncertain (e.g. a mixture of p and q , rather than p or q separately)
then decisions made under higher uncertainty will suffer a higher loss.
D EFINITION AND T HEORETICAL P ROPERTIES 3. H-J ENSEN S HANNON D IVERGENCE
As a warm up, we present a special case of our divergence.
Deﬁnition 1 (H-Jensen Shannon divergence) .
[Equation]
D JS (cid:96) is always non-negative because H-entropy is concave (Lemma 1), and clearly D JS (cid:96) ( p, q )[-= 
-]
= whenever p = q .
Therefore, D JS (cid:96) is a valid probability divergence.
In particular, if we choose H (cid:96) as the Shannon entropy, Deﬁnition 1 recovers the Jensen Shannon divergence.
Other special loss function choices can recover deﬁnitions in (Burbea & Rao, 1982).
3. G ENERAL H- DIVERGENCE
In addition to the H-Jensen Shannon divergence, there are other functions based on the H-entropy that satisfy the requirements of a divergence.
For example,
[Equation]
is also a valid divergence (this will be proved later as a special case of Lemma 2).
We can deﬁne a general set of divergences that includes the above two divergences with the following deﬁnition:
Deﬁnition 2 (H-divergence) .
For two distributions p, q on X , given any continuous function φ : R→ R such that φ ( θ, λ ) > {+
+} whenever θ + λ > 0 and φ (0 , 0) = 0 , deﬁne
[Equation]
Intuitively H (cid:96) (cid:0) p + q 2 (cid:1) − H (cid:96) ( p ) and H (cid:96) (cid:0) p + q 2 (cid:1) − H (cid:96) ( q ) measure how much more difﬁcult it is to minimize loss on the mixture distribution ( p + q ) / 2 than on p and q respectively.
φ is a general class of functions that map these differences into a scalar divergence, while satisfying some desirable properties described in the next section.
The following proposition shows that the H-divergence generalizes the previous deﬁnitions (1) and (2).
Therefore, any property of H-divergence is inherited by e.g. the H-Jensen Shannon divergence.
Proposition 1.
If φ ( θ, λ ) = θ + λ 2 then D φ(cid:96) ( p, q ) is the H-Jensen Shannon divergence in Eq.(1).
If φ ( θ, λ ) = max( θ, λ ) then D φ(cid:96) ( p, q ) is the H-Min divergence in Eq.(2).
3. P ROPERTIES OF THE H- DIVERGENCE
We ﬁrst verify that D φ(cid:96) is indeed a (strict or non-strict) probability divergence.
Lemma 2.
For any choice of (cid:96) and for any choice of φ that satisfy Deﬁnition 2, D φ(cid:96) is non-negative and D φ(cid:96)
( p, q ) =
whenever p = q .
Furthermore, D φ(cid:96) is symmetric whenever φ is symmetric.
Depending on the choice of (cid:96) , H -divergence may or may not be strict (i.e. whenever p (cid:54) = q ,
D ( p (cid:107) q ) > 0 ).
The following proposition characterizes conditions for a strict divergence.
Proposition 2 (Strict Divergence) .
For any choice of φ the following are equivalent 1) ∀ p (cid:54)
= q ,
D (cid:96) ( p (cid:107) q ) > 0 . 2) The H-entropy H (cid:96) ( p ) : = inf a E p [ (cid:96) ( X, a )] is strictly convex in p . 3) ∀ p (cid:54) = q ,
[Equation] In particular, this proposition can be used to characterize all strict H-divergences, because the set of all losses (cid:96) that induces strict H-entropy functions H (cid:96) can be characterized by Fenchel duality (Duchi et al., 2018).
One important property of the H-divergence is that two distributions have non-zero divergence if and only if they have different optimal actions, i.e. the optimal solutions for their respective H-entropy are different.
This is shown in the following proposition (proof in Appendix A).
Proposition 3.
D φ(cid:96)
( p (cid:107) q ) >
if and only if arg inf a E p [ (cid:96) ( X, a )] ∩ arg inf a E q [ (cid:96) ( X, a )] = ∅ .
Intuitively, D φ(cid:96) only takes into account differences between distributions that lead to different optimal action choices.
This property allows us to incorporate prior domain knowledge.
By choosing A and (cid:96) we can specify which differences between distributions lead to different optimal actions, and which differences do not.
For example, we can choose A as a set of generative models (e.g., mixture of Gaussians) and (cid:96) ( x, a ) as the negative log likelihood of x under generative model a .
If under two distributions we end up learning the same generative model (by maximizing log likelihood), the H-divergence between them is zero.
3. R ELATIONSHIP TO MMD
An important special case of the H-divergence is the set of squared Maximum Mean Discrepencies (MMD), as shown by the following theorem:
Theorem 1.
The set of H-Jensen Shannon Divergences is strictly larger than the MMD 2 distances.
To prove this theorem, we show that for each choice of kernel k : X × X
→ R , there exists an action space A and loss (cid:96) such that the corresponding squared MMD distance and H-divergence are the same (see proof in Appendix A).
In particular, this equivalence can be achieved by choosing A to be the RKHS H of k ( · , · ) , and (cid:96) ( x, a ) = 4 (cid:107) k
( x, · ) − a (cid:107) 2 H .
Inclusion is strict because the Jensen Shannon divergence is a H-Jensen Shannon Divergence but not a squared MMD distance.
3. E STIMATION AND C ONVERGENCE
Many machine learning tasks can be reduced to the problem of estimating the divergence between two distributions given samples.
Speciﬁcally, suppose we are provided with a set of m i.i.d.
samples ˆ p m =
( x 1 , · · · , x m ) drawn from distribution p and ˆ q m =
( x (cid:48) 1 , · · · , x (cid:48) m ) drawn from distribution q , and would like to obtain an estimate of D φ(cid:96) ( p (cid:107) q ) based on the samples.
Here, ˆ p m and ˆ q m denote empirical distributions drawn from p and q respectively.
In this section we propose an empirical estimator for the H-divergence and show that it has favorable convergence properties.
Let ˆ D φ(cid:96)
(ˆ p m (cid:107) ˆ q m ) be the empirical (random) estimator for D φ(cid:96)
( p (cid:107) q ) deﬁned by
[Equation]
where x (cid:48)(cid:48) i
= x i b i + x (cid:48)
i (1 − b i ) and b i are i.i.d uniformly sampled from { 0 , 1 } , so that x (cid:48)(cid:48)
i is a sample from the mixture distribution ( p + q ) / 2 of size m .
Using x (cid:48)(cid:48) i as deﬁned above is crucial for the convergence properties we will prove in Theorem 2.
It might be tempting to replace the term 1 m (cid:80) mi =1 (cid:96) ( x (cid:48)(cid:48)
i , a ) with 12 m (cid:80) mi =1 ( (cid:96) ( x i , a ) + (cid:96) ( x (cid:48)
i , a )) to use all the available samples.
However, optimizing the action based on a ﬁnite set of samples (instead of in expectation) is prone to overﬁtting, and introduces bias.
Intuitively, using m samples ( x (cid:48)(cid:48) i ) ensures the bias for the mixture is comparable to that of p and q .
Without this, Theorem 2 is no longer true, and empirical performance also degrades.
Before presenting the convergence results, we ﬁrst must deﬁne several assumptions that make convergence possible.
In particular, we are going to assume that the loss function (cid:96) is C -bounded, i.e. there exists some C such that 0 ≤ (cid:96) ( x, a ) ≤ C, ∀ a, x .
This assumption seemingly exclude important special cases such as the Jensen-Shannon divergence (which is associated with the unbounded log loss).
However, we show in the appendix that the Jensen-Shannon divergence cannot be consistently estimated in general, hence correctly excluded by our theorem.
One practical solution is to clip the log likelihood, which is the approach adopted in (Song & Ermon, 2019) for improved divergence estimation (for a similar KL divergence estimation problem).
In addition, we assume that φ is 1 -Lipschitz under the ∞ -norm, i.e. | φ ( θ + dθ, λ + dλ ) − φ ( θ, λ ) | ≤ max( dθ, dλ ) , ∀ θ, λ, dθ, dλ ∈ R .
Both φ ( θ, λ )
=
θ + λ 2 and φ ( θ, λ ) =
max { θ + λ } are 1 -Lipschitz under the ∞ -norm.
This is a mild assumption because if φ is not 1 -Lipschitz we can rescale φ to make it 1 -Lipschitz.
Finally, deﬁne the Radamacher complexity
[Equation]
We deﬁne R qm ( (cid:96) ) analogously.
Based on these assumptions and deﬁnitions we can bound the difference between ˆ D φ (cid:96) (ˆ p m (cid:107) ˆ q m ) and D φ (cid:96) ( p (cid:107) q ) .
Theorem 2.
If (cid:96) is C -bounded, and φ is 1 -Lipschitz under the ∞ -norm, for any choice of distribution p, q ∈ P ( X ) and t > 0 we have
Pr[ ˆ D φ(cid:96) (ˆ p m (cid:107) ˆ q m ) ≥ t ] ≤ 4 e − t 2 m 2 C 2 if p = q . 2.
Pr (cid:104)(cid:12)(cid:12)(cid:12) ˆ D φ(cid:96) (ˆ p m (cid:107) ˆ q m ) − D φ(cid:96) ( p (cid:107) q ) (cid:12)(cid:12)(cid:12) ≥ 4 max( R pm ( (cid:96) ) , R qm ( (cid:96) ))
+ t (cid:105) ≤ 4 e − t 2 m 2 C
Corollary 1.
(cid:113)
Var
[Equation]
m
For proof see Appendix A. Note that when p = q , the convergence of ˆ D φ(cid:96) (ˆ p m (cid:107) ˆ q m ) does not depend on the Radamacher complexity of (cid:96) , and converges to 0 very quickly.
When p (cid:54) =
q the estimator ˆ D φ(cid:96) (ˆ p m (cid:107) ˆ q m ) is still consistent (under regularity assumptions)
Corollary 2.
[Consistency] Under the condition of Theorem 2, if additionally either 1.
A is a ﬁnite set 2.
A is a bounded subset of R d for some d ∈ N and (cid:96) is Lipschitz w.r.t.
a , then almost surely lim m →∞ ˆ D φ(cid:96) (ˆ p m (cid:107) ˆ q m )
=
D φ(cid:96)
( p (cid:107) q ) .
For both cases in Corollary 2 the Radamacher complexity R pm ( (cid:96) ) goes to zero (as sample size m → ∞ ) at a rate of O (1 / √ m ) .
In other words we can conclude that the estimation error in Theorem 2 is bounded by O (1 / √ m ) and the variance of the estimator is also bounded by O (1 / √ m ) when the sample size m → ∞ .
E XPERIMENT : T WO S AMPLE T EST
The ﬁrst application is to design more powerful two sample tests.
We aim to show that H-divergence allow us to leverage inductive biases for each data type (e.g. image, bio, text) by choosing suitable actions A and loss (cid:96) , which leads to improved test power.
4. T WO S AMPLE T EST
For the task of two sample test, we would like to decide if two sets of samples are drawn from the same distribution or not.
Speciﬁcally, given two sets of samples ˆ p m :=
( x 1 , · · · , x m ) i.i.d.
∼ p and ˆ q m :=
( x (cid:48) 1 , · · · , x (cid:48) m ) i.i.d.
∼ q we would like to decide if p = q .
Typical approaches estimate a divergence ˆ D (ˆ p m (cid:107) ˆ q m ) and output p (cid:54) =
q if the divergence exceeds some threshold.
There are two types of errors: Type I error happens when the algorithm incorrectly outputs p (cid:54) =
q ; the probability of type I errors is called the signiﬁcance level.
Type II error happens when the algorithm incorrectly outputs p = q ; the probability of not making a Type II error is called the test power (higher is better).
Note that both the signiﬁcance level and the test power are relative to distributions p and q .
We follow the typical setup where we guarantee a certain signiﬁcance level while empirically measuring the test power.
In particular, the signiﬁcance level can be guaranteed with a permutation test (Ernst et al., 2004).
In a permutation test, in addition to the original set of samples ˆ p m and ˆ q m , we also uniformly randomly swap elements between ˆ p m and ˆ q m , and sample multiple randomly swapped datasets (ˆ p 1 m , ˆ q 1 m ) , (ˆ p 2 m , ˆ q 2 m ) , · · · .
The testing algorithm outputs p (cid:54) =
q if ˆ D (ˆ p m (cid:107) ˆ q m ) is in the top
α -quantile among { ˆ D (ˆ p 1 m (cid:107)
ˆ q 1 m ) , ˆ D (ˆ p 2 m (cid:107) ˆ q 2 m ) , · · · } .
Permutation test guarantees the signiﬁcance level (i.e. low Type I error) because if p = q then swapping elements between ˆ p m and ˆ q m should not change its distribution, so each pair (ˆ p m , ˆ q m ) , (ˆ p 1 m , ˆ q 1 m ) , · · · should have the same distribution.
Therefore, ˆ D (ˆ p m (cid:107) ˆ q m ) happens to be in the top α -quantile with at most α probability.
Note that the signiﬁcance level guarantee does not rely on accurate estimation of H-divergence in
Theorem 2 (accurate H-divergence estimation is still important because the test power does depends on it).
When the choice of D (cid:96) is not a strict divergence (See Proposition 2) we may falsely conclude p =
q ( D ( p (cid:107) q ) = 0 ) when in reality p (cid:54) =
q .
This is true but inconsequential in ﬁnite data scenarios.
With ﬁnite data, it is generally impossible to guarantee the test power (i.e. bounding the probability of concluding p =
q when in reality p (cid:54) =
q for any p, q ) and prior literature do not provide such guarantees.
Hence our guarantee is no weaker than prior two sample test literature.
4. E XPERIMENT S ETUP
Baselines We compare our proposed approach with six other divergences.
All methods are based on the permutation test explained in Section 4.1.
MMD-D (Liu et al., 2020) measures the MMD distance with a deep kernel, while MMD-O (Gretton et al., 2012) measures the MMD distance with a Gaussian kernel.
Mean embedding (ME) and smoothed characteristic functions (SCF) (Chwialkowski et al., 2015; Jitkrittum et al., 2016) are distances based on the difference in Gaussian kernel mean embedding at a set of optimized points, or a set of optimized frequencies.
C2STS-S & C2ST-L (Lopez-Paz & Oquab, 2017; Cheng & Cloninger, 2019) use a classiﬁer’s accuracy distinguishing between the two distributions.
Comparison Metrics and Setup All methods have the same signiﬁcance level (which is provably equal to α = 0 .
because of the permutation test), therefore we only consider the test power.
We follow Liu et al.
and consider four datasets: Blob (Liu et al., 2020), HDGM (Liu et al.,
, HIGGS (Adam-Bourdarios et al., 2014) and MNIST (LeCun & Cortes, 2010).
Our method and all the baseline methods have hyper-parameters.
To ensure fair comparison, we follow the same evaluation setup as (Liu et al., 2020) for all methods.
We split each dataset into two equal partitions: a training set to tune hyper-parameters, and a validation set to compute the ﬁnal test output.
Implementation Details We choose φ ( θ, λ ) =
(cid:0) θ s + λ s(cid:1) 1 /s for s > 1 (which includes the HJensen Shannon divergence when s = 1 and the H-Min divergence when s = ∞ ).
We deﬁne l ( x, a ) as the negative log likelihood of x under distribution a , where a is in a certain model family A .
We experiment with mixture of Gaussian distributions, Parzen density estimtor and Variational Autoencoder (Kingma & Welling, 2013).
Our hyper-parameters consist of the best parameter s and also the best generative model family.
Choosing these hyper-parameters might seem cumbersome, but compared to the second best baseline (MMD-D which chooses thousands of deep kernel parameters), we have much fewer hyper-parameters.
We use α = 0 .
in all two-sample test experiments.
Each permutation test uses 100 permutations, and we run each test 100 times to compute the test power (i.e. the percent of times it correctly outputs p (cid:54) = q ).
Finally we plot and report the performance standard deviation by repeating the entire experiment 10 times.
4. E XPERIMENT R ESULTS
The average test powers are reported in Figure 4, Figure 2, Table 1 and Table 3.
Our approach achieves superior test power across the board .
Notably on Higgs we achieve the same test power with 2x fewer samples than the second best test, and on MNIST we can achieve perfect test power even on the smallest sample size evaluated in (Liu et al., 2020).
Following (Liu et al., 2020) we also evaluate the test power as the dimension of the problem increases (Figure 2).
Our test power decreases gracefully as the dimension of the problem increases .
We hypothesize that the test power improvements come from leveraging progress in generative model research: for each type of data (e [Figure] g. bio, image, text) there has been decades of research ﬁnding suitable generative models; we use commonly used generative models (in modern literature) for each data type (e.g. KDE for low dimensional physics/bio data, VAE for simple images). All tests have high test power for low data dimensions
[Figure]
but our method scales better for higher data dimensions. [Figure] [Table]
E XPERIMENT : D ECISION D EPENDENT D ISCREPANCY M EASUREMENT 5. A SSESSING C LIMATE C HANGE
As an illustrative example of how H-divergence can facilitate decision making, we use climate data and study how climate change affects decision making through the lens of H-divergence.
Scientists and policy makers are often interested in how climate change disparately affect different geographical locations.
Existing methods (Preston et al., 2011) focus on one aspect of climate change (such as the expected economic loss (Burke et al., 2018)) using tailor-designed analysis, while Hdivergence provides a general tool for hypothesis testing and visualization for different aspects of climate change.
In our example, we choose suitable loss functions to quantitatively measure aspects of climate change that are relevant to decision making in agriculture and renewable energy production.
Setup We use the NOAA database which contains daily weather from thousands of weather stations at different geographical locations.
For each location, we summarize the weather sequence of each year into a few summary statistics (average yearly temperature, humidity, wind speed and rainy days).
We are interested in assessing changes in weather over this period at each location, from the perspective of agriculture and renewable energy activities.
Further details of these experiments are in Appendix C.2.
Example: Agriculture It is known that climate changes affect crop suitability (Lobell et al., 2008).
Let A denote the set of possible crops to plant at each location (e.g. wheat/barley/rice), and (cid:96) ( x, a ) denote the loss of planting crop a if the yearly weather is x .
We estimate the function (cid:96) by matching geographical locations in the FAO crop yield dataset (FAOSTAT et al., 2006) to weather stations in the NOAA database, and learn a function to predict crop yield from weather data with kernel ridge regression.
The H-divergence has a natural interpretation: a geographical location could either (1) plant the same crop for the entire period 1981-2019 that is optimal for the local climate (i.e. choose a ∗ =
arg min a ∈A E ( p + q ) /
[ (cid:96) ( X, a )] ); (2) plant the optimal crops for 1981-1999 and for 2000respectively.
H divergence measures the additional loss of option (1) compared to option (2).
In other words, it is the excess loss of not adapting crop type to climate change.
For each geographical location we can compute the H-divergence D JS (cid:96) for the estimated (cid:96) (plotted in Figure 3 left).
Example: Energy production Changes in weather also affect electricity generation, since climate change could affect the amount of wind/solar energy available.
Let A denote the number of wind/solar/fossil fuel power plants built, and (cid:96) ( x, a ) denote the loss (negative utility) when the weather is x .
We obtain the function (cid:96) using empirical formulas for energy production (Npower, 2012).
The H divergence for this loss function is shown in Figure 3 (right)
Intuitively the di- [Table] vergence measures the excess loss of using the same energy generation infrastructure for the entire time period vs. using different infrastructure that adapts to climate change.
While this is only an illustrative example, comparing the two maps we see that regions and industries are affected by climate change in different ways – H divergence provides a quantitative framework for this kind of assessments.
5. F EATURE S ELECTION
In a feature selection task, we wish to know which input features are most predictive of the label.
Feature selection provides information on which features have the biggest inﬂuence on the label, and can be used in scientiﬁc discovery (Jovi´c et al., 2015; Zhang et al., 2015).
Off-the-shelf feature selection algorithms often do not take into account problem speciﬁc requirements.
For example, denote the input features as X 1 , · · · , X K and label as Y , the mutual information feature selection algorithms estimate the Shannon mutual information
I ( X i , Y ) :=
KL( p ( X i , Y ) (cid:107) p ( X i ) p ( Y )) and select features with largest mutual information.
However, scientists and policies makers often need ﬁne-grained control to answer their speciﬁc scientiﬁc or policy questions.
For example, social scientists might want to know which features are more important for high-income as compared to low-income groups (e.g. to understand potential glass ceilings).
With H-Divergence we can select features with large D φ(cid:96)
( p ( X i , Y ) (cid:107) p
( X i ) p
( Y ))
(i.e. the optimal action is different under the joint p ( X i , Y ) and the product of marginals p
( X i ) p
( Y ) ).
By choosing different loss functions (cid:96) we can get different feature selection results, each reﬂecting important features for that decision problem.
For example, in Table 2 we show the features selected for the UCI income prediction dataset (Blake, 1998).
For this dataset, we choose A as the set of logistic regression functions, l ( x, a ) as the cross entropy loss for regression function a on the sample x and φ ( θ, λ ) =
max( θ, λ ) .
If we want to focus on high income groups, we can assign a higher weight to the loss of high income samples, and vice versa.
We observe that gender/race is more predictive of income for high-income groups, while relationship or marital status is more predictive of income for lower-income groups.
This can help us identify potential inequality or suggest further investigation into the cause of low income and poverty.
For example, our results suggest a connection between family and relationship status and poverty, and a connection between gender/race and high income.
These connections merit further investigation into the cause and policy remedy.
A CKNOWLEDGEMENTS
SE acknowledges support by NSF(#1651565, #1522054, #1733686), ONR (N000141912145),
AFOSR (FA95501910024), ARO (W911NF-21-1-0125) and Sloan Fellowship.
A P ROOFS
Lemma 2.
For any choice of (cid:96) and for any choice of φ that satisfy Deﬁnition 2, D φ(cid:96) is non-negative and D φ(cid:96)
( p, q )[-= 
-]
=whenever p = q .
Furthermore, D φ(cid:96) is symmetric whenever φ is symmetric.
Proof of Lemma 2.
For any choice of p, q by the concavity of the H-entropy in Lemma 1 we have
[Equation]
Therefore by summing the two inequalities we have
[Equation]
By the requirement on φ we know that D φ(cid:96) ( p (cid:107) q ) ≥ 0 .
In addition when p =
q
since ( p + q ) / 2 = p
=
q we have
D φ(cid:96) ( p (cid:107) q ) =
φ (0 , 0) = 0 .
To show it is symmetric, note that
[Equation]
q (cid:19) − H (cid:96) ( q ) (cid:19) =
φ (cid:18)
H (cid:96) (cid:18) p + q (cid:19) − H (cid:96) ( q ) , H (cid:96) (cid:18) p + q (cid:19) − H (cid:96) ( p ) (cid:19) =
D φ(cid:96)
( q (cid:107) p ) whenever φ is symmetric.
Proposition 3.
D φ(cid:96)
( p (cid:107) q ) >
if and only if arg inf a E p [ (cid:96) ( X, a )] ∩ arg inf a E q [ (cid:96) ( X, a )] = ∅ .
Proof of Proposition 3.
Denote A ∗ p = arg inf a E p [ (cid:96) ( X, a )] and A ∗ q = arg inf a E q [ (cid:96) ( X, a )] .
Also compute
[Equation]
(cid:19) (3)
If A ∗ p ∩ A ∗ q = ∅ , for any action a (cid:48) such that E p [ (cid:96) ( X, a (cid:48) )]
= H (cid:96) ( p ) , we must have a (cid:48) ∈
A ∗ p so a (cid:48) (cid:54)∈ A ∗ q and E q [ (cid:96) ( X, a (cid:48) )] >
H (cid:96) ( q ) .
Similar if we choose a (cid:48)(cid:48) such that E q [ (cid:96) ( X, a (cid:48)(cid:48) )]
= H (cid:96) ( q ) we have similarly have E p
[ (cid:96) ( X, a (cid:48)(cid:48) )] > H (cid:96) ( p ) .
In other words, for any choice of action a ∈ A either a / ∈
A ∗ p and E p [ l ( X, a )] > H (cid:96) ( p ) or a ∈
A ∗ p and E q [ l ( X, a )] > H (cid:96) ( q ) .
Therefore inf a
[Equation]
Combining Eq.(3) and Eq.(4) we have
[Equation]
By Deﬁnition 2 this implies (for any choice of φ that satisﬁes the requirements in Deﬁnition 2) that D φ(cid:96) ( p (cid:107) q ) > 0 .
To prove the converse simply obverse that if A ∗ p ∩ A ∗ q (cid:54)
=
φ , let a ∗ ∈
A ∗ p ∩ A ∗ q we have a ∗ = arg inf a ∈A E p + q 2 [ l ( X, a )] .
This implies that
[Equation]
( X, a ∗ )] − E p [ l ( X, a ∗ )] =
By Deﬁnition 2 we can conclude that D φ(cid:96) ( p (cid:107) q ) = 0 .
Theorem 1.
The set of H-Jensen Shannon Divergences is strictly larger than the MMD 2 distances.
Proof of Theorem 1.
Let k ( x, y ) be some kernel on an input space X , and let H be the RKHS induced by the kernel.
The (squared) MMD distance is deﬁned by
[Equation]
which we write more compactly as MMD 2 ( p, q )
=
E p,p k ( X, Y ) + E q,q k ( X, Y ) − 2 E p,q k ( X, Y ) .
Deﬁne φ ( x, y )
= (cid:107) k
( x, · ) − k
( y, · ) (cid:107) 2 H .
We can rewrite this in the following form:
[Equation]
We also observe an algebraic relationship for any function f ( x, y ) such that f ( x, y ) =
f ( y, x ) for all x, y :
[Equation]
Furthermore, we have that
[Equation]
Based on the above, noting that φ ( x, y ) =
φ ( y, x )
[Equation]
we can derive
[Equation]
mean def.
Therefore we can deﬁne a loss (cid:96) :
X × H → R where
[Equation]
Under the new notation we have
[Equation]
Conversely we want to show that not every H-Jensen Shannon divergence is a MMD.
For example, take H (cid:96) to be the Shannon entropy, then the corresponding D JS (cid:96) is the Jensen-Shannon divergence, which is not a MMD.
This is because the JS divergence is a type of f -divergence, and the only f -divergence that is also an IPM is total variation distance Sriperumbudur et al.
Therefore, the set of H-Jensen Shannon Divergences is strictly larger than the set of MMDs.
Theorem 2.
If (cid:96) is C -bounded, and φ is 1 -Lipschitz under the ∞ -norm, for any choice of distribution p, q ∈ P ( X ) and t >we have
Pr[ ˆ D φ(cid:96) (ˆ p m (cid:107) ˆ q m ) ≥ t ] ≤ 4 e − t 2 m 2 C 2 if p = q . 2.
Pr (cid:104)(cid:12)(cid:12)(cid:12) ˆ D φ(cid:96) (ˆ p m (cid:107) ˆ q m ) − D φ(cid:96) ( p (cid:107) q ) (cid:12)(cid:12)(cid:12) ≥ 4 max( R p m ( (cid:96) ) , R q m ( (cid:96) ))
+ t (cid:105) ≤ 4 e − t 2 m 2 C
Proof of Theorem 2.
Let ˆ p m be a sequence of n samples ( x 1 , · · · , x m ) drawn from p , and ˆ q m a sequence of n samples ( x (cid:48) 1 , · · · , x (cid:48) m ) drawn from q .
Let ˆ r m the sub-sampling mixture ( x (cid:48)(cid:48) 1 , · · · , x (cid:48)(cid:48) m ) deﬁned in Section 3.5 (i.e. x (cid:48)(cid:48)
i
= x i b i + x (cid:48)
i (1 − b i ) where b i is uniformly sampled from { 0 , 1 } ).
We also overload the notation H (cid:96) by deﬁning H (cid:96) (ˆ p m )
= inf a ∈A 1 m (cid:80) mi =1 l ( x i , a ) , and deﬁne H (cid:96) (ˆ q m ) , H (cid:96) (ˆ r m ) similarly.
Before proving this theorem we need the following Lemmas
Lemma 3.
Under the assumptions of Theorem
[Equation]
Lemma 4.
Under the assumptions of Theorem
[Equation]
To prove the ﬁrst statement of the Theorem, when p = q we can denote µ =
E [ H (cid:96) (ˆ p m )]
=
E [ H (cid:96) (ˆ q m )]
=
E [ H (cid:96) (ˆ r m )] , and we have
[Equation]
Def≤
[Equation]
φ 1-Lipschitz ≤ Pr [ H (cid:96) (ˆ r m ) − H (cid:96) (ˆ p m ) ≥ t ] + Pr [ H (cid:96) (ˆ r m ) − H (cid:96) (ˆ q m ) ≥ t ]
Union bound
[Equation]
Union bound
[Equation]
Lemma
The third inequality is because if H (cid:96) (ˆ r m ) − H (cid:96) (ˆ p m ) ≥ t then it must be either H (cid:96) (ˆ p m ) − µ ≤ − t/ 2 or H (cid:96) (ˆ r m ) − µ ≥ t/ 2 .
Similarly if H (cid:96) (ˆ r m ) − H (cid:96) (ˆ q m ) ≥ t then it must be either H (cid:96) (ˆ q m ) − µ ≤ − t/ 2 and H (cid:96) (ˆ r m ) − µ ≥ t/ 2 .
To prove the second statement of the Theorem, we observe that
[Equation]
[Table]
Jensen
Therefore, the event | ˆ D φ(cid:96) ( p m (cid:107) q m
[Equation]
−
[Equation]
φ(cid:96) ( p (cid:107)
[Equation]
|
[Equation]
max( R pm ( (cid:96) ) , R qm ( (cid:96) ))
+ t happens only if at least one of the following events happen
[Equation]
Based on Lemma 4 each of these events only happen with probability at most e − t 2 m 2 C 2 .
Therefore we can conclude by union bound that
[Equation]
Finally we prove the two Lemmas used in the theorem.
Lemma 4 is a standard result in the Radamacher complexity literature.
For a proof, see e.g. Section 26.1 in (Shalev-Shwartz & Ben-David, 2014).
Lemma 3 can also be proved by standard techniques.
We provide the proof here.
Proof of Lemma 3.
Consider two sets of samples x 1 , · · · , x j , · · · , x m and x (cid:48) 1 , · · · , x (cid:48) j , · · · , x (cid:48)
m where x i =
x (cid:48)
i for every index i = 1 , · · · , m except index j .
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Then we can conclude by Mcdiarmid inequality that
[Equation]
[Table]
[Equation]
[Table]
Corollary 1.
[Equation]
Proof of Corollary 1.
For notation convenience denote B = 4 max( R pm ( (cid:96) ) , R qm ( (cid:96) ))
[Equation]
Corollary 2.
[Consistency] Under the condition of Theorem 2, if additionally either 1.
A is a ﬁnite set 2.
A is a bounded subset of R d for some d ∈ N and (cid:96) is Lipschitz w.r.t.
a , then almost surely lim m →∞ ˆ D φ(cid:96) (ˆ p m (cid:107) ˆ q m )
=
D φ(cid:96)
( p (cid:107) q ) .
Proof of Corollary 2.
We can prove the consistency results from Theorem 2 by observing that the expected Radamacher complexity goes to 0 when m → ∞ .
The ﬁrst statement is a simple consequence of Massart’s Lemma, (e.g. see Eq.(8.44) in (Ma, 2021)).
In particular, because A is ﬁnite we have
[Equation]
To prove the second statement, ﬁrst observe that because A is bounded, there must exist some r ∈ R such that A ⊂ B r :=
{ a, (cid:107) a (cid:107) 2 ≤ r } .
In addition, without loss of generality we can assume that there exists L ∈ R such that ∀ x ∈ X and a
, a (cid:48) ∈
A
[Equation]
There is no loss of generality because in ﬁnite dimensions all norms are equivalent, so if f is Lipschitz under any norm, then (cid:96) is Lipschitz under the 2-norm.
We can apply the results on Radamacher complexity for smoothly parameterized class proved in (Bartlett, 2013), and conclude that lim m →∞ R pm ( (cid:96) )
= 0 .
B A DDITIONAL E XPERIMENTAL R ESULTS B. B LOB DATASET
B. E VALUATING S AMPLE Q UALITY
The gold standard for evaluating image generative models requires human decision, which is nevertheless expensive.
Several surrogate measurements are commonly used, such as the Frechet Inception Distance (FID) (Heusel et al., 2017) or the inception score.
Here by formulating such evaluation as an estimation of the discrepancy between the generated and the real images, we can quantify the quality of image samples by calculating the corresponding H-Divergences.
We choose A as the set of Gaussian mixture distributions on the inception feature space, l ( x, a ) as the negative log likelihood of x under distribution a and φ ( θ, λ )
= max( θ, λ ) .
To evaluate the performance, we use the same setup as (Heusel et al., 2017), where we add corruption from (Hendrycks & Dietterich, 2019) to a set of images.
Intuitively, adding more corruption degrades the sample quality, so a good measurement of sample quality should assign a lower quality score (higher divergence from clean images).
The results are plotted in Figure 5.
The remaining plots of other perturbations are in Appendix B.2.
Both FID and H-divergence are generally monotonically increasing as we increase the amount of corruption.
Our method is slightly better on some perturbations (such as “snow”), where the FID fails to be monotonically increasing, but our method is still monotonic, better aligning with human intuition.
C A DDITIONAL T HEORY AND E XPERIMENT D ETAILS C. C ONNECTION TO O PTIMAL T RANSPORT
We ﬁrst show that H-divergence can also have a transportation interpretation.
For all the intuitive interpretations we avoid technical difﬁculty by assuming X is a ﬁnite set, even though all the formulas are applicable when X has inﬁnite cardinality.
[Table]
Setup Choose A = X , and let l ( x, a ) be a symmetric function ( l ( x, a )
=
l ( a, x ) ) that denotes the cost of transporting a unit of goods from x to a .
When we say that a unit of goods is located according to p , we mean that there is 1 unit of goods dispersed in X locations, where p ( x ) is the amount of goods at location x .
Optimal Transport Distance
The optimal transport distance is deﬁned by
[Equation]
Intuitively the optimal transport distance measures the following cost: initially the goods are located according to p , we would like to move them to locate according to q ; O ( p, q ) denote the minimum cost to accomplish this transportation task.
H-Divergence as Optimal Storage
We ﬁrst consider the intuitive interpretation to the H-entropy
[Equation]
Suppose we want to move goods located according to p to a storage location (for example, we want to collect all the mail in a city to a package center), then a ∗ is the optimal location to build the storage location, and H-entropy measures the minimum cost to transport all goods to the storage location.
Similarly 2 H (cid:96) (cid:0) p + q 2 (cid:1) measures the minimum cost to transport both goods located according to p and goods located according to q to the same storage location.
The H-divergence
[Equation]
measures the reduction of transportation cost with two storage locations (one for p and one for q ) rather than a single storage location (for both p and q ).
The H-Divergence is related to the optimal transport distance by the following inequality.
Proposition 4.
If (cid:96) satisﬁes the triangle inequality ∀ x, y, z ∈ X , l ( x, y )
+
l ( y, z ) ≥ l ( x, z ) then D JS (cid:96) ( p (cid:107) q ) ≤ 12 O ( p, q )
Proof of Proposition 4.
Let a ∗ q = arg [Equation] E [Equation]
[Equation]
( X, a )] then we have
[Equation]
Intuitively
[Equation] to move goods located according to p and goods located according to q to some storage location [Equation] one option is to ﬁrst transport all goods from p to q (so that the goods at location x will be 2 q ( x ) )
[Equation]
then move the goods located according to 2 q to the optimal storage location.
Similarly we have
[Equation]
which combined we have
[Equation]
C. I MPOSSIBILITY OF J ENSEN -S HANNON D IVERGENCE E STIMATION
Suppose we have a consistent estimator for the Jenson-Shannon divergence, i.e. a function ˆJS such that for any pair of distribution p, q and given N i.i.d.
samples p N ∼ p and q N ∼ q we have lim N →∞ ˆJS( p N (cid:107) q N ) =
ˆJS( p (cid:107) q ) , then we prove a contradiction by the probabilistic method.
Let p be a standard Gaussian distribution, let Q M be a uniform distribution on a set of M i.i.d.
samples from p (hence Q M is itself a random variable that depends on the i.i.d.
samples).
Let Q ∗ be the limit of Q M when M → ∞ (i.e. it is the uniform distribution on an inﬁnite set of samples).
Let q ∗ denote a value that Q ∗ can take.
Because q ∗ is always supported on countably many points, hence JS( p (cid:107) q ∗ ) = 1 .
Note that for any N the following two sampling process leads to identical distribution on p N , q N :
[Equation]
Hence, the expectation of any function (including ˆJS ) is also identical.
[Equation]
Hence the limit when N → ∞ must be identical
[Equation]
Because the Jenson Shannon divergence is always bounded, any consistent estimator must also be bounded for sufﬁciently large N .
By the dominated convergence theorem we can exchange the expectation and the limit.
[Equation]
By the probabilistic method (i.e. for any function f
if E Q ∗ ∼ p
[ f ( Q ∗ )]
=
there must exist some q ∗ such that f ( q ∗ ) ≤ 0 ) there must exist some q ∗ such that
[Equation]
Therefore ˆJS cannot be consistent.
C. C LIMATE C HANGE E XPERIMENT D ETAILS
Setup Details
In this experiment, we extract the statistics of yearly weather for each year from 1981-2019.
We use the NOAA dataset, which contains daily weather data from thousands of weather stations across the globe.
For each year we compute the following summary statistics: average yearly temperature, average yearly humidity, average yearly wind speed and average number of rainy days in an year.
For example x 1990 is a 4 dimensional vector where each dimension correspond to one of the summary statistics above.
Let p denote the uniform distribution over { x 1981 , · · · , x 1999 } and q denote the uniform distribution over { x 2000 , · · · , x 2019 } .
For example E p [ (cid:96) ( X, a )] denote the expected loss of action a for a random year sampled from 1981-1999.
Note that for many decision problems, it is possible to make yearly decisions (e.g. decide the best crop to plant for each year).
However, because we want to measure the difference between two time periods 1981-1999 vs. 2000-2019, we choose the action space
A to be a single crop selection that will be used for the entire time period (rather than a different crop selection for each year).
Similarly for energy production we choose the action space
A to be the proportion of different energy production methods that will be used for the entire time period.
Crop yield We obtain the crop yield loss function (cid:96) ( x, a ) with the following procedure
We obtain the crop yield dataset from (FAOSTAT et al., 2006), each entry we extract is the following tuple: (country code, year, crop type, yield per hectare (kg/ha)) 2.
We associate each country code with the central coordinate (i
e. the average latitude and longitude) of the country.
For each central coordinate we ﬁnd the nearest weather station in the NOAA database.
We use data for the nearest weather station as the weather data for the country.
Based on step 2 for each (country code, year) pair we can associate a weather statistics (i
e. the 4 dimensional vector described in Setup Details).
We update each entry in step 1 to be (weather statistics, crop type, yield per hectare).
Based on the data entries we obtain in step 3 we train a Kernel Ridge regression model to learn the function (cid:96) ( x, a ) where x is the weather statistics, a is the crop type, and (cid:96) ( x, a ) is learned to predict the yield (normalized by market price) of the weather x for crop type a .
Energy production We consider three types of energy production methods: solar, wind and traditional (such as fossil fuel).
Solar energy and wind energy both depend heavily on weather, while traditional energy does not.
In particular, we use empirical formulas for solar and wind energy calculation:
[Equation]
D D ISCUSSIONS
Future Work
In this paper we explored the applications of H-divergence to two sample testing.
Future work can explore other applications of divergences.
Potential applications include
• Generative model training.
Many generative models learning algorithm minimize divergences (Nowozin et al 2016; Arjovsky et al., 2017), and future work can explore if the new divergence family leads to new generative model learning algorithms.
• Independence tests.
Independence tests are two sample tests between the joint distribution p XY and the product of marginal distributions p X p Y , therefore the two sample test results from this paper are applicable to independence tests.
• Robustness.
Many robust optimization, estimation or prediction methods aim to achieve good performance even when the data distribution is perturbed.
Typically perturbation is measured by e g. the KL divergence or the Lp distances.
Future work can measure perturbation with the H-divergence H (cid:96) by choosing loss functions (cid:96) that are tailored for the problem.
Computation Issues There are several situations where estimating the H-divergence is (provably) computationally feasible:
• When A is a small ﬁnite set, in which case we can enumerate all possible values of a ∈ A .
• When the loss function (cid:96) ( y, a ) is convex in a , in which case we can accurately estimate the H-divergence in polynomial time by solving the optimization problem inf a E [ (cid:96) ( Y, a )] by gradient descent.
In general, while it is difﬁcult to guarantee computational feasibility, we use a practical technique that works well in our experiments: we use the same number of gradient descent steps for evaluating H (cid:96) (cid:0) p + q 2 (cid:1) and H (cid:96) ( p ) , H (cid:96) ( q ) .
Intuitively, the “sub-optimality” when estimating the three terms is approximately the same in expectation and cancels out.