Identifying the Instances Associated with Distribution Shifts using the Max-Sliced Bures Divergence
Abstract
We investigate an interpretable approach to compare two distributions.
The approach, max-sliced Bures divergence, approximates the max-sliced Wasserstein distance and projects the distributions into a one-dimensional subspace deﬁned by a ‘slicing’ vector.
Unlike heuristic algorithms for the max-sliced Wassersteindistance that are not guaranteed to ﬁnd the optimal slice, we detail a tractable algorithm that ﬁnds the global optimal slice and scales to large sample sizes, due to its expression in terms of second moments.
However, it is unable to detect changes in higher-order statistics.
To overcome this, we explore using a non-linear mapping provided by the internal representation of a pre-trained neural network (Inception
Net).
Our approach provides an interpretation of the Fréchet Inception distance by identifying the instances that are either overrepresented or underrepresented with respect to the other sample.
We apply the proposed measure to detect class imbalances and underrepresentation within data sets.
Introduction
Divergence measures quantify the dissimilarity between probability distributions and are useful for detecting and correcting covariate shift [Shimodaira, 2000, Quionero-Candela et al., 2009, Lipton et al., 2018].
Although a variety of divergences exist, not all are easy to both interpret and estimate.
We consider a divergence to be interpretable if it can be expressed in terms of a ‘witness function’ that can be used to identify subsets that contribute most to the divergence between two distributions.
Additionally, rather than allow subsets to come from across the domain, the witness function should identify localized, ideally contiguous, subsets.
This allows the user to pinpoint and understand speciﬁc examples from a subset of the differences between two samples or distributions.
A natural choice for a witness function is the log of the ratio of the probability density functions, as in the Kullback-Leibler divergence [Kullback and Leibler, 1951] and other f -divergences [Ali and Silvey, 1966, Rényi, 1961].
However, estimating the densities from samples is challenging [Vapnik, 2013], which motivates alternative approaches such as directly estimating the density ratio [Nguyen et al., 2008, Kanamori et al., 2009, Yamada et al., 2011, 2013, Saito et al., 2018, Lee et al., 2019] or via variational formulations [Nguyen et al., 2010, Nowozin et al., 2016].
Another class of divergences are the integral probability metrics (IPMs) [Müller, 1997].
An IPM can be expressed as the maximal difference between the ﬁrst moments of witness function evaluations under each distribution for a given function family Ω , D Ω ( µ, ν )
=
sup ω ∈ Ω (cid:12)(cid:12)
E X ∼ µ [ ω ( X )] − E Y ∼ ν [ ω ( Y )] (cid:12)(cid:12) .
Different choices of Ω yield different IPMs; examples include total variation, the dual formulation of the Wasserstein-1 distance, maximum mean discrepancy (MMD) [Gretton et al., 2007], and covariance feature matching [Mroueh et al., 2017].
Since ω ∗ ( · ) ∈ arg max ω ∈ Ω E X ∼ µ [ ω ( X )] − E
Y ∼ ν [ ω ( Y )] , relatively large evaluations of ω ∗ ( · ) are associated to subsets with higher probability under µ .
Restricting Ω to the class of functions whose large evaluations are localized would yield an interpretable divergence.
An alternative option for interpretable witness functions is to consider restricted subspaces (slices) of the domain.
Before answering the question, ‘Do existing interpretable divergences exist?’
we ﬁrst consider why they may not be useful for other tasks—beyond interpretability.
A divergence that is restricted to localized witness functions makes inefﬁcient use of all the instances in a sample in learning and hypothesis testing (two-sample tests).
For the latter, the witness function should collect evidence across the domain: focusing on a localized subset or low-dimensional subspace limits the evidence collection.
For batch sampling to train a generative model, a localized witness would only provide feedback for one speciﬁc subset at a time, wasting points that fall outside of this subset.
For example, restricting the witness function to a rank-1 subspace would waste differences normal to this subspace.
Nonetheless, such a divergence exists: it is the max-sliced Wasserstein-
p distance [Kolouri et al., 2019, Deshpande et al., 2019], D W p , Ω ( µ, ν ) = sup ω ∈ Ω W p ( ω (cid:93) µ, ω (cid:93) ν ) for an appropriate function family Ω .
The limitations of max-slicing for generative modeling and learning have already been noted and alternatives proposed [Nguyen et al., 2021].
While not as simple to interpret as the general case, the distributional form of the sliced Wasserstein distance [Nguyen et al., 2021], which optimizes the distribution over the slices, can also be used to investigate discrepancies across different subspaces.
A difﬁculty with the max-sliced Wasserstein- p distance is its computation, since it is a saddle-point optimization problem that requires solving a one-dimensional optimal transport problem at each evaluation.
Previous works have approximated the max-sliced Wasserstein distance using a ﬁrst moment approximation [Deshpande et al., 2019] or a ﬁnite number of steps of a local optimizer [Kolouri et al., 2019].
Neither guarantees obtaining an optimal witness function.
As an alternative we propose the “max-sliced Bures divergence” which relaxes the Wasserstein- (W2) distance with a second-moment approximation that relies on the Bures distance [Dowson and Landau, 1982, Gelbrich, 1990]. The Bures distance is a distance metric between positive semideﬁnite matrices, which is equivalent to the W2 distance for zero-mean Gaussians [Muzellec and Cuturi, 2018, Zhang et al., 2020, Oh et al., 2020, De Meulemeester et al., 2020, Janati et al., 2020].
We derive a one-sided version of the max-sliced Bures divergence that
• provides an interpretable witness function that highlight regions of over- and underrepresentation in one distribution with respect to the other, especially useful for covariate shift detection; • is expressed as an optimization problem that has a globally optimal solution, which can be computed efﬁciently (after initial computation of the means and covariance matrices, calculating the divergence and the optimal witness function is independent of the sample sizes, enabling it to be applied to large data sets and high-dimensional settings); • and can be easily extended beyond second-order moments by using a nonlinear mapping, for example the hidden-layer activations of a pre-trained neural network can be used to create a task-relevant mapping
Methodology
We consider the set of probability distributions over the domain X and denote them P X .
Let µ, ν ∈ P X denote two probability measures, and let X ∼ µ and Y ∼ ν denote two random variables X, Y ∈ X .
We restrict our attention to probability distributions in the d -dimensional real-valued vector space X ⊆
R
d with ﬁnite second moments and denote them as P L 2 ( X ) .
Denote by m X =
E
X ∼ µ [ X ] ∈ X and m Y =
E Y ∼ ν
[ Y ] ∈ X the ﬁrst moments of the random variables.
The family of probability mass functions over the discrete set [ n ] = { 1 , . . . , n } is denoted P [ n ] .
Two samples from µ and ν , { x i = x i } mi =1 and { y i =
y i } ni
=1 , are denoted by their empirical measures ˆ µ and ˆ ν .
The max-sliced Bures divergence requires the matrices ρ X and ρ Y , which can be estimated from sample means m X , m Y and covariance matrices Σ X , Σ Y : ρ X = m
X m (cid:62) X
+ m − 1 m Σ X and ρ Y =
m Y m (cid:62) Y + n − 1 n Σ Y , assuming unbiased covariance estimates.
Let Ω = { ω ( · ) = (cid:104)· , w (cid:105) 2 : w ∈ S} , where S denotes the unit hypersphere S d − 1 .
Using E [ ω ( X )] =
E [ (cid:104) X, w (cid:105) 2 ] = w (cid:62) E [ XX (cid:62) ] w = w (cid:62) ρ X w , the max-sliced Bures divergence (MSB) is
[Equation]
where ω µ>ν ( · ) = (cid:104)· , w µ>ν (cid:105) 2 and ω µ<ν ( · ) = (cid:104)· , w µ<ν (cid:105) 2 correspond to the optimal slices w µ>ν and w µ<ν .
In Algorithm 1, we describe how to compute the vector w µ>ν deﬁning the one-sided max sliced Bures divergence witness function ω µ>ν .
It requires solving a one-dimensional bounded optimization, in which each function evaluation requires solving a maximum eigenvalue problems, which can be done efﬁciently using standard linear algebra packages.
As an alternative, ﬁrst-order
Algorithm 1: One-sided max-sliced Bures divergence
Input: ρ X = 1 m (cid:80) mi
=1 x i x (cid:62) i , ρ Y = 1 n (cid:80) ni
=
y i y (cid:62) i ∈ R
d × d , (cid:15) >
Deﬁne the function v ( · ) :
R → R d as v ( γ ) : γ (cid:55)→ arg max w : (cid:107) w (cid:107) 2 ≤ 1 w (cid:62) ( γ ρ X − ρ Y )
w
Solve the 1D bound problem: γ (cid:63) = arg max 0 <γ ≤ 1 (cid:113) v (cid:62) ( γ ) ρ X v ( γ ) − (cid:113) v (cid:62) ( γ ) ρ Y v ( γ )
Output: w µ>ν = v ( γ (cid:63) ) constrained algorithms can be applied after the objective of the maximization is made differentiable by smoothing the negative square root w µ>ν ≈ arg max w ∈S (cid:112) w (cid:62) ρ X w − (cid:112) w (cid:62)
ρ Y w
+
(cid:15) .
In the Appendix, we formally state and show that the max-sliced Bures divergence is a lower bound for the max-sliced Wasserstein-2 distance, see Theorem 1.
Following previous work on max-sliced Wasserstein distance by Deshpande et al. [2019] and Nguyen et al. [2021], we show that the statistical error of approximating the max-sliced Bures divergence from empirical distributions of size n is
O ( (cid:112) d log( n ) /n )
(assuming a compact subset X ⊂ R d ) . This is stated in Theorems 4 and 6.  Experiments We present various examples of using the proposed max-sliced divergences to identify the discrepancies between two samples.
For illustration, we use the MNIST, CIFAR10, and Ofﬁce data sets, creating pseudo-real experiments by taking different subsets of real data sets.
We also use the witness function to interpret samples from GANs trained for the stacked MNIST data set.
3.1 Detecting Non-Uniform Labels Distributions
We consider covariate shift on CIFAR10, created through purposeful class imbalance.
ˆ µ is a uniform sample from the training set and ˆ ν is a sample from the test set with less instances from one class.
Applying the max-sliced Bures witness function ω µ>ν to points from ˆ ν should assign larger values to points from the under-represented class, identifying the top-10 witness points with the largest witness function evaluations ω µ>ν ( y ` σ (1) ) ≥ . . .
≥ ω µ>ν ( y ` σ (10) ) , where ` σ is the required permutation.
For the vector representations of the samples, we use the internal representation of the Inception
Network [Szegedy et al., 2016] as in the Fréchet Inception distance [Heusel et al., 2017] and Inception score [Salimans et al., 2016].
The witness function is ω µ>ν ( x ) =
(cid:104) w µ>ν , φ ( x ) (cid:105) 2 , where φ ( x ) ∈ R 2048 is the Inception code.
The results are reported in Figure 1.
When the class probabilities differ by 2% (10.2% for majority and 8.2% for minority), the mean average precision (MAP) is 0.94.
This is compared to a MAP of 0.82 for the ﬁrst-moment-based surrogate of the max-sliced W distance [Deshpande et al., 2019].
3.2 Detecting Mode Dropping
We use the one-sided max-sliced Bures divergence to monitor mode dropping during GAN training for the stacked MNIST data set.
We create a 3-channel “Stacked MNIST” data set with 500, images from the MNIST training set to test mode dropping detection throughout GAN training (DCGAN architecture).
The training set ˆ µ =
Real has 1000 possible modes corresponding to all 3-digit combinations.
At the end of each training epoch (1000 iterations), a sample ˆ ν = Fake of n = 10 4 synthetic images is generated.
Figure 3 shows the top-5 fake and real images for both both witness functions after 10 epochs.
To verify mode coverage we use a 4-layer convolutional network trained on single-channel MNIST; any missing combination of 3-digit labels is considered a dropped mode.
Figure 2 shows the precision of the approach in detecting the dropped modes.
[Figure]
3.3 Interpreting the Fréchet Inception Distance
We use a witness function to identify mismatches in domain transfer tasks, speciﬁcally the Ofﬁce dataset introduced by Saenko et al.
The data set contains three domains: images from
Amazon.com (A), high-resolution DSLR images (D), and low-resolution webcam images (W).
There are 31 classes and a varying number of images per class.
The images are represented by internal activations of the Inception object classifying network [Szegedy et al., 2016].
We applied both of witness functions corresponding to the two one-sided max-sliced Bures divergences.
Figure 4 shows the images identiﬁed by the witness functions.
The results make sense as bicycles and notebooks with plain white backgrounds are much more common in A than in W. The closest examples are bicycles in front of white walls.
Likewise binders with a wooden table and other background are common in webcam images but uncommon on Amazon.com.  Limitations
While the proposed approach yields an interpretable and precise witness function, the utility of a localized witness function that assesses the differences in second moments along a 1-dimensional subspace is limited.
It is ill-suited to be used in learning, as it focuses only on the worst discrepancy— ignoring the other discrepancies.
Furthermore, the reliance on second moments requires a suitable non-linear learning representation, such as a pre-trained neural network or random Fourier features.
We note that previous work has shown that MMD operating on top of an untrained convolutional encoding network can detect covariate shift [Rabanser et al., 2019].
Since MMD with a Gaussian kernel can be approximated using the norm of the difference of means after random Fourier feature embedding, we are hopeful that the ability to pinpoint the worst discrepancies will prove useful in practice. On the practical side, further testing is needed to see how the method performs on high-dimensional tabular data. A speciﬁc example is to consider sparse bag-of-words representations of corpus of documents. In this case, the sparsity of the representation may be exploited when computing the largest eigenvector of the difference of the Gram matrices ρ X and ρ Y .
On the theoretical side, while the max-sliced Bures divergence does not suffer from the curse of dimensionality, it is only a probability metric amongst distributions completely characterized by their ﬁrst and second moments.
Further analysis is needed for the convergence rate of the proposed algorithm and its dependence on the eigenstructure of the Gram matrices.
We propose the max-sliced Bures divergence, a lower-bound on the max-sliced W2 distance.
The underlying one-sided max-sliced Bures divergences enable direct interpretation of under- and overrepresentation between two samples via the witness functions, which can be used to identify discrepancies. The optimal witness functions can be found through optimization problems posed on the unit interval, as opposed to the saddle point optimization problem involved in the max-sliced Wasserstein distance.
We show the potential of applying this approach to covariate shift detection. Acknowledgments and Disclosure of Funding
Luis G. Sanchez Giraldo would like to acknowledge the support from the NVIDIA Academic
Hardware Grant Program.
The authors gratefully acknowledge the support of Dr. Tory Cobb and
ONR321. Austin J. Brockmeier’s effort were sponsored by the Department of the Navy, Ofﬁce of Naval Research under ONR award number N00014-21-1-2300. Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the Ofﬁce of Naval Research. A Appendix
These appendices detail the derivation of the proposed divergences; formal results relating them to existing divergences; and additional experimental results.
First, we detail the relationship between Wasserstein-2 distance the Fréchet distance/divergence and the Bures distance/divergence. Then we detail statistical guarantees on the convergence of the estimator.
Finally, we detail additional synthetic results.
A.1 Bures, Fréchet, and Wasserstein Distances
The Bures distance generalizes the Hellinger distance to positive semideﬁnite matrices [Fuchs and Van De Graaf, 1999, Bhatia et al., 2019].
For a pair of positive semideﬁnite matrices A , B ∈ R d × d the Bures distance is deﬁned as
[Equation]
where (cid:107)·(cid:107) 1 denotes the trace norm (sum of the singular values), (cid:107) A 12 B 12 (cid:107) 1 = tr
( A 12 BA 12 ) 12 = tr( B 12 AV 12 ) 12 .
d B ( A , A ) ≤ (cid:107)
A 12 − B 12 (cid:107) 22 = tr( A 12 − B 12 ) 2 , with equality if the matrices commute.
When the matrices commute they share the same eigenbasis and the Bures distance is proportional to the Hellinger distance between the eigenvalues.
While the Bures distance is a metric between positive semideﬁnite matrices, the corresponding divergence,
[Equation]
is a semi-metric since it relies on the uncentered second moments.
Thus, it is a probability metric only on the subset of distributions deﬁned completely by their ﬁrst or second moments, such as Gaussian distributions or other elliptically symmetric distributions.
To compare two Gaussian distributions, it is natural to compare the means and covariance parameters as in the Fréchet distance [Fréchet, 1957, Dowson and Landau, 1982],
[Equation]
The Fréchet distance is also known as the Wasserstein-Bures [Janati et al., 2020] or Bures-Wasserstein distance [Bhatia et al., 2019], since it is equal to the Wasserstein-2 distance ( W 2 ) between Gaussian measures, and generally lower bounds the W2 distance [Gelbrich, 1990],
[Equation]
6) where
[Equation]
and Γ( µ, ν ) deﬁnes the set of all joint distributions with marginals µ and ν .
The sliced Wasserstein distance [Wu et al., 2019, Deshpande et al., 2018, Kolouri et al., 2018] and the max-sliced Wasserstein distance [Deshpande et al., 2019], and its generalizations [Kolouri et al., 2019], evaluate discrepancies along one-dimensional linear, or non-linear, subspaces.
They can be expressed by the Radon transform of integrable functions [Bonneel et al., 2015].
The linear max-sliced Wasserstein- 2 distance (squared) is
[Equation]
where S can be either the unit hypersphere S d − 1 or its convex hull { w ∈ R d : (cid:107) w (cid:107) 2 ≤ 1 } (in terms of the optimization problem the two are equivalent as the optimal solution necessarily lies on the unit hypersphere).
Slicing is motivated by the relative ease of computing the Wasserstein- p distance in one dimension, since it has a closed form [Santambrogio, 2015, Lemma 2.8] and in the case of samples only requires a sorting procedure.
While the Bures divergence already has a closed-form solution, the Bures divergence is not easily interpretable as it cannot be written in terms of witness function.
We propose a sliced version based on projecting a positive semideﬁnite matrix A ∈ R n × n to a rank-1 matrix Ω w = ww (cid:62) .
For compactness we denote the rank-1 projection as
[Equation]
When w ∈ S d [Equation] (cid:107)
[Equation]
Aw , and
[Equation]
Let ρ X, w = (cid:104) Ω w , ρ X (cid:105)
Ω w and ρ Y, w = (cid:104) Ω w , ρ Y (cid:105)
Ω w .
The max-sliced Bures divergence (MSB) is
[Equation]
The max-sliced Bures divergence is a lower-bound on the max-sliced Wasserstein-2 distance, since the latter is lower bounded by the max-sliced Fréchet distance.
This is stated in the following theorem.
Theorem 1.
For µ, ν ∈ P L 2 ( X ) , the max-sliced Bures divergence is a lower bound on the max-sliced
Fréchet distance and the max-sliced W2 distance, D MSB ( µ, ν ) ≤ D MSF ( µ, ν ) ≤ D MSW 2 ( µ, ν ) .
To prove this theorem, we will develop some deﬁnitions and intermediate results.
Recall that P L 2 ( X ) denotes the set of probability distributions in the d -dimensional real-valued vector space X ⊆ R d , with ﬁnite second moments.
The means of the random variables are denoted m X =
E
X ∼ µ [ X ] ∈ X
and m Y =
E Y ∼ ν
[ Y ] ∈ X .
The uncentered second moments are denoted ρ X =
E [ XX (cid:62) ]
∈ X × X and ρ Y = E
[ Y Y (cid:62) ] ∈ X × X .
The covariances are denoted Σ X =
ρ X − m X m (cid:62) X and Σ Y =
ρ Y − m Y m (cid:62) Y .
Let w ∈ R d denote a slicing vector.
Denote the sliced means and rank-1 covariances of the random variables (cid:104) w , X (cid:105) w and (cid:104) w , Y (cid:105) w as m X, w = (cid:104) w , m X (cid:105) w ∈ R d
and m Y, w = (cid:104) w , m Y (cid:105) w ∈ R d , and Σ X, w = (cid:104) w , Σ X w (cid:105) ( ww (cid:62) )
∈ R
d × d and Σ Y, w = (cid:104) w , Σ Y w (cid:105) ( ww (cid:62) )
∈ R d × d , respectively.
The Fréchet difference between the sliced random variables is
[Equation]
The second equality follows from the fact that trace norm (cid:107)·(cid:107) 1 is equal to the Frobenius norm (cid:107)·(cid:107) F for rank-1 matrices.
The last equality uses the fact that (cid:107) ( ww (cid:62) )
[Equation]
[Equation]
We now state the max-sliced Fréchet distance and an upper bound.
[Equation]
where σ X, w = (cid:112) E
[( (cid:104) w , X (cid:105) − (cid:104) w , m X (cid:105) ) 2 ] is the standard deviation of (cid:104) w , X (cid:105) , and, similarly, σ Y, w = (cid:112) E [
[Equation]
[Equation]
−
[Equation]
Y
[Equation]
The following upper bound uses a separate slice for the means and covariances:
[Equation]
Clearly, D MSF ( µ, ν ) ≤ D U MSF ( µ, ν ) .
We now consider the relationship between the Gauss-Wasserstein or Fréchet distance—which combines the distances between the ﬁrst and second-order moments—with the max-sliced Bures when it is applied directly to the uncentered covariance matrices.
For this we need the following lemma that states the reverse triangle inequality in R 2 .
Lemma 2 (Reverse triangle inequality) .
For two vectors in u , v ∈ R 2 , the difference between their
Euclidean norms is less than or equal to the Euclidean norm of their differences |(cid:107) u (cid:107) − (cid:107) v (cid:107)| ≤ (cid:107) u − v (cid:107) .
For u = [ a, b ] (cid:62) , v = [ c, d ] [Equation] Theorem 3 (Sliced Fréchet and sliced Bures divergence inequality) .
The max-sliced Bures divergence based on the uncentered covariance matrices is less than or equal to the max-sliced Fréchet distance,
D MSB ( µ, ν ) ≤ D MSF ( µ, ν ) .
Proof.
To relate the sliced Bures divergence and sliced Fréchet distances, we note that E
X ∼ µ [ (cid:104) X, w (cid:105) 2 ]
= (cid:104) m
X , w (cid:105) 2 + σ 2 X, w and, likewise, E Y ∼ ν
[ (cid:104) Y, w (cid:105) 2 ]
= (cid:104) m
Y , w (cid:105) 2 + σ 2 Y, w .
Then, (cid:12)(cid:12)(cid:12)(cid:12)(cid:113) E X ∼ µ [ (cid:104) X, w (cid:105) 2 ] − (cid:112) E Y ∼
[Equation]
where the inequality relies on the reverse triangle inequality (Lemma 2), with u
=
[ a, b ] (cid:62) =
[ (cid:104) m
X , w (cid:105) , σ X, w ] (cid:62) , v = [ c, d ] (cid:62) =
[ (cid:104) m
Y , w (cid:105) ,
σ Y, w ] (cid:62) .
Taking the supremum over slices of both sides yields the desired inequality,
[Equation]
Using these results we can now state the proof of Theorem 1.
Proof of Theorem 1.
For w ∈ R d , let w ( · ) =
(cid:104)· , w (cid:105) .
Let X ∼ µ and Y ∼ ν , then deﬁne the zero-mean random variables ˜ w ( X )
=
(cid:104)
X − m X , w (cid:105) and ˜ w ( Y ) =
(cid:104)
Y − m Y , w (cid:105) .
By deﬁnition, E [ ˜ w 2 ( X )]
= σ 2 X, w and E
[ ˜ w 2 ( Y )]
= σ 2 Y, w .
The squared Fréchet distance between random variables w ( X ) =
(cid:104) m
X , w (cid:105) + ˜ w ( X ) and w ( Y ) =
(cid:104) m
Y , w (cid:105) + ˜ w ( Y ) is
D 2 F ( w (cid:93) µ, w (cid:93) ν ) =
( E [ w ( X )] − E [ w ( Y )]) 2 + ( σ X, w − σ Y, w ) 2 = (cid:104) m
{+X − m Y , w (cid:105) 
+}
+ ( σ X, w − σ Y, w )=
(cid:104) m
X , w (cid:105) 2 − 2 (cid:104) m
X , w (cid:105)(cid:104) m Y , w (cid:105) + (cid:104) m Y , w (cid:105) 2 + σ 2 X, w − 2 σ X, w σ Y, w + σ 2 Y, w = (cid:104) m
X , w (cid:105) 2 + σ 2 X, w (cid:124) (cid:123)(cid:122) (cid:125) E [ w 2 ( X )]
+ (cid:104) m
Y , w (cid:105) 2 + σ 2 Y, w (cid:124) (cid:123)(cid:122) (cid:125) E
[ w 2 ( Y )] −  (cid:104) m
X , w (cid:105) (cid:124) (cid:123)(cid:122) (cid:125) E [ w ( X )] (cid:104) m Y , w (cid:105) (cid:124) (cid:123)(cid:122) (cid:125) E [ w ( Y )]
+ σ X, w σ Y, w (cid:124) (cid:123)(cid:122) (cid:125) √ E
[ ˜ w 2 ( X )]
E [ ˜ w 2 ( Y )]  
=
E [ w 2 ( X )] + E [ w 2 ( Y )] − 2 (cid:16) E
[ w ( X )]
E [ w ( Y )]
+ (cid:112)
[Equation]
which follows from Hölder’s inequality,
[Equation]
As this holds for any joint distribution, taking the inﬁmum over all possible joint distributions ( X, Y ) ∼ γ that are within the coupling distribution γ ∈ Γ( µ, ν ) , yields the Wasserstein-2 distance (squared) between the sliced distributions
[Equation]
Maximizing over possible slices, yields
[Equation]
Combining with Theorem 3 yields the desired result.
Theorem 4.
Given a probability measure µ ∈ P L 2 ( X ) supported on a compact subset X ⊂ R d .
Assume that { X i } ni =1 are i.i.d.
random variables with distribution µ .
For the empirical measure µ n
[Equation]
where c > 0 is some universal constant.
Proof.
First we note that for all µ, ν ∈ P L 2 ( X ) , D MSW 2 ( µ, ν ) ≤ D MSW 1 ( µ, ν ) due to the inequality between the L 1 -norm and the L 2 -norm.
Then Nguyen et al.
[2021] show that
[Equation]
with probability at least 1 − δ , for any δ ∈ (0 , 1) , which yields E [ D MSW 1 ( µ, µ n )] ≤ c (cid:113) d log n n for some universal constant c > 0 .
From Theorem 1, we have that E [ D MSB ( µ, µ n )] ≤ E
[ D MSW 2 ( µ, µ n )] .
Combing these two results using E [ D MSW 2 ( µ, ν )] ≤ E [ D MSW 1 ( µ, ν )] yields the desired result.
Lemma 5.
The max-sliced Bures divergence obeys the triangle inequality, D MSB ( µ, ν ) ≤
D MSB ( µ, ξ ) + D MSB ( ξ, ν ) for any µ, ν, ξ ∈ P L 2 ( X ) .
Proof.
Using | a − c | ≤ | a − b | + | b − c | , it is trivial to note that the sliced Bures distance obeys the triangle inequality
[Equation]
Let A =
ρ X = E X ∼ µ [ XX (cid:62) ] , B =
ρ Y =
E Y ∼ ν
[ Y Y (cid:62) ] , and C
=
ρ Z = E
Z ∼ ξ [ ZZ (cid:62) ] , then
[Equation]
Using the triangle inequality and Theorem 4, we have the following result directly.
Theorem 6.
Given probability measures µ, ν ∈ P L 2 ( X ) supported on a compact subset X ⊂ R d . Assume that { X i } ni =1 are i.i.d. random variables with distribution µ and { Y i } ni =1 are i.i.d.
random variables with distribution ν .
For the empirical measures µ n = 1 n n (cid:80)
[Equation]
with probability at least 1 − δ , for any δ ∈ (0 , 1) . A.2 Synthetic Data We start by comparing the proposed max-sliced Bures divergence to the max-sliced W2 distance for two-dimensional data. We compare Algorithm 1 for solving each one-sided max-sliced Bures divergence with gradient-based approaches for it and the max-sliced W2 distance using ADAM with parameters lr = 1e-3, beta1 = 0.9, beta2 = 0.999, epsilon = 1e-08 , capping the number of iterations at 1000 or until the change in the slice is minimal (cid:107) w − w old (cid:107) ∞ < 10 − 6 .
In two-dimensions, a near optimal slice can be obtained by a ﬁne grid search of the sliced Bures and sliced W2 distance as shown in Figure 5 for two zero-mean Gaussian distributions.
A.3 Pseudo-real Data: Detecting Non-Uniform Labels Distributions
We investigate the precision of the witness functions in detecting covariate shift caused by class imbalances.
ˆ µ is a uniform sample from the training set and ˆ ν is a sample from the test set with less instances from one class, the minority class.
Using the one-sided max-sliced Bures divergence we obtain the optimal witness function ω µ>ν , which should have larger evaluations for at least a subset of points in the under-represented class.
We can use this witness function to identify the minority class in the test set by examining the training set instances and their labels with the largest witness function evaluations.
We apply ω µ>ν to the sample ˆ µ , identifying the top-10 witness points with the largest witness function evaluations
[Equation] ( i ) }
Ki =1 are the K = 10 points from ˆ µ with the largest norm after projection to this subspace deﬁned by w µ>ν .
The performance is quantiﬁed by the precision of the labels of these instances.
Ideally, these points should all be from the minority class.
We illustrate this simple case of covariate shift detection on the MNIST data set.
Figure A.3 details a comparison of the max-sliced versions of the Bures, Fréchet, and W2 distances in terms of the divergence value estimates, the runtime, and the witness function’s precision. [Figure] [Table]
[Figure] [Figure]