Revisiting Sliced Wasserstein on Images: From Vectorization to Convolution
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The conventional sliced Wasserstein is deﬁned between two probability measuresthat have realizations as vectors .
When comparing two probability measuresover images, practitioners ﬁrst need to vectorize images and then project themto one-dimensional space by using matrix multiplication between the samplematrix and the projection matrix.
After that, the sliced Wasserstein is evaluated byaveraging the two corresponding one-dimensional projected probability measures.
However, this approach has two limitations.
The ﬁrst limitation is that the spatialstructure of images is not captured efﬁciently by the vectorization step; therefore,the later slicing process becomes harder to gather the discrepancy information.
The second limitation is memory inefﬁciency since each slicing direction is avector that has the same dimension as the images.
To address these limitations,we propose novel slicing methods for sliced Wasserstein between probabilitymeasures over images that are based on the convolution operators.
We deriveconvolution sliced Wasserstein (CSW) and its variants via incorporating stride,dilation, and non-linear activation function into the convolution operators.
Weinvestigate the metricity of CSW as well as its sample complexity, its computationalcomplexity, and its connection to conventional sliced Wasserstein distances.
Finally,we demonstrate the favorable performance of CSW over the conventional sliced
Wasserstein in comparing probability measures over images and in training deepgenerative modeling on images.
Introduction
Optimal transport and Wasserstein distance [55, 47] have become popular tools in machine learningand data science.
For example, optimal transport has been utilized in generative modeling tasksto generate realistic images [2, 54], in domain adaptation applications to transfer knowledge fromsource to target domains [9, 3], in clustering applications to capture the heterogeneity of data [21],and in other applications [29, 58, 59].
Despite having appealing performance, Wasserstein distancehas been known to suffer from high computational complexity, namely, its computational complexityis at the order of O ( m 3 log m ) [45] when the probability measures have at most m supports.
Inaddition, Wasserstein distance also suffers from the curse of dimensionality, namely, its samplecomplexity is at the order of O ( n  1 /d )
[15] where n is the sample size.
A popular line of work toimprove the speed of computation and the sample complexity of the Wasserstein distance is by addingan entropic regularization term to the Wasserstein distance [10].
This variant is known as entropicregularized optimal transport (or equivalently entropic regularized Wasserstein).
By using the entropicversion, we can approximate the value of Wasserstein distance with the computational complexities being at the order of O ( m 2 ) [1, 35, 36, 34] (up to some polynomial orders of approximation errors).
Furthermore, the sample complexity of the entropic version had also been shown to be at the order of
O ( n  1 / 2 ) [39], which indicates that it does not suffer from the curse of dimensionality.
Another useful line of work to improve both the computational and sample complexities of the
Wasserstein distance is based on the closed-form solution of optimal transport in one dimension.
Anotable distance along this direction is sliced Wasserstein (SW) distance [6].
Due to the fast computational complexity O ( m log 2 m ) and no curse of dimensionality O ( n  1 / 2 ) , the sliced Wasserstein hasbeen applied successfully in several applications, such as generative modeling [57, 13, 25], domainadaptation [31], and clustering [26].
The sliced Wasserstein is deﬁned between two probabilitymeasures that have supports belonging to a vector space, e.g, R d .
As deﬁned in [6], the sliced
Wasserstein is written as the expectation of one-dimensional Wasserstein distance between twoprojected measures over the uniform distribution on the unit sphere.
Due to the intractability of theexpectation, Monte Carlo samples from the uniform distribution over the unit sphere are used toapproximate the sliced Wasserstein distance.
The number of samples is often called the number ofprojections and it is denoted as L .
On the computational side, the computation of sliced Wassersteincan be decomposed into two steps.
In the ﬁrst step, L projecting directions are ﬁrst sampled andthen stacked as a matrix (the projection matrix).
After that, the projection matrix is multiplied by thetwo data matrices resulting in two matrices that represent L one-dimensional projected probabilitymeasures.
In the second step, L one-dimensional Wasserstein distances are computed between thetwo corresponding projected measures with the same projecting direction.
Finally, the average ofthose distances is yielded as the value of the sliced Wasserstein.
Despite being applied widely in tasks that deal with probability measures over images [57, 13],the conventional formulation of sliced Wasserstein is not well-deﬁned to the nature of images.
Inparticular, an image is not a vector but is a tensor.
Therefore, a probability measure over imagesshould be deﬁned over the space of tensors instead of vectors.
The conventional formulation leads toan extra step in using the sliced Wasserstein on the domain of images which is vectorization.
Namely,all images (supports of two probability measures) are transformed into vectors by a deterministicone-one mapping which is the "reshape" operator.
This extra step does not keep the spatial structuresof the supports, which are crucial information of images.
Furthermore, the vectorization step alsoposes certain challenges to design efﬁcient ways of projecting (slicing) samples to one dimensionbased on prior knowledge about the domain of samples.
Finally, prior empirical investigationsindicate that there are several slices in the conventional Wasserstein collapsing the two probabilitymeasures to the Dirac Delta at zero [13, 12, 24].
Therefore, these slices do not contribute to theoverall discrepancy.
These works suggest that the space of projecting directions in the conventionalsliced Wasserstein (the unit hyper-sphere) is potentially not optimal, at least for images.
Contribution.
To address these issues of the sliced Wasserstein over images, we propose to replacethe conventional formulation of the sliced Wasserstein with a new formulation that is deﬁned on thespace of probability measures over tensors.
Moreover, we also propose a novel slicing process bychanging the conventional matrix multiplication to the convolution operators [16, 18].
In summary,our main contributions are two-fold: 1.
We leverage the beneﬁts of the convolution operators onimages, including their efﬁcient parameter sharing and memory saving as well as their superiorperformance in several tasks on images [28, 19], to introduce efﬁcient slicing methods on sliced
Wasserstein, named convolution slicers .
With the convolution slicers, we derive a novel variantof sliced Wasserstein, named convolution sliced Wasserstein (CSW).
We investigate the metricityof CSW, its sample and computational complexities, and its connection to other variants of sliced
Wasserstein.
We then illustrate the favorable performance of CSW in comparing probabilitymeasures over images.
In particular, we show that CSW provides an almost identical discrepancybetween MNIST’s digits compared to that of the SW while having much less slicing memory.
Furthermore, we compare SW and CSW in training deep generative models on standard benchmarkimage datasets, including CIFAR10, CelebA, STL10, and CeleA-HQ.
By considering the quality ofthe trained generative models, training speed, and training memory of CSW and SW, we observe thatthe CSW has more favorable performance than the vanilla SW.
Organization.
The remainder of the paper is organized as follows.
We ﬁrst provide backgroundabout Wasserstein distance, the conventional slicing process in the sliced Wasserstein distance, andthe convolution operator in Section 2.
In Section 3, we propose the convolution slicing and theconvolution sliced Wasserstein, and analyze some of its theoretical properties.
Section 4 containsthe application of CSW to generative models, qualitative experimental results, and quantitativeexperimental results on standard benchmarks.
We conclude the paper In Section 5.
Finally, we deferthe proofs of key results and extra materials in the Appendices.
Notation.
For any d  2 , S d  1 := { ✓ 2 R d | || ✓ || 22 = 1 } denotes the d dimensional unit hyper-spherein L 2 norm, and U ( S d  1 ) is the uniform measure over S d  1 .
Moreover,  denotes the Dirac deltafunction.
For p  1 , P p ( R d ) is the set of all probability measures on R d that have ﬁnite p -moments.
For µ, ⌫ 2 P p ( R d ) , ⇧ ( µ, ⌫ ) := { ⇡ 2 P p ( R d ⇥ R d ) |
R
R d ⇡ ( x, y ) dx = ⌫ ,
R
R d ⇡ ( x, y ) dy = µ }is the set of transportation plans between µ and ⌫ .
For m  1 , we denotes µ ⌦ m as the productmeasure which has the supports are the joint vector of m random variables that follows µ .
For avector X 2 R dm , X :=
( x 1 , . . .
, x m ) , P X denotes the empirical measures 1 m
P mi =1  x i .
For anytwo sequences a n and b n , the notation a n = O ( b n ) means that a n  Cb n for all n  1 where C issome universal constant.
Background
In this section, we ﬁrst review the deﬁnitions of the Wasserstein distance, the conventional slicing,and the sliced Wasserstein distance, and discuss its limitation.
We then review the convolution andthe padding operators on images.
Sliced Wasserstein: For any p  1 and dimension d 0  1 , we ﬁrst deﬁne the Wasserstein- pdistance [55, 46] between two probability measures µ 2 P p ( R d 0 ) and ⌫ 2 P p ( R d 0 ) , which is givenby W p ( µ, ⌫ ) :=
[Equation]
⌘ 1 p .
When d 0 = 1 , the Wasserstein distancehas a closed form which is W p ( µ, ⌫ ) = (
R [Equation] | F  1 µ ( z )  F  1 ⌫ ( z ) | p dz ) 1 /p where F µ and F ⌫ are thecumulative distribution function (CDF) of µ and ⌫ respectively.
Given this closed-form property of Wasserstein distance in one dimension, the sliced Wasserstein distance [6] between µ and ⌫ had been introduced and admitted the following formulation:
SW
[Equation]
[Equation]
W pp ( ✓] µ, ✓]⌫ ) d ✓ , where ✓] µ is the push-forward probability measure of µthrough the function T ✓ : R d 0 !
R with T ✓ ( x ) = ✓ > x .
For each ✓ 2 S d 0  1 , W pp ( ✓] µ, ✓]⌫ ) can becomputed in linear time O ( m log 2 m ) where m is the number of supports of µ and ⌫ .
However, theintegration over the unit sphere in the sliced Wasserstein distance is intractable to compute.
Therefore,
Monte Carlo scheme is employed to approximate the integration, namely, ✓ 1 , . .
. , ✓ L ⇠ U ( S d 0  1 )are drawn uniformly from the unit sphere and the approximation of the sliced Wasserstein distance isgiven by:
[Equation]
✓ i ]⌫ ) .
In practice, L should be chosen to be sufﬁcientlylarge compared to the dimension d 0 , which can be undesirable.
Sliced Wasserstein on Images: Now, we focus on two probability measures over images: µ, ⌫
P p
( R c ⇥ d ⇥ d ) for number of channels
c  1 and dimension d  1 .
In this case, the sliced Wassersteinbetween µ and ⌫ is deﬁned as:
[Equation]
where R : R c ⇥ d ⇥ d !
R cd 2 is a deterministic one-to-one "reshape" mapping.
The slicing process: The slicing of sliced Wasserstein distance on probability measures overimages consists of two steps: vectorization and projection.
Suppose that the probability measureµ 2 P ( R c ⇥ d ⇥ d ) has n supports.
Then the supports of µ are transformed into vectors in R cd 2 and arestacked as a matrix of size n ⇥ cd 2 .
A projection matrix of size L ⇥ cd 2 is then sampled and haseach column as a random vector following the uniform measure over the unit hyper-sphere.
Finally,the multiplication of those two matrices returns L projected probability measures of n supports inone dimension.
We illustrate this process in Figure 3 in Appendix C.
Limitation of the conventional slicing: First of all, images contain spatial relations across channelsand local information.
Therefore, transforming images into vectors makes it challenging to obtain that information.
Second, vectorization leads to the usage of projecting directions from the unithyper-sphere, which can have several directions that do not have good discriminative power.
Finally,sampling projecting directions in high-dimension is also time-consuming and memory-consuming.
As a consequence, avoiding the vectorization step can improve the efﬁciency of the whole process.
Convolution operator: We now deﬁne the convolution operator on tensors [16], which will be usedas an alternative way of projecting images to one dimension in the sliced Wasserstein.
The deﬁnitionof the convolution operator with stride and dilation is as follows.
Deﬁnition 1 (Convolution)
Given the number of channels c  1 , the dimension d  1 , thestride size s  1 , the dilation size b  1 , the size of kernel k  1 , the convolution of atensor X 2 R c ⇥ d ⇥ d with a kernel size K 2 R c ⇥ k ⇥ k is X s,b ⇤ K
= Y,
[Equation]
where d 0 = d  b ( k  1)  1 s + 1 .
For i = 1 , . . .
, d 0 and j = 1 , . . .
, d 0 , Y 1 ,i,j is deﬁned as:
[Equation]
From its deﬁnition, we can check that the computational complexity of the convolution operator is
O ✓
[Equation]
Convolution Sliced Wasserstein
In this section, we will deﬁne a convolution slicer that maps a tensor to a scalar by convolutionoperators.
Moreover, we discuss the convolution slicer and some of its speciﬁc forms includingthe convolution-base slicer, the convolution-stride slicer, the convolution-dilation slicer, and theirnon-linear extensions.
After that, we derive the convolution sliced Wasserstein (CSW), a family ofvariants of sliced Wasserstein, that utilizes a convolution slicer as the projecting method.
Finally, wediscuss some theoretical properties of CSW, namely, its metricity, its computational complexity, itssample complexity, and its connection to other variants of sliced Wasserstein.
3.1 Convolution Slicer
We ﬁrst start with the deﬁnition of the convolution slicer, which plays an important role in deﬁningconvolution sliced Wasserstein.
Deﬁnition 2 (Convolution Slicer)
For N  1 , given a sequence of kernels K (1)
R c (1) ⇥
d (1) ⇥
d (1) , . .
, K ( N ) 2 R c ( N ) ⇥
d ( N ) ⇥
d ( N ) , a convolution slicer S ( ·| K (1) , . .
, K ( N ) ) on
R c ⇥ d ⇥ d is a composition of N convolution functions with kernels K (1) , . .
, K ( N ) (with strideor dilation if needed) such that S ( X | K (1) , . .
, K ( N ) )
2 R 8 X 2 R c ⇥ d ⇥ d .
As indicated in Deﬁnition 2, the idea of the convolution slicer is to progressively map a given data
X to a one-dimensional subspace through a sequence of convolution kernels, which capture spatialrelations across channels as well as local information of the data.
It is starkly different from thevectorization step in standard sliced Wasserstein on images (1).
The illustration of the convolutionslicer is given in Figure 4 in Appendix C.
We consider three particular types of convolution slicers based on using linear function on theconvolution operator, named convolution-base, convolution-stride, and convolution-dilation slicers.
We defer the deﬁnition of convolution-dilation slicers to Deﬁnition 6 in Appendix B. We ﬁrst startwith the deﬁnition of the convolution-base slicer.
Deﬁnition 3 (Convolution-base Slicer)
Given X 2 R c ⇥ d ⇥ d ( d  2 ),1.
When d is even, N is the biggest integer that satisﬁes d = 2 N  1 a with a is also an integer,sliced kernels are deﬁned as K (1) 2 R c ⇥ (2  1 d +1) ⇥ (2  1 d +1) and K ( h ) 2 R 1 ⇥ (2  h d +1) ⇥ (2  h d +1)for h = 2 , . .
. , N  1 , and K ( N ) 2 R 1 ⇥ a ⇥ a where a = d 2 N  1 .
Then, the convolution-base slicer
CS -b ( X | K (1) , . .
, K ( N ) ) is deﬁned as:
[Equation]
When d is odd, the convolution-base slicer CS -b ( X | K (1) , . .
, K ( N ) ) takes the form:
[Equation]
where K (1) 2 R c ⇥ 2 ⇥ 2 and K (2) , . . .
, K ( N ) are the corresponding sliced kernels that are deﬁned onthe dimension d  1 .
The idea of the convolution-base slicer in Deﬁnition 3 is to reduce the width and the height of theimage by half after each convolution operator.
If the width and the height of the image are odd, theﬁrst convolution operator is to reduce the size of the image by one via convolution with kernels ofsize 2 ⇥ 2 , and then the same procedure as that of the even case is applied.
We would like to remarkthat the conventional slicing of sliced Wasserstein in Section 2 is equivalent to a convolution-baseslicer S ( ·| K (1) ) where K (1) 2 R c ⇥ d ⇥ d that satisﬁes the constraint
[Equation]
We now discuss the second variant of the convolution slicer, named convolution-stride slicer, wherewe further incorporate stride into the convolution operators.
Its deﬁnition is as follows.
Deﬁnition 4 (Convolution-stride Slicer)
Given X 2 R c ⇥ d ⇥ d ( d  2 ),1.
When d is even, N is the biggest integer that satisﬁes d = 2 N  1 a with a is also an integer,sliced kernels are deﬁned as K (1) 2 R c ⇥ 2 ⇥ 2 and K ( h ) 2 R 1 ⇥ 2 ⇥ 2 for h = 2 , . .
. , N  1 , and
K ( N ) 2 R 1 ⇥ a ⇥ a where a = d 2 N  1 .
Then, the convolution-stride slicer CS -s ( X | K (1) , . .
, K ( N ) ) isdeﬁned as:
[Equation]
When d is odd, the convolution-stride slicer CS
-s ( X | K (1) , . . .
, K ( N ) ) takes the form:
[Equation]
where K (1) 2 R c ⇥ 2 ⇥ 2 and K (2) , . . .
, K ( N ) are the corresponding sliced kernels that are deﬁned onthe dimension d  1 .
Similar to the convolution-base slicer in Deﬁnition 3, the convolution-stride slicer reduces the widthand the height of the image by half after each convolution operator.
We use the same procedure ofreducing the height and the width of the image by one when the height and the width of the imageare odd.
The beneﬁt of the convolution-stride slicer is that the size of its kernels does not depend onthe width and the height of images as that of the convolution-base slicer.
This difference improvesthe computational complexity and time complexity of the convolution-stride slicer over those of theconvolution-base slicer (cf.
Proposition 3 in Appendix B).
Non-linear convolution-base slicer:
The composition of convolution functions in the linearconvolution slicer and its linear variants is still a linear function, which may not be effective whenthe data lie in a complex and highly non-linear low-dimensional subspace.
A natural generalizationof linear convolution slicers to enhance the ability of the slicers to capture the non-linearity of thedata is to apply a non-linear activation function after convolution operators.
This enables us to deﬁnea non-linear slicer in Deﬁnition 7 in Appendix
E. The non-linear slicer can be seen as a deﬁningfunction in generalized Radon Transform [48] which was used in generalized sliced Wasserstein [24].
3.2 Convolution Sliced Wasserstein
Given the deﬁnition of convolution slicers, we now state general deﬁnition of convolution sliced
Wasserstein.
An illustration of the convolution sliced Wasserstein is given in Figure 4 in Appendix C.
Deﬁnition
For any p  1 , the convolution sliced Wasserstein (CSW) of order p > 0 between twogiven probability measures µ, ⌫ 2 P p ( R c ⇥ d ⇥ d ) is given by:
[Equation]
⌘i⌘
[Equation]
where the expectation is taken with respect to K (1) ⇠ U ( K (1) ) , . .
, K ( N )
⇠ U ( K ( N ) ) .
Here,
S ( ·| K (1) , . . .
, K ( N ) ) is a convolution slicer with K ( l ) 2 R c ( l ) ⇥ k ( l ) ⇥ k ( l ) for any l 2 [ N ] and U ( K ( l ) )is the uniform distribution with the realizations being in the set K ( l ) which is deﬁned as K ( l ) :
=n
K ( l ) 2 R c ( l ) ⇥ k ( l ) ⇥ k ( l [Equation] |
P c [Equation] l
[Equation] P k ( l ) [Equation] =
[Equation] P k ( l ) [Equation] 0 =1 K ( i )2 h,i 0 ,j
= o , namely, the set K ( l ) consists of tensors
K ( l ) whose squared ` 2 norm is 1.
The constraint that ` 2 norms of K ( l ) is 1 is for guaranteeing the distances between projected supportsare bounded.
When we speciﬁcally consider the convolution slicer as convolution-base slicer ( CS -b),convolution-stride slicer ( CS -s), and convolution-dilation slicer ( CS -d), we have the correspondingnotions of convolution-base sliced Wasserstein (CSW-b), convolution-stride sliced Wasserstein(CSW-s), and convolution-dilation sliced Wasserstein (CSW-d).
Monte Carlo estimation and implementation: Similar to the conventional sliced Wasserstein, theexpectation with respect to kernels K (1) , . .
, K ( N ) uniformly drawn from the sets K (1) , . .
, K ( N ) inthe convolution sliced Wasserstein is intractable to compute.
Therefore, we also make use of Monte
Carlo method to approximate the expectation, which leads to the following approximation of theconvolution sliced Wasserstein:
[Equation]
[Equation] where K ( ` ) i are uniform samples from the sets K ( ` ) (which is equivalent to sample uniformly from S c ( l ) · k ( l )2 then applying the one-to-one reshape mapping) for any ` 2 [ N ] andi 2 [ L ] .
Since each of the convolution slicer S ( ·| K (1) i , . . .
, K ( N ) i ) is in one dimension, wecan utilize the closed-form expression of Wasserstein metric in one dimension to compute
[Equation]
p
[Equation]
with a complexity of O ( m log 2 m ) for eachi
[ L ] where m is the maximum number of supports of µ and ⌫ .
Therefore, the total computationalcomplexity of computing the Monte Carlo approximation (6) is O
( Lm log 2 m ) when the probabilitymeasures µ and ⌫ have at most m supports.
It is comparable to the computational complexity of sliced
Wasserstein on images (1) where we directly vectorize the images and apply the Radon transform tothese ﬂatten images.
Finally, for the implementation, we would like to remark that L convolutionslicers in equation (6) can be computed independently and parallelly using the group convolutionimplementation which is supported in almost all libraries.
Properties of convolution sliced Wasserstein:
We ﬁrst have the following result for the metricity ofthe convolution sliced Wasserstein.
Theorem 1 For any p  1 , the convolution sliced Wasserstein CSW p ( ., . ) is a pseudo-metric on thespace of probability measures on R c ⇥ d ⇥ d , namely, it is symmetric, satisﬁes the triangle inequality,and CSW p ( µ, ⌫ ) =6() µ = ⌫ .
Proof of Theorem 1 is in Appendix D.1.
Our next result establishes the connection between theconvolution sliced Wasserstein and max-sliced Wasserstein and Wasserstein distances.
Proposition 1 For any p  1 , we ﬁnd that CSW p ( µ, ⌫ )  Max-SW p ( µ, ⌫ )  W p ( µ, ⌫ ) , where
Max-SW p ( µ, ⌫ ) :
= max ✓ 2 R cd 2 : k ✓ k 1 W p ( ✓] µ, ✓]⌫ ) is max-sliced Wasserstein of order p .
Proof of Proposition 1 is in Appendix D.2.
Given the bounds in Proposition 1, we demonstratethat the convolution sliced Wasserstein does not suffer from the curse of dimensionality for theinference purpose, namely, the sample complexity for the empirical distribution from i.i.d.
samplesto approximate their underlying distribution is at the order of O ( n  1 / 2 ) .
Proposition 2 Assume that P is a probability measure supported on compact set of R c ⇥ d ⇥ d .
Let
X 1 , X 2 , . . .
, X n be i.i.d.
samples from P and we denote P n = 1 n
P n i =
X i as the empirical measureof these data.
Then, for any p  1 , there exists a universal constant C > 0 such that
[Equation]
where the outer expectation is taken with respect to the data X 1 , X 2 , . .
X n .
Proof of Proposition 2 is in Appendix D.3.
The result of Proposition 2 indicates that the samplecomplexity of the convolution sliced Wasserstein is comparable to that of the sliced Wasserstein onimages (1), which is at the order of O ( n  1 / 2 ) [4], and better than that of the Wasserstein metric,which is at the order of O ( n  1 / (2 cd 2 ) )
Extension to non-linear convolution sliced Wasserstein:
In Appendix E, we provide a non-linearversion of the convolution sliced Wasserstein, named non-linear convolution sliced Wasserstein.
The high-level idea of the non-linear version is to incorporate non-linear activation functions to theconvolution-base, convolution-stride, and convolution-dilation slicers.
The inclusion of non-linearactivation functions is to enhance the ability of slicers to capture the non-linearity of the data.
Byplugging these non-linear convolution slicers into the general deﬁnition of the convolution sliced
Wasserstein in Deﬁnition 5, we obtain the non-linear variants of convolution sliced Wasserstein.
Experiments
In this section, we focus on comparing the sliced Wasserstein (SW) (with the conventional slicing), theconvolution-base sliced Wasserstein (CSW-b), the convolution sliced Wasserstein with stride (CSW-s),and
the convolution sliced Wassersstein with dilation (CSW-d) (see Deﬁnition 6 in Appendix B)) intraining generative models on standard benchmark image datasets such as CIFAR10 (32x32) [27],
STL10 (96x96) [8], CelebA (64x64), and CelebA-HQ (128x128) [37].
We recall that the numberof projections in SW and CSW’s variants is denoted as L .
Finally, we also show the values of the
SW and the CSW variants between probability measures over digits of the MNIST dataset [30] in
Appendix F.1.
From experiments on MNIST, we observe that values of CSW variants are similar tovalues of SW while having better projection complexities.
In generative modeling, we follow the framework of the sliced Wasserstein generator in [13] with somemodiﬁcations of neural network architectures.
The details of the training are given in Appendix F.2.
We train the above model on standard benchmarks such as CIFAR10 (32x32) [27], STL10 (96x96) [8],
CelebA (64x64), and CelebAHQ (128x128) [37].
To compare models, we use the FID score [20]and the Inception score (IS) [51].
The detailed settings about architectures, hyperparameters, andevaluation of FID and IS are given in Appendix G.
We ﬁrst show the FID scores and IS scoresof generative models trained by SW and CSW’s variants with the number of projections L 2{ 1 , 100 , 1000 } in Table 1.
In the table, we report the performance of models at the last training epoch.
We do not report the IS scores on CelebA and CelebA-HQ since the IS scores are not suitable forface images.
We then demonstrate the FID scores and IS scores across training epochs in Figure 1for investigating the convergence of generative models trained by SW and CSW’s variants.
After
[Table] [Table]
that, we report the training time and training memory of SW and CSW variants in Table 2.
Finally,we show randomly generated images from SW’s models and CSW-s’ models on CelebA dataset in
Figure 2.
Generated images of all models on all datasets are given in Figures 5-8 in Appendix F.2.
Summary of FID scores and IS scores: According to Table 1, on CIFAR10, CSW-d gives the lowestvalues of FID scores and IS scores when L = 1 while CSW-s gives the lowest FID scores
when
L = 100 and L = 1000 .
Compared to CSW-s, CSW-d and CSW-b yield higher FID scores and lower
IS scores.
However, CSW-d and CSW-b are still better than SW.
On CelebA, CSW-s performs thebest in all settings.
On STL10, CSW’s variants are also better than the vanilla SW; however, it isunclear which is the best variant.
On CelebA-HQ, SW gives the lowest FID score when
L = 1 .
Incontrast, when L = 100 and L = 1000 , CSW-s is the best choice for training the generative model.
Since the FID scores of L = 1 are very high on CelebA-HQ and STL10, the scores are not verymeaningful for comparing SW and CSW’s variants.
For all models, increasing L leads to betterquality.
Overall, we observe that CSW’s variants enhance the performance of generative models.
FID scores and IS scores across epochs: From Figure 1, we observe that CSW’s variants help thegenerative models converge faster than SW
when L = 100 and L = 1000 .
Increasing the numberof projections from 100 to 1000 , the generative models from both SW and CSW’s variants becomebetter.
Overall, CSW-s is the best option for training generative models among CSW’s variants sinceits FID curves are the lowest and its IS curves are the highest.
Training time and training memory: We report in Table 2 the training speed in the number ofiterations per second and the training memory in megabytes (MBs).
We would like to recall thatthe time complexity and the projection memory complexity of CSW-s and CSW-d are the same.
Therefore, we measure the training time and the training memory of CSW-s as the result for both
CSW-s and CSW-d.
We can see that increasing the number of projections L costs more memory andalso slows down the training speed.
However, the rate of increasing memory of CSW is smaller than
SW.
For CSW-s and CSW-d, the extent of saving memory is even better.
As an example, L = 1000in CSW-s and CSW-d costs less memory than SW with L = 100 while the performance is better(see Table 1).
In terms of training time, CSW-s and CSW-d are comparable to SW and they can
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
be computed faster than CSW.
We refer the readers to Section 3 for a detailed discussion about thecomputational time and projection memory complexity of CSW’s variants.
Generated images: We show randomly generated images on CelebA dataset in Figure 2 and Figure 6(Appendix F), and generated images on CIFAR10, CelebA, STL10, and CelebA-HQ in Figures 5-8 asqualitative comparison between SW and CSW variants.
From the ﬁgures, we can see that generatedimages of CSW-s is more realistic than ones of SW.
The difference is visually clear when the numberof projections L is small e.g., L = 1 and L = 100 .
When L = 1000 , we can still ﬁgure out that
CSW-s is better than SW by looking at the sharpness of the generated images.
Also, we can visuallyobserve the improvement of SW and CSW-s when increasing the number of projections.
In summary,the qualitative results are consistent with the quantitative results (FID scores and IS scores) in Table 1.
For the generated images of CSW-b and CSW-d, we also observe the improvement compared to the
SW which is consistent with the improvement of FID scores and IS scores.
Non-linear convolution sliced Wasserstein: We also compare non-linear extensions of SW and
CSW variants in training generative models on CIFAR10 in Appendix F. For details of non-linearextensions, we refer to Appendix E. From experiments, we observe that convolution can also improvethe performance of sliced Wasserstein in non-linear projecting cases.
Compared to linear versions,non-linear versions can enhance the quality of the generative model or yield comparable results.
Conclusion
We have addressed the issue of the conventional slicing process of sliced Wasserstein when workingwith probability measures over images.
In particular, sliced Wasserstein is deﬁned on probability measures over vectors which leads to the step of vectorization for images.
As a result, the conventional slicing process cannot exploit the spatial structure of data for designing the space of projecting directions and projecting operators.
To address the issue, we propose a new slicing process by using the convolution operator which has been shown to be efﬁcient on images.
Moreover, we investigate the computational complexity and projection memory complexity of the new slicing technique.
We show that convolution slicing is comparable to conventional slicing in terms of computational complexity while being better in terms of projection memory complexity.
By utilizing the new slicing technique, we derive a novel family of sliced Wassersein variants, named convolution sliced Wasserstein.
We investigate the properties of the convolution sliced Wasserstein including its metricity, its computational and sample complexities and its connection to other variants of sliced
Wasserstein in literature.
Finally, we carry out extensive experiments in comparing digits images and training generative models on standard benchmark datasets to demonstrate the favorable performance of the convolution sliced Wasserstein
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](
c ) Did you discuss any potential negative societal impacts of your work?
[Yes](
d ) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]