Revisiting Sliced Wasserstein on Images: From Vectorization to Convolution
Anonymous Author(s)
AffiliationAddress
Abstract
The conventional sliced Wasserstein is defined between two probability measuresthat have realizations as vectors .
When comparing two probability measuresover images, practitioners first need to vectorize images and then project themto one-dimensional space by using matrix multiplication between the samplematrix and the projection matrix.
After that, the sliced Wasserstein is evaluated byaveraging the two corresponding one-dimensional projected probability measures.
However, this approach has two limitations.
The first limitation is that the spatialstructure of images is not captured efficiently by the vectorization step; therefore,the later slicing process becomes harder to gather the discrepancy information.
The second limitation is memory inefficiency since each slicing direction is avector that has the same dimension as the images.
To address these limitations,we propose novel slicing methods for sliced Wasserstein between probabilitymeasures over images that are based on the convolution operators.
We deriveconvolution sliced Wasserstein (CSW) and its variants via incorporating stride,dilation, and non-linear activation function into the convolution operators.
Weinvestigate the metricity of CSW as well as its sample complexity, its computationalcomplexity, and its connection to conventional sliced Wasserstein distances.
Finally,we demonstrate the favorable performance of CSW over the conventional sliced
Wasserstein in comparing probability measures over images and in training deepgenerative modeling on images.
Introduction
Optimal transport and Wasserstein distance [53, 46] have become popular tools in machine learningand data science.
For example, optimal transport has been utilized in generative modeling tasksto generate realistic images [2, 52], in domain adaptation applications to transfer knowledge fromsource to target domains [9, 3], in clustering applications to capture the heterogeneity of data [20],and in other applications [28, 56, 57].
Despite having appealing performance, Wasserstein distancehas been known to suffer from high computational complexity, namely, its computational complexityis at the order of O ( m 3 log m ) [44] when the probability measures have at most m supports.
Inaddition, Wasserstein distance also suffers from the curse of dimensionality, namely, its samplecomplexity is at the order of O ( n − 1 /d )
[14] where n is the sample size.
A popular line of work toimprove the speed of computation and the sample complexity of the Wasserstein distance is by addingan entropic regularization term to the Wasserstein distance [10].
This variant is known as entropicregularized optimal transport (or equivalently entropic regularized Wasserstein).
By using the entropic version, we can approximate the value of Wasserstein distance with the computational complexities being at the order of O ( n 2 ) [1, 34, 35, 33] (up to some polynomial orders of approximation errors).
Furthermore, the sample complexity of the entropic version had also been shown to be at the order of
O ( n − 1 / 2 ) [38], which indicates that it does not suffer from the curse of dimensionality.
Another useful line of work to improve both the computational and sample complexities of the
Wasserstein distance is based on the closed-form solution of optimal transport in one dimension.
Anotable distance along this direction is sliced Wasserstein (SW) distance [6].
Due to the fast computational complexity O ( m log 2 m ) and no curse of dimensionality O ( n − 1 / 2 ) , the sliced Wasserstein hasbeen applied successfully in several applications, such as generative modeling [55, 13, 24], domainadaptation [30], and clustering [25].
The sliced Wasserstein is defined between two probabilitymeasures that have supports belonging to a vector space, e.g, R d .
As defined in [6], the sliced
Wasserstein is written as the expectation of one-dimensional Wasserstein distance between twoprojected measures over the uniform distribution on the unit sphere.
Due to the intractability of theexpectation, Monte Carlo samples from the uniform distribution over the unit sphere are used toapproximate the sliced Wasserstein distance.
The number of samples is often called the number ofprojections and it is denoted as L .
On the computational side, the computation of sliced Wassersteincan be decomposed into two steps.
In the first step, L projecting directions are first sampled andthen stacked as a matrix (the projection matrix).
After that, the projection matrix is multiplied by thetwo data matrices resulting in two matrices that represent L one-dimensional projected probabilitymeasures.
In the second step, L one-dimensional Wasserstein distances are computed between thetwo corresponding projected measures with the same projecting direction.
Finally, the average ofthose distances is yielded as the value of the sliced Wasserstein.
Despite being applied widely in tasks that deal with probability measures over images [55, 13],the conventional formulation of sliced Wasserstein is not well-defined to the nature of images.
Inparticular, an image is not a vector but is a tensor.
Therefore, a probability measure over imagesshould be defined over the space of tensors instead of images.
The conventional formulation leads toan extra step in using the sliced Wasserstein on the domain of images which is vectorization.
Namely,all images (supports of two probability measures) are transformed into vectors by a deterministicone-one mapping which is the "reshape" operator.
This extra step does not keep the spatial structuresof the supports, which are crucial information of images.
Furthermore, the vectorization step alsoposes certain challenges to design efficient ways of projecting (slicing) samples to one dimensionbased on prior knowledge about the domain of samples.
Finally, prior empirical investigationsindicate that there are several slices in the conventional Wasserstein collapsing the two probabilitymeasures to the Dirac Delta at zero [13, 12, 23].
Therefore, these slices do not contribute to theoverall discrepancy.
These works suggest that the space of projecting directions in the conventionalsliced Wasserstein (the unit hyper-sphere) is potentially not optimal, at least for images.
Contribution.
To address these issues of the sliced Wasserstein over images, we propose to replacethe conventional formulation of the sliced Wasserstein with a new formulation that is defined on thespace of probability measures over tensors.
Moreover, we also propose a novel slicing process bychanging the conventional matrix multiplication to the convolution operators [15, 17].
In summary,our main contributions are two-fold: 1.
We leverage the benefits of the convolution operators onimages, including their efficient parameter sharing and memory saving as well as their superiorperformance in several tasks on images [27, 18], to introduce efficient slicing methods on sliced
Wasserstein, named convolution slicers .
With the convolution slicers, we derive a novel variantof sliced Wasserstein, named convolution sliced Wasserstein (CSW).
We investigate the metricityof CSW, its sample and computational complexities, and its connection to other variants of sliced
Wasserstein.
We then illustrate the favorable performance of CSW in comparing probabilitymeasures over images.
In particular, we show that CSW provides an almost identical discrepancybetween MNIST’s digits compared to that of the SW while having much less slicing memory.
Furthermore, we compare SW and CSW in training deep generative models on standard benchmarkimage datasets, including CIFAR10, CelebA, STL10, and CeleA-HQ.
By considering the quality ofthe trained generative models, training speed, and training memory of CSW and SW, we observe that the CSW has more favorable performance than the vanilla SW.
Organization.
The remainder of the paper is organized as follows.
We first provide backgroundabout Wasserstein distance, the conventional slicing process in the sliced Wasserstein distance, andthe convolution operator in Section 2.
In Section 3, we propose the convolution slicing and theconvolution sliced Wasserstein, and analyze some of its theoretical properties.
Section 4 containsthe application of CSW to generative models, qualitative experimental results, and quantitativeexperimental results on standard benchmarks.
We conclude the paper In Section 5.
Finally, we deferthe proofs of key results and extra materials in the Appendices.
Notation.
For any d ≥ 2 , S d − 1 := { θ ∈ R d | || θ || 22 = 1 } denotes the d dimensional unit hyper-spherein L 2 norm, and U ( S d − 1 ) is the uniform measure over S d − 1 .
Moreover, δ denotes the Dirac deltafunction.
For p ≥ 1 , P p ( R d ) is the set of all probability measures on R d that have finite p -moments.
For µ, ν ∈ P p ( R d ) , Π( µ, ν ) := { π ∈ P p ( R d × R d ) | (cid:82) R d π ( x, y ) dx =
ν, (cid:82) R d π ( x, y ) dy = µ } is theset of transportation plans between µ and ν .
For m ≥ 1 , we denotes µ ⊗ m as the product measurewhich has the support is m random variables follows µ .
For a vector X ∈ R dm , X :=
( x 1 , . . .
, x m ) , P X denotes the empirical measures 1 m {+(cid:80) mi =
+}
δ x i .
For any two sequences a n and b n , the notationa n = O ( b n ) means that a n ≤ Cb n for all n ≥ 1 where C is some universal constant.
Background
In this section, we first review the definitions of the Wasserstein distance, the conventional slicing,and the sliced Wasserstein distance, and discuss its limitation.
We then review the convolution andthe padding operators on images.
[Table] : For any p ≥ 1 and dimension d ′ ≥ 1 , we first define the Wasserstein- pdistance [53, 45] between two probability measures µ ∈ P p ( R d ′ ) and ν ∈ P p ( R d ′ ) , which is givenby W p ( µ, ν ) := (cid:16) inf π ∈ Π( µ,ν ) (cid:82) R d ′ × R d ′ ∥ x − y ∥
pp dπ ( x, y ) (cid:17) 1 p .
When d ′ = 1 , the Wasserstein distancehas a closed form which is W p ( µ, ν )
=
( (cid:82) 1 0 | F − 1 µ ( z ) − F − 1 ν ( z ) | p dz ) 1 /p where F µ and F ν are thecumulative distribution function (CDF) of µ and ν respectively.
Given this closed-form property of Wasserstein distance in one dimension, the sliced Wasserstein distance [6] between µ and ν had been introduced and admitted the following formulation:
SW pp ( µ, ν ) :
= (cid:82) S d − 1 W pp ( θ♯µ, θ♯ν ) dθ , where θ♯µ is the push-forward probability measure of µthrough the function T θ : R d ′ → R with T θ ( x )
= θ ⊤ x .
For each θ ∈ S d ′ − 1 , W pp ( θ♯µ, θ♯ν ) can becomputed in linear time O ( m log 2 m ) where m is the number of supports of µ and ν .
However, theintegration over the unit sphere in the sliced Wasserstein distance is intractable to compute.
Therefore,
Monte Carlo scheme is employed to approximate the integration, namely, θ 1 , . . .
, θ L ∼ U ( S d ′ − 1 )are drawn uniformly from the unit sphere and the approximation of the sliced Wasserstein distance isgiven by: (cid:100)
SW p p ( µ, ν ) ≈ 1 L (cid:80) Li =1 W
pp ( θ i ♯µ, θ i ♯ν ) .
In practice, L should be chosen to be sufficientlylarge compared to the dimension d ′ , which can be undesirable.
Sliced Wasserstein on Images: Now, we focus on two probability measures over images: µ, [-⌫ 
-]ν ∈
P p
( R c × d × d ) for number of channels c ≥ 1 and dimension d ≥ 1 .
In this case, the sliced Wassersteinbetween µ and ν is defined as:
[Equation]
where R : R c × d × d → R cd 2 is a deterministic one-to-one "reshape" mapping.
The slicing process: The slicing of sliced Wasserstein distance on probability measures over imagesconsists of two steps: vectorization and projection.
Suppose that the probability measure µ R c ×× dhas n supports.
Then the supports of µ are transformed into vectors in R cd 2 and are stacked as amatrix of size n × cd 2 .
A projection matrix of size L × cd 2 is then sampled and has each column as arandom vector following the uniform measure over the unit hyper-sphere.
Finally, the multiplicationof those two matrices returns L projected probability measures of n supports in one dimension.
Weillustrate this process in Figure 3 in Appendix C.
Limitation of the conventional slicing: First of all, images contain spatial relations across channelsand local information.
Therefore, transforming images into vectors makes it challenging to obtainthat information.
Second, vectorization leads to the usage of projecting directions from the unit hyper-sphere, which can have several directions that do not have good discriminative power.
Finally,sampling projecting directions in high-dimension is also time-consuming and memory-consuming.
As a consequence, avoiding the vectorization step can improve the efficiency of the whole process.
Convolution operator: We now define the convolution operator on tensors [15], which will be usedas an alternative way of projecting images to one dimension in the sliced Wasserstein.
The definitionof the convolution operator with stride and dilation is as follows.
Definition 1 (Convolution)
Given the number of channels c ≥ 1 , the dimension d ≥ 1 , thestride size s ≥ 1 , the dilation size b ≥ 1 , the size of kernel k ≥ 1 , the convolution of atensor X ∈ R c × d × d with a kernel size K ∈ R c × k × k is X s,b ∗ K =
Y, Y ∈ R [Equation] d
[Equation]
where d ′ = d − b ( k − 1) − 1 s + 1 .
For i = 1 , . . .
, d ′ and j = 1 , . . .
, d ′ , Y 1 ,i,j is defined as:
[Equation]
From its definition, we can check that the computational complexity of the convolution operator is
O
[Equation]
Convolution Sliced Wasserstein
In this section, we will define a convolution slicer that maps a tensor to a scalar by convolutionoperators.
Moreover, we discuss the convolution slicer and some of its specific forms includingthe convolution-base slicer, the convolution-stride slicer, the convolution-dilation slicer, and theirnon-linear extensions.
After that, we derive the convolution sliced Wasserstein (CSW), a family ofvariants of sliced Wasserstein, that utilizes a convolution slicer as the projecting method.
Finally, wediscuss some theoretical properties of CSW, namely, its metricity, its computational complexity, itssample complexity, and its connection to other variants of sliced Wasserstein.
3.1 Convolution Slicer
We first start with the definition of the convolution slicer, which plays an important role in definingconvolution sliced Wasserstein.
Definition 2 (Convolution Slicer)
For N ≥ 1 , given a sequence of kernels K (1) [-
-]∈
R c (1) × d (1) × d (1) , . . .
, K ( N ) ∈ R c
( N ) × d ( N ) × d ( N ) , a convolution slicer S ( ·| K (1) , . . .
, K ( N ) ) on
R c × d × d is a composition of N convolution functions with kernels K (1) , . .
, K ( N ) (with strideor dilation if needed) such that S ( X | K (1) , . .
, K ( N ) )
∈ R ∀ X ∈ R c × d × d .
As indicated in Definition 2, the idea of the convolution slicer is to progressively map a given data
X to a one-dimensional subspace through a sequence of convolution kernels, which capture spatialrelations across channels as well as local information of the data.
It is starkly different from thevectorization step in standard sliced Wasserstein on images (1).
The illustration of the convolutionslicer is given in Figure 4 in Appendix C.
We consider three particular types of convolution slicers based on using linear function on theconvolution operator, named convolution-base, convolution-stride, and convolution-dilation slicers.
We defer the definition of convolution-dilation slicers to Definition 6 in Appendix B. We first startwith the definition of the convolution-base slicer.
Definition 3 (Convolution-base Slicer)
Given X ∈ R c × d × d ( d ≥ 2 ),1.
When d is even, N = [log 2 d ] , sliced kernels are defined as K (1) ∈ R c × (2 − 1 d +1) × (2 − 1 d +1) and K ( h ) ∈ R 1 × (2 − h d +1)
× (2 − h d +1) for h = 2 , . . . , N − 1 , and K ( N ) ∈ R 1 × a × a where a = d 2 N − 1 .
Then,the convolution-base slicer CS -b ( X | K (1) , . .
, K ( N ) ) is defined as:
[Equation]
When d is odd, the convolution-base slicer CS -b ( X | K (1) , . .
, K ( N ) ) takes the form:
[Equation]
[Equation]
where K (1) ∈ R c × 2 × 2 and K (2) , . .
, K ( N ) are the corresponding sliced kernels that are defined onthe dimension d − 1 .
The idea of the convolution-base slicer in Definition 3 is to reduce the width and the height of theimage by half after each convolution operator.
If the width and the height of the image are odd, thefirst convolution operator is to reduce the size of the image by one via convolution with kernels ofsize 2 × 2 , and then the same procedure as that of the even case is applied.
We would like to remarkthat the conventional slicing of sliced Wasserstein in Section 2 is equivalent to a convolution-baseslicer S ( ·| K (1) ) where K (1) ∈ R c × d × d that satisfies the constraint (cid:80) ch
=1 (cid:80) di =1 (cid:80) dj =1 K (1)2 h,i,j = 1 .
We now discuss the second variant of the convolution slicer, named convolution-stride slicer, wherewe further incorporate stride into the convolution operators.
Its definition is as follows.
Definition 4 (Convolution-stride Slicer)
Given X ∈ R c × d × d ( d ≥ 2 ),1.
When d is even, N = [log 2 d ] , sliced kernels are defined as K (1) ∈ R c × 2 × 2 and K ( h ) ∈ R 1 × 2 × 2for h = 2 , . .
, N − 1 , and K ( N ) ∈ R 1 × a × a where a = d 2 N − 1 .
Then, the convolution-stride slicer CS -s ( X | K (1) , . .
, K ( N ) ) is defined as:
[Equation]
When d is odd, the convolution-stride slicer CS
-s ( X | K (1) , . . .
, K ( N ) ) takes the form:
[Equation]
where K (1) ∈ R c × 2 × 2 and K (2) , . .
, K ( N ) are the corresponding sliced kernels that are defined onthe dimension d − 1 .
Similar to the convolution-base slicer in Definition 3, the convolution-stride slicer reduces the widthand the height of the image by half after each convolution operator.
We use the same procedure ofreducing the height and the width of the image by one when the height and the width of the imageare odd.
The benefit of the convolution-stride slicer is that the size of its kernels does not depend onthe width and the height of images as that of the convolution-base slicer.
This difference improvesthe computational complexity and time complexity of the convolution-stride slicer over those of theconvolution-base slicer (cf.
Proposition 3 in Appendix B).
Non-linear convolution-base slicer:
The composition of convolution functions in the linearconvolution slicer and its linear variants is still a linear function, which may not be effective whenthe data lie in a complex and highly non-linear low-dimensional subspace.
A natural generalizationof linear convolution slicers to enhance the ability of the slicers to capture the non-linearity of thedata is to apply a non-linear activation function after convolution operators.
This enables us to definea non-linear slicer in Definition 7 in Appendix
E. The non-linear slicer can be seen as a definingfunction in generalized Radon Transform [47] which was used in generalized sliced Wasserstein [23].
3.2 Convolution Sliced Wasserstein
Given the definition of convolution slicers, we now state general definition of convolution sliced
Wasserstein.
An illustration of the convolution sliced Wasserstein is given in Figure 4 in Appendix C.
Definition
For any p ≥ 1 , the convolution sliced Wasserstein (CSW) of order p > 0 between twogiven probability measures µ, ν ∈ P p ( R c × d × d ) is given by:
[Equation]
where the expectation is taken with respect to K (1) ∼ U ( K (1) ) , . . .
, K ( N ) ∼ U ( K ( N ) ) .
Here,
S ( ·| K (1) , . . .
, K ( N ) ) is a convolution slicer with K ( l ) ∈ R c ( l ) × k ( l ) × k ( l ) for any l ∈ [ N ] and U ( K ( l ) )is the uniform distribution with the realizations being in the set K ( l ) which is defined as K ( l ) :
= 215 (cid:110)
K ( l ) ∈ R c
( l ) × k ( l ) × k ( l ) | (cid:80) c ( l ) h =1 (cid:80) k
( l ) i ′
=
1 (cid:80) k
( [-i )2 h,i 0 ,j 
-]l ) j ′
=
1 K ( i )2 h,i ′ ,j ′ = 1 (cid:111) , namely, the set K ( l ) consists of tensors
K ( l ) whose squared ℓ 2 norm is 1.
The constraint that ℓ 2 norms of K ( l ) is 1 is for guaranteeing the distances between projected supportsare bounded.
When we specifically consider the convolution slicer as convolution-base slicer ( CS -b),convolution-stride slicer ( CS -s), and convolution-dilation slicer ( CS -d), we have the correspondingnotions of convolution-base sliced Wasserstein (CSW-b), convolution-stride sliced Wasserstein(CSW-s), and convolution-dilation sliced Wasserstein (CSW-d).
Monte Carlo estimation and implementation: Similar to the conventional sliced Wasserstein, theexpectation with respect to kernels K (1) , . .
, K ( N ) uniformly drawn from the sets K (1) , . .
, K ( N ) inthe convolution sliced Wasserstein is intractable to compute.
Therefore, we also make use of Monte
Carlo method to approximate the expectation, which leads to the following approximation of theconvolution sliced Wasserstein:
[Equation]
where K ( ℓ ) i are uniform samples from the sets K ( ℓ ) (which is equivalent to sample uniformly from S c ( l ) · k ( l )2 then applying the one-to-one reshape mapping) for any ℓ ∈ [ N ] andi ∈ [ L ] .
Since each of the convolution slicer S ( ·| K (1) i , . . .
, K ( N ) i ) is in one dimension, wecan utilize the closed-form expression of Wasserstein metric in one dimension to compute
W p (cid:16) S ( ·| K (1) i , . . . , K ( N ) i ) ♯µ, S ( ·| K (1) i , . . , K ( N
[Equation]
i ) ♯ν (cid:17) with a complexity of O ( m log 2 m ) for eachi [-
-]∈
[ L ] where m is the maximum number of supports of µ and ν .
Therefore, the total computationalcomplexity of computing the Monte Carlo approximation (6) is O
( Lm log 2 m ) when the probabilitymeasures µ and ν have at most m supports.
It is comparable to the computational complexity of sliced
Wasserstein on images (1) where we directly vectorize the images and apply the Radon transform tothese flatten images.
Finally, for the implementation, we would like to remark that L convolutionslicers in equation (6) can be computed independently and parallelly using the group convolutionimplementation which is supported in almost all libraries.
Properties of convolution sliced Wasserstein: We first have the following result for the metricity ofthe convolution sliced Wasserstein.
Theorem 1 For any p ≥ 1 , the convolution sliced Wasserstein CSW p ( ., . ) is a pseudo-metric on thespace of probability measures on R c × d × d , namely, it is symmetric, satisfies the triangle inequality,and CSW p ( µ, ν ) {+= 
 ̸⇐⇒+} µ = ν .
Proof of Theorem 1 is in Appendix D.1.
Our next result establishes the connection between theconvolution sliced Wasserstein and max-sliced Wasserstein and Wasserstein distances.
Proposition 1 For any p ≥ 1 , we find that CSW p ( µ, ν ) ≤ Max-SW p ( µ, ν ) ≤ W p ( µ, ν ) , where
Max-SW p ( µ, ν ) :
= max θ ∈ R cd 2 : ∥ θ ∥≤ 1 W p ( θ♯µ, θ♯ν ) is max-sliced Wasserstein of order p .
Proof of Proposition 1 is in Appendix D.2.
Given the bounds in Proposition 1, we demonstratethat the convolution sliced Wasserstein does not suffer from the curse of dimensionality for theinference purpose, namely, the sample complexity for the empirical distribution from i.i.d.
samplesto approximate their underlying distribution is at the order of O ( n − 1 / 2 ) .
Proposition 2 Assume that P is a probability measure supported on compact set of R c × d × d .
Let
X 1 , X 2 , . . .
, X n be i.i.d.
samples from P and we denote P n = 1 n (cid:80) ni =
δ X i as the empirical measureof these data.
Then, for any p ≥ 1 , there exists a universal constant C > 0 such that
[Equation]
where the outer expectation is taken with respect to the data X 1 , X 2 , . .
X n .
[Figure]
variants usually converge faster
Proof of Proposition 2 is in Appendix D.3.
The result of Proposition 2 indicates that the samplecomplexity of the convolution sliced Wasserstein is comparable to that of the sliced Wasserstein onimages (1), which is at the order of O ( n − 1 / 2 ) [4], and better than that of the Wasserstein metric,which is at the order of O ( n − 1 / (2 cd 2 ) )
Extension to non-linear convolution sliced Wasserstein:
In Appendix E, we provide a non-linearversion of the convolution sliced Wasserstein, named non-linear convolution sliced Wasserstein.
The high-level idea of the non-linear version is to incorporate non-linear activation functions to theconvolution-base, convolution-stride, and convolution-dilation slicers.
The inclusion of non-linearactivation functions is to enhance the ability of slicers to capture the non-linearity of the data.
Byplugging these non-linear convolution slicers into the general definition of the convolution sliced
Wasserstein in Definition 5, we obtain the non-linear variants of convolution sliced Wasserstein.
Experiments
In this section, we focus on comparing the sliced Wasserstein (SW) (with the conventional slicing), theconvolution-base sliced Wasserstein (CSW-b), the convolution sliced Wasserstein with stride (CSW-s),and
the convolution sliced Wassersstein with dilation (CSW-d) (see Definition 6 in Appendix B)) intraining generative models on standard benchmark image datasets such as CIFAR10 (32x32) [26],
STL10 (96x96) [8], CelebA (64x64), and CelebA-HQ (128x128) [36].
We recall that the numberof projections in SW and CSW’s variants is denoted as L .
Finally, we also show the values of the
SW and the CSW variants between probability measures over digits of the MNIST dataset [29] in
Appendix F.1.
From experiments on MNIST, we observe that values of CSW variants are similar tovalues of SW while having better projection complexities.
In generative modeling, we follow the framework of the sliced Wasserstein generator in [13] with somemodifications of neural network architectures.
The details of the training are given in Appendix F.2.
We train the above model on standard benchmarks such as CIFAR10 (32x32) [26], STL10 (96x96) [8],
CelebA (64x64), and CelebAHQ (128x128) [36].
To compare models, we use the FID score [19]and the Inception score (IS) [50].
The detailed settings about architectures, hyperparameters, andevaluation of FID and IS are given in Appendix G. We first show the FID scores and IS scoresof generative models trained by SW and CSW’s variants with the number of projections L ∈{ 1 , 100 , 1000 } in Table 1.
In the table, we report the performance of models at the last training epoch.
We do not report the IS scores on CelebA and CelebA-HQ since the IS scores are not suitable forface images.
We then demonstrate the FID scores and IS scores across training epochs in Figure 1for investigating the convergence of generative models trained by SW and CSW’s variants.
After
[Table] [Table]
that, we report the training time and training memory of SW and CSW variants in Table 2.
Finally,we show randomly generated images from SW’s models and CSW-s’ models on CelebA dataset in
Figure 2.
Generated images of all models on all datasets are given in Figures 5-8 in Appendix F.2.
Summary of FID scores and IS scores: According to Table 1, on CIFAR10, CSW-d gives the lowestvalues of FID scores and IS scores when L = 1 while CSW-s gives the lowest FID scores
when
L = 100 and L = 1000 .
Compared to CSW-s, CSW-d and CSW yield higher FID scores and lower
IS scores.
However, CSW-d and CSW are still better than SW.
On CelebA, CSW-s performs the bestin all settings.
On STL10, CSW’s variants are also better than the vanilla SW; however, it is unclearwhich is the best variant.
On CelebA-HQ, SW gives the lowest FID score when L = 1 .
In contrast,when L = 100 and L = 1000 , CSW-s is the best choice for training the generative model.
Since the
FID scores of L = 1 are very high on CelebA-HQ and STL10, the scores are not very meaningful forcomparing SW and CSW’s variants.
For all models, increasing L leads to better generative quality.
Overall, we observe that CSW’s variants enhance the performance of the generative models well.
FID scores and IS scores across epochs: From Figure 1, we observe that CSW’s variants help thegenerative models converge faster than SW
when L = 100 and L = 1000 .
Increasing the numberof projections from 100 to 1000 , the generative models from both SW and CSW’s variants becomebetter.
Overall, CSW-s is the best option for training generative models among CSW’s variants sinceits FID curves are the lowest and its IS curves are the highest.
Training time and training memory: We report in Table 2 the training speed in the number ofiterations per second and the training memory in megabytes (MBs).
We would like to recall thatthe time complexity and the projection memory complexity of CSW-s and CSW-d are the same.
Therefore, we measure the training time and the training memory of CSW-s as the result for both
CSW-s and CSW-d.
We can see that increasing the number of projections L costs more memory andalso slows down the training speed.
However, the rate of increasing memory of CSW is smaller than
SW.
For CSW-s and CSW-d, the extent of saving memory is even better.
As an example, L = 1000in CSW-s and CSW-d costs less memory than SW with L = 100 while the performance is better(see Table 1).
In terms of training time, CSW-s and CSW-d are comparable to SW and they can
[Equation]
be computed faster than CSW.
We refer the readers to Section 3 for a detailed discussion about thecomputational time and projection memory complexity of CSW’s variants.
Generated images: We show randomly generated images on CelebA dataset in Figure 2 and Figure 6(Appendix F), and generated images on CIFAR10, CelebA, STL10, and CelebA-HQ in Figures 5-8 asqualitative comparison between SW and CSW variants.
From the figures, we can see that generatedimages of CSW-s is more realistic than ones of SW.
The difference is visually clear when the numberof projections L is small e.g., L = 1 and L = 100 .
When L = 1000 , we can still figure out that
CSW-s is better than SW by looking at the sharpness of the generated images.
Also, we can visuallyobserve the improvement of SW and CSW-s when increasing the number of projections.
In summary,the qualitative results are consistent with the quantitative results (FID scores and IS scores) in Table 1.
For the generated images of CSW-b and CSW-d, we also observe the improvement compared to the
SW which is consistent with the improvement of FID scores and IS scores.
Non-linear convolution sliced Wasserstein: We also compare non-linear extensions of SW and
CSW variants in training generative models on CIFAR10 in Appendix F. For details of non-linearextensions, we refer to Appendix E. From experiments, we observe that convolution can also improvethe performance of sliced Wasserstein in non-linear projecting cases.
Compared to linear versions,non-linear versions can enhance the quality of the generative model or yield comparable results.
Conclusion
We have addressed the issue of the conventional slicing process of sliced Wasserstein when workingwith probability measures over images.
In particular, sliced Wasserstein is defined on probability measures over vectors which leads to the step of vectorization for images.
As a result, the conventionalslicing process cannot exploit the spatial structure of data for designing the space of projectingdirections and projecting operators.
To address the issue, we propose a new slicing process byusing the convolution operator which has been shown to be efficient on images.
Moreover, weinvestigate the computational complexity and projection memory complexity of the new slicingtechnique.
We show that convolution slicing is comparable to conventional slicing in terms ofcomputational complexity while being better in terms of projection memory complexity.
By utilizingthe new slicing technique, we derive a novel family of sliced Wassersein variants, named convolutionsliced Wasserstein.
We investigate the properties of the convolution sliced Wasserstein including itsmetricity, its computational and sample complexities, and its connection to other variants of sliced
Wasserstein in literature.
Finally, we carry out extensive experiments in comparing digits images andtraining generative models on standard benchmark datasets to demonstrate the favorable performanceof the convolution sliced Wasserstein.