Sliced Iterative Normalizing Flows
Abstract
We develop an iterative (greedy) deep learning algorithm which is able to transform an arbitrary probability distribution function (PDF) into the target PDF.
The model is based on iterative Optimal Transport of a series of 1D slices, matching on each slice the marginal PDF to the target.
As special cases of this algorithm, we introduce two
Sliced Iterative Normalizing Flows (SINF), which map from the data to the latent space (GIS) and vice versa (SIG).
We show that SIG is able to generate high quality samples that match the GAN benchmarks.
GIS obtains better results on small dataset density estimation tasks compared to the density trained NFs.
SINF approach deviates signiﬁcantly from the current DL paradigm, as it is greedy and does not use concepts such as minibatching, stochastic gradient descent and gradient back-propagation through deep layers.
1. Introduction
Latent variable generative models such as Normalizing
Flows (NFs) (Rezende & Mohamed, 2015; Dinh et al., 2014; 2017; Kingma & Dhariwal, 2018), Variational AutoEncoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Radford et al., 2016) aim to model the distribution p ( x ) of high-dimensional input data x by introducing a mapping from a latent variable z to x where z is assumed to follow a given prior distribution π ( z ) .
These models usually parameterize the mapping using neural networks and the training of these models typically consists of minimizing a dissimilarity measure between the model distribution and the target distribution.
In this work we adopt a different approach to build the map from latent variable z to data x .
We approach this problem from the Optimal Transport (OT) point of view.
OT studies whether the transport maps exist between two probability distributions, and if they do, how to construct the map to minimize the transport cost.
We propose to decompose the high dimensional problem into a succession of 1D transport problems, where the OT solution is known.
The mapping is iteratively augmented, and it has a NF structure that allows explicit density estimation and efﬁcient sampling.
We name the algorithm Sliced Iterative Normalizing Flow (SINF).
Our objective function is inspired by the Wasserstein distance, which is deﬁned as the minimal transport cost and has been widely used in the loss functions of generative models (Arjovsky & Bottou, 2017; Tolstikhin et al., 2018).
We propose a new metric, max K-sliced Wasserstein distance, which enables the algorithm to scale well to high dimensions.
2. Background
The p-Wasserstein distance, p ∈ [1 , ∞ ) , between two probability distributions p 1 and p 2 is deﬁned as:
[Equation]
where Π( p 1 , p 2 ) is the set of all possible joint distributions γ ( x, y ) with marginalized distributions p 1 and p 2 .
In 1D the Wasserstein distance has a closed form solution via Cumulative Distribution Functions (CDFs), but this evaluation is intractable in high dimension.
An alternative metric, the
Sliced p-Wasserstein Distance (SWD) (Rabin et al., 2011; Bonneel et al., 2015), is deﬁned as:
[Equation]
where S d − 1 denotes the unit sphere θ 21 + · · · θ 2 n = 1 in
R d , dθ is the normalized uniform measure on S d − 1 , and
R denotes the Radon transform.
The deﬁnition of Radon transform can be found in the appendix.
For a given θ , the function ( R p )( · , θ ) : R → R is essentially the slice (or projection) of p ( x ) on axis θ .
The SWD can be calculated by approximating the high dimensional integral with Monte Carlo samples.
However, in high dimensions a large number of projections is required to accurately estimate SWD.
This motivates to use the maximum Sliced p-Wasserstein Distance (max SWD):
[Equation]
which is the maximum of the Wasserstein distance of the 1D marginalized distributions of all possible directions.
3. Sliced Iterative Normalizing Flows
We consider the general problem of building a NF that maps an arbitrary PDF p 1 ( x ) to another arbitrary PDF p 2 ( x ) of the same dimensionality.
We ﬁrstly introduce our objective function in Section 3.1.
The general SINF algorithm is presented in Section 3.2.
We then consider the special cases of p 1 and p 2 being standard Normal distributions in Section 3.3 and Section 3.4, respectively.
3.1. Maximum K-sliced Wasserstein distance
We generalize the idea of maximum SWD and propose maximum K-Sliced p-Wasserstein Distance (max K-SWD): max [Equation] K
[Equation]
In this work we ﬁx p = 2 .
The max K-SWD deﬁnes K orthogonal axes { θ 1 , · · · , θ K } where the marginal distributions of p 1 and p 2 are most different, providing a natural choice for performing 1D marginal matching in our algorithm (see Section 3.2).
The proof that max K-SWD is a proper distance and the details of its estimation are provided in the appendix.
3.2. Proposed SINF algorithm
The SINF algorithm is based on iteratively matching the 1D marginalized distribution of p 1 to p 2 .
This is motivated by the inverse Radon Transform (see Appendix) and CramérWold theorem, which suggest that matching the high dimensional distributions is equivalent to matching the 1D slices on all possible directions, decomposing the high dimensional problem into a series of 1D problems.
Given a set of i.i.d.
samples X drawn from p 1 , in each iteration, a set of 1D marginal transformations { Ψ k } Kk =1 1 ( K ≤ d where d is the dimensionality of the dataset) are applied to the samples on orthogonal axes { θ k } Kk =1 to match the 1D marginalized
PDF of p 2 along those axes.
Let A
=
[ θ 1 , · · · , θ K ] be the
Algorithm 1 Sliced
Flow Input: { x i ∼ p 1 } Ni =1 , { y i ∼ p 2 } Ni =1 , K , number of iteration L iter for l = 1 to L iter do θ 1 , · · · , θ K = arg max K-SWD ( x i , y i , K ) for k = 1 to K do
Compute ˆ x i = θ k · x i and ˆ y i
= θ k · y i for each i ˜ x m = quantiles ( PDF (ˆ x i ))
˜
y m
=
quantiles ( PDF (ˆ y i ))
Ψ l,k = RationalQuadraticSpline
(˜ x m , ˜ y m ) end for Ψ l =
[Ψ 1 , · · · , Ψ K ] , A l = [ θ 1 , · · · , θ K ]
Update x i = x
i − A l A Tl x i + A l Ψ l ( A Tl x i ) end for
[Table]
Iteratively maps Iteratively maps
Gaussian to p data p data to Gaussian NF structure
Yes Yes
Good density estimation weight matrix ( A T A
= I K ), the transformation at iteration l of samples X l can be written as
[Equation]
where X ⊥ l = X l −
A l A Tl X l .
Ψ l
=
[Ψ l 1 , · · · , Ψ lK ] T is the marginal mapping of each dimension of A Tl X l , and its components are required to be monotonic and differentiable.
The Inverse and Jacobian determinant of transformationcan be easily evaluated (see appendix).
The weight matrix A l and the marginal transformations Ψ l are determined by iteratively minimizing the max K-SWD (Equation 4) between the transformed p 1 and p 2 .
Specifically, we propose to iteratively solving for the orthogonal axes { θ 1 , · · · , θ K } in max K-SWD, and then apply 1D marginal matching on those axes to minimize max K-SWD.
Let p 1 ,l be the transformed p 1 at iteration l .
The k th component of Ψ l , Ψ l,k , maps the 1D marginalized PDF of p 1 ,l to p 2 and has an OT solution:
[Equation]
where G l,k ( x ) = (cid:82) x −∞ ( R p 1 ,l )( t, θ k ) dt and F k ( x ) =
(cid:82) x
−∞ ( R p 2 )( t, θ k ) dt are the CDFs of p 1 ,l and p 2 on axis θ k , respectively.
The CDFs can be estimated using the quantiles of the samples (in SIG Section 3.3), or using Kernel Density
[Figure] Estimation (KDE, in GIS Section 3.4).
Equation 6 is monotonic and therefore invertible.
We choose to parametrize it with monotonic rational quadratic splines (Gregory &
Delbourgo, 1982; Durkan et al., 2019).
Details about the splines are shown in the appendix.
The proposed algorithm iteratively minimizes the max KSWD between the transformed p 1 and p 2 .
The orthonomal vectors { θ 1 , · · · , θ K } specify K axes along which the marginalized PDF between p 1 ,l and p 2 are most different, thus maximizes the gain at each iteration and improves the efﬁciency of the algorithm.
The model is able to converge with two orders of magnitude fewer iterations than random axes, and it also leads to better sample quality.
This is because as the dimensionality d grows, the number of slices ( R p )( · , θ ) required to approximate p ( x ) using inverse Radon formula scales as L d − 1 (Kolouri et al., 2015), where L is the number of slices needed to approximate a similar smooth 2D distribution.
Therefore, if θ are randomly chosen, it takes a large number of iterations to converge in high dimensions due to the curse of dimensionality.
Our objective function reduces the curse of dimensionality in high dimensions by identifying the most relevant directions.
Unlike KL-divergence which is invariant under ﬂow transformations, max K-SWD is different in data space and in latent space.
Therefore the direction of building the ﬂow model is of key importance.
In the next two sections we discuss two different ways of building the ﬂow, which are good at sample generation and density estimation, respectively.
[Figure] 3.3. Sliced Iterative Generator (SIG)
For Sliced Iterative Generator (SIG) p 1 is a standard Normal distribution, and p 2 is the target distribution.
The model iteratively maps the Normal distribution to the target distribution using 1D slice transformations.
Speciﬁcally, one ﬁrstly draw a set of samples from the standard Normal distribution, and then iteratively updates the samples following
Equation 5.
SIG directly minimizes the max K-SWD between the generated distribution and the target distribution, and is able to generate high quality samples.
3.4. Gaussianizing Iterative Slicing (GIS)
For Gaussianizing Iterative Slicing (GIS) p 1 is the target distribution and p 2 is a standard Normal distribution.
The model iteratively Gaussianizes the target distribution, and the mapping is learned in the reverse direction of SIG.
In
GIS the max K-SWD between latent data and the Normal distribution is minimized, thus the model performs well in density estimation, even though its objective is not p ( x ) .
A comparison between SIG and GIS is shown in Table 1. [Table]
[Table]
RQ-NSF (AR)  55 .38 .
1 Training failures.
4. Experiments 4.1. Generative modeling of images
We evaluate SIG as a generative model of images using the following 4 datasets: MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky et al., 2009) and Celeb-A (Liu et al., 2015).
In Figure 1 we show samples of these four datasets.
For MNIST, Fashion-MNIST and CelebA dataset we show samples from the model with reduced temperature T = 0 .
85 (i.e., sampling from a Gaussian distribution with standard deviation T = 0 .
85 in latent space), which slightly improves the sample quality (Parmar et al., 2018; Kingma & Dhariwal, 2018).
We report the ﬁnal
FID score (calculated using temperature T=1) in Table 2, where we compare our results with similar algorithms SWF (Liutkus et al., 2019) and Flow-Gan (ADV) (Grover et al., 2018).
We also list the FID scores of some other generative models for comparison, including models using slice-based distance SWAEs (Wu et al., 2019; Kolouri et al., 2018) and CWAE (Knop et al., 2018), Wasserstein GAN models (Arjovsky et al., 2017; Gulrajani et al., 2017), and other
GANs and AE-based models PAE (Böhm & Seljak, 2020) and two-stage VAE (Dai & Wipf, 2019; Xiao et al., 2019).
We notice that previous iterative algorithms were unable to produce good samples on high dimensional image datasets.
In contrast, SIG obtains the best FID scores on MNIST and
Fashion-MNIST, while on CIFAR-10 and CelebA
it also outperforms similar algorithms and AE-based models, and gets comparable results with GANs.
4.2. Density estimation p ( x ) of small datasets
We perform density estimation with GIS on four UCI datasets (Lichman et al., 2013) and BSDS300 (Martin et al., 2001).
The data preprocessing of UCI datasets and
BSDS300 follows Papamakarios et al.
We vary the size of the training set N train from 10 2 to 10 5 to test the model performance on a wide range of dataset size. For GIS we consider two hyperparameter settings: large regularization α (see appendix for more details) for better log p performance, and small regularization α for faster training. In Figure 2 we compare GIS to other NF models GF (Meng et al., 2020), FFJORD (Grathwohl et al., 2019), MAF (Papamakarios et al., 2017) and RQ-NSF (AR)(Durkan et al., 2019), as well as KDE.
Some non-GIS NF models collapsed during training or used more memory than our GPU, and are not shown in the plot.
The results in Figure 2 show that GIS is more stable compared to other NFs and outperforms them on small training sets.
This highlights that
GIS is less sensitive to hyper-parameter optimization and achieves good performance out of the box.
GIS training time varies with data size, but is generally lower than other
NFs for small training sets.
We report the training time for 100 training data in Table 3.
GIS with small regularization α requires signiﬁcantly less time than other NFs, while still outperforming them at 100 training size.
5. Conclusions
We introduce sliced iterative normalizing ﬂow (SINF) that iteratively transforms data distribution to a Gaussian (GIS) or the other way around (SIG) using OT.
To the best of our knowledge, SIG is the ﬁrst greedy deep learning algorithm that is competitive with the SOTA generators in high dimensions, while GIS achieves comparable results on density estimation with current NF models, but is more stable, faster to train, and achieves higher p ( x ) when trained on small training sets even though it does not train on p ( x ) .
SINF has very few hyperparameters, and is very insensitive to their choice.
SINF is related to several previous models, which is discussed in Appendix.
SINF has deep neural network architecture, but its approach deviates signiﬁcantly from the current Deep Learning paradigm, as it does not use concepts such as mini-batching, stochastic gradient descent and gradient back-propagation through deep layers.
SINF is an existence proof that greedy Deep Learning without these ingredients can be SOTA for modern high dimensional ML applications.
SINF may be of particular interest in applications where robustness, insensitivity to hyperparameters, small data size, and speed are of primary importance. Acknowledgements
We thank He Jia for providing his code on Iterative Gaussianization, and for helpful discussions. We thank Vanessa Boehm and Jascha Sohl-Dickstein for comments on the manuscript. This material is based upon work supported by the National Science Foundation under Grant Numbers 1814370 and NSF 1839217, and by NASA under Grant
Number 80NSSC18K1274.
in Neural Information Processing
id=rJxgknCcK
AAAI
URL https://www.aaai.org/ocs/index.php/
AAAI/AAAI18/paper/view/17409 .
NeurIPS
Vancouver
BC
Canada
Chiappa
23rd
International Conference on Artiﬁcial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , volume 108 of Proceedings of Machine Learning Research , pp.
PMLR, 2020.
URL http://proceedings.mlr.press/ v108/meng20b.html .
Nalisnick
T
A
Teh
Y
W
Görür
D and Lakshminarayanan, B. Do deep generative models know what they don’t know?
In 7th International
Conference on Learning Representations, ICLR
New Orleans, LA, USA, May 6-9, 2019 .
OpenReview.net, 2019.
URL https://openreview.net/forum?
id=H1xwNhCcYm .
.mlr
press openreview
HkL7n
Sliced Iterative Normalizing Flows
SUPPLEMENTARY DOCUMENT
A. Radon transform
Let L 1 ( X ) be the space of absolute integrable functions on
X .
The Radon transform R : L 1 ( R d ) → L 1 ( R × S d − 1 ) is deﬁned as
[Equation]
where S d − 1 denotes the unit sphere θ 21 + · · · θ 2 n = 1 in R d , δ ( · ) is the Dirac delta function, and (cid:104)· , ·(cid:105) is the standard inner product in R d .
For a given θ , the function ( R p )( · , θ ) :
R → R is essentially the slice (or projection) of p ( x ) on axis θ .
Note that the Radon transform R is invertible.
Its inverse, also known as the ﬁltered back-projection formula, is given by (Helgason, 2010; Kolouri et al., 2019)
[Equation]
where ∗ is the convolution operator, and the convolution kernel h has the Fourier transform ˆ h ( k )
=
c | k | d − 1 .
The inverse
Radon transform provides a practical way to reconstruct the original function p ( x ) using its 1D slices ( R p )( · , θ ) , and is widely used in medical imaging.
This inverse formula implies that if the 1D slices of two functions are the same in all axes, these two functions are identical.
This is also known as Carmér-Wold theorem (Cramér & Wold, 1936).
B. max K-SWD
The optimization in max K-SWD is performed under the constraints that { θ 1 , · · · , θ K } are orthonormal vectors, or equivalently,
A T A
= I K
where A =
[ θ 1 , · · · , θ K ] is the matrix whose i-th column vector is θ i .
Mathematically, the set of all possible A matrices is called Stiefel Manifold
V K ( R d ) =
{ A ∈ R d × K : A T A
= I K } .
As suggested by Tagare (2011), the optimization of matrix A can be performed by doing gradient ascent on the Stiefel Manifold:
[Equation]
where A ( j ) is the weight matrix at gradient descent iteration j (which is different from the iteration l of the algorithm), τ is the learning rate, which is determined by backtracking line search, B = GA T − AG T , and G is the negative gradient matrix G =
[ − ∂ F ∂A p,q ] ∈ R d × K .
Equation 9 has the properties
that A ( j +1) ∈
V K ( R d ) , and that the tangent vector dA ( j +1) dτ | τ =0 is the projection of gradient [ ∂ F ∂A p,q ] onto
T A ( j ) ( V K ( R d ))
(the tangent space of V K ( R d ) at A ( j ) ) under the canonical inner product (Tagare, 2011).
However, Equation 9 requires the inversion of a d × d matrix, which is computationally expensive in high dimensions.
The matrix inversion can be simpliﬁed using the ShermanMorrison-Woodbury formula, which results in the following equation (Tagare, 2011): [Equation] where U = [ G, A ] (the concatenation of columns of G and W ) and V =
[ A, − G ] .
Equation 10 only involves the inversion of a 2 K × 2 K matrix, where K is the number of axes to apply marginal transformation in each iteration.
For high dimensional data (e.g. images), we use a relatively small K to avoid the inversion of large matrices. A large K leads to faster training, but one would converge to similar results with a small K using more iterations. The procedure for estimating max K-SWD and A is summarized in Algorithm 2. Proposition 1.
Let P p (Ω) be the set of Borel probability measures with ﬁnite p’th moment on metric space (Ω , d ) . The maximum K-sliced p-Wasserstein distance is a metric over P p (Ω) . Proof. We ﬁrstly prove the triangle inequality.
Let µ 1 , µ 2 and µ 3 be probability measures in P p (Ω) with probability density function p 1 , p 2 and p 3 , respectively.
Let
[Equation]
Algorithm Input: { x i ∼ p 1 } Ni =1 , { y i ∼ p 2 } Ni =1 , K , order p , max iteration J maxiter
Randomly initialize A ∈ V K ( R d ) for j = 1 to J maxiter do
Initialize D =for k
= 1 to K do θ k
=
A [: , k ]
Compute ˆ x i = θ k · x i and ˆ y i
= θ k · y i for each
i
Sort ˆ x i and ˆ x j in ascending order s.t.
ˆ x i [ n ] ≤ ˆ x i [ n +1] and ˆ y j [ n ] ≤ ˆ y j
[ n +1]
D = D + 1 KN (cid:80) Ni
{+=
+}
| ˆ x i [ n ] − ˆ y j [ n ] | p end for
G = [ − ∂D∂A i,j ] , U =
[ G, A ] , V =
[ A, − G ]
Determine learning rate τ with backtracking line search
A = A − τU ( I 2 K + τ 2 V T U ) − 1 V T A
if A has converged then
Early stop end if end for
Output: D 1 p ≈ max - K - SW p , A ≈ [ θ 1 , · · · , θ K ] (cid:16) 1 K (cid:80)
Kk =1 W pp (( R p 1 )( · , θ k ) , ( R p 3 )( · , θ k )) (cid:17) 1 p ; then max - K - SW p ( p 1 , p 3 )
=
max { θ 1 , ··· ,θ K } orthonormal (cid:32)
K
K (cid:88) k
=
W pp (( R p 1 )( · , θ k ) , ( R p 3 )( · , θ k )) (cid:33) 1 p
= (cid:32)
K
K (cid:88) k
=
W pp (( R p 1 )( · , θ ∗ k ) , ( R p 3 )( · , θ ∗ k )) (cid:33) 1 p ≤ (cid:32)
K
K (cid:88) k
= [ W p (( R p 1 )( · , θ ∗ k ) , ( R p 2 )( · , θ ∗ k ))
+ W p
( R p 2 )( · , θ ∗ k ) , ( R p 3 )( · , θ ∗ k ))] p (cid:33) 1 p ≤ (cid:32)
K
K (cid:88) k
=
W pp (( R p 1 )( · , θ ∗ k ) , ( R p 2 )( · , θ ∗ k )) (cid:33) 1 p + (cid:32) K K (cid:88) k
=
W pp (( R p 2 )( · , θ ∗ k ) , ( R p 3 )( · , θ ∗ k )) (cid:33) 1 p ≤ max { θ 1 , ··· ,θ K } orthonormal (cid:32)
K
K (cid:88) k
=
W pp (( R p 1 )( · , θ k ) , ( R p 2 )( · , θ k )) (cid:33) 1 p + max { θ 1 , ··· ,θ K } orthonormal (cid:32) K K (cid:88) k
= W pp (( R p 2 )( · , θ k ) , ( R p 3 )( · , θ k )) (cid:33) 1 p = max - K - SW p p 1 , p 2 ) + max - K - SW p
( p 2 , p 3 ) , (11) where the ﬁrst inequality comes from the triangle inequality of Wasserstein distance, and the second inequality follows
Minkowski inequality.
Therefore max - K - SW p satisﬁes the triangle inequality.
Now we prove the identity of indiscernibles.
For any probability measures µ 1 and µ 2 in P p (Ω) with probability density function p 1 and p 2 , let ˆ
[Equation]
p 1 )
[Equation]
· , θ ) , ( R p
[Equation]
· , θ )) , and
[Equation]
1 p , we have
[Equation]
On the other hand, let { ˆ θ, ˜ θ 2 , · · · , ˜ θ K } be a set of orthonormal vectors in S d − 1 where the ﬁrst element is ˆ θ , we have
[Equation]
Therefore we have ( 1 K ) 1 p max - SW p ( p 1 , p 2 ) ≤ max - K - SW p ( p 1 , p 2 ) ≤ max - SW p ( p 1 , p 2 ) .
Thus max
[Equation]
µ 1 = µ 2 , where we use the non-negativity and identity of indiscernibles of max - SW p .
Finally, the symmetry of max - K - SW p can be proven using [Figure] the fact that p-Wasserstein distance is symmetric:
[Equation]
C. More details of SINF
We show an illustration of the SINF algorithm in Figure 3. C.1. Inverse and Jacobian determinant
The SINF transformation of Equation 5 can be easily inverted:
[Equation]
where X ⊥ l = X l −
A l A Tl X l = X l +1 − A l A Tl X l +1 .
The
Jacobian determinant of the transformation is also efﬁcient to calculate:
[Equation]
Proof of Equation 16.
Let { θ 1 , · · · , θ K , · · · , θ d } be a set of orthonormal basis in R d where the ﬁrst K vectors are θ 1 , · · · , θ K , respectively.
Let R l
= [ θ 1 , · · · , θ d ] be an orthogonal matrix whose i-th column vector is θ i ,
U l
=
[ θ K +1 , · · · , θ d ] .
Since A l = [ θ 1 , · · · , θ K ] , we have
R l =
[ A l , U l ] (the concatenation of columns of A and
U ).
Let I d − K
=
[id 1 , · · · , id d − K ] T be a marginal transformation that consists of d − K 1D identity transformation, [Equation]
, we have
[Equation]
Since R l is an orthogonal matrix with determinant ± 1 , and the Jacobian of the marginal transformation ˆ Ψ l is diagonal [Equation] the Jacobian determinant of the above equation can be written as
[Equation]
C.2. Objective
At iteration l , the objective of SINF can be written as:
[Equation]
The algorithm ﬁrstly optimize θ lk to maximize the objective, with Ψ lk ﬁxed to identical transformations (equivalent to
Equation 4).
Then the axes θ lk are ﬁxed and the objective is minimized with marginal matching Ψ l .
The samples are updated, and this process repeats until convergence.
C.3. Monotonic Rational Quadratic Spline
Monotonic Rational Quadratic Splines (Gregory & Delbourgo, 1982; Durkan et al., 2019) approximate the function in each bin with the quotient of two quadratic polynomials.
They are monotonic, contineously differentiable, and can be inverted analytically.
The splines are parametrized by the coordinates and derivatives of M knots: { ( x m , y m , y (cid:48) m ) }
Mm =1 , with x m +1 > x m , y m +1 > y m and y (cid:48) m > 0 .
Given these parameters, the function in bin m can be written as (Durkan et al., 2019)
[Equation]
[Figure]
where s m =
( y m +1 − y m ) / ( x m +1 − x m ) , σ m = y [Equation] m +
+ y (cid:48)
m − 2 s m and ξ = ( x − x m ) / ( x m +1 − x m ) .
The derivative is given by
[Equation]
Finally, the inverse can be calculated with
[Equation]
where a = ( s m − y (cid:48) m ) + ζσ m , b
= y (cid:48)
m − ζσ m , c = − s m ζ and ζ =
( y − y m ) /
( y m +1 − y m ) .
The derivation of these formula can be found in Appendix A of Durkan et al.
In our algorithm the coordinates of the knots are determined by the quantiles of the marginalized PDF (see Algorithm 1).
The derivative y (cid:48) m (1 < m < M ) is determined by ﬁtting a local quadratic polynomial to the neighboring knots ( x m − 1 , y m − 1 ) , ( x m , y m ) , and ( x m +1 , y m +1 ) :
[Equation]
The function outside [ x 1 , x M ] is linearly extrapolated with slopes y (cid:48)
and y (cid:48)
M .
In SIG, y (cid:48) 1 and y (cid:48)
M are ﬁxed to 1, while in
GIS they are ﬁtted to the samples that fall outside [ x 1 , x M ] .
We use M = 400 knots in SIG to interpolate each Ψ l,k , while in GIS we set M = min( √ N train , 200) .
The performance is insensitive to these choices, as long as M is large enough to fully characterize the 1D transformation Ψ l,k .
C.4. Patch-based hierarchical approach for SIG
Generally speaking, the neighboring pixels in images have stronger correlations than pixels that are far apart.
This fact has been taken advantage by convolutional neural networks, which outperform Fully Connected Neural Networks (FCNNs) and have become standard building blocks in computer vision tasks.
Like FCNNs, vanilla SIG and GIS make no assumption about the structure of the data and cannot model high dimensional images very well.
Recently, Meng et al.
(2020) proposed a patch-based approach, providing a different way to improve the modeling of the local correlations for NF models.
The patch-based approach decomposes an S × S image into p × p patches, with q × q neighboring pixels in each patch ( S = pq ).
In each iteration the marginalized distribution of each patch is modeled separately without considering the correlations between different patches.
This approach effectively reduces the dimensionality from S 2 to q 2 , at the cost of ignoring the long range correlations.
To reduce the effects of ignoring long range correlations, we propose a hierarchical model.
In SIG, we start from modeling the entire images, which corresponds to q = S and p = 1 .
After some iterations the samples show correct structures, indicating the long range correlations have been modeled well.
We then gradually decrease the patch size q until q = 2 , which allows us to gradually focus on the smaller scales.
Assuming a periodic boundary condition, we let the patches randomly shift in each iteration.
If the patch size q does not divide S , we set p = (cid:98) S/q (cid:99) and the rest of the pixels are kept unchanged.
C.5. Regularization of GIS
We add regularization to GIS for density estimation tasks to further improve the performance and reduce overﬁtting.
The regularization is added in the following two aspects:
The weight matrix W l is regularized by limiting the maximum number of iteration J maxiter (see Algorithm 2).
We set J maxiter =
N/d , where N is the number of training data and d is the dimensionality.
Thus for very small datasets ( N/d → 1 ) the axes of marginal transformation are almost random.
This has no effect on datasets of regular size.
The CDFs in Equation 6 are estimated using KDE, and the 1D marginal transformation is regularized with:
[Equation]
where α ∈ [0 , 1)
is the regularization parameter.
˜ ψ l k is the regularized transformation.
As α increases, the performance improves, but more iterations are needed to converge.
Thus α controls the trade-off between performance and speed.
D. Related work
Iterative normalizing ﬂow models called RBIG (Chen &
Gopinath, 2000; Laparra et al., 2011) are simpliﬁed versions of GIS, as they are based on a succession of rotations followed by 1D marginal Gaussianizations.
Iterative Distribution Transfer (IDT) (Pitié et al., 2007) is a similar algorithm but does not require the base distribution to be
a
Gaussian.
These models do not scale well to high dimensions because they do not have a good way of choosing the axes, and they are not competitive against modern NFs trained on p ( x ) (Meng et al., 2020).
Meng et al.
(2019) use a similar algorithm Projection Pursuit Monge Map (PPMM) to construct OT maps.
They propose to ﬁnd the most informative axis using Projection Pursuit (PP) in each iteration, and show that PPMM works well in low-dimensional bottleneck settings ( d = 8 ).
However, it has yet to be proven that PPMM scales to high dimensions, considering that PP scales as O ( d 3 ) .
A DL, non-iterative version of these models is Gaussianization Flow (GF) (Meng et al., 2020), which trains on p ( x ) and achieves good density estimation results in low dimensions, but does not have good sampling properties in high dimensions.
RBIG, GIS and GF have similar architectures but are trained differently.
We compare their density estimation results in Section 4.2.
Another iterative generative model is Sliced Wasserstein
Flow (SWF) (Liutkus et al., 2019).
Similar to SIG, SWF tries to minimize the SWD between the distributions of samples and the data, and transforms this problem into solving a d dimensional PDE.
The PDE is solved iteratively by doing a gradient ﬂow in the Wasserstein space, and they show
SWF works well for low dimensional bottleneck features.
However, in each iteration the algorithm requires evaluating an integral over the d dimensional unit sphere approximated with Monte Carlo integration, which does not scale well to high dimensions.
Another difference with SIG is that SWF does not have a ﬂow structure, cannot be inverted, and does not provide the likelihood.
We compare the sample qualities between SWF and SIG in Section 4.1.
SWD, max SWD and other slice-based distance (e.g.
Cramér-Wold distance) have been widely used in training generative models (Deshpande et al., 2018; 2019;
Wu et al., 2019; Kolouri et al., 2018; Knop et al., 2018; Nguyen et al., 2020b;a; Nadjahi et al., 2020).
Wu et al.
(2019) propose a differentiable SWD block composed of a rotation followed by marginalized Gaussianizations, but unlike RBIG, the rotation matrix is trained in an end-to-end DL fashion.
They propose Sliced Wasserstein AutoEncoder (SWAE) by adding SWD blocks to an AE to regularize the latent variables, and show that its sample quality outperforms
VAE and AE + RBIG.
Nguyen et al.
(2020b;a) generalize the max-sliced approach using parametrized distributions over projection axes.
Nguyen et al.
(2020b) propose Mixture Spherical Sliced Fused Gromov Wasserstein (MSSFG), which samples the slice axes around a few informative directions following Von Mises-Fisher distribution.
They apply
MSSFG to training Deterministic Relational regularized AutoEncoder (DRAE) and name it mixture spherical DRAE (ms-DRAE).
Nguyen et al.
(2020a) go further and propose
Distributional Sliced Wasserstein distance (DSW), which tries to ﬁnd the optimal axes distribution by parametrizing it with a neural network.
They apply DSW to the training of
GANs, and we will refer to their model as DSWGAN in this paper.
We compare the sample qualities between SWAE and SIG in Section 4.1. Besides GANs and AE-based models,
OT has also been used in constructing NFs (Zhang et al., 2018; Finlay et al., 2020b; Onken et al., 2020; Finlay et al., 2020a; Huang et al., 2020).
Grover et al.
(2018) propose Flow-GAN using a NF as the generator of a GAN, so the model can perform likelihood evaluation, and allows both maximum likelihood and adversarial training.
Similar to our work they ﬁnd that adversarial training gives good samples but poor p ( x ) , while training by maximum likelihood results in bad samples.
Similar to
SIG, the adversarial version of Flow-GAN minimizes the
Wasserstein distance between samples and data, and has a
NF structure.
We compare their samples in Section 4.1.
E. Hyperparameter study and ablation analysis
Here we study the sensitivity of SINF to hyperparameters and perform ablation analyses.
E.1. Hyperparameter K , objective function, and patch based approach
We ﬁrstly test the convergence of SIG on MNIST dataset with different K choices. We measure the SWD (Equation 2) and max SWD (Equation 3) between the test data and model samples for different iterations (without patch based hierarchical modeling). The results are presented in Figure 6. The SWD is measured with 10000 Monte Carlo samples and averaged over 10 times.
The max SWD is measured with Algorithm 2 ( K = 1 ) using different starting points in order to ﬁnd the global maximum.
We also measure the SWD and max SWD between the training data and test data, which gives an estimate of the noise level arising from the ﬁnite number of test data.
For the range of K we consider ( 1 ≤ K ≤ 128 ), all tests we perform converges to the noise level, and the convergence is insensitive to the choice of K , but mostly depends on the total number of 1D transformations ( N iter · K ).
As a comparison, we also try running SIG with random orthogonal axes per iteration, and
Total number of 1D transformations ( N iter ⋅ K ) 0.
S W D
K =
K =
K =
K = random
K =128 random
{+K =784 random training data 
+}
Total number of 1D transformations ( N iter ⋅ K ) 1010m a x
S W D for MNIST, our greedy algorithm converges with two orders of magnitude fewer marginal transformations than random orthogonal axes (Figure 6).
For K = 1 , the objective function (Equation 19) is the same as max SWD, so one would expect that the max SWD between the data and the model distribution keep decreasing as the iteration number increases.
For K > 1 , the max
K-SWD is bounded by max SWD (Equation 12 and 13) so one would also expect similar behavior.
However, from
Figure 6 we ﬁnd that max SWD stays constant in the ﬁrst 400 iterations.
This is because SIG fails to ﬁnd the global maximum of the objective function in those iterations, i.e., the algorithm converges at some local maximum that is almost perpendicular to the global maximum in the high dimensional space, and therefore the max SWD is almost unchanged.
This suggests that our algorithm does not re- [Figure] [Figure] quire global optimization of A at each iteration: even if we ﬁnd only a local maximum, it can be compensated with subsequent iterations.
Therefore our model is insensitive to the initialization and random seeds.
This is very different from the standard non-convex loss function optimization in deep learning with a ﬁxed number of layers, where the random seeds often make a big difference (Lucic et al., 2018).
In Figure 7 we show the samples of SIG of random axes, optimized axes and hierarchical approach.
On the one hand, the sample quality of SIG with optimized axes is better than that of random axes, suggesting that our proposed objective max K-SWD improves both the efﬁciency and the accuracy of the modeling.
On the other hand, SIG with optimized axes has reached the noise level on both SWD and max SWD at around 2000 marginal transformations
(Figure 6), but the samples are not good at that point, and further increasing the number of 1D transformations from 2000 to 200000 does not signiﬁcantly improve the sample quality.
At this stage the objective function of Equationis dominated by the noise from ﬁnite sample size, and the optimized axes are nearly random, which signiﬁcantly limits the efﬁciency of our algorithm.
To better understand this noise, we do a simple experiment by sampling two sets of samples from the standard normal distribution N (0 , I ) and measuring the max SWD using the samples.
The true distance should be zero, and any nonzero value is caused by the ﬁnite number of samples.
In Figure 8 we show the measured max SWD as a function of sample size and dimensionality.
For small number of samples and high dimensionality, the measured max SWD is quite large, suggesting that we can easily ﬁnd an axis where the marginalized PDF of the two sets of samples are signiﬁcantly different, while their underlying distribution are actually the same.
Because of this sample noise, once the generated and the target distribution are close to each other (the max K-SWD reached the noise level), the optimized axes becomes random and the algorithm becomes inefﬁcient.
To reduce the noise level, one needs to either increase the size of training data or decrease the dimensionality of the problem.
The former can be achieved with data augmentation.
In this study we adopt the second approach, i.e., we effectively reduce the dimensionality of the modeling with a patch based hierarchical approach.
The corresponding samples are shown in the bottom panel of Figure 7.
We see that the sample quality keeps improving after 2000 marginal transformations, because the patch based approach reduces the effective noise level.
E.2. Effects of regularization parameter α in density estimation
[Figure]
To explore the effect of regularization parameter α , we train
GIS on POWER dataset with different α .
We keep adding iterations until the log-likelihood of validation set stops improving.
The ﬁnal test log p and the number of iterations are shown in Figure 9.
We see that with a larger α , the algorithm gets better density estimation performance, at the cost of taking more iterations to converge.
Setting the regu-
[Figure] larization parameter α is a trade-off between performance and computational cost. F. Other experiments F.1. Density estimation of full datasets We perform density estimation with GIS on four UCI datasets (Lichman et al., 2013) and BSDS300 (Martin et al., 2001), as well as image datasets MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017).
The data preprocessing of UCI datasets and BSDS300 follows Papamakarios et al.
In Table 4 we compare our results with RBIG (Laparra et al., 2011) and GF (Meng et al., 2020).
The former can be seen as GIS with random axes to apply 1D gaussianization, while the latter can be seen as training non-iterative GIS with MLE training on p ( x ) .
We also list other NF models Real NVP (Dinh et al., 2017),
Glow (Kingma & Dhariwal, 2018), FFJORD (Grathwohl et al., 2019), MAF (Papamakarios et al., 2017) and RQ-NSF (AR)(Durkan et al., 2019) for comparison.
We observe that RBIG performs signiﬁcantly worse than current SOTA.
GIS outperforms RBIG and is the ﬁrst iterative algorithm that achieves comparable performance compared to maximum likelihood models.
This is even more impressive given that GIS is not trained on p ( x ) , yet it outperforms
GF on p ( x ) on GAS, BSDS300 and Fashion-MNIST.
F.2. More samples from SIG
In Figure 10 we show the SIG samples at different iterations.
In Figure 11 we display interpolations between SIG samples, and the nearest training data, to verify we are not
[Table]
memorizing the training data.
F.3. Out of Distribution (OoD) detection
[Table] OoD detection with generative models has recently attracted a lot of attention, since the log p estimates of NF and VAE have been shown to be poor OoD detectors: different generative models can assign higher probabilities to OoD data than to In Distribution (InD) training data (Nalisnick et al., 2019).
One combination of datasets for which this has been observed is Fashion-MNIST and MNIST, where a model trained on the former assigns higher density to the latter.
SINF does not train on the likelihood p ( x ) , which is an advantage for OoD. Likelihood is sensitive to the smallest variance directions (Ren et al., 2019): for example, a zero variance pixel leads to an inﬁnite p ( x ) , and noise must be added to regularize it.
But zero variance directions contain little or no information on the global structure of the image.
SINF objective is more sensitive to the meaningful global structures that can separate between OoD and
InD. Because the patch based approach ignores the long range correlations and results in bad OoD, we use vanilla
SINF without patch based approach.
We train the models on F-MNIST, and then evaluate anomaly detection on test data of MNIST and OMNIGLOT (Lake et al., 2015).
In
Table 5 we compare our results to maximum likelihood p ( x ) models PixelCNN++(Salimans et al., 2017; Ren et al., 2019), and IWAE (Choi et al., 2018).
Other models that perform well include VIB and WAIC (Choi et al., 2018), which achieve 0.941, 0.943 and 0.766, 0.796, for MNIST and OMNIGLOT, respectively (below our SIG results).
For the MNIST case Ren et al.
(2019) obtained 0.996 using the likelihood ratio between the model and its perturbed version, but they require ﬁne-tuning on some additional OoD dataset, which may not be available in OoD applications.
Lower dimensional latent space PAE (Böhm & Seljak, 2020) achieves 0.997 and 0.981 for MNIST and OMNIGLOT, respectively, while VAE based likelihood regret (Xiao et al., 2020) achieves 0.988 on MNIST, but requires additional (expensive) processing.
F.4. Improving the samples of other generative models
Since SIG is able to transform any distribution to the target distribution, it can also be used as a "Plug-and-Play" tool to improve the samples of other generative models.
To demonstrate this, we train SWF, Flow-GAN(ADV) and MAF(5) on
Fashion-MNIST with the default architectures in their papers, and then we apply 240 SIG iterations ( 30% of the total number of iterations in Section 4.1) to improve the samples.
In Figure 12 we compare the samples before and after SIG improvement.
Their FID scores improve from 207 .
6 , 216 .and 81 .
2 to 23 .
9 , 21 . 2 and 16 . 6 , respectively.
These results can be further improved by adding more iterations.