I MPLICIT N ORMALIZING F LOWS
Anonymous authors
Paper under double-blind review
A BSTRACT
Normalizing ﬂows deﬁne a probability distribution by an explicit invertible transformation z
= f ( x ) .
In this work, we present implicit normalizing ﬂows (ImpFlows), which generalize normalizing ﬂows by allowing the mapping to be implicitly deﬁned by the roots of an equation F ( z , x ) = 0 .
ImpFlows build on residual ﬂows (ResFlows) with a proper balance between expressiveness and tractability.
Through theoretical analysis, we show that the function space of
ImpFlow is strictly richer than that of ResFlows.
Furthermore, for any ResFlow with a ﬁxed number of blocks, there exists some function that ResFlow has a nonnegligible approximation error.
However, the function is exactly representable by a single-block ImpFlow.
We propose a scalable algorithm to train and draw samples from ImpFlows.
Empirically, we evaluate ImpFlow on several classiﬁcation and density modeling tasks, and ImpFlow outperforms ResFlow with a comparable amount of parameters on all the benchmarks.
I NTRODUCTION
Normalizing ﬂows (NFs) (Rezende & Mohamed, 2015; Dinh et al., 2014) are promising methods for density modeling.
NFs deﬁne a model distribution p x ( x ) by specifying an invertible transformation f ( x ) from x to another random variable z .
By change-of-variable formula, the model density is
[Equation]
(1) where p z ( z ) follows a simple distribution, such as Gaussian.
NFs are particularly attractive due to their tractability, i.e., the model density p x ( x ) can be directly evaluated as Eqn.
To achieve such tractability, NF models should satisfy two requirements: (i) the mapping between x and z is invertible; (ii) the log-determinant of the Jacobian J f ( x ) is tractable.
Searching for rich model families that satisfy these tractability constraints is crucial for the advance of normalizing ﬂow research.
For the second requirement, earlier works such as inverse autoregressive ﬂow (Kingma et al., 2016) and
RealNVP (Dinh et al., 2017) restrict the model family to those with triangular Jacobian matrices.
More recently, there emerge some free-form Jacobian approaches, such as Residual Flows (ResFlows)
(Behrmann et al., 2019; Chen et al., 2019).
They relax the triangular Jacobian constraint by utilizing a stochastic estimator of the log-determinant, enriching the model family.
However, the
Lipschitz constant of each transformation block is constrained for invertibility.
In general, this is not preferable because mapping a simple prior distribution to a potentially complex data distribution may require a transformation with a very large Lipschitz constant (See Fig.
3 for a 2D example).
Moreover, all the aforementioned methods assume that there exists an explicit forward mapping z
= f ( x ) .
Bijections with explicit forward mapping only covers a fraction of the broad class of invertible functions suggested by the ﬁrst requirement, which may limit the model capacity.
In this paper, we propose implicit ﬂows (ImpFlows) to generalize NFs, allowing the transformation to be implicitly deﬁned by an equation F ( z , x ) = 0 .
Given x (or z ), the other variable can be computed by an implicit root-ﬁnding procedure z = RootFind( F ( · , x )) .
An explicit mapping z = f ( x ) used in prior NFs can viewed as a special case of ImpFlows in the form of F ( z , x ) =
f ( x ) − z = 0 .
To balance between expressiveness and tractability, we present a speciﬁc from of ImpFlows, where each block is the composition of a ResFlow block and the inverse of another ResFlow block.
We theoretically study the model capacity of ResFlows and ImpFlows in the function space.
We show that the function family of single-block ImpFlows is strictly richer than that of two-block
ResFlows by relaxing the Lipschitz constraints.
Furthermore, for any ResFlow with a ﬁxed number of blocks, there exists some invertible function that ResFlow has non-negligible approximation error, but ImpFlow can exactly model.
On the practical side, we develop a scalable algorithm to estimate the probability density and its gradients, and draw samples from ImpFlows.
The algorithm leverages the implicit differentiation formula.
Despite being more powerful, the gradient computation of ImpFlow is mostly similar with that of ResFlows, except some additional overhead on root ﬁnding.
We test the effectiveness of
ImpFlow on several classiﬁcation and generative modeling tasks.
ImpFlow outperforms ResFlow on all the benchmarks, with comparable model sizes and computational cost.
R ELATED W ORK
Expressive Normalizing Flows There are many works focusing on improving the capacity of NFs.
For example, Dinh et al.
(2014; 2017); Kingma & Dhariwal (2018);
Ho et al.
; Song et al.
; Hoogeboom et al.
De Cao et al.
(2020); Durkan et al.
(2019) design dedicated model architectures with tractable Jacobian.
More recently, Grathwohl et al.
; Behrmann et al.
; Chen et al.
(2019) propose NFs with free-form Jacobian, which approximate the determinant with stochastic estimators.
In parallel with architecture design, Chen et al.
; Huang et al.
(2020); Cornish et al.
(2020); Nielsen et al.
(2020) improve the capacity of NFs by operating in a higher-dimensional space.
As mentioned in the introduction, all these existing works adopt explicit forward mappings, which is only a subset of the broad class of invertible functions.
In contrast, the implicit function family we consider is richer.
While we primarily discuss the implicit generalization of ResFlows (Chen et al., 2019) in this paper, the general idea of utilizing implicit invertible functions could be potentially applied to other models as well.
Finally, Zhang et al.
(2020) formally prove that the model capacity of ResFlows is restricted by the dimension of the residual blocks.
In comparison, we study another limitation of ResFlows in terms of the bounded Lipschitz constant, and compare the function family of ResFlows and ImpFlows with a comparable depth.
Continuous Time Flows (CTFs) (Chen et al., 2018b; Grathwohl et al., 2019; Chen et al., 2018a) are ﬂexible alternative to discrete time ﬂows for generative modeling.
They typically treat the invertible transformation as a dynamical system, which is approximately simulated by ordinary differential equation (ODE) solvers.
In contrast, the implicit function family considered in this paper does not contain differential equations, and only requires ﬁxed point solvers.
Moreover, the theoretical guarantee is different.
While CTFs typically study the universal approximation capacity under the continuous time case (i.e., “ inﬁnite depth ” limit), we consider the model capacity of ImpFlows and
ResFlows under a ﬁnite number of transformation steps.
Finally, while CTFs are ﬂexible, their learning is challenging due to instability (Liu et al., 2020; Massaroli et al., 2020) and exceedingly many
ODE solver steps (Finlay et al., 2020), making their large-scale application still an open problem.
Implicit Deep Learning Utilizing implicit functions enhances the ﬂexibility of neural networks, enabling the design of network layers in a problem-speciﬁc way.
For instance, Bai et al.
(2019) propose a deep equilibrium model as a compact replacement of recurrent networks; Amos & Kolter (2017) generalize each layer to solve an optimization problem; Wang et al.
(2019) integrate logical reasoning into neural networks; Reshniak & Webster (2019) utilize the implicit Euler method to improve the stability of both forward and backward processes for residual blocks; and Sitzmann et al.
(2020) incorporate periodic functions for representation learning.
Different from these works, which consider implicit functions as a replacement to feed-forward networks, we develop invertible implicit functions for normalizing ﬂows, discuss the conditions of the existence of such functions, and theoretically study the model capacity of our proposed ImpFlow in the function space.
I MPLICIT N ORMALIZING F LOWS
We now present implicit normalizing ﬂows, by starting with a brief overview of existing work.
3. N ORMALIZING F LOWS
As shown in Eqn.
(1), a normalizing ﬂow f : x (cid:55)→ z is an invertible function that deﬁnes a probability distribution with the change-of-variable formula.
The modeling capacity of normalizing ﬂows depends on the expressiveness of the invertible function f .
Residual ﬂows (ResFlows) (Chen et al.,
R (cid:36)
[Figure]
(cid:36)
[Figure]
2019; Behrmann et al., 2019) are a particular powerful class of NFs due to their
ResFlows use f = f L ◦ · · · ◦ f 1 to construct the invertible mapping, where each layer f l is an invertible residual network with Lipschitz constraints bounded by a ﬁxed constant κ :
[Equation]
where Lip( g ) is the Lipschitz constant of a function g (see Sec.
4.1 for details).
Despite their free-form Jacobian, the model capacity of ResFlows is still limited by the Lipschitz constant of the invertible function.
The Lipschitz constant of each ResFlow block f l cannot exceed 2 (Behrmann et al., 2019), so the Lipschitz constant of an L -block ResFlow cannot exceed 2 L .
However, to transfer a simple prior distribution to a potentially complex data distribution, the Lipschitz constant of the transformation can be required to be sufﬁciently large in general.
Therefore, ResFlows can be undesirably deep simply to meet the Lipschitz constraints (see Fig.
3 for a 2D example).
Below, we present implicit ﬂows (ImpFlows) to relax the Lipschitz constraints.
3. M ODEL S PECIFICATION
In general, an implicit ﬂow (ImpFlow) is deﬁned as an invertible mapping between random variables x and z of dimension d by ﬁnding the roots of F ( z , x ) = 0 , where F is a function from R 2 d to R d .
In particular, the explicit mappings z
= f ( x ) used in prior ﬂow instances (Chen et al., 2019; Kingma & Dhariwal, 2018) can be expressed as an implicit function in the form F ( z , x )
= f
( x ) − z = 0 .
While ImpFlows are a powerful family to explore, generally they are not guaranteed to satisfy the invertibility and the tractability of the log-determinant as required by NFs.
In this paper, we focus on the following speciﬁc form, which achieves a good balance between expressiveness and tractability, and leave other possibilities for future studies.
Deﬁnition 1.
Let g z : R d → R d and g x : R d → R d be two functions such that Lip( g x ) < 1 and
Lip( g z ) < 1 , where Lip( g ) is the Lipschitz constant of a function g .
A speciﬁc form of ImpFlows is deﬁned by
[Equation]
The root pairs of Eqn.
(3) form a subset in R d × R d , which actually deﬁnes the assignment rule of a unique invertible function f .
To see this, for any x 0 , according to Deﬁnition 1, we can construct a contraction h x 0 ( z )
=
F ( z , x 0 )
+ z with a unique ﬁxed point, which corresponds to a unique root (w.r.t.
z ) of F ( z , x 0 ) = 0 , denoted by f ( x 0 ) .
Similarly, in the reverse process, given a z 0 , the root (w.r.t. x ) of F ( z 0 , x ) = 0 also exists and is unique, denoted by f − 1 ( z 0 ) .
These two properties are sufﬁcient to ensure the existence and the invertibility of f , as summarized in the following theorem.
Theorem 1.
Eqn.
(3) deﬁnes a unique mapping f : R d → R d , z
= f ( x ) , and f is invertible.
See proof in Appendix A.1.
Theorem 1 characterizes the validness of the ImpFlows introduced in
Deﬁnition 1.
In fact, a single ImpFlow is a stack of a single ResFlow and the inverse of another single ResFlow, which will be formally stated in Sec 4.
We will investigate the expressiveness of the function family of the ImpFlows in Sec 4, and present a scalable algorithm to learn a deep generative model built upon ImpFlows in Sec. 5.
− 1 .− 0 .0 .0 .
.data
(a) Target function − 1 .− 0 .0 .
L=1 L=
L= (b) ResFlow − 1 .− 0 .0 .0 .
L= (c) ImpFlow − 1 .− 0 .0 .0 .
.2 .f
1 = f 2 f −
[Figure]
Composition
E XPRESSIVENESS P OWER
We ﬁrst present some preliminaries on Lipschitz continuous functions in Sec.
4.1 and then formally study the expressiveness power of ImpFlows, especially in comparison to ResFlows.
In particular, we prove that the function space of ImpFlows is strictly richer than that of ResFlows in Sec. 4.
(see an illustration in Fig.
1 (a)).
Furthermore, for any ResFlow with a ﬁxed number of blocks, there exists some function that ResFlow has a non-negligible approximation error.
However, the function is exactly representable by a single-block ImpFlow.
The results are illustrated in Fig.
1 (b) and formally presented in Sec. 4.3.
F UNCTIONS
For any differentiable function f : R d → R d and any x ∈ R d , we denote the Jacobian matrix of f at x as J f ( x ) ∈ R d × d .
Deﬁnition 2.
A function R d → R d is called Lipschitz continuous if there exists a constant L , s.t.
[Equation]
The smallest L that satisﬁes the inequality is called the Lipschitz constant of f , denoted as Lip( f ) .
Generally, the deﬁnition of Lip( f ) depends on the choice of the norm ||
· || , while we use L 2 -norm by default in this paper for simplicity.
Deﬁnition 3.
A function R d → R d is called bi-Lipschitz continuous if it is Lipschitz continuous and has an inverse mapping f − 1 which is also Lipschitz continuous.
It is useful to consider an equivalent deﬁnition of the Lipschitz constant in our following analysis.
Proposition 1.
(Rademacher (Federer (1969), Theorem 3.1.6))
If f : R d → R d is Lipschitz continuous, then f is differentiable almost everywhere, and
[Equation]
where (cid:107) M (cid:107) 2 = sup { v : (cid:107) v (cid:107) 2 =1 } (cid:107) M v (cid:107) 2 is the operator norm of the matrix M ∈ R d × d .
C OMPARISON TO TWO - BLOCK R ES F LOWS
We formally compare the expressive power of a single-block ImpFlow and a two-block ResFlow.
We highlight the structure of the theoretical results in this subsection in Fig.
1 (a) and present a 1D motivating example in Fig.
All the proofs can be found in Appendix.
A. On the one hand, according to the deﬁnition of ResFlow, the function family of the single-block
ResFlow is
[Equation]
where C 1 ( R d , R d ) consists of all functions from R d to R d with continuous derivatives and Id denotes the identity map.
Besides, the function family of (cid:96) -block
ResFlows is deﬁned by composition:
[Equation]
By deﬁnition of Eqn. (4) and Eqn. (5), R 1 = R .
On the other hand, according to the deﬁnition of the ImpFlow in Eqn.
(3), we can obtain ( g x +
Id)( x ) =
g x ( x ) + x = g z
( z ) + z =
( g z + Id)( z ) , where ◦ denotes the composition of functions.
Equivalently, we have z = (cid:0) ( g z + Id) − 1 ◦ ( g x + Id) (cid:1) ( x ) , which implies the function family of the single-block ImpFlow is
[Equation]
Intuitively, a single-block ImpFlow can be interpreted as the composition of a ResFlow block and the inverse function of another ResFlow block, which may not have an explicit form (see Fig.
2 (c) and (d) for a 1D example).
Therefore, it is natural to investigate the relationship between I and R 2 .
Before that, we ﬁrst introduce a family of “monotonically increasing functions” that does not have an explicit Lipschitz constraint, and show that it is strictly larger than R .
Lemma 1.
[Equation]
where D is the set of all bi-Lipschitz C 1 -diffeomorphisms from R d to R d , and A (cid:36) B means A is a proper subset of B .
Note that it follows from Behrmann et al.
(2019, Lemma 2) that all functions in R are bi-Lipschitz, so
R (cid:36) D .
In the 1D input case, we can get R = { f ∈ C 1 ( R ) : inf x ∈ R f (cid:48) ( x ) > 0 , sup x ∈ R f (cid:48)
( x ) < 2 } , and F = { f ∈ C 1 ( R ) : inf x ∈ R f (cid:48) ( x ) > 0 } .
In the high dimensional cases, R and F are hard to illustrate.
Nevertheless, the Lipschitz constants of the functions in R is less than 2 (Behrmann et al., 2019), but those of the functions in F can be arbitrarily large.
Based on Lemma 1, we prove that the function family of ImpFlows I consists of the compositions of two functions in F , and therefore is a strictly larger than R 2 , as summarized in the following theorem.
Theorem 2.
(Equivalent form of the function family of a single-block ImpFlow).
[Equation]
Note that the identity mapping Id ∈ F , and it is easy to get F ⊂ I .
Thus, the Lipschitz constant of a single ImpFlow (and its reverse) can be arbitrarily large.
Because R (cid:36) F and there exists some functions in I \ R 2 (see a constructed example in Sec. 4.3), we can get the following corollary.
Corollary 1.
R (cid:36) R 2 (cid:36) F 2 = I .
The results on the 1D example in Fig.
2 (b) and (c) accord with Corollary 1.
Besides, Corollary 1 can be generalized to the cases with 2 (cid:96) -block
ResFlows and (cid:96) -block
ImpFlows, which strongly motivates the usage of implicit layers in normalizing ﬂows.
4. C OMPARISON WITH MULTI - BLOCK R ES F LOWS
We further investigate the relationship between R (cid:96) for (cid:96) >
and I , as illustrated in Fig.
1 (b).
For a ﬁxed (cid:96) , the Lipschitz constant of functions in R (cid:96) is still bounded, and there exist inﬁnite functions that are not in R (cid:96) but in I .
We construct one such function family: for any L, r ∈ R + , deﬁne
[Equation]
where B r is an d -dimensional ball with radius of r .
Obviously, P ( L, r ) is an inﬁnite set.
Below, we will show that ∀ 0 < (cid:96) < log 2 ( L ) , R (cid:96) has a non-negligible approximation error for functions in
P ( L, r ) .
However, they are exactly representable by functions in I .
Theorem 3.
Given L > 0 and r > 0 , we have
• P ( L, r ) ⊂ I .
• ∀ 0 < (cid:96) < log 2 ( L ) , P ( L, r ) ∩R (cid:96) = ∅ .
Moreover, for any f ∈ P ( L, r ) with d -dimensional ball B r , the minimal error for ﬁtting f in B r by functions in R (cid:96) satisﬁes
[Equation]
It follows Theorem 3 that to model f ∈ P ( L, r ) , we need only a single-block ImpFlow but at least a log 2 ( L ) -block ResFlow.
In Fig.
2 (b), we show a 1D case where a 3-block ResFlow cannot ﬁt a function that is exactly representable by a single-block ImpFlow.
In addition, we also prove some other properties of ImpFlows.
In particular, R 3 (cid:54)⊂ I .
We formally present the results in Appendix B.
G ENERATIVE M ODELING WITH I MP F LOWS
ImpFlows can be parameterized by neural networks and stacked to form a deep generative model to model high-dimensional data distributions.
We develop a scalable algorithm to perform inference, sampling and learning in such models.
For simplicity, we focus on a single-block during derivation.
Formally, a parametric ImpFlow block z = f ( x ; θ ) is deﬁned by
F ( z , x ; θ ) = 0 , where F ( z , x ; θ )
=
g x ( x ; θ ) − g z ( z ; θ ) + x − z , (11) and Lip( g x ) < 1 , Lip( g z ) < 1 .
Let θ denote all the parameters in g x and g z (which does NOT mean g x and g z share parameters).
Note that x refers to the input of the layer, not the input data.
The inference process to compute z given x in a single ImpFlow block is solved by ﬁnding the root of F ( z , x ; θ ) =
w.r.t.
z , which cannot be explicitly computed because of the implicit formulation.
Instead, we adopt a quasi-Newton method (i.e. Broyden’s method (Broyden, 1965)) to solve this problem iteratively, as follows:
[Equation]
where B is a low-rank approximation of the Jacobian inverse 1 and α is the step size which we use line search method to dynamically compute.
The stop criterion is (cid:107) F ( z [ i ] , x ; θ ) (cid:107) 2 < (cid:15) f , where (cid:15) f is a hyperparameter that balances the computation time and precision.
As Theorem 1 guarantees the existence and uniqueness of the root, the convergence of the Broyden’s method is also guaranteed, which is typically faster than a linear rate.
Another inference problem is to estimate the log-likelihood.
Assume that z ∼ p ( z ) where p ( z ) is a simple prior distribution (e.g. standard Gaussian).
The log-likelihood of x can be written by
[Equation]
where J f ( x ) denotes the Jacobian matrix of a function f at x . See Appendix.
A.4 for the detailed derivation.
Exact calculation of the log-determinant term requires O ( d 3 ) time cost and is hard to scale up to high-dimensional data.
Instead, we propose the following unbiased estimator of ln p ( x ) using the same technique in Chen et al.
with Skilling-Hutchinson trace estimator (Skilling, 1989; Hutchinson, 1989):
[Equation]
where p ( N ) is a distribution supported over the positive integers.
The sampling process to compute x given z can also be solved by the Broyden’s method, and the hyperparameters are shared with the inference process.
In the learning process, we perform stochastic gradient descent to minimize the negative loglikelihood of the data, denoted as L .
For efﬁciency, we estimate the gradient w.r.t.
the model parameters in the backpropagation manner.
According to the chain rule and the additivity of the log-determinant, in each layer we need to estimate the gradients w.r.t.
x and θ of Eqn.
In particular, the gradients computation involves two terms: one is ∂∂ ( · ) ln det( I + J g ( x ; θ )) and the [Table] [Table] other is ∂ L ∂ z ∂ z ∂ ( · ) , where g is a function satisfying Lip( g ) <
1 and ( · ) denotes x or θ .
On the one hand, for the log-determinant term, we can use the same technique as Chen et al.
(2019), and obtain an unbiased gradient estimator as follows.
[Equation]
where p ( N ) is a distribution supported over the positive integers.
On the other hand, ∂ L ∂ z ∂ z ∂ ( · ) can be computed according to the implicit function theorem as follows (See details in Appendix A.5):
[Equation]
, where G ( z ; θ ) =
g z ( z ; θ ) + z .
In comparision to directly calculate the gradient through the quasi-Newton iterations of the forward pass, the implicit gradient above is simple and memory-efﬁcient, treating the root solvers as a blackbox.
Following Bai et al.
(2019), we compute ∂ L ∂
z J − 1 G ( z ) by solving a linear system iteratively, as detailed in Appendix C.1.
The training algorithm is formally presented in Appendix C.4.
E XPERIMENTS
We demonstrate the model capacity of ImpFlows on the classiﬁcation and density modeling tasks 2 .
In all experiments, we use spectral normalization (Miyato et al., 2018) to enforce the Lipschitz constrants, where the Lipschitz constant upper bound of each layer (called Lipschitz coefﬁcient) is denoted as c .
For the Broyden’s method, we use (cid:15) f
= 10 − 6 and (cid:15) b = 10 − 10 for training and testing to numerically ensure the invertibility and the stability during training.
Please see other detailed settings including the method of estimating the log-determinant, the network architecture, learning rate, batch size, and so on in Appendix D.
6. V ERIFYING C APACITY ON C LASSIFICATION
We ﬁrst empirically compare ResFlows and ImpFlows on classiﬁcation tasks.
Compared with generative modeling, classiﬁcation is a more direct measure of the richness of the functional family, because it isolates the function ﬁtting from generative modeling subtleties, such as log-determinant estimation.
We train both models in the same settings on CIFAR10 and CIFAR100 (Krizhevsky [Table] et al., 2009).
Speciﬁcally, we use an architecture similar to ResNet-18 (He et al., 2016).
Overall, the amount of parameters of ResNet-18 with vanilla ResBlocks, ResFlows and ImpFlows are the same of 6 .
5 M.
The detailed network structure can be found in Appendix D.
The classiﬁcation results are shown in Table 1.
To see the impact of the Lipschitz constraints, we vary the Lipschitz coefﬁcient c to show the difference between ResFlows and ImpFlows under the condition of a ﬁxed Lipschitz upper bound.
Given different values of c , the classiﬁcation results of ImpFlows are consistently better than those of ResFlows.
These results empirically validate Corollary 1, which claims that the functional family of ImpFlows is richer than ResFlows.
Besides, for a large Lipschitz constant upper bound c , ImpFlow blocks are comparable with the vanilla ResBlocks in terms of classiﬁcation.
[Figure]
For the density modeling tasks, we ﬁrst evaluate ImpFlows on the Checkerboard data whose density is multi-modal, as shown in Fig.
3 (a).
For fairness, we follow the same experiment settings as Chen et al.
(2019) (which are speciﬁed in Appendix D), except that we adopt a Sine (Sitzmann et al., 2020) activation function for all models.
We note that the data distribution has a bounded support while we want to ﬁt a transformation f
mapping it to the standard Gaussian distribution, whose support is unbounded.
A perfect f requires a sufﬁciently large (cid:107)
J f ( x ) (cid:107) 2 for some x mapped far from the mean of the Gaussian.
Therefore, the Lipschtiz constant of such f is too large to be ﬁtted by a ResFlow with 8 blocks (See Fig.
3 (b)).
A 4 -block ImpFlow can achieve a result of 5 .
05 bits, which outperforms the 5 .
08 bits of a 8 -block ResFlow with the same number of parameters.
Such results accord with our theoretical results in Theorem 2 and strongly motivate ImpFlows.
6. D ENSITY M ODELING O N R EAL D ATA
We also train ImpFlows on some real density modeling datasets, including the tabular datasets (used by Papamakarios et al.
(2017)), CIFAR10 and 5-bit 64 × 64 CelebA (Kingma & Dhariwal, 2018).
For all the real datasets, we use the scalable algorithm proposed in Sec. 5.
We test our performance on ﬁve tabular datasets: POWER ( d = 6 ), GAS ( d = 8 ), HEPMASS ( d = 21 ), MINIBOONE ( d = 43 ) and BSDS300 ( d = 63 ) from the UCI repository (Dua & Graff, 2017), where d is the data dimension.
For a fair comparison, on each dataset we use a 10-block ResFlow and a 5-block ImpFlow with the same amount of parameters, and a 20-block ImpFlow for a better result.
The detailed network architecture and hyperparameters can be found in Appendix D. Table 2 shows the average test log-likelihood for ResFlows and ImpFlows.
ImpFlows achieves better density estimation performance than ResFlow consistently on all datasets.
Again, the results demonstrate the effectiveness of ImpFlows.
Then we test ImpFlows on the CIFAR10 dataset.
We train a multi-scale convolutional version for both ImpFlows and ResFlows, following the same settings as Chen et al.
except that we use a smaller network of 5.5M parameters for both ImpFlows and ResFlows (see details in Appendix D).
As shown in Table 3, Impﬂow achieves better results than ResFlow consistently given different values of the Lipschitz coefﬁcient c .
Moreover, the computation time of ImpFlow is comparable to that of ResFlow.
See Appendix C.2 for detailed results.
Besides, there is a trade-off between the expressiveness and the numerical optimization of ImpFlows in larger models.
Based on the above experiments, we believe that advances including an lower-variance estimate of the log-determinant can beneﬁt ImpFlows in larger models, which is left for future work.
We also train ImpFlows on the 5-bit 64 × 64 CelebA. For a fair comparison, we use the same settings as Chen et al.
The samples from our model are shown in Appendix E.
C ONCLUSIONS
We propose implicit normalizing ﬂows (ImpFlows), which generalize normalizing ﬂows via utilizing an implicit invertible mapping deﬁned by the roots of the equation F ( z , x ) = 0 .
ImpFlows build on
Residual Flows (ResFlows) with a good balance between tractability and expressiveness.
We show that the functional family of ImpFlows is richer than that of ResFlows, particularly for modeling functions with large Lipschitz constants.
Based on the implicit differentiation formula, we present a scalable algorithm to train and evaluate ImpFlows.
Empirically, ImpFlows outperform ResFlows on several classiﬁcation and density modeling benchmarks.
Finally, while this paper mostly focuses on the implicit generalization of ResFlows, the general idea of utilizing implicit functions for NFs could be extended to a wider scope.
We leave it as a future work.
A A DDITIONAL L EMMAS AND P ROOFS A. P ROOF F OR T HEOREM
Proof.
( Theorem 1 )
Firstly, ∀ x 0 ∈ R d , the mapping
[Equation]
is a contrative mapping, which can be shown by Lipschitz condition of g z :
[Equation]
Therefore, h x 0 ( z ) has an unique ﬁxed point, denoted by f ( x 0 ) :
[Equation]
Similarly, we also have: ∀ z 0 ∈ R d , there exists an unique g ( z 0 ) satisfying F ( z 0 , g ( z 0 ))
= 0 .
Moreover, Let z
= f
( x 0 ) , we have F ( f ( x 0 ) , g ( f ( x 0 )))
= 0 .
By the uniqueness, we have g ( f ( x 0 ))
= x 0 , ∀ x 0 ∈ R d .
Similarly, f ( g ( x 0 ))
= x 0 , ∀ x 0 ∈ R d .
Therefore, f is unique and invertible.
A. P ROOF F OR T HEOREM
We denote D as the set of all bi-Lipschitz C 1 -diffeomorphisms from R d to R d .
Firstly, we prove Lemma 1 in the main text.
Proof.
( Lemma 1 ).
∀ f ∈ R , we have
[Equation]
which is equivalent to
[Equation]
Deﬁnition of operator norm. )
[Equation]
Note that J f ( x ) is nonsingular, so ∀ x , v ∈ R d , (cid:107) v (cid:107) 2 = 1 , we have v T J Tf ( x ) J f ( x ) v > 0 .
Thus, 0 >
[Equation]
So we have
[Equation]
Note that the converse is not true, because v T J f ( x ) v > 0 does not restrict the upper bound of
Lipschitz constant of f .
For example, when f ( x ) = m x where m is a positive real number, we have
[Equation]
However, m can be any large positive number.
So we have R (cid:36) F .
Lemma 2.
∀ f ∈ D , if
[Equation]
(17) then
[Equation]
Proof.
(Proof of Lemma 2).
By Inverse Function Theorem ,
[Equation]
Because f is from R d to R d , we have
[Equation]
Let u =
J − 1 f ( x ) v and v
= u (cid:107) u (cid:107) 2 , we have (cid:107) v 0 (cid:107) 2 = 1 , and
[Equation]
The above equation uses this fact: for a real d × d matrix A , ∀ x ∈ R d , x T A x = ( x T A x ) T = x T A T x because x T A x ∈ R .
Note that f is Lipschitz continuous, (cid:107)
J f ( x ) (cid:107) 2 ≤ Lip( f ) .
So
[Equation]
Lip
[Equation]
f )
[Equation]
u (cid:107) 2 , which means (cid:107) u (cid:107) 2 ≥ Lip( f ) . Thus, inf x
[Equation]
v
[Equation]
[Table] [Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
Lemma 3.
∀ f ∈ D , if f − 1 ∈ R , we have
[Equation]
Proof.
(Proof of Lemma 3).
∀ f ∈ D , if f − 1 ∈ R , then from Lemma 1, we have [Equation]
[Equation]
Note that f − 1 ∈ D , from Lemma 2 we have
[Equation]
Lemma 4.
∀ f ∈ D , if
[Equation]
d [Equation] d
[Equation]
0 , then ∃ α 0 > 0 , s.t. ∀ 0 < α < α 0 , sup x ∈ R d (cid:107) αJ f ( x ) − I (cid:107) [Equation] < 1 .
Proof.
(Proof of Lemma 4).
Note that f is Lipschitz continuous, so Lip( f ) =
sup x ∈ R d (cid:107) J f ( x ) (cid:107) 2 .
Denote β =
[Equation]
And let
[Equation]
∀ 0 < α < α 0 , we have
[Equation]
The above equation uses this fact: for a real d × d matrix A , ∀ x ∈ R d , x T A x = ( x T A x ) T = x T A T x because x T A x ∈ R .
Proof.
( Theorem 2 )
Denote
P = { f ∈ D | ∃ f 1 , f 2 ∈ D , f = f 2 ◦ f 1 , where
[Equation]
Firstly, we show that I ⊂ P .
∀ f ∈ I , assume f
= f 2 ◦ f 1 , where f 1 ∈ R and f − 1∈ R .
By
Lemma 1 and Lemma 3, we have
[Equation]
Thus, f ∈ P .
So I ⊂ P .
Next, we show that P ⊂ I .
∀ f ∈ P , assume f
= f 2 ◦ f 1 , where
[Equation]
From Lemma 2, we have
[Equation]
From Lemma 4, ∃ α 1 > 0 , α 2 > 0 , s.t. ∀ 0 < α < min { α 1 , α 2 } ,
[Equation]
− I
[Equation] < [Equation] Let α = 12 min { α 1 , α 2 } .
Let g
= g 2 ◦ g 1 , where g 1 ( x
[Equation]
= αf 1 ( x ) , g 2 ( x ) =
f 2 ( x α ) .
We have g ( x ) =
f 2 ( αf 1 ( x ) α ) = f
( x ) , and
J g 1 ( x ) =
αJ f 1 ( x ) , g − 1 2 ( x ) =
αf
[Equation]
So we have
[Equation]
Thus, g 1 ∈ R and g − 1 2 ∈ R and f = g 2 ◦ g 1 .
So f ∈ I .
Therefore, P ⊂ I .
In conclusion, I = P .
A. P ROOF F OR T HEOREM
Firstly, we prove a lemma of bi-Lipschitz continuous functions.
Lemma 5.
If f : ( R d , (cid:107) · (cid:107) )
→ ( R d , (cid:107) · (cid:107) ) is bi-Lipschitz continuous, then
Lip( f − 1 ) ≤ (cid:107) f ( x 1 ) − f ( x 2 ) (cid:107) (cid:107) x 1 − x 2 (cid:107)
[Equation]
[Equation]
, x
[Equation]
[Equation] = x 2 .
Proof.
(Proof of Lemma 5).
∀ x 1 , x 2 ∈ R d , x 1 (cid:54) =
x 2 , we have (cid:107) f ( x 1 ) − f ( x 2 ) (cid:107) ≤ Lip( f ) (cid:107) x 1 − x 2 (cid:107) (cid:107) x 1 − x 2 (cid:107) = (cid:107) f − 1 ( f ( x 1 )) − f − 1 ( f ( x 2 )) (cid:107) ≤ Lip
[Equation]
f − 1 ) (cid:107) f
( x 1 ) − f ( x 2 ) (cid:107)
Thus, we get the results.
Assume a residual ﬂow f = f L ◦ · · · ◦ f 1 where each layer f l is an invertible residual network:
[Equation]
Thus, each layer f l is bi-Lipschitz and it follows by Behrmann et al.
and Lemma 5 that
[Equation]
By multiplying all the inequalities, we can get a bound of the bi-Lipschitz property for ResFlows, as shown in Lemma 6.
Lemma 6.
For ResFlows built by f = f L ◦ · · · ◦ f 1 , where f l ( x ) =
x + g l ( x ) , Lip( g l ) ≤ κ < 1 , then
[Equation]
Next, we prove Theorem 3.
Proof.
( Theorem 3 )
According to the deﬁnition of P ( L, r ) , we have P ( L, r ) ⊂
F ⊂ I .
∀ 0 < (cid:96) < log 2 ( L ) , we have L − 2 (cid:96) > 0 .
∀ g ∈ R (cid:96) , by Lemma 6, we have (cid:107) g
x ) − g ( y ) (cid:107) 2 ≤ 2 (cid:96) (cid:107) x − y (cid:107) 2 , ∀ x , y ∈ B r .
Thus, ∀ x 0 ∈ B r , we have
[Equation]
So
[Equation]
Notice that the inequality above is true for any x 0 ∈ B r , so we have
[Equation]
Therefore,
[Equation]
So we get
[Equation]
Because ∀ f ∈ P ( L, r ) , inf g ∈R (cid:96) sup x ∈B r (cid:107) f
( x ) − g ( x ) (cid:107) 2 > 0 , we have R (cid:96) ∩ P ( L, r ) = ∅ .
A. P ROOF FOR E QUATION
Proof.
( Equation 13 ) By Change of Variable formula:
[Equation]
Since z = f ( x ) is deﬁned by the equation
[Equation]
by Implicit function theorem, we have
[Equation]
Thus,
[Equation]
Note that any eigenvalue λ of J g x ( x ) satisﬁes | λ | < σ ( J g x ( x ))
= (cid:107)
J g x ( x ) (cid:107) 2 < 1 , so λ ∈ ( − 1 , 1) .
Thus, det( I + J g x ( x )) > 0 .
Similarly, det( I + J g z
( z )) > 0 .
Therefore,
[Equation]
z ))
A. P ROOF FOR E QUATION
Proof.
( Equation 16 )
By implicitly differentiating two sides of F ( z , x ; θ ) = 0 by x , we have
[Equation]
So we have
[Equation]
∂ x By implicitly differentiating two sides of F ( z , x ; θ ) =
by θ , we have
[Equation]
So we have
[Equation]
Therefore, the gradient from z to ( · ) is
[Equation]
B O THER P ROPERTIES OF I MPLICIT F LOWS
In this section, we propose some other properties of ImpFlows. Lemma 7. For a single implicit ﬂow f ∈ I , assume that f = f − 1◦ f 1 , where [Equation]
[Equation]
(22) then
[Equation] Proof.
(Proof of Lemma 7 ) According to Eqn. (20), we have
[Equation]
By multiplying these two inequalities, we can get the results.
Theorem 4.
(Limitation of the single ImpFlow).
[Equation]
where λ ( A ) denotes the set of all eigenvalues of matrix A .
Proof.
(Proof of Theorem 4 )
Proof by contradiction.
Assume ∃ f ∈
I and x ∈ R d , s.t.
∃ λ ∈ λ
( J f ( x )) , λ < 0 .
There exists a vector u (cid:54) = 0 , J f ( x ) u = λ u .
By Theorem 2, ∃ f 1 , f 2 ∈ F , f = f 2 ◦ f 1 , hence
J f ( x ) =
J f 2 ( f 1 ( x ))
J f 1 ( x ) .
We denote A :=
J f 2 ( f 1 ( x )) , B :=
J f 1 ( x ) .
Since f 1 , f 2 ∈ F , we have
[Equation]
Note that B is the Jacobian of a bi-Lipschitz function at a single point, so B is non-singular.
As u (cid:54) = 0 , we have B u (cid:54) = 0 .
Thus,
[Equation]
The last equation uses this fact: for a real d × d matrix A , ∀ x ∈ R d , x T A x = ( x T A x ) T = x T A T x because x T A x ∈ R .
Note that the left side is positive, and the right side is negative.
It’s a contradiction.
Therefore, I cannot include all the bi-Lipschitz C 1 -diffeomorphisms.
As a corollary, we have R 3 (cid:54)⊂
I .
Corollary 2.
R 3 (cid:54)⊂ I .
Proof.
(Proof for Corollary 2 ) Consider three linear functions in R :
[Equation]
x
We can get that f = f 1 ◦ f 2 ◦ f 3 is in R 3 , and f is also a linear function with Jacobian (cid:18) 0 .
2776 − 0 .0 .
5290 − 0 .(cid:19)
However, this is a matrix with two negative eigenvalues: -0.1881, -0.2100.
Hence f is not in I .
Therefore, R 3 (cid:54)⊂ I .
C C OMPUTATION C. A PPROXIMATE I NVERSE J ACOBIAN
The exact computation for the Jacobian inverse term costs much for high dimension tasks.
We use the similar technique in Bai et al.
(2019) to compute ∂ L ∂
z J − 1 G ( z ) : solving the linear system of variable y :
[Equation]
[Equation] where the left hand side is a vector-Jacobian product and it can be efﬁciently computed by autograd packages foy any y without computing the Jacobian matrix.
In this work, we also use Broyden’s method to solve the root, the same as methods in the forward pass, where the tolerance bound for the stop criterion is (cid:15) b . [Table] Backward Training
Sample
[Table]
[Table] Remark.
Although the forward, inverse and backward pass of ImpFlows all need to solve the root of some equation, we can choose small enough (cid:15) f and (cid:15) b to ensure the approximation error is small enough.
Thus, there is a trade-off between computation costs and approximation error.
In practice, we use (cid:15) f
= 10 − 6 and (cid:15) b = 10 − 10 and empirically does not observe any error accumulation.
Note that such approximation is rather different from the variational inference technique in Chen et al.
(2020); Nielsen et al.
(2020), because we only focus on the exact log density itself.
C. C OMPUTATION T IME
We evaluate the average computation time for the model trained on CIFAR10 in Table 3 for c = 0 .on
a single Tesla P100 (SXM2-16GB).
See Table 4 for the details.
For a fair comparision, the forward (inference) time in the training phase of ImpFlow is comparable to that of ResFlow because the log-determinant term is the main cost.
The backward time of ImpFlow costs more than that of
ResFlow because it requires to rewrite the backward method in PyTorch to solve the linear equation.
The training time includes forward, backward and other operations (such as the Lipschitz iterations for spectral normalization).
The sample time of ImpFlow is less than that of ResFlow because the inverse of L -block ImpFlow needs to solve L ﬁxed points while the inverse of 2 L -block ResFlow needs to solve 2 L ﬁxed points.
Also, we evaluate the average Broyden’s method iterations and the average function evaluation times during the Broyden’s method.
See Table 5 for the details.
Note that we use line search in the Broyden’s method, so the function evaluation times is more than the Broyden’s method iterations. C. N UMERICAL S ENSITIVITY We train a 20 -block ImpFlow on POWER dataset with (cid:15) f
= 10 − 6 (see Appendix. D for detailed settings), and then test this model with different (cid:15) f to see the numerical sensitivity of the ﬁxed-point iterations.
Table 6 shows that our model is not sensitive to (cid:15)
f in a fair range.
C. T RAINING ALGORITHM
We state the training algorithms for both forward and backward processes in Algorithm 1 and Algorithm
Require: g x ; θ , g z ; θ in Eqn.
(3), stop criterion (cid:15) f .
Input: x .
Output: z = f ( x ) and ln p ( x ) , where f is the implicit function deﬁned by g x ; θ and g z ; θ .
Deﬁne h
( z ) =
F ( z , x ; θ ) z ←while (cid:107) h ( z ) (cid:107) 2 ≥ (cid:15) f do
B ← The estimated inverse Jacobian of h ( z ) (e.g. by Broyden’s method) α ← LineSearch( z , h, B )
z ← z − αBh ( z ) if training then
Esitamate ln det( I + J g x
( x ; θ )) by Eqn.
Esitamate ln det( I + J g z
( z ; θ )) by Eqn.
(15) else
Esitamate ln det( I + J g x
( x ; θ )) by Eqn.
Esitamate ln det( I + J g z
( z ; θ )) by Eqn.
Compute ln p ( x ) by Eqn.
Algorithm
Require: g x ; θ , g z ; θ in Eqn.
(3), stop criterion (cid:15) b .
Input: x , z , ∂ L ∂ z . Output
Deﬁne G ( z ; θ ) =
g z ( z ; θ ) + z and h ( y ) =
y J G ( z ) − ∂ L ∂
z y ←while (cid:107) h ( y ) (cid:107) 2 ≥ (cid:15) b do
B ← The estimated inverse Jacobian of h ( y ) (e.g. by Broyden’s method) α ← LineSearch( y ,
h, B ) y ← y − αBh ( y )
Compute
∂F ( z , x ; θ ) ∂ x and ∂F ( z , x ; θ ) ∂θ by autograd packages
∂ L ∂ z ∂ z ∂ x ← y ∂F ( z , x ; θ ) ∂ x ∂ L ∂ z ∂
z ∂θ ← y ∂F ( z , x ; θ ) ∂θ [Table]
D N ETWORK S TRUCTURES D. 1-D EXAMPLE
We specify the function (data) to be ﬁtted is [Equation] For I , we can construct a fully-connected neural network with ReLU activation and 3 parameters as following: [Equation]
The two networks can be implemented by spectral normalization.
Assume the implicit function deﬁned by Eqn.
(3) using the above g x ( x ) and g z ( z ) is f I . Next we show that f = f I . Let f 1 ( x ) = x + ReLU( − 0 . 9 x ) and f 2 ( x ) = x − √ 0 .
9ReLU( √ 0 .
9 x ) , we have f − 1 2 ( x ) = x +
ReLU(9 x ) .
Therefore, f I = f − 1◦ f 1 = f .
For every residual block of R , R 2 and R 3 , we train a 4 -layer MLP with ReLU activation andhidden units, and the Lipschitz coefﬁcient for the spectral normalization is 0 . 99 , and the iteration number for the spectral computation is 200 .
The objective function is
[Equation]
where f θ is the function of 1 or 2 or 3 residual blocks.
We use a batch size of 5000 .
We use the Adam optimizer, with learning rate 10 − 3 and weight decay 10 − 5 .
We train the model until convergence, on a single NVIDIA GeForce GTX 1080Ti.
The losses for R , R 2 and R 3 are 5 .
32 , respectively.
D. C LASSIFICATION
For the classiﬁcation tasks, we remove all the BatchNorm layers which are inside of a certain ResBlock, and only maintain the BatchNorm layer in the downsampling layer.
Moreover, as a single
ImpFlow consists of two residual blocks with the same dimension of input and output, we replace the downsampling shortcut by a identity shortcut in each scale of ResNet-18, and add a downsampling layer (a convolutional layer) with BatchNorm after the two residual blocks of each scale.
Thus, each scale consists of two ResBlocks with the same dimension of input and output, which ( 6 . 5 M parameters) is different from the vanilla ResNet-18 architecture ( 11 .
2 M parameters).
Note that the “vanilla ResNet-18” in our main text is refered to the 6 .
5 M-parameter architecture, which is the same as the versions for ResFlow and ImpFlow.
We use the comman settings: batch size of 128 , Adam optimizer with learning rate 10 − 3 and no weight decay, and total epoch of 150 .
For the spectral normalization iterations, we use a error bound of 10 − 3 , the same as Chen et al.
We train every experiment on a single NVIDIA GeForce
GTX 2080Ti.
D. D ENSITY M ODELING ON T OY 2D D ATA
Following the same settings as Chen et al.
(2019), we use 4-layer multilayer perceptrons (MLP) with fully-connected layers of 128 hidden units.
We use the Adam optimizer with learning rate of 10 − 3 and weight decay of 10 − 5 .
Moreover, we ﬁnd that 1 2 π sin(2 π x ) is a better activation for this task while maintain the property of 1 -Lipschitz constant, so we use this activation function for all experiments, which can lead to faster convergence and better log-likelihood for both ResFlows and
ImpFlows, as shown in Fig.
We do not use any ActNorm or BatchNorm layers.
For the log-determinant term, we use brute-force computation as in Chen et al.
For the forward and backward, we use the Broyden’s method to compute the roots, with (cid:15) f
= 10 − 6 .
The Lipschitz coefﬁcient for spectral normalization is 0 . 999 , and the iteration number for spectral normalization is 20 .
The batch size is 5000 , and we trainiterations.
The test batch size is 10000 .
Also, we vary the network depth to see the difference between ImpFlow and ResFlow.
For every depth L , we use an L -block ImpFlow and a 2 L -block ResFlow with the same settings as stated above, and train 3 times with different random seeds.
As shown in Figure 5, the gap between
ImpFlow and ResFlow shrinks as the depth grows deep, because the Lipschitz constant of ResFlow grows exponentially.
Note that the dashed line is a 200-block ResFlow in Chen et al.
, and we tune our model better so that our models perform better with lower depth. [Figure] [Figure] D. D ENSITY M ODELING ON T ABULAR D ATASETS
We use the same data preprocessing as Papamakarios et al.
(2017), including the train/valid/test datasets splits.
For all models, we use a batch size of 1000 (both training and testing) and learning rate of 10 − 3 for the Adam optimizer.
The main settings are the same as Chen et al.
(2019) on the toy2D dataset.
The residual blocks are 4-layer MLPs with 128 hidden units.
The ResFlows use blocks and ImpFlows use 5 blocks to ensure the same amount of parameters. And we use a 20-block ImpFlow for a better result.
Also, we use the Sine activation as 12 π sin(2 π x ) .
We do not use any ActNorm or BatchNorm layers.
For the Lipschitz coefﬁcient, we use c = 0 .
9 and the iteration error bound for spectral normalization is 10 − 3 .
For the settings of our scalable algorithms, we use brute-force computation of the log-determinant term for POWER and GAS datasets and use the same estimation settings as Chen et al. (2019) for HEPMASS, MINIBOONE and BSDS300 datasets.
In particular, for the estimation settings, we always exactly compute 2 terms in training process and 20 terms in testing process for the logdeterminant series.
We use a geometric distribution of p = 0 .
5 for the distribution p ( N ) for the log-determinant term.
We use a single sample of ( n, v ) for the log-determinant estimators for both training and testing.
We train each expeirment on a single NVIDIA GeForce GTX 2080Ti for about 4 days for ResFlows and 6 days for ImpFlows. For 20-block ImpFlow, we train our model for about 2 weeks. However, we ﬁnd that the 20-block ImpFlow will overﬁt the training dastaset for MINIBOONE because this dataset is quite small, so we use the early-stopping technique.
D. D ENSITY M ODELING ON I MAGE D ATASETS
For the CIFAR10 dataset, we follow the same settings and architectures as Chen et al.
In particular, every convolutional residual block is
LipSwish
[Equation]
→ LipSwish
[Equation]
LipSwish
[Equation]
× 3 Conv .
The total architecture is
Image → LogitTransform( α ) → k × ConvBlock → [Squeeze → k × ConvBlock] × 2 , where ConvBlock is i-ResBlock for ResFlows and ImpBlock for ImpBlock, and k = 4 for ResFlows and k = 2 for ImpFlows.
And the ﬁrst ConvBlock does not have LipSwish as pre-activation, followed as Chen et al.
We use ActNorm2d after every ConvBlock.
We do not use the FC layers (Chen et al., 2019).
We use hidden channels as 512 .
We use batch size of 64 and the Adam optimizer of learning rate 10 − 3 .
The iteration error bound for spectral normalization is 10 − 3 .
We use α = 0 .
for CIFAR10.
For the settings of our scalable algorithms, we use the same as Chen et al.
for the logdeterminant terms.
In particular, we always exactly compute 10 terms in training process andterms in testing process for the log-determinant series.
We use a possion distribution for the distribution p ( N ) for the log-determinant term.
We use a single sample of ( n, v ) for the log-determinant estimators for both training and testing.
We train each ResFlow on a single NVIDIA GeForce GTX 2080Ti and each ImpFlow on two cards of NVIDIA GeForce GTX 2080Ti for about 6 days for ResFlows and 8 days for ImpFlows.
Although the amount of parameters are the same, ImpFlows need more GPU memory due to the implementation of PyTorch for the backward pass of implicit function.
For the CelebA dataset, we use exactly the same settings as the ﬁnal version of ResFlows in Chen et al.
(2019), except that we use the Sine activation of the form as 12 π sin(2 π x ) .
E I MP F LOW S AMPLES