On the expressivity of bi-Lipschitz normalizing ﬂows
Abstract
An invertible function is bi-Lipschitz if both the function and its inverse have bounded Lipschitz constants.
Nowadays, most Normalizing Flows are bi-Lipschitz by design or by training to limit numerical errors (among other things).
In this paper, we discuss the expressivity of bi-Lipschitz Normalizing Flows and identify several target distributions that are difﬁcult to approximate using such models.
Then, we characterize the expressivity of bi-Lipschitz Normalizing Flows by giving several lower bounds on the Total Variation distance between these particularly unfavorable distributions and their best possible approximation.
Finally, we discuss potential remedies which include using more complex latent distributions.
1. Introduction
A number of recent publications have demonstrated the beneﬁts of constructing machine learning models with a small
Lipschitz constant.
First, models with a small Lipschitz constant have been linked with better generalization capabilities, both in terms of true risk (Bartlett et al., 2017), and adversarial risk (Farnia et al., 2018).
In addition, models with a small Lipschitz constraint are more stable during training, and are less prone to numerical errors, a property which is particularly important in the context of invertible neural networks and normalizing ﬂows (Behrmann et al., 2021).
Unfortunately, enforcing a small Lipschitz constant, either by design or using regularization during training, can impede the ability of a model to ﬁt the data distribution.
Based on this observation, several researchers have studied the limitations of neural networks with bounded Lipschitz con- .psl.eu > .
Third workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models (ICML 2021).
Copyright 2021 by the author(s).
stant.
In particular Tanielian et al.
(2020) was able identify a family of target distributions with disconnected support that cannot be ﬁtted with a GAN with a bounded Lipschitz constant.
In this paper we focus on the impact of the Lipschitz constraints on normalizing ﬂows.
Normalizing ﬂows are often not only Lipschitz, but bi-Lipschitz , meaning that both the mapping function and its inverse have bounded Lipschitz constant.
For example, Additive Coupling, neural ODE and Residual Networks are bi-Lipschitz by design.
Other types of normalizing ﬂows, can also be trained to be biLipschitz, in order to avoid exploding inverses (Behrmann et al., 2021).
We study the expressivity of normalizing ﬂows with bounded Lipschitz constant and discuss the impact of the bi-Lipschitz constant on the Total Tariation distance.
More precisely, we give several lower bounds on the total variation distance between the generated distribution and the target distribution, in some (particularly unfavorable) training settings.
2. Background
A normalizing ﬂow is an invertible density model in which both density estimation and sampling can be done efﬁciently.
In short, training a normalizing ﬂow consists in learning an invertible mapping between a data space X and a latent space Z .
Typically, the forward direction F : X → Z (i.e. the normalizing direction) is tractable and exact and the inverse direction F − 1 : Z → X (i.e. the generative direction) either has a closed form, or can be approximated using an iterative algorithm.
Suppose that P ∗ is the true data distribution over X , and that P ∗ admits a density function denoted p ∗ that we wish to approximate.
We ﬁrst chose a d -dimensional Gaussian distribution Q over Z (a.k.a. the latent space), and its density function q ( z ) =
[Equation]
(cid:107) 22 . Then, we can deﬁne ˆ p , the approximation of p ∗ , based on q and the mapping F : X → Z , using a simple change of variable formula:
[Equation]
Note the estimated probability (cid:98) P ( A ) of any event A ⊆ X can be retrieved as follows:
[Equation]
As seen in Equation 1, performing density estimates requires computing the determinant of the Jacobian matrix which can be large in practice, thus most normalizing ﬂows have been speciﬁcally designed to make this computation efﬁcient.
2.1. Bi-Lipschitz Normalizing Flows
In this paper, we focus on bi-Lipschitz normalizing ﬂows, which is a mapping F whose Lipschitz constants are bounded in both directions.
More speciﬁcally, we deﬁne the bi-Lipschitz property as follows.
Deﬁnition 2.1.
A bijective function F : X ⊂ R d → Z ⊂
R d is said to be ( L 1 , L 2 ) -bi-Lipschitz if F is L 1 -Lipschitz and its inverse F − 1 is L 2 -Lipschitz, i.e.: ∀ x 1 , x 2 ∈ X , (cid:107) F ( x 1 ) − F ( x 2 ) (cid:107) ≤ L 1 (cid:107) x 1 − x 2 (cid:107) and ∀ z 1 , z 2 ∈ Z , (cid:107) F [Equation] [Equation]
[Equation] F [Equation] [Equation]
) (cid:107) ≤ L 2 (cid:107) z 1 − z 2 (cid:107) Alternatively, since the mapping F is bijective, the biLipschitz continuity can be expressed over F only as: [Equation]
However, enforcing the bi-Lipschitz continuity of F results in a bounded determinant for the Jacobian matrix:
Proposition 2.1.
Jac F satisﬁes for all x ∈ X :
[Equation]
As we will show in the rest of this paper, this can limit the expressivity of normalizing ﬂows.
This is relevant, because many normalizing ﬂows are biLipschitz in practice, for example, the i-ResNet (Behrmann et al., 2019) and the Residual Flow (Chen et al., 2020) are both based on residual atomic blocks f i = I
d
+ g i .
Their invertibility is ensured by the Lipschitz constant
Lip( g i ) ≤ L < 1 .
If F is composed of m residual blocks such that F = f m ◦ · · · ◦ f 1 , then the overall bi-Lipschitz constants satisfy Lip( F ) ≤ (1 + L ) m and Lip( F − 1 ) ≤ 1 / (1 − L ) m .
Alternatively, in Glow (Kingma & Dhariwal, 2018) with atomic blocks W i = P i L i ( U i + diag ( s i )) , the bi-Lipschitz constants statisfy:
Lip( F ) ≤ (cid:81) mi (cid:107) W i (cid:107)and Lip( F − 1 ) ≤ (cid:81) m i (cid:107) W − 1 i (cid:107) 2 .
Consequently, the biLipschitzness constraints on either the function or its Jacobian determinant can be released by increasing the depth of the network but, by doing so, the stability of the inverse can be affected (Behrmann et al., 2021).
2.2. Assessing the learning abilities
Our goal is to understand how the bi-Lipschitz property affects the approximation ability of the network.
To do so, we will compare the true data distribution P ∗ and its density p ∗ with the learned distribution (cid:98) P and its density ˆ p .
To evaluate how the true distribution P ∗ and the generated distribution (cid:98) P differ, we use the Total Variation (TV) distance deﬁned as:
[Equation]
3. Lower Bounds on the TV Distance 3.1. A bound on bi-Lipschitz normalizing ﬂow for any subset A
The ﬁrst theorem is a lower bound on the TV distance between the learned distribution and the target distribution in a general setting.
Intuitively, the idea is to ﬁnd an arbitrary subset A that is sufﬁciently concentrated so that the
Lipschitz constrained mapping can not concentrate enough weight form the Gaussian distribution onto this subset.
Theorem 3.1 (bi-Lipschitz mappings fail to capture high density subset) .
Let F be ( L 1 , L 2 ) -bi-Lipschitz and η A
= P ∗ ( A ) vol( A ) be the average density over any subset A ⊂ R d .
Then:
[Equation]
Therefore, if there is a subset A that satisﬁes η A
[Equation]
d , then the TV is necessarily strictly positive.
The proof of this Theorem is given in appendix A.1.
Remark that the bound in Theorem 3.1 depends on both
Lipschitz constraints L 1 and L 2 .
If a subset A is found to be very dense, the mapping will not be able to expand the given volume of A to match the lower density of the Gaussian density because of L 1 .
On the other hand, the point with the highest density within A will be matched with the highest point on the Gaussian density but all its neighbourhood has to me moved by a factor of 1 /L 2 .
The main advantage of this formulation is to apply to any subset of the data space, but at the expense of a loose bound on the TV.
3.2. Bounds for speciﬁc subset B R, x
The bound in Theorem 3.1 can be further improved by making assumptions on the structure of the subset A .
We choose to focus on l 2 balls instead of arbitrary subsets.
Let B R, x 0 be the l 2 ball with center x 0 and radius R (i.e.
B R, x 0 = { x ∈ X : (cid:107) x − x 0 (cid:107) 2 ≤ R } ).
Then we can show that both high density balls and low density ones are difﬁcult to ﬁt properly, the former because of the Lipschitz constraint of F , the latter because of the Lipschitz constraint of F − 1 .
We ﬁrst consider high density balls.
Theorem 3.2 (NF with a L 1 -Lipschitz mapping F fails to capture high density balls) .
Let F be L 1 -Lipschitz.
Then:
[Equation]
Therefore, if we ﬁnd a ball for which the true measure satisﬁes
[Equation]
L 1 , then the
TV is necessarily strictly positive.
The theorem 3.2 highlights the effect of the Lipschitz constraint of the forward mapping F .
If a ball has a high probability in the data space P ∗ ( B R ) , then the probability assigned to this ball is at most Q ( B RL 1 ) in the latent space and is upper bounded by RL 1 / √ π (Ball, 1993).
A one dimensional representation of a pathological case for theorem 3.2 is shown on ﬁgure 1.
In other words no ball with a high enough density in data space can be expended sufﬁciently to have a matching probability in the latent space.
Note that we could use a closed-form of Q ( B RL 1 ) but it is less open to interpretation than the approximations we have made.
Conversely, the mapping being bi-Lipschitz, the mapping can not contract arbitrarily.
If there is a low density zone mapped on the maximum of the Gaussian density, then the
Normalizing Flow cannot reduce enough the probability of the corresponding zone.
Notice that the assumption of a low density zone is strong but fairly reasonable. For instance, one can observe a multi-modal density with fairly well separated modes.
If the modes are roughly equiprobable, we expect a mapping to assign those modes in balanced way around the mode of the Gaussian distribution in the latent space.
Therefore, the low density ball is mapped on a zone wider than the ball B R/L 2 and consequently the Gaussian measure associated is lower bounded by Q ( B R/L 2 ) as illustrated on the one dimensional example on ﬁgure 2.
Despite the lower bounds established by (Pinelis, 2020), there is no reasonably interpretable bounds, therefore we use the closed-form that is expressed with the Gamma function Γ and the incomplete gamma function γ .
The numerical approximations of the closed form are given in ﬁgure 3.
We can observe that the higher the dimension is, the larger the l 2 distance between two modes can be.
Theorem 3.3 (NF with L 2 -Lipschitz inverse mappings F −fail to capture low density balls) .
Let F − 1 be L 2 -Lipschitz.
We consider the balls centered on F − 1 (0) , we have the lower bound:
[Equation]
Therefore, if we ﬁnd a ball for which the the true measure satisﬁes P ∗ [Equation] R,
[Equation]
, then the TV is necessarily strictly positive.
Both formal proofs are detailed in appendix A.2 and A.3.
3.3. Comparison to related work
A related set up is used in (Tanielian et al., 2020).
The authors consider two disconnected subsets M 1 and M 2 separated by a distance D , with equal probabilities in the latent space, i.e. (cid:98) P ( M 1 )
= (cid:98) P ( M 1 ) = 1 / 2 .
As a consequence,
F − 1 (0) is equidistant from M 1 and M 2 as illustrated in
Figure 4.
Corollary 3.3.1 (NF with L 2 -Lipschitz inverse mapping) .
If F − 1 is L 2 -Lipschitz, then we have a lower bound on the
TV distance based on the distance D between M 1 and M 2 :
[Equation]
Note that, the TV distance being deﬁned as the sup on any subspace A , we can accumulate the failures made by the network, and therefore take into account the error when the two manifolds M 1 and M 2 are too dense.
This set up is then a appropriate pathological case to study the effect of the bi-Lipschitzness of the mapping.
The original work assesses the learning abilities of their generative model, a GAN (Goodfellow et al., 2014), with a deﬁnition of precision and recall given by (Sajjadi et al., 2018) and improved by (Kynk ¨ a ¨ anniemi et al., 2019).
The main advantage of this metric is that is well ﬁtted to be used with the Gaussian Isoperimetric Inequality and therefore gives a result independent from the dimension.
By using, the TV distance or any distance for that matter, it can be applied on distributions with any support.
The details of the Precision and Recall and the comparison between both methods can be found in appendix B and the proof of corollary 3.3.1 is in appendix A.4.
4. Potential remedies & Discussion
As mentioned earlier, increasing the Lipschitz constants of the entire network (for example, by adding extra layers) may impact invertibility and stability during training (Behrmann et al., 2021), and thus is not a suitable approach to improve the expressivity.
Alternatively, one can consider learning the parameters of the latent Gaussian distribution µ and Σ = diag ( σ i ) . However, this is equivalent to changing the Lipschitz constants of F from ( L 1 , L 2 ) to ( L 1 σ , L 2 σ ) , thus this results in trading off the expected error on very dense subsets (Theorem 3.2) with the expected error on subsets with low densities (Theorems 3.3) or vice-versa.
In other words this can lead to a better approximation for a some particular data distributions, but it does not generally improve the expressivity of the normalizing ﬂow. To improve expressivity, a Gaussian Mixture latent distribution can be considered.
Indeed, Khayatkhoei et al. and Izmailov et al. (2019) have shown that such distributions can learn disconnected manifolds.
When the latent distribution is a Gaussian Mixture, Theorem 3.3 does not hold anymore. Limitations similar to the ones highlighted in
Theorem 3.2 still apply, but can be mitigated using learnable parameters. We can trivially adapt the lower bound from Theorem 3. to the Gaussian Mixture with K equally distributed modes with learnable mean µ j and diagonal covariance matrix Σ
[Equation] [Equation]
As we can see here, the lower bound depends on the inverse of the number of modes K in the mixture. Thus, this approach can solve the limitations highlighted in Theorem 3. only if K is small enough (so that the probability mapped onto a dense subset can be sufﬁciently large). In addition, further investigation is required to understand how to tune or learn the parameters σ ij and K of the Gaussian Mixture in order to obtain satisfying training results.
5. Conclusion
We have established the bi-Lipschitz constraints reduce the expressivity of Normalizing ﬂows.
When the dataset meets some particular conditions such as a high density zone or a low density zone between two high density zones, the reduced expressivity fails to capture the real distribution of the dataset.
To compensate, this lack of learning ability of the mapping, a more complex, i e. expressive latent distribution can be implemented.
However, this method suffers from training difﬁculties and should be further studied.
A. Proofs A.1. Proof of theorem 3.
By deﬁnition we have (cid:98) P ( A ) = (cid:82)
A ˆ p ( x ) d x , then with the change of variable formula we obtain:
[Equation]
As F − 1 is L 2 -Lipschitz, F satisﬁes: ∀ x 1 , x 2 ∈ X ,
[Equation]
and in particular, we have:
[Equation]
Consequently ∀ x ∈ X q ( F ( x )) =
[Equation]
where T is the afﬁne mapping given by
[Equation]
As F is L 1 -Lipschitz we have | Jac F ( x ) | <
L d 1 and thus (cid:98) P ( A ) ≤ (cid:16)
L
[Equation]
(cid:17) d (cid:82) A
[Equation]
T ( x ) (cid:107) 22 d x ≤ (cid:16) L 1 √ 2 π (cid:17) d (cid:82) T ( A ) 1 (4 L 2 ) d d x ≤ (cid:16) L 1 4 L 2 √ 2 π (cid:17) d vol( A ) , and thus TV ( P ∗ , (cid:98) P ) = sup A | P ∗ ( A ) − (cid:98) P ( A ) | implies
TV ( P ∗ , (cid:98) P ) ≥ sup A (cid:32) P ∗ ( A ) − (cid:18) L4 L 2 √
[Equation]
(cid:19) d vol( A ) (cid:33)
A.2. Proof of theorem 3.
By deﬁnition of the TV distance, we have
[Equation]
where B R, x 0 is the ball of a radius R centered in x 0 .
Then, the idea is to show that the image of a ball B R by a L 1Lipschitz function is in a ball of radius L 1 R , and then use a reverse isoperimetric inequality the ﬁnd an upper bound of the measure of a ball of a radius L 1 R .
Proof of F ( B R, x 0 ) ⊂ B L 1 R,F ( x 0 )
First of all, for every z ∈ F ( B R, x 0 ) , there exist x ∈ B R such that F − 1 ( z ) =
x , we have:
[Equation]
Upper bound of Q ( B L 1 R )
First of all, it can be easily establish that Q ( B L 1 R ( F ( x 0 ))) is at a maximum when
F ( x 0 ) = 0 .
From now on, we will only consider B L 1 R the ball centered on 0 .
Therefore the objective is to ﬁnd an upper bound on:
[Equation]
We can use the polar coordinates system to get another expression of the Gaussian measure with S d − 1 ( r ) =
2 π d/ 2 r d − 1 Q ( d/ 2) being the volume of the hypersphere:
[Equation]
However r d − 1 e − r 2 / 2 has a maximum value reached for r = √ d −
[Equation]
we can have an upper bound:
[Equation]
Then, with the Stirling approximation of the Gamma function:
[Equation]
We obtain:
[Equation]
Using the bound 1 √
[Equation]
d −, we have
[Equation]
Otherwise, an inequality from (Ball, 1993) give for the hypersphere in R d − 1 , an upper bound of the Gaussian Measure over a convex set C in R d :
[Equation]
Therefore, for a Ball B L 1 R :
[Equation]
Lower Bound of the TV As soon as we have an upper bound on Q ( B L 1 R ) , we have:
[Equation]
With the second upper bound, the theorem can be formulated as:
[Equation]
A.3. Proof of theorem 3.
In this section, we denote B R =
B R,F − 1 (0) .
As F − 1 is
L 2 -Lipschitz, F − 1 ( B R/L 2 , 0 ) ⊂
B R and thus
[Equation]
By construction
[Equation]
when z follows the standard Gaussian distribution in R d .
This quantity can be computed using the cumulative distribution function of the chi-square distribution, i.e.
[Equation]
where γ is the lower incomplete gamma function given by
[Equation]
A Proof of Corollary 3.3.
Since M 1 and M 2 are separated by a distance D the ball centered on F − 1 (0) has a radius at least as big as D that we might call B D to simplify the notation.
Therefore:
[Equation]
And since P ∗ ( B D [Equation] =
[Equation]
B. The link with the Precision and Recall for generative models B.1. Deﬁnitions of the precision and the recall
The precision and the recall are deﬁned as such :
Deﬁnition B.1.
For α, β ∈ [0 , 1] , the distributions (cid:98) P is said to have a precision α at recall β with respect to P ∗ if there exist the distributions ν, ˆ ν, ν ∗ , such that (cid:98) P and P ∗ can be decomposed as such :
[Equation]
and
[Equation]
The distribution ν deﬁned on Supp( (cid:98) P ) ∪ Supp( P ∗ ) while
Supp(ˆ ν ) =
Supp( (cid:98) P ) and Supp( P ∗ ) =
Supp( ν ∗ )
It can be interpreted as such in (Sajjadi et al., 2018):
ν represent the part of P ∗ that (cid:98) P correctly models, ˆ ν is simultaneously the part of P ∗ that (cid:98) P misses on their joint support and all the points that should not be represented by (cid:98) P .
Finally, ν ∗ cover the points of P ∗ that (cid:98) P could not model and the difference between ν and ˆ ν on their joint support.
Among all the potential decompositions, i.e. the pairs ( α, β ) , the focus is set on the maximum precision ¯ α and the maximum recall ¯ β .
Proposition B.1.
The maximum precision and the maximum recall satisfy :
[Equation]
and [Equation] An improved version of the precision and the recall are deﬁned in (Kynk ¨ a ¨ anniemi et al., 2019).
B.2. The link between the maximum precision, the maximum recall and the TV distance
The link between the TV and the maximum precision ¯ α and the maximum recall ¯ β is :
[Equation]
B.3. Corollary 3.3.1 given in terms of maximum precision
Corollary B.0.1 ( F − 1 L 2 -Lipschitz) .
If F − 1 is L 2Lipschitz, therefore we have an upper bound on the maximum precision based on the distance D between M 1 and
[Equation]
This result is to be compared with the actual upper bound on the maximum precision:
[Equation]
Our result may depend on the dimension but an upper bound of the precision can be directly computed
[Equation]