Generalization of the Change of Variables Formula with Applications to Residual Flows
Abstract
Normalizing ﬂows leverage the Change of Variables Formula (CVF) to deﬁne ﬂexible density models.
Yet, the requirement of smooth transformations (diffeomorphisms) in the CVF poses a signiﬁcant challenge in the construction of these models.
To enlarge the design space of ﬂows, we introduce L -diffeomorphisms as generalized transformations which may violate these requirements on zero Lebesgue-measure sets.
This relaxation allows e.g. the use of non-smooth activation functions such as ReLU.
Finally, we apply the obtained results to planar, radial, and contractive residual ﬂows.
1. Introduction
The term normalizing ﬂow refers to a concatenation of arbitrarily many simple transformations such that together they describe a transformation of desired ﬂexibility and expressiveness.
Formally, a transformation f : Z → X denotes a diffeomorphism , i.e., a bijective mapping where both f and f - 1 are continuously differentiable.
The crucial reason why transformations are considered in normalizing ﬂows is the validity of the Change of Variables Formula (CVF) for a probability density p Z on Z described by (cid:90)
[Equation]
where p f ( x ) :
= p Z ( f - 1 ( x )) | det J f - 1 ( x ) | and A ⊆ X .
This formula provides an explicit expression of the density p f induced by f on the target space X , which includes the determinant of the Jacobian as a volume correction term.
Based on this deﬁnition of a transformation, however, it is generally not accurate to use non-smooth activations, workshop on Invertible Networks, Normalizing Flows and Models (ICML 2021) such as ReLU, Leaky ReLU, or ELU with α (cid:54) = 1 , in the design of normalizing ﬂows.
These usually cause the ﬂow to become non-differentiable on a set with no volume w.r.t.
the Lebesgue measure λ n , hence no diffeomorphism.
In measure theory, these sets are called λ n -null sets or only null sets for short and are negligible in integration.
We demonstrate that the requirements for a ﬂow can be signiﬁcantly weakened by excluding null sets from the base and target space while preserving the validity of the CVF.
There are remarks on using almost everywhere (a.e.) differentiable activation functions in Kobyzev et al.
(2020) or
Kong & Chaudhuri (2020), yet both works lack a proof of the validity of such transformations to deﬁne ﬂows.
In our work, we provide such proofs for an even more general statement.
At the same time, we discuss the probabilistic background of normalizing ﬂows to induce a well-deﬁned density in the end.
Furthermore, we point out that not every generalization of the CVF found in the mathematical literature is immediately suitable for ﬂows.
Finally, we put a special emphasis on the applications to residual ﬂows.
In doing so, we prove that non-smooth activations are also valid for both planar and radial ﬂows (Rezende & Mohamed, 2015), as well as for contractive residual ﬂows (Behrmann et al., 2019).
2. Background on CVF in Probability Theory
The basic idea behind normalizing ﬂows is to transform a known and tractable probability space into a more complex one.
Mathematically, a probability space ( Z, A Z , P Z ) is composed of a set Z equipped with a σ -algebra and a probability measure P Z .
In the target space ( X, A X , P data ) , only the set and σ -algebra are ﬁxed, and the data distribution P data is unknown.
For simplicity, we only consider open subsets of R n and trace σ -algebras of the Lebesgue algebra
L in the following, i.e., A Z =
L ( Z ) and A X
=
L ( X ) .
The trace σ -Algebra is a restricted σ -Algebra on a subset deﬁned by L ( Z ) :
= { A ∩ Z | A ∈ L} .
Besides, we assume that the distribution P Z is absolutely continuous w.r.t.
the n -dimensional Lebesgue measure λ n , i.e., λ n -null sets have a P Z -probability of zero.
Therefore, the existence of a probability density p Z follows by Radon-Nikodym’s theorem (Bogachev, 2006, Sec. 3.2).
For more information on measure theory, see Bogachev (2006) or Elstrodt (2013).
At the lowest level, a transformation f : Z → X has to be at least an A Z - A X -measurable mapping (i.e., a random variable) in order to induce a distribution on the target space by the so-called pushforward measure :
[Equation]
Under the assumption that the base distribution P Z has a density function p Z , there also exists an integral representation for the pushforward measure, i.e.,
[Equation]
Assumption 1 (Transformations for the CVF) .
The function f : Z → X between two open sets Z,
X ⊆ R n is a diffeomorphism; or equivalently expressed by the inverse function theorem, f is bijective, continuously differentiable and without critical points.
We note that z ∈ Z is a critical point if the Jacobian-determinant vanishes in this point, i.e., det J f ( z ) = 0 .
In particular, a critical point z indicates that the inverse is nondifferentiable or non-continuously differentiable in f ( z ) .
If the mapping f satisﬁes Assumption 1, the CVF from eq.
(1) holds, and we can extend the expression (2) of the distribution P f by
[Equation]
Since the equality (3) is valid for all A X -measurable sets, the λ n -unique probability density of P f is given by
[Equation]
In order to unify the existing deﬁnitions of ﬂows in the literature, we will speak of a ﬂow or a proper ﬂow when a density on X of the form as in eq.
(4) is induced.
3. Generalization of the CVF
If it is argued that a ﬂow indeed induces a probability density, often the CVF is merely named, and only in rare cases reference is made to sources like Rudin (1987) or Bogachev (2006).
In most of the mathematical literature, it is proved in the following form
[Equation]
where f : U → R n is injective, continuously differentiable with U ⊆ R n open, Z ⊂ U measurable, and ψ Lebesgue integrable.
Moreover, there are even broader formulations of this statement where f is only differentiable or even only Lipschitz continuous everywhere and injective almost everywhere (cf.
Varberg, 1971).
In Bogachev (2006, Thm.
5.8.30) and Hajłasz (1993)
, generalizations are discussed where injectivity is not even required by considering the cardinality of the preimage set.
The identity (5) is, however, rather analytically motivated for solving integrals and does not aim to provide a representation of the density induced by f .
In the common case where f satisﬁes Assumption 1, these two variants (eq. (1) and (5)) are valid, since both f and f - 1 form diffeomorphisms.
Nevertheless, when we consider generalizations, it is usually no longer clear how and whether they can also be applied for the purposes of normalizing ﬂows.
In the following sections, we derive a similar strong generalization of the CVF as in (5), which is more suited to transforming probability densities, i.e., more suited for normalizing ﬂows.
L - Diffeomorphism
The basic idea is to require the conditions for a diffeomorphism only almost everywhere, since these sets do not affect integration.
This idea leads to the following deﬁnition of a generalized transformation, providing a weaker set of conditions than in Assumption 1.
Deﬁnition 2 (Lebesgue-Diffeomorphism) .
A mapping f : Z → X between two open sets Z,
X ⊆ R n is called
Lebesgue-diffeomorphism ( L -diffeomorphism for short), if there are λ n -null sets N Z , N X with N Z closed such that the restriction f : Z \ N Z → X \ N X is bijective, continuously differentiable, and the set of critical points is a null set.
Examples.
In the following, we list a few L -diffeomorphism and their corresponding null sets:
A diffeomorphism f : Z → X forms an L -diffeomorphism where both N Z and N X are empty sets.
The cubic function f : R → R with f ( x ) = x 3 is an L -diffeomorphism with N Z = N X = ∅ and one critical point { 0 } , which is a null set.
In particular, the plane polar coordinates transformation f : R + ×
[0 , 2 π ] → R 2 given by f ( r, φ ) =
( r cos( φ ) , r sin( φ )) is an L -diffeomorphism with N Z = { 0 } ×
[0 , 2 π ] ∪ R > 0 × { 0 , 2 π } and N X
= R + × { 0 } .
In appendix
Lemma
A. we show that a
L -diffeomorphism is a measurable mapping with respect to the corresponding trace σ -algebras of the Lebesgue algebra, thus inducing a distribution on X by the pushforward measure.
Moreover, the name ’diffeomorphism’ does justice to the Deﬁnition 2, which we state in the following lemma (see Appendix A for the proof):
Lemma 3.
Let f : Z → X be an L -diffeomorphism.
Then there are λ n -null sets N Z , N X with N Z closed such that the restriction f : Z \ N Z → X \ N X is a diffeomorphism.
The reasoning behind the assumptions of an Ldiffeomorphism can be understood as follows: We can remove negligible sets from the domain and the target space such that the restriction is bijective and continuously differentiable.
To show that the restricted inverse is continuously differentiable, we use the inverse function theorem (Rudin, 1976, Thm. 9.24).
Nevertheless, for the inverse function theorem to apply, all critical points and their image must still be removable, i.e., measure-theoretically negligible.
That the set of critical points is a null set was assumed, and for the image, we use Sard’s theorem (see Appendix A for the detailed proof).
Furthermore, it is necessary to suppose that the set of critical points is a null set; because there are examples of continuously differentiable and bijective functions whose set of critical points does not have measure zero.
These points cause the inverse function to be non-continuously differentiable on a set with a positive measure (see Appendix A. for an example).
3.2. CVF for L -Diffeomorphism
The previously deﬁned L -diffeomorphisms form a reasonable generalization of diffeomorphisms and are comparable to those transformations discussed in the introduction of this section.
Furthermore, the following theorem justiﬁes the validity of CVF as well for L -diffeomorphisms (see Appendix
A for the proof):
Theorem 4 (CVF for L -Diffeomorphism) .
Let f : Z → X be an L -diffeomorphism and P Z a distribution on Z with probability density p Z w.r.t.
λ n .
Then the CVF (see eq.
(3) ) holds for f .
In particular f induces a distribution on X with density p f given by
[Equation]
This theorem legitimizes the use of functions as proper ﬂows that are not everywhere bijective, continuous, differentiable, or continuously differentiable.
Even more, the inverse does not have to fulﬁll these properties everywhere either.
In short, we can apply L -diffeomorphisms as ﬂows.
3.3. Invariance under Composition
The strength and tremendous upswing of normalizing ﬂows mainly occurred because simple ﬂows can be chained together.
Thus, we can achieve the desired degree of complexity and expressiveness by increasing the number of simple ﬂows.
For this purpose, ﬂows are often considered on the same base and target space.
This crucial property is also retained for L -diffeomorphisms (see Appendix A for the proof):
Lemma 5 (Composition) .
Let Ω ⊆ R n be an open set and f 1 , f 2 : Ω → Ω L -diffeomorphisms.
Then the composition f 2 ◦ f 1 is also an L -diffeomorphism on Ω .
From this lemma, it results inductively that the concatenation f = f K ◦ . . .
◦
f 1 of K L -diffeomorphisms f 1 , . . .
f K on Ω forms an L -diffeomorphism.
Hence, common formulas from the normalizing ﬂow literature, e.g., presented in
Kobyzev et al.
(2020) or Papamakarios et al.
(2021), also apply to L -diffeomorphisms or can be extended to them.
For example, the following holds for the Jacobian-determinant of f with x i := f - 1 i +1 ◦ . . .
◦ f - 1 K ( x ) and x K := x
det J f - 1 ( x ) =
K (cid:89) i = det J f - 1 i ( x i ) ( a.e. x ∈ Ω) .
Despite the legitimate use of L -diffeomorphisms mathematically, it is essential to note that these can lead to numerical instabilities.
In some points, the Jacobian-determinant or the inverse function does not need to exist.
Nevertheless, these values can be set meaningfully or ignored in some situations.
4. Non-smooth Activations in Residual Flows
In this section, we apply the previous results to residual mappings, which are perturbations of the identity of the form f ( x ) =
x + g ( x ) .
This justiﬁes the use of non-smooth activations in planar, radial, and contractive residual ﬂows.
4.1. Planar Flows
The term planar ﬂow was ﬁrst introduced by Rezende &
Mohamed (2015), which refers to functions of the form
[Equation]
with non-linearity h and w, u ∈ R n , b ∈ R .
They describe a plane-wise expansion or contraction of all hyperplanes orthogonal to w .
In order to admit also non-smooth activations, we generalize the results from Rezende & Mohamed (2015) in the following theorem and obtain a sufﬁcient criterion for the bijectivity of a planar ﬂow f P (see Appendix
B.1 for the proof).
Theorem 6.
Let f P be a planar ﬂow with activation h .
If the one-dimensional mapping ψ : R → R with ψ ( λ ) =
λ + w T uh ( λ ) is bijective, then f P is also bijective.
However, bijectivity is not sufﬁcient for the planar ﬂow to induce a density of the desired form.
But similar to Theorem
[Table] LU [Table] 6, conditions can be imposed on a one-dimensional mapping such that f P describes an L -diffeomorphism (see Appendix
B.1 for the proof):
Theorem 7.
Let f P be a bijective planar ﬂow.
If there is a countable and closed set N ⊂ R such that the activation function h is continuously differentiable on R \ N and
[Equation]
is countable, then f P is an L -diffeomorphism.
In particular, f P is a proper ﬂow.
Using Theorem 7, conditions for different activations such that the resulting planar ﬂow describes a ﬂow can be found via reducing it to a one-dimensional problem.
The constraints for the most popular non-linearities are summarized in Table 1.
4.2. Radial Flows
Another intuitive way to perturb the identity in R n is to expand or contract spherically around a centering point.
This type of transformation was initially studied by Tabak & Turner (2013) and subsequently by Rezende & Mohamed (2015).
These transformations of the form
[Equation]
are called radial ﬂows , where h : R + → R + is a localization function, x 0 ∈ R n the center, and β ∈ R .
When β is negative, a contraction occurs, and positive values lead to an expansion around the center x 0 .
The following theorem provides a sufﬁcient criterion for the bijectivity of a radial ﬂow if we consider non-smooth functions h (see Appendix
B.2 for the proof):
Theorem 8.
Let f R be a radial ﬂow with localization h .
If the one-dimensional mapping ψ : R + → R + with ψ ( r ) :
= r + βh ( r ) r is bijective, then f R is also bijective.
Again, bijectivity is sufﬁcient only for the existence of the inverse which means that f R , in general, does neither describe an L -diffeomorphism nor a ﬂow.
However, this property is ensured by the conditions of the following theorem (see
Appendix B.2 for the proof):
Theorem 9.
Let f R be a bijective radial ﬂow.
If there is a countable, closed set N ⊂ R > 0 such that the localization function h is continuously differentiable on R > 0 \ N and
C := { r ∈ R > 0 \ N | 1 + β ( h ( r ) + rh (cid:48) ( r ) = 0) } is countable, then f R is an L -diffeomorphism.
In particular, f R is a proper ﬂow.
4.3. Contractive Residual Flows
Contractive mappings provide a more general type of perturbations of the identity.
A function g is called contractive if there exists a constant L < 1 such that for all x, y ∈ R n it holds for any norm on the vector space R n [Equation]
In Behrmann et al.
and
Chen et al.
(2019), these kinds of residual ﬂows are called (contractive) residual ﬂows and are denoted henceforth by f C .
On the one hand, this strong condition on g gives the bijectivity of f C by the Banach’s ﬁxed point theorem; on the other hand, it follows that f C has no critical points (Behrmann, 2019, Lem. 5.4).
Thus, only a few assumptions are required to guarantee that a residual ﬂow forms an L -diffeomorphism.
Theorem 10.
Let f C be a residual ﬂow with contractive perturbation g .
If there is a closed λ n -null set N ⊂ R n such that g is continuously differentiable on R n \ N , then f C is an L -diffeomorphism.
In particular, f C is a proper ﬂow.
The proof of this statement follows directly from the property that Lipschitz continuous functions map λ n -null sets to sets of λ n -measure zero (Rudin, 1987, Lem. 7.25) and the fact that Lip ( f C ) = 1 + L .
5. Conclusion
In this paper, we have shown that the conditions on a ﬂow need not be strictly satisﬁed everywhere, and a certain degree of freedom is permitted on measure-theoretically negligible sets.
With this, we have justiﬁed using L -diffeomorphism instead of a normal diffeomorphism as ﬂows.
This gain signiﬁcantly increases the possibilities in the design of normalizing ﬂows and, in particular, allows the usage of non-smooth activations.
Nevertheless, we have only justiﬁed their existence and usage mathematically, far from leading to a successful practical application.
Thus, future work should investigate whether and in which situations non-smooth activations provide an actual gain.
Moreover, we only applied these generalizations to simple residual ﬂows.
For this reason, it remains open to what extent other ﬂows such as more general planar ﬂows, like Sylvester ﬂows (Van Den Berg et al., 2018), or autoregressive ﬂows (Huang et al., 2018; Jaini et al., 2019) also beneﬁt from this. Acknowledgements
Niklas Koenen and Marvin N. Wright gratefully acknowledge the funding from the German Research Foundation (DFG) in the context of the Emmy Noether Grant
A. Generalization of the Change of Variables Formula
Lemma A.1.
Let f : Z → X be an L -diffeomorphism from the measurable space ( Z, A Z ) into the target space ( X, A X ) .
Then f is an A Z - A X -measurable mapping, i.e., a random variable.
Proof.
According to the deﬁnition of measurable functions, we have to show that every preimage of a measurable set contained in A X is also an element of A Z .
Let B ∈
A X .
Since f is an L -diffeomorphism, there exist λ n -null sets N Z , N X such that the restriction f : Z \ N Z → X \ N X is bijective and continuously differentiable.
We can decompose the preimage of B as
[Equation]
Since N Z is a null set, the subset N Z ∩ f - 1 ( B ) also has a measure of zero.
We can represent the other set of the equation above by the bijectivity of f on Z \ N Z as
[Equation]
Thus it results from the continuity of the restricted mapping, which is in particular L ( Z \ N Z ) - L ( X \ N X ) -measurable, that the set (7) is contained in the trace σ -algebra L ( Z \ N Z ) , hence an element of A Z .
In the end, we can represent f - 1 ( B ) as a union of a null set and a A Z -measurable set, which shows the claim.
Proof of Lemma 3 .
From the Deﬁnition 2 of an L -diffeomorphism, we obtain the existence of null sets N Z , N X with N Z closed such that the restricted function f : Z \ N Z → X \ N X is bijective, continuously differentiable, and the set of critical points C has measure zero.
At ﬁrst, we show that C is a closed set.
Consider the mapping g : Z \ N Z
→ R with g ( x ) = det( J f ( x )) .
Because of the continuity of the determinant and the continuous differentiability of the restricted f , the mapping g is also continuous.
Therefore the preimage under g of closed sets is also closed according to the topological deﬁnition of continuity, i.e.,
[Equation]
is a closed set; thus, Z \ ( N Z ∪ C ) is open in R n .
It follows by the inverse function theorem and its consequences (Rudin, 1976, Thm. 9.24 and 9.25) that the restriction
[Equation]
forms a diffeomorphism.
By assumption, N C and C are λ n -null sets, and so N Z ∪ C .
From Sard’s theorem (Milnor, 1965, §2), it follows that f ( C ) , hence especially N X ∪ f ( C ) , is a set of measure zero.
Proof of Theorem 4 .
We obtain from Lemma 3 the existence of two null sets N Z , N X so that the restriction f : Z \ N Z → X \ N X of an L -diffeomorphism forms a diffeomorphism.
Let A ∈
A X be a measurable set in the target space X .
Because of the additivity of P Z and the deﬁnition of the pushforwad measure P f , it follows
[Equation]
Since P Z is absolutely continuous w.r.t.
the Lebesgue measure and f - 1 ( B ) ∩ N Z is a subset of a λ n -null set, the right summand of term (8) vanishes.
For the other term, we note the following set equality:
[Equation]
Because the restriction f : Z \ N Z → X \ N X is a diffeomorphism and B \ N X is an element of the trace σ -algebra
L ( X \ N X ) , the equation (8) can be extended by the CVF from eq.
(1) as
[Equation]
We deﬁne a function p f : X → R + with
[Equation]
Since B ∩ N X ⊆ N X is a λ n -null set, we can additionally integrate in (10) over this set and obtain the expression of the distribution induced by f
[Equation]
hence p f forms a density of distribution P f .
Finally, by the Radon-Nikodym theorem (Bogachev, 2006, Sec. 3.2), the density function is λ n -unique, which shows the claim.
Proof of Lemma 5 .
Since f 1 and f 2 are L -diffeomorphisms, by Lemma 3 there are λ n -null sets N 1 Z , N 2 Z , N 1 X , N 2 X ⊂ Ω with N 1 Z and N 2 Z closed
such that both
[Equation]
and
[Equation]
[Equation] are diffeomorphisms.
We remove from the domain of f 1 all elements mapping to the set
N 2 Z since f 2 is not a diffeomorphism on it.
Hence we deﬁne
[Equation]
and
[Equation]
[Equation]
It is relatively easy to see that both f 1 (Ω \ N Z ) =
Ω \ ( N 1 X ∪ N 2 Z ) and f 2 ( f 1 (Ω \ N Z ))
=
Ω \ N X are valid.
Moreover, from the continuity of f 1 and since N 2 Z ∩ Ω \ N 1 X is closed in the subspace topology on Ω \ N 1 X , it follows that f - 1 ( N 2 Z ∩ Ω \ N 1 X ) is closed in Ω \ N 1 Z .
Hence there exists a closed set
A ⊂ Ω such that (see Sec.
6 in Willard, 1970 for more information)
[Equation]
This results with eq.
(13) in
[Equation]
which is a closed set in Ω .
In summary, the mapping f 2 ◦ f 1 : Ω \ N Z
→
Ω \ N X is well-deﬁned, continuously differentiable, bijective, and has no critical points, since we have merely further restricted the diffeomorphisms from (12).
In addition, N Z is a closed set in Ω .
To complete the proof, we still need to show that N Z and N X have a λ n -measure of zero.
However, this follows directly from the fact that diffeomorphisms map null sets to null sets.
The following example illustrates why we assume that the set of critical points is a null set.
Indeed, one can construct bijective and continuously differentiable functions whose inverse is not differentiable on a set with positive measure.
Example A.2.
As an example, consider the following recursively deﬁned set on the interval [0 , 1] called the Smith-VolterraCantor set C SV .
The deﬁnition and properties of this set are only brieﬂy sketched here.
For more information and detailed proofs, the reader is referred to Bressoud (2008) or DiMartino & Urbina (2014).
Another example can also be found in
Floret (1981, Sec. 17.15).
The recursive deﬁnition of C SV on [0 , 1] starts by removing the open middle quarter from the interval.
Then we create subsequent sets S n by removing an open interval of length 14 n from the center of each interval in S n - 1 , i.e., S 1 =
[Figure]
0 , 38 (cid:105) ∪ (cid:104)
[Figure] S 2 = (cid:104) 0 , 532 (cid:105) ∪ (cid:104) [Figure] 38 (cid:105) ∪ (cid:104) 58 , 2532 (cid:105) ∪ (cid:104)
[Figure]
(cid:105)
S 3 = (cid:104) 0 , 116 (cid:105) ∪ (cid:104) 332 , 532 (cid:105) ∪ [Figure] ∪ [Figure]
[Figure]
Now the Smith-Volterra-Cantor set is deﬁned as the following intersection of all S n
[Equation]
which is closed because of the countable intersection of closed sets.
In addition, in each recursion step in each of the 2 nintervals, the middle pieces with the Lebesgue measure of 14 n are removed.
Hence it holds
[Equation]
due to the geometric series.
In order to construct a counterexample, we consider the function d : [0 , 1] → R with d ( x ) =
inf c ∈ C SV | x − c | .
This function is obviously continuous and d ( x ) = 0 holds if and only if x ∈ C SV , otherwise it only takes positive values.
Because of the continuity, this function is integrable and it follows from the fundamental theorem of calculus that
[Equation]
is continuously differentiable.
Moreover, f is injective resulting from the construction of C SV and the fact that d is not constant zero on any interval with positive measure.
But the set of critical points of f is the Smith-Volterra-Cantor set, which is not a λ -null set.
B. Non-smooth Activations in Residual Flows B.1. Planar Flows
Theorem 6 presented here corresponds to a generalization of the proof given by Rezende & Mohamed (2015) since not only the smooth activation function h ( x ) =
tanh( x ) is considered.
However, the argument in the proof is very similar.
Proof of Theorem 6 .
Let y ∈ R n be arbitrary.
Then we have to show that the following equation has a unique solution
[Equation]
If w = 0 holds, then a unique solution is given by x = y − uh ( b ) , wherefrom bijectivity results.
For this reason, let w (cid:54) =
in the following, thus w spans a one-dimensional linear subspace W in R n .
Consequently, each element x ∈ R n has a unique orthogonal decomposition x = x (cid:107) + x ⊥ with x (cid:107) ∈ W and x ⊥ ∈ W ⊥ :
= { x ∈ R n | w T x = 0 } .
Due to the orthogonality of x ⊥ and w , the following solution of the orthogonal component depending on the parallel one can be inferred from eq.
[Equation]
Since W is a one-dimensional linear subspace, there is a unique λ ∈ R with x (cid:107) =
w λw T w .
By using this representation, the original equation multiplied by w T from the left yields
[Equation]
where the last equation in (16) follows again by w T x ⊥ = 0 .
The assumed bijectivity of ψ leads to the unique existence of a λ y ∈ R , which solves the equation above.
In addition, this implies the existences of x (cid:107) and x ⊥ , thus
[Equation]
is the unique solution of the initial equation (14) and consequently the bijectivity of f P follows.
Proof of Theorem 7 .
In case w = 0 , f P represents an afﬁne linear mapping describing a diffeomorphism, hence an
L -diffeomorphism.
For this reason, let w (cid:54) = 0 .
Consequently, w spans a one-dimensional linear subspace W , from which follows a unique orthogonal decomposition of the vector space R n = W + W ⊥ as a direct sum.
This leads to a characterization of the vector space as a disjoint union of hyperplanes, i.e.,
[Equation]
Consider the function τ : R n → R with τ ( x ) =
w T x + b mapping every element of a hyperplane H λ to the same value λ + b .
Since the activation function h is not continuously differentiable on the countable set N , we remove all hyperplanes mapping to N under τ .
So we deﬁne
[Equation]
such that
[Equation]
Because τ is continuous and N is closed in R , it follows that H N = τ - 1 ( N ) is closed in R n .
Moreover, H N as a countable union of hyperplanes is also a null set w.r.t.
the Lebesgue measure λ n due to the subadditivity.
Accordingly, the restriction f P : R n \ H N → R n \ f P
H N ) is bijective and continuously differentiable.
Furthermore, H N is a closed λ n -null set.
We note that the image of a hyperplane H λ under f P is also a hyperplane; hence f P ( H N ) forms a countable union of hyperplanes which is a null set.
Finally, it is left to show that the set of critical points of the restricted planar ﬂow is also a null set.
By the matrix determinant lemma we get for the set of critical points the set equality
[Equation]
This gives τ ( C f P ) =
C , which we have assumed to be a countable set.
Hence
[Equation]
which is as a countable union of hyperplanes a λ n -null set.
B.1.1. E XAMPLES FOR SOME A CTIVATIONS
In the following, we infer conditions for particular choices of activation functions for a planar ﬂow f P .
A visualization of the crucial function for each proof can be found in Figure 2, where the conditions on the ﬂow are fulﬁlled, just (not) fulﬁlled and not fulﬁlled anymore.
Lemma B.1 (ReLU) .
If w T u > − 1 holds, then f P with activation ReLU is an L -diffeomorphism.
Proof.
Consider the mapping ψ : R
→ R given by
[Equation]
Because of the inequality w T u > − 1 , this mapping is strictly monotonically increasing and thus obviously bijective.
Consequently, the bijectivity of the planar ﬂow f P follows from Theorem 6.
Furthermore, the ReLU activation is continuously differentiable on R \ { 0 } and it holds for all λ ∈ R \ { 0 }
[Equation]
Therefore, the planar ﬂow f P with activation ReLU has no critical points; thus, the claim follows from Theorem 7.
Lemma B.2 ( tanh ) .
If w T u ≥ − 1 holds, then f P with activation tanh is an L -diffeomorphism.
Proof.
Consider the function ψ : R → R given by ψ ( λ )
= λ + w T u tanh( λ ) with derivative ψ (cid:48) ( λ ) = 1 + w T u cosh( λ ) 2 .
Since the hyperbolic cosine is 1 only at 0 and otherwise always greater than 1 , we get with the assumed inequality ψ (cid:48)
( λ ) ≥and equality only if λ = 0 and w T u = − 1 .
Therefore, the function ψ is strictly monotonically increasing, hence injective.
Moreover, the surjectivity follows from the boundedness of the hyperbolic tangent.
Consequently, the bijectivity of the planar ﬂow f P follows from Theorem 6.
The activation function is continuously differentiable, and as seen earlier, the equation 1+ w T u tanh( λ ) = 0 is only satisﬁed if w T u
= − 1 and λ = 0 .
In any case, the set of critical points is a countable set, so the claim follows from Theorem 7.
Lemma B.3 (ELU) .
If w T u > max( − 1 , − 1 α ) holds, then f P with activation ELU ( α > 0) is an L -diffeomorphism.
Proof.
Consider the function ψ : R → R with
[Equation]
This function is continuously differentiable on R \ { b } with derivative given by
[Equation]
By the assumed inequality and e λ + b ∈ (0 , 1) for λ < − b , we obtain for λ < − b
[Equation]
In addition, we get the positivity for the other case λ > − b
[Equation]
Because of the limit lim λ (cid:38)− b ψ ( λ ) = ψ ( − b ) = − b , the function ψ is strictly monotonically increasing, thus injrective.
Furthermore, surjectivity results from the mean value theorem; hence f P is bijective by Theorem 6.
Additionally, the activation function ELU is continuously differentiable on R \ { b } , and similar to (17) and (18) the inequality ψ (cid:48) ( λ )
= 1 + w T u ELU ( λ ) > 0 follows for all λ (cid:54) = 0 .
Therefore we can apply Theorem 7, which shows the claim.
Lemma B.4 (Softplus) .
If w T u > − 1 holds, then f P with activation Softplus is an L -diffeomorphism.
Proof.
For the function ψ : R → R given by ψ ( λ ) = λ + w T u
Softplus ( λ + b ) with Softplus ( λ ) =
log (cid:0) 1 + e λ + b (cid:1) the derivative is
[Equation]
Since the range of the factor 1 1+ e - ( λ + b ) is in the interval (0 , 1) , the assumed inequality results for all λ ∈ R in
[Equation]
Consequently, the function ψ is strictly monotonically increasing, hence injective.
By continuity, the limit lim λ →−∞ =
−∞ results.
Without loss of generality, we consider λ > − b in the following.
In this case, the inequality log(1 + e λ + b ) ≤ log(2 e λ + b ) holds, and we obtain for w T u <
[Equation]
For the other case w T u ≥ 0 , results lim λ →∞ ψ ( λ ) ≥ lim λ →∞ λ = ∞ .
Thus, the surjectivity follows from the mean value theorem.
Therefore, we get the bijectivity from Theorem 6.
In addition, the activation function Softplus is everywhere continuously differentiable and has not critical points, so the claim follows from Theorem 7.
B.2. Radial Flows
Theorem 8 presented here corresponds to a generalization of the proof given by Rezende & Mohamed (2015), since not only the localization function h ( r ) =
α + r is considered.
However, the arguments in the proof are very similar.
−−−   −−−   λ ψ ( λ ) w T u =w T u = −w T u = −(a) ReLU activation −−−   −−−   λ ψ ( λ ) w T u =w T u
=
−w T u = −(b) tanh activation −−−   −−−   λ ψ ( λ ) w T u =w T u = − 0 .w
T u = − 0 .(c) ELU activation with α =−−−
−−−   λ ψ ( λ ) w T u = 0 .w T u = −w T u = −(d)
[Figure]
activation
Proof of Theorem 8 .
Let y ∈ R n arbitrary.
We have to show for bijectivity that the following equation has a unique solution
[Equation]
where
[Equation]
If y = x 0 , we then obtain the equality
[Equation]
) (cid:107)
[Equation]
r +
[Equation] r ) [Equation] r ) after rearranging equation (20) and taking the Euclidean norm.
Since ψ (0) = 0 and ψ was assumed to be bijective, it follows that (cid:107) x − x 0 (cid:107) = 0 which is equivalent to x = x 0 because of the deﬁniteness of the Euclidean distance.
Hereafter let y ∈ R n \ { x 0 } , i.e., r = (cid:107) x − x 0 (cid:107) 2 > 0 .
In this case, each element x − x 0 with x ∈ R n \ { x 0 } can be expressed unambiguously as the product of its projection on the unit sphere and its Euclidean distance.
Thus, for x ∈ R n \ { x 0 } there exists a unique normalized direction vector ˆ x ∈ S 1 ( x 0 ) :
= { x ∈ R n | (cid:107) x − x 0 (cid:107) 2 = 1 }
such that x = x 0 + r ˆ x where r := (cid:107) x − x 0 (cid:107) 2 .
Substituting this representation into equation (20), one obtains after conversion
[Equation]
and by taking of the norm of this
[Equation]
Because of the bijectivity of ψ , the unique existence of a radius r y > 0 around the centering point results.
Due to the fact that ψ ( r ) > 0 remains valid for r > 0 , equation (21) gives the following unambiguous expression of the direction vector ˆ x
[Equation]
Hence x = x 0 + r y ˆ x y is the unique solution of the original equation (20), concluding ﬁnally that f R is bijective.
Proof of Theorem 9 .
According to the requirement, the localization function is not continuously differentiable for all radii, so spheres with such distances around the centering point must be removed from the domain of f R .
Furthermore, the point x 0 corresponding to a radius of 0 must be eliminated in order to restrict the localization function to an open set, thus allowing us to verily speak of differentiability.
For this purpose we deﬁne
[Equation]
where
[Equation]
Since the shifted Euclidean norm τ ( x ) := (cid:107) x − x 0 (cid:107) 2 is continuous, it follows that the preimage of the closed set N ∪ { 0 } is also closed.
Therefore, the set of eliminated points
[Equation]
is closed; moreover, it is a set of measure zero because a countable union of spheres and points is a λ n -null set.
In summary, the restriction f R : R n \ S
→ R n \ f R ( S ) describes a bijective and continuously differentiable mapping.
Besides, the following is valid for every x ∈ S r ( x 0 )
[Equation]
This indicates that f R maps spheres with radius r to spheres with radius r + βh ( r ) r around the centering point x 0 ; thus, f R ( S ) is also a countable union of null sets, therefore, itself a null set.
Finally, it remains to show that the set of critical points of this restriction is also a Lebesgue null set.
From the matrix determinant lemma and the higher dimensional differentiation rules, the Jacobian-determinant of f R at position x ∈ R n \ S is given by
[Equation]
Since f R ( x 0 ) = x 0 holds and f R is bijective, both (cid:107) f
R ( x ) − x 0 (cid:107) 2 > 0 and r = (cid:107) x − x 0 (cid:107) 2 > 0 hold for all x ∈ R n \ S .
Thus equation (22) yields
[Equation]
Knowing that this term does not vanish for any x ∈ R n \ S , the set of critical points can be represented using equation (23) like
[Equation]
This gives τ ( C f R ) =
C , which we have assumed to be a countable set.
Hence
[Equation]