U NIVERSAL J OINT A PPROXIMATION OF M ANIFOLDS AND D ENSITIES BY S IMPLE I NJECTIVE F LOWS
Anonymous authors
Paper under double-blind review
A BSTRACT
We analyze neural networks composed of bijective ﬂows and injective expansive elements.
We ﬁnd that such networks universally approximate a large class of manifolds simultaneously with densities supported on them.
Among others, our results apply to the well-known coupling and autoregressive ﬂows.
We build on the work of Teshima et al.
2020 on bijective ﬂows and study injective architectures proposed in Brehmer et al.
2020 and Kothari et al.
Our results leverage a new theoretical device called the embedding gap, which measures how far one continuous manifold is from embedding another.
We relate the embedding gap to a relaxation of universally we call the manifold embedding property, capturing the geometric part of universality.
Our proof also establishes that optimality of a network can be established “in reverse,” resolving a conjecture made in Brehmer et al.
2020 and opening the door for simple layer-wise training schemes.
Finally, we show that the studied networks admit an exact layer-wise projection result,
Bayesian uncertainty quantiﬁcation, and black-box recovery of network weights.
I NTRODUCTION
In the past several years, invertible ﬂow networks emerged as powerful deep learning models to learn maps between distributions (Durkan et al., 2019a; Grathwohl et al., 2018; Huang et al., 2018;
Jaini et al., 2019; Kingma et al., 2016; Kingma & Dhariwal, 2018;
Kobyzev et al., 2020; Kruse et al., 2019; Papamakarios et al., 2019).
They generate excellent samples (Kingma & Dhariwal, 2018) and facilitate solving scientiﬁc inference problems (Brehmer & Cranmer, 2020; Kruse et al., 2021).
By design, invertible ﬂows are bijective and, hence, may not be a natural choice when the target distribution has low-dimensional support.
This problem can be overcome by combining bijective ﬂows with expansive, injective layers, which map to higher dimensions (Brehmer & Cranmer, 2020;
Cunningham et al., 2020; Kothari et al., 2021).
Despite their empirical success, the theoretical aspects of such globally injective architectures are not well understood.
In this work, we present approximation-theoretic properties of injective ﬂows whose architecture combines bijective ﬂows and expansive, injective layers, with an emphasis on approximating measures with low-dimensional support.
We state conditions under which these networks are universal approximators and describe how their design enables applications to inference and inverse problems.
1. P RIOR W ORK
The idea to combine invertible (coupling) layers with expansive layers has been explored by
Brehmer & Cranmer (2020) and Kothari et al.
Brehmer & Cranmer (2020) combine two ﬂow networks with a simple expansive element (in the sense made precise in Section 2.1) and obtain a network that paramterizes probability distributions supported on manifolds.
They suggest that such constructions may be universal but neither they not Kothari et al.
(2021) derive theoretical results.
We discuss in detail the connection between these two empirical works and the approximation results derived here in Appendix B.1.
Kothari et al.
(2021) propose expansive coupling layers and build networks similar to that of
Brehmer & Cranmer (2020) but with an arbitrary number of expressive and expansive elements.
They observe that the resulting network trains much faster with a smaller memory footprint, while producing high-quality samples on a variety of benchmark datasets.
While to the best of our knowledge, there are no approximation-theoretic results for injective ﬂows, there exists a body of work on universality of invertible ﬂows; see Kobyzev et al.
(2020) for an overview.
Several works show that certain bijective ﬂow architectures are distributionally universal.
This was proved for autoregressive ﬂows with sigmoidal activations by Huang et al.
(2018) and for sum-of-squares polynomial ﬂows by Jaini et al.
Teshima et al.
(2020) show that several ﬂow networks including those from Huang et al.
(2018) and Jaini et al.
(2019) are also universal approximators of diffeomorphisms.
The injective ﬂows considered here have key applications in inference and inverse problems; for an overview of deep learning approaches to inverse problems, see Arridge et al.
Bora et al.
(2017) proposed to regularize compressed sensing problems by constraining the recovery to the range of (pre-trained) generative models.
Injective ﬂows with efﬁcient inverses as generative models gives an algorithmic projection 1 on the range, which facilitates implementation of reconstruction algorithms.
An alternative approach is Bayesian, where ﬂows are used to obtain tractable variational approximations of posterior distributions over parameters of interest, via supervised training on labeled input-output data pairs.
Ardizzone et al.
(2018) encode the dimension-reducing forward process by an invertible neural network (INN), with additional outputs used to encode posterior variability.
Invertibility guarantees that a model of the inverse process is learned implicitly.
For a given measurement, the inverse pass of the INN approximates the posterior over parameters.
Sun & Bouman (2020) propose variational approximations of the posterior using an untrained deep generative model.
They train a normalizing ﬂow which produces samples from the posterior, with the prior and the noise model given implicitly by the regularized misﬁt functional.
In Kothari et al.
this procedure is adapted to priors speciﬁed by injective ﬂows which yields signiﬁcant improvements in computational efﬁciency.
1. O UR C ONTRIBUTION
We derive new approximation results for neural networks composed of bijective ﬂows and injective expansive layers, including those introduced by Brehmer & Cranmer (2020) and Kothari et al.
We show that these networks universally jointly approximate a large class of manifolds and densities supported on them.
We build on the results of Teshima et al.
and develop a new theoretical device which we refer to as the embedding gap.
This gap is a measure of how nearly a mapping from R o
→ R m embeds an n -dimensional manifold in R m , where n ≤ o .
We ﬁnd a natural relationship between the embedding gap and the problem of approximating probability measures with low-dimensional support.
We then relate the embedding gap to a relaxation of universality we call manifold embedding property .
We show that this property captures the essential geometric aspects of universality and uncover important topological restrictions on the approximation power of these networks, to our knowledge, heretofore unknown in the literature.
We give an example of an absolutely continuous measure µ and embedding f : R 2 → R 3 such that f # µ can not be approximated with combinations of ﬂow layers and linear expansive layers.
This may be surprising since it was previously thought that networks such as those of Brehmer & Cranmer (2020) can approximate any “nice” density supported on a “nice” manifold.
We establish universality for manifolds with suitable topology, described in terms of extendable embeddings .
Our proof shows that optimality of the approximating network can be established in reverse: optimality of a given layer can be established without optimality of preceding layers.
This settles a (generalization of a) conjecture posed for a two-layer case in (Brehmer & Cranmer, 2020).
Finally, we show that these universal architectures are also practical and admit exact layer-wise projections, as well as other properties discussed in Section 3.4.
D ESCRIPTION OF THE A RCHITECTURE
Let C ( X, Y ) denote the space of continuous functions
X → Y .
We study networks in F ⊂ C
( X, Y ) that are of the form:
[Equation]
where R n (cid:96) − 1 ,n (cid:96) (cid:96) ⊂
C ( R n (cid:96) − 1 , R n (cid:96) ) , T n (cid:96) (cid:96) ⊂
C ( R n (cid:96) , R n (cid:96) ) , L ∈ N is the number of networks, n 0 = n , n L = m , and n (cid:96) ≥ n (cid:96) − 1 for (cid:96) = 1 , . . .
, L .
We introduce a well-tuned shorthand notation and write H ◦ G :=
{ h ◦ g : h ∈ H , g ∈ G} throughout the paper.
We identify R with the expansive layers and T with the bijective ﬂows.
Loosely speaking, the purpose of the expansive layers is to allow the network to parameterize high-dimensional functions by low-dimensional coordinates in an injective way.
The ﬂow networks give the network the expressivity necessary for universal approximation of manifold-supported distributions.
2. E XPANSIVE L AYERS
The expansive elements transform an n -dimensional manifold M embedded in R n (cid:96) − 1 , and embed it in a higher dimensional space R n (cid:96) .
To preserve the topology of the manifold, this must be done injectively.
We thus make the following assumptions about the expansive elements:
Deﬁnition 1 (Expansive Element) .
Let (cid:96) = 1 , . . .
, L , and R n (cid:96) − 1 ,n (cid:96) (cid:96) be a family of functions from
[Equation]
is a family of expansive elements if every R ∈ R n
[Equation]
[Equation]
n
[Equation]
is injective and
Lipschitz.
Examples of expansive elements include
(R1)
Zero padding
[Equation]
T where 0 ( m − n ) is the zero vector (Brehmer & Cranmer, 2020).
(R2) Multiplication by an arbitrary full-rank matrix, or one-by-one convolution:
[Equation]
where W ∈ R m × n and rank( W ) =
n (Cunningham et al., 2020), and w is a convolution kernel (cid:63) denotes convolution Kingma & Dhariwal (2018).
(R3)
Injective ReLU layers: R = ReLU( Wx ) , W = (cid:2) B T , − DB T , M T (cid:3) T , R ( x )
= ReLU (cid:0)(cid:2) w T , − w T (cid:3) (cid:63) x (cid:1) for matrix B ∈ GL n ( R ) , positive diagonal matrix D ∈ R n × n , and arbitrary matrix M ∈ R ( m − 2 n ) × n (Puthawala et al., 2020).
(R4) Injective ReLU networks (Puthawala et al., 2020, Theorem 15).
These are functions R : R n → R m of the form R ( x ) =
W L +1 ReLU( . . .
ReLU( W 1 x + b 1 ) . . . )
+ b L where W (cid:96) are n (cid:96) +1 × n (cid:96) matrices and b (cid:96) are the bias vectors in R n (cid:96) +1 .
The weight matrices W L satisfy the Directed Spanning Set (DSS) condition for (cid:96) ≤ L (that make all layers injective) and W L +1 is a generic matrix which makes the map R : R n → R m injective.
Note that the DSS condition requires that n (cid:96) ≥ 2 n (cid:96) − 1 + 1 for (cid:96) ≤ L and we have n 1 = n and n L +1 = m.
2. B IJECTIVE F LOW N ETWORKS
The bulk of our theoretical analysis is devoted to expressive elements.
The expressive elements bend the range of the expansive elements into the correct shape.
We make the following assumptions about the expressive elements:
Deﬁnition 2 (Bijective Flow Network) .
Let (cid:96) = 0 , . . .
, L be given and let T n (cid:96) (cid:96) ⊂
C ( R n (cid:96) , R n (cid:96)
[Equation]
T n (cid:96) (cid:96) is a family of bijective ﬂow networks if every T ∈ T n (cid:96) (cid:96) is Lipschitz continuous and bijective.
Examples of bijective ﬂow networks include (T1) Coupling ﬂows , introduced by Dinh et al.
(2014) consider R ( x ) =
H k ◦ · · · ◦ H 1 ( x ) where
[Equation]
In Eqn. 3, h i : R d × R e → R d is invertible w.r.t.
the ﬁrst argument given the second, and g i : R n − d → R e is arbitrary.
Typically in practice the operation in Eqn.
3 is combined with additional invertible operations such as permutations, masking or convolutions Dinh et al.
(2014; 2016); Kingma & Dhariwal (2018).
(T2) Autoregressive ﬂows , introduced by Kingma et al.
(2016) are generalizations of triangular ﬂows
A : R n → R n where for i = 1 , . .
. , n the i ’th value of A is given by of the form
[Equation]
In Eqn. 4, h i : R × R m → R where again
h
i is invertible w.r.t.
the ﬁrst argument given the second, and g i : R i − 1 → R m is arbitrary except for g 1 = 0 .
In Huang et al.
, the authors choose h i ( x, y ) ,
where y ∈ R m , to be a multi-layer perceptron (MLP) of the form
[Equation]
where φ is a sigmoidal increasing non-linear activation function.
M AIN R ESULTS 3. E MBEDDING G AP
We call a function f an embedding and denote it by f ∈ emb( X, Y ) if f : X → Y is continuous, injective, and f − 1 : f ( X ) → X is continuous 2 .
Also we denote emb k
( R n , R m )
=
emb( R n , R m ) ∩
C k ( R n , R m ) .
In order to set up our result concerning embedding of manifolds, we ﬁrst need a way to measure the degree to which a mapping g ∈ emb( R o , R m ) nearly embeds a manifold M = f ( K ) for compact K ⊂ R n and f ∈ emb( K, R m ) .
With this in mind we introduce embedding gap
B K,W ( f, g ) , a non-symmetric notion of distance between f and g .
Later in the paper, f will be the function to be approximated, and g a ﬂow-network to be the approximator.
Deﬁnition 3 (Embedding Gap) .
Let K ⊂ R n be compact and non-empty, W ⊂ R
o contain the closure of set U which is open in the subspace topology of some vector subspace V of dimension p , where n ≤ p ≤ o ≤ m , f ∈ emb( K, R m ) , and
g ∈ emb( W, R m ) .
Then we deﬁne the Embedding
Gap between f and g on sets K and W as
[Equation]
where I : f ( K ) → f ( K ) is the identity function and (cid:107) h (cid:107) L ∞ ( X )
=
ess sup x ∈ X (cid:107) h ( x ) (cid:107) 2 for h : X →
Y .
We refer to the embedding gap between f and g without specifying K and W when it is clear from context.
Remark 1 .
As W ⊂ R o contains U , an open set in V , there is an afﬁne map A : R n → V such that
A ( K )
⊂ W .
Then, the map r 0 = g ◦ A ◦ f − 1 : f ( K ) → g ( W ) is an injective continuous map from a compact set to its range and
hence r 0 ∈ emb( f ( K ) , g ( W )) .
Thus, in the above inﬁmum the set emb( f ( K ) , g ( W )) is non-empty.
Before giving properties of B K,W ( f, g ) , we brieﬂy describe its interpretation and meaning.
We denote by P ( X ) the set of probability measures over X .
If the embedding gap between two functions is small, then g is nearly an embedding of the range of f into R o .
B K,W ( f, g ) is constructed expressly to serve as an upper bound
[Equation]
,W ( f, g ) where µ n ∈ P ( K ) is given, and W 2 ( ν 1 , ν 2 ) denotes the Wasserstein-2 distance with (cid:96) 2 ground metric (Villani, 2008), as shown in Lemma 7 part 5.
The above result has a simple meaning in the context of machine learning.
Suppose we want to learn a generative model g to (approximately) sample from a probability measure ν with low-dimensional support, by applying g to samples from a base distribution µ o .
Suppose further that ν is a pushforward of some (known or unknown) distribution µ n via f .
The embedding gap B K,W ( f, g ) then upper bounds the 2-Wasserstein distance between ν and g # µ 0 for the best possible choice of µ o .
The embedding r can be interpreted as a candidate transport map from any measure pushed forward by f , that can be pulled back through g .
Loosely speaking, for µ (cid:48)
o = g − 1 ◦ r ◦ f # µ n , r is a valid
Wasserstein transport map that transports f # µ n to g # µ (cid:48)
o with cost no more than (cid:107) I − r (cid:107) L ∞ ( f ( K )) .
See Fig.
1 for a visualization of the embedding gap between two toy functions.
Further, the embedding gap satisﬁes inequalities useful for studying networks of the form of Eqn. 1, see Lemma 7.
3. M ANIFOLD E MBEDDING P ROPERTY
We now introduce a central concept, the manifold embedding property (MEP).
A family of networks satisﬁes the MEP if it nearly embeds a large class of manifolds of certain dimension and regularity, as measured by the embedding gap.
The MEP is a property of a family of functions
E o,m ⊂ emb( W, R m ) where W ⊂ R o .
For this manuscript, E o,m will always be formed by taking
E o,m :=
T m ◦ R o,m .
We note here that the MEP is closely related to the question of whether or not a given n -dimensional manifold M that is an image of an embedding f : K → R m , that is, M = f ( K ) , K ⊂ R n , can be approximated by the images of neural networks E : K
→ R m of a given type.
In particular, we will consider neural networks that are compositions where R : R o
→ R m are ‘simple’ non-universal functions, and ﬂow-maps T : R m → R m that are diffeomorphism.
This choice of applying nonuniversal expansive layers, and then diffromorphisms has some topological consequences, which we discuss below.
T OPOLOGICAL OBSTRUCTIONS TO MANIFOLD LEARNING WITH NEURAL NETWORKS
We ﬁnd that using non-universal expansive layers, followed by layers imposes some topological conditions on what can be approximated.
We illustrate this with the following problem.
When n = 2 , m = 3 , and K = S 1 ⊂ R 2 is the circle.
We consider maps E = T ◦ R : R n → R m where
R : R n → R m is, e.g., a linear map of rank n , and T : R m → R m is a coupling ﬂow which is a homeomorphism.
Let f ∈ emb
( K, R 3 ) be an embedding that maps K to a trefoil knot M =
f ( S 1 ) , see Fig.
Such a function f can not be written as a restriction T ◦ R on S 1 .
In Sec.
C.2.2 we prove this fact and build a related example where a measure, µ ∈ P ( R 2 ) , supported on an annulus is pushed forward to a measure supported on a knotted ribbon in R 3 by an embedding g : R 2 → R 3 .
For this measure, there are no E :=
T ◦ R , with linear injective R and embedding T , such that g # µ = E # µ .
To sidestep this fundamental difﬁculty, we deﬁne the MEP property for a certain subclass of manifolds { f ( K ) : f ∈ F} .
Finally, when considering ﬂow networks which are universal approximators of C 2 diffeomorphisms, we restrict the class of manifolds to be approximated even further.
This is because manifolds that are homeomorphic are not necessarily diffeomorphic; for example exotic spheres.
These are topological structures that are homeomorphic, but not diffeomorphic, to the sphere Milnor (1956).
Moreover, it is known that general homeomorphisms F : R m → R m can
[Figure]
not be approximated in C 0 -topology by C 2 -smooth diffeomorphisms T : R m → R m .
See M¨uller (2014) for a precise statement.
However, all C 1 -smooth diffeomorphims F : R m → R m can be approximated in the strong topology of C 1 by C 2 -smooth diffeomorphims ˜ F : R m → R m , (cid:96) ≥ k see Hirsch (2012), Ch. 2, Theorem 2.7.
Because of this, we have to pay attention to the smoothness of the maps in the subset F ⊂ emb( K, R m ) .
For these reasons, when we refer to the MEP, we consider it with respect to a class of functions F ⊂ emb( R n , R m ) .
The MEP can be interpreted as a density statement, saying that our networks E o,m are dense in the ‘ B K,W distance’ in the set F ⊂ emb( R n , R m ) .
Two examples we are particularly interested in are the following.
First, when F = Φ ◦ A where A : R n → R m are linear maps of rank n and Φ : R m → R m are C k diffeomorphisms with k ≥ 1 .
Second, when F ∈ emb( R n , R m ) .
Deﬁnition 4.
Let E o,m ⊂ emb
( R o , R m ) and F n,m ⊂ emb
( R n , R m ) be two families of functions.
We say that E o,m has the m, n, o Manifold Embedding Property (MEP) w.r.t.
F n,m if for every compact non-empty set K ⊂ R n , f ∈ F n,m , and (cid:15) > 0 , there is an E ∈ E
o,m and a compact set
W ⊂ R o such that the restriction of f in K and the restriction of E in W satisfy
[Equation]
When it is clear from the context, we abbreviate the m, n, o MEP w.r.t.
F n,m simply by the m, n, o
MEP, or simply the MEP.
We also note here that if a model from R o → R m is a uniform universal approximator of F o,m =
C 0 ( R n , R m ) on compact sets, such as those considered in Yarotsky (2017; 2018), then it has the m, n, o MEP w.r.t F n,m for any n ≤ o .
Thus, networks that are uniform universal approximators automatically possess the MEP, as shown in Lemma 1.
However, we have a result that applies to network which are uniform universal approximators.
Deﬁnition 5 (Uniform Universal Approximator) .
For a non-empty subset F n,m ⊂ C ( R n , R m ) , a family E n,m ⊂ C ( R n , R m ) is said to be a uniform universal approximator of F n,m if for every f ∈ F n,m , every non-empty compact K ⊂ R n , and each (cid:15) > 0 , there is an E ∈ E n,m satisfying:
[Equation]
We also have the following function class, which is key to our analysis.
Deﬁnition 6 (Extendable Embeddings) .
Let I ( R n , R m ) be the set of functions F : R n → R m that are compositions F = Φ ◦ R where R : R n → R m is a linear map of rank n and Φ : R m → R m is a C 1 -smooth diffeomorphism.
We call I ( R n , R m ) be the set of extendable embeddings.
We also denote I k
( R n , R m )
=
I ( R n , R m ) ∩ C k
( R n , R m ) .
We observe that I ( R n , R m ) ⊂ emb( R n , R m ) .
Lemma 1.
(i)
If R o,m is a universal approximator of C ( R n , R m ) and I ∈
T m where I is the identity map, then E o,m :=
T m ◦ R o,m has the MEP w.r.t.
emb( R n , R m ) .
(ii)
If R o,m is such that there is an injective R ∈ R o,m and open set U
⊂ R o such that R (cid:12)(cid:12)
U is linear, and T m is a sup universal approximator in the space of Diff 2 ( R m , R m ) , in the sense of Teshima et al.
(2020), of the C 2 -smooth diffeomorphisms, then E o,m :=
T m ◦ R o,m has the MEP w.r.t.
F = I ( R n , R m ) .
The proof of Lemma 1 is in Appendix C.2.1.
It has the following implications for the architectures studied in Section 2.
Example 1.
Let E : =
T m ◦ R o,m , then
(i)
If T m is either (T1) or (T2) and R o,m is (R4), then E o,m has the m, n, o MEP w.r.t.
emb( R n , R m ) .
(ii)
If T m is (T2) with sigmoidal activations Huang et al.
(2018), then if R o,m is any of (R1), ..., (R4), then E o,m has the m, n, o MEP w.r.t.
I ( R n , R m ) .
The proof of Example 1 is in Appendix C.2.3.
We add a remark here showing that if m is sufﬁciently larger than n , and subject to some regularity assumptions, The two classes considered in Example 1 coincide.
Lemma 2.
When m ≥ 3 n + 1 and k ≥ 1 , for any C k embedding f ∈ emb k ( R n , R m ) and compact set K ⊂ R n , there is a map in the closures of the ﬂow type neural network E ∈
I k
( R n , R m ) such that E ( K ) =
f ( K ) .
Moreover,
I
[Equation]
This means that the topological structure of a manifold to approximate M in R m is diffeomorphic to the set K ⊂ R n , and thus some ﬂow type network can approximate M .
The proof of Lemma 2 in Appendix C.2.4.
We ﬁnally have two more lemmas to present before moving on to our discussion of universality as it ties to the MEP.
Lemma 3.
Let F n,o ⊂ emb( R n , R o ) and F o,m ⊂ emb( R 0 , R m ) E p,o⊂ emb( R p , R o ) have the o, n, p MEP w.r.t.
F n,o ⊂ emb( R n , R o ) and E o,m⊂ emb( R o , R m ) have the m, o, o MEP w.r.t.
F o,m ⊂ emb( R o , R m ) .
If each E o,m∈ E o,mis locally Lipschitz, then E o,m◦ E p,ohas the m, n, p MEP w.r.t.
F o,m ◦ F n,o .
The proof of Lemma 3 is in Appendix C.2.5.
We note that when the elements of E o,mare differentiable, local Lipschitzness is automatic, and need not be assumed, see e.g. (Tao, 2009, Ex. 10.2.6).
We also record a near-converse (proved in C.2.6) of Lemma 3 that shows that if E o,m◦ E p,ohas the m, n, p MEP, then E o,mhas the m, n, o MEP.
Lemma 4.
Let F ⊂ emb( R n , R m ) .
Let E p,o⊂ emb( R p , R o ) and E o,m⊂ emb( R o , R m ) be such that E o,m◦E p,ohas the m, n, p MEP with respect to family F .
Then E o,mhas the m, n, o MEP with respect to family F .
3. U NIVERSALITY
We now present our universal approximation result for networks given in Eqn. Eq. 1 and a decoupling property.
Theorem 1 (Qualitative Universality for Embeddings) .
Let n 0 = n, n L = m , µ ∈ P ( K ) be an absolutely continuous measure w.r.t.
Lebesgue measure.
Further let, for each (cid:96) = 1 , . . .
, L ,
E n
[Equation]
:=
[Equation]
where R n (cid:96) −
[Equation]
[Equation]
[Table] (cid:96) is a family of injective expansive elements that contains a linear map, and T n (cid:96) (cid:96) is a family of bijective family networks.
Finally let T n 0 be distributionally universal, i.e. for any absolutely continuous µ ∈ P ( R n ) and ν ∈ P ( R n ) , there is a { T i } ∞ i =1 such that T i # µ → ν in distribution.
Suppose that one of the following two cases hold:
(i) Let F ∈ F n L − 1 ,m L ◦· · ·◦F n,n 1and E n (cid:96) − 1 ,n (cid:96) (cid:96) have the the n (cid:96) , n, n (cid:96) − 1 MEP for (cid:96) = 1 , . . .
, L with respect to F n (cid:96) − 1 ,n (cid:96) (cid:96) .
(ii) Let F ∈ emb 1 ( R n , R m ) be a C 1 -smooth embedding, and assume that for (cid:96) = 1 , . . .
, L it holds that n (cid:96) ≥ 3 n (cid:96) − 1 + 1 and the families T n (cid:96) (cid:96) are dense in the space of C 2 -diffeomorphism Diff 2 ( R n (cid:96)
Then, there is a sequence of { E i } i =1 ,..., ∞ ⊂
E n L − 1 ,m L ◦ · · · ◦ E n 1 ,n◦ T n 0 such that
[Equation]
The proof of Theorem 1 is in Appendix C.3.1.
As discussed in above and in Figure 2, there are topological obstructions for Theorem 1 with a general embedding F : R n → R m .
When n = 2 , m = 3 , L = 1 , and µ is the uniform measure on an annulus K ⊂ R 2 target measure F # µ is the uniform measure on a knotted ribbon M = F ( K )
⊂ R 3 .
There are no injective linear maps
R : R 2 → R 3 and diffeomorphisms T : R 3 → R 3 such that E
= T ◦ R would satisfy M = E ( K ) and E
# µ = F # µ .
This happen even though the set of diffeomorphism T : R 3 → R 3 are universal distributional approximators.
In this case the condition m = n 1 ≥ 3 n + 1 is not valid.
We remark here that our networks are designed expressly to approximate manifolds, and hence injectivity is key.
This separates our results from, e.g. (Lee et al., 2017, Theorem 3.1) or (Lu & Lu, 2020, Theorem 2.1), where universality results of ReLU networks are also obtained.
The previous theorem shows that the entire network is universalif it can be broken into pieces that have the MEP.
The following lemma, proved in Appendix C.3.2, shows that if E n,m = H o,m ◦ G n,o , then H o,m must have the m, n, o MEP if E n,m is universal.
Lemma 5.
Suppose that E n,m =
H
o,m ◦ G n,o where E n,m ⊂ emb( R n , R m ) , H o,m ⊂ emb
( R o , R m ) , and G n,o ⊂ emb( R n , R o ) .
If H o,m does not have the m, n, o MEP w.r.t.
F , then there exists a f ∈ F , compact K ⊂ R n and (cid:15) > 0 such that for all E ∈ E n,m , and r ∈ emb( f ( K ) , E ( W ))
[Equation]
The proof of Theorem 1 also implies that, loosely speaking, later layers decouple from earlier ones.
That is, given a sequence of functions that has the MEP on the last L − ˜ (cid:96) layers, there are always functions in the ﬁrst ˜ (cid:96) layers so that the entire network is end-to-end optimal.
Corollary 1.
Let F n,o ⊂ emb( R n , R o ) , F o,m ⊂ emb
( R o , R m ) , and let E o,m ⊂ emb
( R o , R m ) have the m, n, o MEP w.r.t.
F o,m ◦ F n,o .
For every F ∈ F o,m ◦ F n,o then there is a compact
K ⊂ R n and { E i } ∞ i =1 ⊂ E o,m such that
[Equation]
[Equation]
Further, if E n,o ⊂ emb( W (cid:48) , R o ) has the o, n, n MEP w.r.t.
F n,o , and T n is a universal approximator for distributions, then for any µ ∈ P ( K ) where K ⊂ R n is compact, there is a sequence { E (cid:48) i } i =1 ,..., ∞ ⊂ E n,o and { T i } i =1 ,..., ∞ ⊂ T n so that
[Equation]
The proof of Corollary 1 is in Appendix C.3.3.
Approximation results for neural networks are typically given in terms of the network end-to-end.
Corollary 1 shows that the layers of approximating networks can in fact be built one at a time.
It is related to an observation made in (Brehmer & Cranmer, 2020, Section B) about training strategies, where the authors remark that they ‘expect faster and more robust training of a network’ of the form in Eqn.
when L = 1 , that is F = T m 1 ◦R n,m
[Equation]
Corollary 1 shows that there exists a minimizing sequence in T m 1 that need only minimize Eqn. 12; the T n 0 layers can be minimized after.
We can further combine Lemma 5 and Cor.
1 to prove that not only can the network from Brehmer & Cranmer (2020) be trained layerwise, but that any universal network can necessarily be trained layerwise in reverse order, provided that it can be written as a composition of two smaller layers.
3. L AYER - WISE INVERSION , UNCERTAINTY QUANTIFICATION AND RECOVERY OF WEIGHTS
In this subsection, we describe how our network can be augmented with more useful properties if the architecture satisﬁes a few more assumptions without affecting universal approximation.
We focus on a new layerwise projection result, with a further discussion of Bayesian uncertainty quantiﬁcation, and black-box recovery of our network’s weights in Appendices D.2 & D.3.
Given a point y ∈ R m that does not lie in the range of the network, projecting y onto the range of the network is a practical problem without an obvious answer.
The crux of the problem is inverting the injective (but non-invertible) R layers when R .
When R contains only full-rank matrices as in (R1) or (R2) then we can compute a least-squares solution.
If, however, R contains layers which are only piecewise linear, as in (R3), then the problem of computing a least squares solution is more difﬁcult, see Fig.
Nevertheless, we ﬁnd that if R is (R3) we can still compute a least-squares solution.
Assumption 1.
Let R be given by one of (R1) or (R2), or else (R3) when m = 2 n .
If R only contains linear operators, then the least-squares problem can be computed by solving the normal equations (see (Golub, 1996, Section 5.3).)
This includes cases (R1) or (R2).
For (R3) we have the following result for D =
I n × n and M is an empty matrix.
Deﬁnition 7.
Let W = (cid:2) B t − DB t (cid:3) t ∈ R 2 n × n and y ∈ R 2 n be given, and let R ( x ) =
ReLU( Wx ) .
Then deﬁne
[Equation]
where the max in Eqn.
14 is taken element-wise.
Theorem 2.
Let y ∈ R 2 n .
If for i = 1 , . . . , n , [ y ] i (cid:54) =
[ y ] i + n then
[Equation]
Further, if there is a i ∈ { 1 , . . . , n } such that [ y ] i =
[ y ] i + n , then there are multiple minimizers of (cid:107) y − R ( x ) (cid:107) 2 , one of which is R † ( y ) .
The proof of Theorem 2 is given in Appendix D.1.
Remark 2 .
We note that Theorem 2 is different from many of the existing work on inverting expansive layers, e.g. Aberdam et al.
Bora et al.
Lei et al.
(2019), our result gives a direct inversion algorithm that is provably the least-squares minimizer.
Further, if each expansive layer is any combination of (R1), (R2), or (R3) then the entire network can be inverted end-to-end by using either the above result or solving the normal equations directly.
C ONCLUSION
Bijective ﬂow networks are a powerful tool for learning a push-forward mappings in a space of ﬁxed dimension.
Increasingly, these ﬂow networks have been used in combination with networks that increase dimension in order to produce networks which are purportedly universal.
In this work, we have studied the theory underpinning these ﬂow and expansive networks by introducing two new notions, the embedding gap and the manifold embedding property.
We show that these notions are both necessary and sufﬁcient for proving universality, but require important topological and geometrical considerations which are, heretofore, under-explored in the literature.
We also ﬁnd that optimality of the studied networks can be established ‘in reverse,’ by minimizing the embedding gap, which we expect opens the door to convergence of layer-wise training schemes.
Without compromising universality, we can also use speciﬁc expansive layers with a new layerwise projection result.
Moreover, we show that the studied networks provide Bayesian uncertainty quantiﬁcation and allow black-box recovery of their weights.
A PPENDIX A S UMMARY OF N OTATION
Throughout the paper we make heavy use of the following notation.
Unless otherwise stated, X and Y always refer to subsets of Euclidean space, and K and W always refer to compact subsets of Euclidian space.
f ∈ C ( X, Y ) means that f : X → Y is continuous.
For families of functions F and G where each F (cid:51) f : X → Y and G (cid:51) g :
Y → Z , then we deﬁne G ◦ F = { g ◦ f : X → Z : f ∈ F , g ∈ G} .
f ∈ emb( X, Y ) means that f ∈ C ( X, Y ) is continuous and injective on the range of f , i.e. an embedding, and furthermore that f − 1 : f ( X ) → X is continuous.
µ ∈ P ( X ) means that µ is a probability measure over X . 6.
W 2 ( µ, ν ) for µ, ν ∈ P ( X ) refers to the Wasserstein-2 distance, always with (cid:96) 2 ground metric.
(cid:107)·(cid:107)
L p ( X ) refers to the L p norm of functions, from X to R . 8.
For vector-valued f : X → Y , (cid:107) f (cid:107) L ∞ ( X ) =
ess sup x ∈X (cid:107) f (cid:107) 2 .
Note that Y is always ﬁnite dimensional, and so all discrete 1 ≤ q ≤ ∞ norms are equivalent.
Lip( g ) refers to the Lipschitz constant of f . 10.
For x ∈ R n , [ x ] i ∈ R is the i ’th component of x .
Similarly, for matrix A ∈ R m × n , [ A ] ij refers to the j ’th element in the i ’th column.
A PPENDIX B D ETAILED C OMPARISON TO P RIOR WORK B. C ONNECTION TO B REHMER & C RANMER (2020)
In Brehmer & Cranmer (2020), the authors introduce manifold-learning ﬂows as an invertible method for learning probability density supported on a low-dimensional manifold.
Their model can be written as
[Equation]
where T m 1 ⊂ C
( R m , R m ) , T m 0 ⊂ C
( R n , R n ) , and R = (cid:26)(cid:20)
I n × n 0 ( m − n ) × n (cid:21)(cid:27) is a zero-padding (R1).
They invert F ∈ F in two different ways.
For manifold-learning ﬂows ( M -ﬂows) they restrict T m 1 to be an invertible ﬂow, and for manifold-learning ﬂows with separate encoder (
M e -ﬂows) they place no such restrictions on
T m 1 and instead train a separate neural network e to invert elements of
T m 1 .
Our results apply out-of-the-box to the architectures used in Experiment A of Brehmer & Cranmer (2020).
The architecture described in Eqn.
18 is of the form of Eqn.
where L = 1 .
Further, although they are not studied here, our analysis can also be applied to quadratic ﬂows.
The network used in (Brehmer & Cranmer, 2020, Experiment 4.A) uses coupling networks, (T1), where T m 1 and T n 0 are both 5 layers deep.
For (Brehmer & Cranmer, 2020, Experiments 4.B and 4.C) the authors choose expressive elements T that are rational quadratic ﬂows Durkan et al.
(2019b) for both T m 1 and T n 0 .
In Experiment 4.B they let T 1 and T 0 again be 5 layers deep, and in 4.C they again let T 1 by 20 layers deep and T 0 15 layers.
For the ﬁnal experiment, 4.D, the choose more complicated expressive elements that combine Glow Kingma & Dhariwal (2018) and Real
NVP Dinh et al.
(2016) architectures.
These elements include the actnorm, 1 × 1 convolutions and rational-quadratic coupling transformations along with a multi-scale transformation.
The authors mention universality of their network without our proof, but our universality results in
Theorem 1 apply to their networks from Experiment A wholesale.
Further in their work the authors describe how training can be split into a manifold phase and density phase, wherein the manifold phase T m 1 is trained to learn the manifold, and in the density phase T m 1 if ﬁxed and T n 0 is trained to learn the density thereupon.
This statement is made formal and proven by our Cor. 1.
B. C ONNECTION TO K OTHARI ET AL . (2021)
In Kothari et al.
(2021), the authors introduce the ‘Trumpet’ architecture, for it’s architecture, which has many alternating ﬂow networks & expansive layers with many ﬂow-networks in the lowdimensional early stages of the network, which gives the architecture a shape similar to the titular instrument.
The architecture studied in Kothari et al.
(2021) is precisely of the form of Eqn.
1, where the bijective ﬂow networks are revnets Gomez et al.
; Jacobsen et al.
(2018) architecture, and the expansive elements are 1 × 1 convolutions, as in (R2).
To out knowledge, there are no results that show that the revnets used are universal approximators, but if they revnets are substituted with either (T1) or (T2), then the, we could apply Theorem 1 to the resulting architecture.
The authors of Kothari et al. (2021) also remark that their entire network can be designed in order to facilitate uncertainty quantiﬁcation, a remark that we too can apply to our networks, as discussed further in Section D.2.
A PPENDIX C P ROOFS C. M AIN R ESULTS C.1. H ELPER L EMMA
Before presenting the proof of Lemma 7, we present the following three helper inequalities
Lemma 6.
For all of the following results, f ∈ emb( K, R m ) and g ∈ emb( W, R m ) and n ≤ o ≤
m .
[Equation] [Equation]
Let X, Y ⊂ W , let g be Lipschitz on W ,
and r ∈ emb( X, Y ) .
Then, there is a r (cid:48) ∈ emb( g ( X ) , g ( Y ))
such that g ◦ r = r (cid:48) ◦ g and (cid:107)
I − r (cid:48) (cid:107) L ∞ ( g ( X ))
≤ (cid:107)
I − r (cid:107) L ∞ ( X ) Lip( g ) .
[Equation]
Let K ⊂ R n , X ⊂ R p and W ⊂ R o be compact sets.
Also, let f ∈ emb
( K, W ) and h ∈ emb( X, W ) , and let g ∈ emb( W, R m ) be a Lipschitz map.
Then
[Equation]
Proof.
Let
r ∈ C ( f ( K ) , g ( W )) , then
[Equation]
2 where
x o =
g −
◦ r ◦ f
( x n
[Equation]
inf
[Equation]
g
[Equation]
g
is injective on X , hence we can deﬁne r (cid:48)
such that r (cid:48) =
g ◦ r ◦ g − 1 : g ( X ) → g ( r ( X ))
⊂ g ( Y ) such that r (cid:48) ∈ emb( g ( X ) , g ( Y )) and thus ∀ x ∈ X ,
[Equation] [Equation]
where we have used (cid:107) r ( x
[Equation]
− x (cid:107) 2 ≤ (cid:107) I − r (cid:107)
[Equation]
X
[Equation]
For every x ∈ r ( K ) , we have a y ∈ K such that x
= r ( y ) , thus ∀ x ∈ r ( K ) ,
[Equation]
But r is clearly surjective onto it’s range, hence taking the supremum over all x ∈ X yields
[Equation]
As g ∈ emb( W, R m ) , the map g : W → g ( W ) is a homeomorphism and there is g − 1 ∈ emb( g ( W ) , W ) .
For a map r ∈ emb( g ◦ f ( K ) , g ◦ h ( X )) , we see that
ˆ r =
g − 1 ◦ r ◦ g ∈ emb( f ( K ) , h ( X )) .
Also, the opposite is valid as if ˆ r ∈ emb( f ( K ) , h ( X )) then r = g ◦ ˆ r ◦ g − 1 ∈ emb( g ◦ f ( K ) , g ◦ h ( X )) .
Thus
[Equation]
Lemma 7.
Let f ∈ emb
( K, R m ) and g ∈ emb( W, R m ) and n ≤ o ≤ m .
Then the following hold:
B K,W ( f, g ) ≤ sup x ∈ K (cid:107) g ◦ h ( x ) − f ( x ) (cid:107) 2 where h ∈ emb( K, R o ) is a map satisfying h ( K )
⊂ W . 2.
For any X that is the closure of an open set , if h ∈ emb( X, W ) then
[Equation] [Equation]
For any r ∈ emb( f ( K ) , R m ) ,
[Equation]
For any r ∈ emb( f ( K ) , g ( W )) and h ∈ emb( X, W ) where X ⊂ R p is the closure of a compact set where n ≤ p ≤ o we have that
[Equation]
where Lip( g ) denotes the Lipschitz constant of g . 5.
For any µ n ∈ P ( K ) , there is a µ o ∈ P ( W ) such that
[Equation]
The proof of Lemma
[Equation]
If
we let r :
=
g ◦ h ◦ f − then r ∈ emb( f ( K ) g
( W )) and
[Equation]
Given that g ◦ h ( X ) ⊂ g ( W ) , we have that emb
f
K
) , g ◦ h ( X )
[Equation]
⊂ emb( f ( K ) , g ( W )) , thus the inﬁmum in Eqn.
6 is taken over a smaller set, thus B K,W ( f, g ) ≤ B K,X ( f, g ◦ h ) .
Note that for any r (cid:48) ∈ emb( r ◦ f ( K ) , g ( W )) , r (cid:48) ◦ r ∈ emb( f ( K ) , g ( W )) , and so we have
[Equation]
where we have used that r is injective for the ﬁnal equality.
This holds for all possible r (cid:48) , hence we have the result.
Recall that f ∈ emb( K, W ) , g ∈ emb
( W, R m ) , h ∈ emb( X, W ) and r ∈ emb( f ( K ) , g ( W )) .
Then g − 1 ∈ emb( g ( W ) , W ) .
As r ◦ f ( K ) ⊂ g ( W ) , we see that
[Equation]
This, Lemma 7, point 3 and Lemma 6, point 4 yield that
[Equation]
g ) B K,X ( g − 1 ◦ r ◦ f, h ) , which proves the claim.
Let r ∈ emb
( f ( K ) , g ( W )) be such that (cid:107) I − r (cid:107) L ∞ (Range( f ))
≤ B K,W ( f, g ) + (cid:15) for some (cid:15) > 0 , then for every x ∈ K , there exists y ∈ W such that g ( y ) =
r ◦ f ( x ) .
From injectivity of g , we have that y
=
g − 1 ◦ r ◦ f ( x ) .
Note that g − 1 ◦ r ◦ f ∈ emb
K, W ) , hence
K (cid:48) :
= g − 1 ◦ r ◦ f
( K ) is compact.
Deﬁne µ (cid:48) :
= ( g − 1 ◦ r ◦ f ) # µ .
Clearly g # µ (cid:48)
=
r ◦ f # µ , and thus
[Equation]
but W 2 ( g # µ (cid:48) , r ◦ f # µ ) =
and
[Equation]
This is true for every (cid:15) > 0 , hence taking the inﬁmum over all r ∈ emb( f ( K ) , g ( K )) establishes W 2 ( g # µ (cid:48) , f # µ ) ≤ B K,W ( f, g ) + (cid:15) for every (cid:15) > 0 , and thus we have that
[Equation]
C. M ANIFOLD E MBEDDING P ROPERTY C.2. T HE PROOF OF L EMMA
The proof of Lemma 1.
(i) Let us consider (cid:15) > 0 , a compact set K ⊂ R n and f ∈ emb( R n , R m ) .
Let W = K
×
0 } o − n and F : R o
→ R m be the map given by F ( x, y )
=
f
( x ) , ( x, y ) ∈ R n × R o − n .
Because R o,m is a uniform universal approximator of C ( R n , R m ) , there is an R ∈ R o,m such that (cid:107) F − R (cid:107) L ∞ ( W ) < (cid:15) .
Then for the map E
= I ◦
R we have that B K,W ( f, E ) <
(cid:15) .
This is true for every (cid:15) > 0 , and so E o,m has the MEP property w.r
t. the family emb( R n , R m ) .
(ii) Recall that f :=
Φ 0 ◦ R 0 for Φ 0 ∈ Diff 1 ( R m , R m ) and linear R 0 : R n → R m , and that R ∈ R is such that R (cid:12)(cid:12) U is linear for open U .
We present the proof in the case when n = o , and we make the assumption that R (cid:12)(cid:12) K is linear.
In this case, we have the existence of an afﬁne map A : R m → R m so that R
=
A ◦ R so that ˜ K :=
R 0 ( K ) =
A ( R ( K )) .
Let (cid:15) > 0 be given.
By (Hirsch, 2012, Chapter 2, Theorem 2.7), the space Diff 2 ( R m , R m ) is dense in the space Diff 1 ( R m , R m )
and so there is some Φ 1 ∈ Diff 2 ( R m , R m ) such that
[Equation]
Then, let T ∈ T m be such that (cid:107) T − Φ 1 ◦ A (cid:107) L ∞ ( R ( K );
R m ) < (cid:15) 2 .
Then we have that
[Equation]
Hence, if we let r =
T ◦ R ◦ f − 1 ∈ emb( f ( K ) , T ◦ R ( K )) then we obtain that B K,K
( f, T ◦ R ) < (cid:15) .
This holds for any (cid:15) , and hence we have that
T ◦ R has the MEP for I ( R n , R m ) .
The proof in the case that o ≥ n follows with minor modiﬁcation, and applying Lemmapoint 1.
C.2. S 1 CAN NOT BE MAPPED EXTENDABLY TO THE TREFOIL KNOT
We ﬁrst show that there are no maps E :=
T ◦ R where R : R 2 → R 3 such that T is a homeomorphism and E ( S 1 ) is a trefoil knot.
We use the fact that the trivial knot S 1 and the trefoil knot
M = f ( S 1 ) are not equivalent, that is, there are no homeomorphisms in R 3 that map S 1 to M .
Indeed, by (Murasugi, 2008, Section 3.2), the trefoil knot M and its mirror image are not equivalent, whereas the trivial knot S 1 and its mirror image are equivalent.
Hence, M and R ( S 1 ) are not equivalent knots in R 3 .
Thus by (Murasugi, 2008, Deﬁnition 1.3.1 and Theorem 1.3.1), we see that there is no orientation preserving homeomorphism T : R 3 → R 3 such that T ( R 3 \ R ( S 1 ))
=
R 3 \ M .
As the orientation of the map T can be changed by composing T with the reﬂection J : R 3 → Racross the plane Range( R ) that deﬁnes a homeomorphim J : R 3 \ R ( S 1 ) → R 3 \ R ( S 1 ) , we see that there is no homeomorphism T : R 3 → R 3 such that T ( R 3 \ R ( S 1 ))
=
R 3 \ M .
This example shows that the composition E
= T ◦ R of a linear map R and a coupling ﬂow T cannot have the property that E ( S 1 ) =
f ( S 1 ) for this embedding f .
Moreover, the complement R\ E ( S 1 ) is never homeomorphic to R 3 \ f ( S 1 ) for any such map E .
We now construct another example, similar to Figure 2, where an annulus that is mapped to a knotted ribbon in R 3 .
To do this, replace the circle S 1 by an annulus K = { x ∈ R 2 : 1 / 2 ≤ | x | ≤ 3 / 2 } , that in the polar coordinates is { ( r, θ ) : 1 / 2 ≤ r ≤ 3 / 2 } and deﬁne a map
F : K → R 3 by deﬁning in the polar coordinates
[Equation]
where f : S 1 → Σ 1 ⊂ R 3 is an smooth embedding of S 1 to a trefoil knot Σ 1 and v ( θ ) ∈ R 3 is a unit vector normal to Σ 1 at the point f
( θ ) such that v ( θ ) is a smooth function of θ , and a > 0 is a small number.
In this case, M 1 = F ( K ) is a 2-dimensional submanifold of R 3 with boundary, which can visualizes M 1 as a knotted ribbon.
We now show that there are no maps E
= T ◦ R such that E ( K )
=
F ( K ) where T : R 3 → R 3 is an embedding, and R : R 2 → R 3 injective and linear.
The key insight is that if such a T existed, then this implies that the trefoil knot is equivalent to S 1 in R 3 , which is known to be false.
Let U ρ ( A ) denote the ρ -neighborhood of the set
A in R 3 .
It is easy to see that R 2 \ ( { 0 } × [ − 1 , 1]) is homeomorphic to R 2 \ B R 2 (0 , 1) , which is further homeomorphic to R 2 \ { 0 } .
Thus, using tubular coordinates near Σ 1 and a sufﬁciently small ρ > 0 , we see that R 3 \ M 1 is homeomorphic to R\ U
ρ (Σ 1 ) , which is further homeomorphic to R 3 \ Σ 1 .
Also, when R : R 2 → R 3 is an injective linear map, we see that M 2 =
R ( K ) is a un-knotted band in R 3 and R 3 \ M 2 is homeomorphic to R 3 \ Σ 2 .
If R 3 \ M 1 and R 3 \ M 2 would be homeomorphic, then also R 3 \ Σ 1 and R 3 \ Σ 2 would be homeomorphic that is not possible by knot theory, see (Murasugi, 2008, Deﬁnition 1.3.1 and Theorem 1.3.1).
This shows that there are no injective linear maps R : R 2 → R 3 and homeomorphisms Φ : R 3 → R 3 such that (Φ ◦ R )( K ) =
M 1 .
Similar examples can be obtained in a higher dimensional case by using a knotted torus S´equin (2011) 4 and their Cartesian products.
C.2. T HE PROOF OF E XAMPLE
Proof.
(i) From (Puthawala et al., 2020, Theorem
) we have that R o,m can approximate any continuous function f ∈ emb( R n , R m ) .
Further, clearly (T1) and (T2) both contain the identity map, thus Lemma 1 (i) applies.
(ii) Let T m be the family autoregressive ﬂows with sigmoidal activations deﬁned in (Huang et al., 2018).
By (Teshima et al., 2020, App. G, Theorem 1 and Proposition 7), T m are sup -universal approximators in the space Diff 2 ( R m , R m ) of C 2 -smooth diffeomorphisms Φ : R m → R m .
When R o,m is one of (R1) or (R2)
the network is always linear, hence the conditions are satisﬁed.
If R o,m is (R4), then R o,m contains linear mappings, and if (R3), then we can shift the origin, so that R ( x ) is linear on K .
In all cases, Lemma 1 part (ii) applies.
C.2. T HE PROOF OF L EMMA
Given an f ∈ emb ∞ ( K, R m ) we ﬁrst show that for m ≥ 2 n +1 there are always a diffeomorphisms Ψ:
R m → R m so that Ψ ◦ f : R n → { 0 } n × R m − n .
The existence of such a Ψ borrows ideas from
Whitney’s embedding theorem (Hirsch, 2012, Theorems 3.4 & 3.5) and is constructed by iteratively constructing an injective projection.
Next if m − n ≥ 2 n + 1 , then we can apply (Madsen et al., 1997, Lemma 7.6), a consequence of the
Tietze extension theorem, to show that Ψ: M → { 0 } n × R m − n can be extended to a diffeomorphism on the entire space, h : R m → R m .
Hence f ( x ) =
Ψ − 1 ◦ h ◦
R ( x ) for diffeomorphism Ψ − 1 ◦ h : R m → R m and zero-padding operator R : R n → R m , and thus f ∈ I ∞ ( K, R m ) .
This fact that for m sufﬁciently large compared to n such a diffeomorphism can always be extended is related to the fact that in 4-dimensions, all knots can be opened.
This can be contrasted with the case in Figure 2.
Now we present our proof.
Proof.
Let use next prove Eq.
when m ≥ 3 n + 1 .Let
[Equation]
be a C k map and M = f ( R n ) be an embedded submanifold of R m .
We have that m ≥ 3 n + 1 > 2 n + 1 .
Let S m − 1 be the unit sphere of R m and let
[Equation]
be the sphere bundle of R m that is a manifold of dimension 2 m − 1 .
By the proof’s of Whitney’s embedding theorem, by Hirsch, (Hirsch, 2012, Chapter 1, Theorems 3.4 and 3.5), there is a set of ‘problem points’ H 1 ⊂ S m − 1 of Hausdorff dimension 2 n such that for all w ∈ R m \ H 1 the orthogonal projection
[Equation]
has a restriction P w | M on M deﬁnes an injective map
[Equation]
Moreover, let T x M be the tangent space of manifold M at the point x and let us deﬁne another set of ‘problem points’ as
[Equation]
For w ∈ S m − 1 \ H 2 the map
[Equation]
m is an immersion, that is, it has an injective differential.
The sphere tangent bundle S M of M has dimension 2 n − 1 , and the set H 2 has the Hausdorff dimension at most 2 n − 1 .
Thus H
= H 1 ∪ Hhas Hausdorff dimension at most 2 n < m − 1 and hence the set S m − 1 \ H is non-empty.
For w ∈ S m − 1 \ H the map P w | M : M → { w } ⊥ is a C k injective immersion and thus
[Equation]
is a C k submanifold.
Let Z : P w ( M ) → M be the C k function deﬁned by
[Equation]
that is it is the inverse of P w | M :
M
→ P w ( M ) , where P w ( M ) ⊂ { w } ⊥ .
Let g : ˜ N = P w ( M ) → R be the function
[Equation]
Then ˜ N is a n -dimensional C k submanifold of ( m − 1) -dimensional
Euclidean space H = { w } ⊥ and g is a C k function deﬁned on it.
By deﬁnition of a C k submanifold of H , any point x ∈ ˜ N has a neighborhood U ⊂ H with local C k coordinates ψ : U → R m such that ψ ( ˜ N ∩ U ) =
( { 0 } m − 1 − n × R n ) ∩ ψ ( U ) .
Using these coordinates, we see that g can be extended to a C k function in U .
Using a suitable partition of unity, we see that there is a C k map G : { w } ⊥ → R that a C k extension of g that is, G | ˜ N = g .
Then the map
[Equation]
is a C k diffeomorphism of R m that maps M to m − 1 dimensional space { w } ⊥ , that is
[Equation]
In the case when m ≥ 3 n + 1 , we can repeat this construction n times.
This is possible as m − n ≥ 2 n + 1 .
Then we obtain C k diffeomorphisms Φ j : R m → R m , j = 1 , . . .
, n such that their composition Φ n ◦ · · · ◦ Φ 1 : R m → R m is a C k -diffeomorphism
such that which
[Equation]
where Y (cid:48)
⊂ R m is a m − n dimensional linear space.
By letting Ψ = Q ◦ Φ n ◦ · · · ◦ Φ 1 for rotation matrix Q ∈ R m × m , we have that Y :=
Q ( Y (cid:48) )
= { 0 } n × R m − n .
Also, letting φ : X = R n × { 0 } m − n → R m be the map
[Equation]
where f is the function given in Eq.
35, we have that φ : X → Y is a C k -diffeomorphism.
We observe that m − n ≥ 2 n +
and so we can apply (Madsen et al., 1997, Lemma 7.6) to extend φ to a C k -diffeomorphism
[Equation]
such that h | X = φ .
Note that (Madsen et al., 1997, Lemma 7.6) concerns an extension of a homeomorphism, but as the extension h is given by an explicit formula which is locally a ﬁnite sum of
C k functions, the same proof gives a C k -diffeomorphic extension h to a diffeomorphism φ .
This technique is called the ‘clean trick’.
Finally, to obtain the claim, we observe that when R : R n → R m , R ( x ) = ( x, 0) ∈ { 0 } n × R m − n is the zero padding operator, we have
[Equation]
As h | X = φ and R ( x ) ∈ X , this yields
[Equation]
that is,
[Equation]
where E =
Ψ − 1 ◦ h : R m → R m is a C k diffeomorphism.
Thus f ∈ I k
( R n , R m ) .
This proves
Eq. 9 when m ≥ 3 n + 1 .
C.2. T HE PROOF OF L EMMA
The proof of Lemma 3.
Let f = F 2 ◦ F 1 where F 2 ∈ F o,m and F 1 ∈ F n,o and (cid:15) > 0 be given, and let E o,m .
Clearly, B K,W ( f, E ) ≤ B K,W ( F 2 , E ) and so by the m, o, o MEP of E o,m with respect to F o,m , we have the existence of an r m ∈ emb( f ( K ) , E o,m ) such that (cid:107) I − r (cid:107) L ∞ ( f ( K )) < (cid:15) .
K o :=
( E o,m ) − 1 ◦ r ◦ f
( K ) is compact, hence E
o,m is Lipschitz on K o , so we can apply Lemmapoint 4, so
[Equation]
,m ◦ E p,o ) ≤ (cid:107) I − r (cid:107) L ∞ ( f ( K ))
+ Lip( E o,m ) B K,W (( E o,m ) − 1 ◦ r ◦ f, E p,o ) .
But, because f ∈ F o,m ◦F n,o , we can choose a E p,o ∈ E p,o 1 so that B K,W (( E o,m ) − 1 ◦ r ◦ f, E p,o ) ≤ (cid:15)
Lip( E o,m ) which, combined with Eqn. 36, proves the result.
C
T HE PROOF OF L EMMA
The proof of Lemma 4.
Recall that F ⊂ emb
( R n , R m ) .
Suppose that E o,mdoes not have the m, n, o MEP with respect to F , then there are some (cid:15) > 0 and f ∈ F so that
[Equation]
From Lemma 7 point 2, we have that
[Equation]
for all E p,o∈ E p,oand for all compact sets W ⊂ R p that satisfy E p,o 1 ( W 1 ) ⊂ W .
We observe that if W (cid:48) ⊂ R p is a compact set such that W (cid:48) ⊂ W , we have
[Equation]
Thus, inequality Eq.
38 holds for all E p,o∈ E p,oand for all compact sets W ⊂ R p .
Summarising, we have seen that there are f ∈ F and (cid:15) > 0 such that for all E p,o∈ E p,oand for all compact sets
W ⊂ R p we have (cid:15) ≤ B K,W ( f, E o,m◦ E p,o 1 ) .
Hence E o,mdoes not have the m, n, o MEP with respect to F , and we have obtained a contradiction, which proves the result.
C. U NIVERSALITY C.3. T HE PROOF OF T HEOREM
The proof of Theorem 1.
First we prove the claim under the assumptions (i). First we prove the claim under assumption (i).
Let W ⊂ R n be an open relatively compact set.
From Lemma 3 we have that
[Equation]
has the m, n, n MEP w.r.t.
F := F n L − 1 ,m L
[Equation]
n,n 1.
Thus for any (cid:15) 1 > 0 , we have an ˜ E ∈ E n,m s.t.
B K,W ( F, ˜ E ) <
(cid:15) 1 .
From Lemma 7 point 5, we have the existence of a µ (cid:48)
∈ P ( W )
so that W 2 (cid:16) F # µ, ˜ E # µ (cid:48) (cid:17) < (cid:15) 1 .
By convolving µ (cid:48) with a suitable molliﬁer φ , we can obtain a measure µ (cid:48)(cid:48)
= µ (cid:48) ∗ φ ∈ P ( W ) that is absolutely continuous with respect to the Lebesgue measure so that
[Equation]
, see (Ambrosio et al., 2008, Lemma 7.1.10.), and so W 2 (cid:16) ˜
E # µ (cid:48) , ˜ E
# µ (cid:48)(cid:48) (cid:17) < (cid:15) 1 .
Hence,
[Equation]
Next, from universality of T n 0 for any (cid:15) 2 > 0 , we have the existence of a T 0 ∈ T n 0 so that W
[Equation]
µ (cid:48)(cid:48) , T 0# µ ) < (cid:15) 2 .
From Lemma 7 points 3 and 4 we have that
[Equation]
[Equation]
For a given (cid:15) > 0 , choosing (cid:15) 1 < (cid:15) 4 and (cid:15) 2 < (cid:15) 2(1+Lip( ˜ E )) yields that the map E =
˜
E ◦ T 0 ∈
E is such that W 2 ( F # µ, E # µ ) < (cid:15) .
This yields the result.
Next we prove the claim under the assumptions (ii).
By our assumptions, in the weak topology of the space C 2 ( R n j , R n j ) , the closure of the set T n j ⊂ C 2 ( R n j , R n j ) contains the space of
Diff 2 ( R n j , R n j ) .
Moreover, by our assumptions R n j − 1 ,n j contains a linear map R .
We observe that as R n j − 1 ,n j is a space of expansive elements, the map R is injective.
and hecne by Lemma 1, the family
[Equation]
has the MEP w.r.t.
F = I 1 ( R n , R m ) .
By Lemma 2, we have that I 1 ( R n , R m ) coincides with the space emb 1 ( R n , R m ) .
Finally, by the assumption that T n 0is dense in the space of C 2diffeomorphism Diff 2 ( R n (cid:96) ) implies that T n 0is a L p -universal approximator for the set of C ∞ - smooth triangular maps for all p < ∞ .
Hence by Lemma 3 in Appendix A of Teshima et al.
T n 0is a distributionally universal.
From these the claim in the case (ii) follows in the same way as the case (i) using the family F = emb 1 ( R n , R m ) .
C.3. T HE PROOF OF L EMMA
The proof of Lemma 5.
The proof follows from taking the logical negation of the MEP for F .
If the
MEP is not satisﬁed, then there is some f ∈ F so that B K,W ( f, E ) is never smaller than (cid:15) > 0 for all E ∈ E .
Applying the deﬁnition of B K,W ( f, E ) from Eqn.
6 yields the result.
C.3. T HE PROOF OF C OR .
The proof of Cor. 1.
The proof of Eqn 12 follows from the deﬁnition of the MEP.
The proof of Eqn.
13 follows from applying Lemma 7 point 4, and applying the same arguments as in the proof of Theorem 1 so that lim i →∞ 1 Lip( E i ) B K,W ( E − 1 i ◦ r ◦ f, E (cid:48) i )
→ 0 , and so that
W 2 ( F # µ, E i ◦ E (cid:48)
i # µ (cid:48) )
= (cid:15) i where (cid:15) i → 0 as i → ∞ ..
From the distributional universality of T n 0 , and continuity of E i ◦ E (cid:48)
i we have the existence of a T i ∈ T n 0 so that W 2 ( F # µ, E i ◦ E (cid:48)
i ◦ T i # µ ) = 2 (cid:15) i .
Choosing (cid:15) i so that lim
i →∞ (cid:15) i yields the result.
A PPENDIX D A DDITIONAL P ROPERTIES D. L AYER - WISE P ROJECTION
Here we provide the details of our closed-form layerwise projection algorithm
The ﬂow layers are injective, and are often implemented to be numerically easy to invert.
Thus, the crux of the algorithm comes from inverting the injective expansive layers, R .
The range of the ReLU layer is piece-wise afﬁne, hence the inversion follows a two-step program.
First, identify which afﬁne piece (described algebraically, onto which sign pattern) to project.
Second, project to this point using a standard least-squares solver.
The second step is always straight-forward to analyze, but the ﬁrst is more complicated.
The key step in our algorithm is the fact that for the speciﬁc choice of weight matrix W = (cid:20) B − DB (cid:21) , given any y ∈ R 2 n , we can always solve the least-squares inversion problem exactly.
We prove this result in several parts given below.
For any y ∈ R 2 n , M y W ∈ R n × n is full-rank.
If [ y ] i (cid:54) =
[ y ] i + n for each i = 1 , . .
. , n , then the argmin in Eqn.
17 is well deﬁned, i.e. that there is a unique minimizer.
Otherwise there are 2 I minimizers, where I is the number of distinct i such that [ y ] i
=
[ y ] i + n .
If ˜ M y = (cid:2) ∆ y
( I n × n − ∆ y ) (cid:3) , then
[Equation]
We verify Eqn. 17.
The proof of Theorem 2.
Using the deﬁnition of M y , we have,
[Equation]
But, ( I n × n − ∆ y − ∆ y D ) is a full-rank diagonal matrix (with entries either 1 or [ D ] i,i ), and B is full rank by assumption, hence M y
[Equation]
is too.
Because B is square and full rank there exists a basis 5 (cid:110)
ˆ
b i (cid:111) i =1 ,...,n of R n such that
[Equation]
For an x ∈ R n , let α i
=
(cid:104)
x, b i (cid:105) for i = 1 , . . . , n be the expansion of x in the ˆ b i basis.
[Equation]
We now consider minizing Eqn. 46 by minimizing the basis expansion in terms of α i , n
[Equation]
[Equation]
Eqn. 47 is clearly minimized by minizing each term in the sum, hence we search for a minimizer of the i ’th term
[Equation]
Noting f
( α i ) as the quantity inside the minimum of Eqn.
48, we consider the positive, negative and zero α i cases of Eqn.
48 separately
and we get
[Equation]
If [ y ] i + n > [ y ] i , then the minimizer of Eqn.
48 is α i = −
[ y ] 2 i + n [ D ] ii < 0 .
Conversely if [ y ] i + n < [ y ] i then the minimzer of Eqn.
48 is α i =
[ y ] i > 0 .
This argument applies all i = 1 , . . . , n , and hence if [ y ] i (cid:54) =
[ y ] i +1 for all i = 1 , . . .
, n then the minimizing x is unique.
If [ y ] i =
[ y ] i +1 then there are exactly two minimizers of f ( α i ) , − [ y ] 2 i + n [ D ] ii
and [ y ] i , for both of which f
( α i ) =
[ y ] 2 i =
[ y ] 2 i + n .
If we suppose that [ y ] i + n − [ y ] i > 0 , then [ c ( y )] i =
and [ c ( y )] i + n > 0 , thus [∆ y ] ii = 1 , hence if we let x min be the minimizing x from part 1, then
[Equation]
If [ y ] i + n − [ y ] i ≤ 0 then we have
[Equation]
Thus combining Eqn.s 45, 46, 54 and 57 for each i = 1 , . .
. , n , we have that
[Equation] [Equation]
For the ﬁnal point, combining all of the above points we have
[Equation] [Equation]
Further we have from Point 1 that M y W is full rank, hence ( M y W ) − 1 M y y
= R †
( y ) is a minimizer of Eqn.
If [ y ] i (cid:54) =
[ y ] i + n for all i
= 1 , . .
. , n then Part 2 applies, and R † ( y ) is the unique minimizer of (cid:107) y − R ( x ) (cid:107) 22 .
In either case, we have that R † ( y ) is a minimizer.
D. B AYESIAN U NCERTAINTY Q UANTIFICATION
We now discuss the process for making the network amenable to Bayesian Uncertainty Quantiﬁcation (Bayesian UQ).
Assumption 2.
Let y 0 ∈ R n be given, let
y 1 =
T 0 y 0 and for each (cid:96) = 2 , . . .
, L , then let y (cid:96) :=
R (cid:96) ( y (cid:96) − 1 / 2 ) and y (cid:96) − 1 / 2 :=
T (cid:96) − 1 ( y (cid:96) − 1 ) .
We assume that for (cid:96) = 1 , . . .
, L − 1 , T (cid:96) ( y (cid:96) ) , R (cid:96) ( y (cid:96) − 1 / 2 ) and T 0 ( y 0 ) are differentiable.
Lemma 8.
If a network F of the form of Eqn. 1 satisﬁes Assumption 2, then we have the following upper bound on the log-likelihood,
[Equation]
Proof.
The proof of Lemma 8 follows from computing the log of,
[Equation]
and then applying deﬁnition 1 and expanding term using the laws of logarithms we obtain.
[Equation]
for (cid:96) = 1 , . . .
, L , see (Kothari et al., 2021, Appendix B) for details.
We note here that in Equation 60 we have an inequality rather than an equality.
The inequality comes from the expansive layers, as the ﬂow-layers are all endomorphisms, i.e. R n (cid:96) → R n (cid:96) .
We now remark on the ease of approximating, or at least bounding, T and R in Eqn. 62.
Values for R when R is (R1) or (R2) are straightforward to compute or approximate 6 .
When R n,m is (R3), R ∈ R are not differentiable, but are differentiable a.e..
Further, ∇ R (cid:96) ( x n (cid:96) ) can a.e. be computed by applying a simple mask 0 and 1 mask to the weight matrices for an exact computation of R .
Additionally, from Hutchinson (1989)
; Kothari et al.
we have the following formula that approximates R ,
[Equation]
where η ∈ R n (cid:96) .
Computation of the T term is often easier, comparatively.
This is because these networks are often designed to make this value as easy to compute as possible.
For example the
NICE Dinh et al.
, Real-NVP Dinh et al.
(2016), and GLOW Kingma & Dhariwal (2018)
architectures are all designed so that T can be computed form a short scalar product.
D. B LACK - BOX RECOVERY
We now discuss assumptions that enable black-box recovery of the weights of our entire network post-training.
Assumption 3.
For each (cid:96) = 1 , . . .
, L , R (cid:96) is an afﬁne ReLU layer.
Each T (cid:96) and T 0 is constructed from a ﬁnite number of afﬁne ReLU layers.
Remark 3 .
If a network F of the form of Eqn. 1 satisﬁes Assumption 3, then given the range of the network, the range of the network can be recovered exactly.
Further, if the linear region assumption from Rolnick & K¨ording (2020) is satisﬁed, then the exact weights are recovered, subject to two natural isometries discussed below.
Remark 4 .
The ReLU part of Assumption 3 is for all examples in Sec. 2.1.
Further it is also satisﬁed by both ﬂows considered in Sec. 2.2, provided that the various g i are given by layers of afﬁne ReLU ’s.
In Rolnick & K¨ording (2020), the authors show that, although ReLU networks depend on the value of their weight matrix in non-linear ways, it is still possible to recover the exact weights of a given
ReLU network in a black-box way, subject to natural isometrics.
The authors show that this is possible not only in theory, but in numerical applications as well.
The works of Rolnick & K¨ording (2020); Bui Thi Mai & Lampert (2020) imply that provided the activation functions of the expressive elements are ReLU then the entire network can be recovered in a black-box way.
Further, provided that either the ‘linear region assumption’ from Rolnick & K¨ording (2020) or the generality assumption from Bui Thi Mai & Lampert (2020) is satisﬁed, then the entire network can be recovered uniquely modulo the natural isometries of rescaling and permutation of weight matrices.
First we describe the two natural isometries of scaling and permutation.
Consider the following function
[Equation]
where φ is coordinate-wise homogeneous degree 1 (such as ReLU ) and W 1 ∈ R n 1 × n 2 and W 2 ∈ R n 2 × n 3 .
If we let P ∈ R n 2 × n 2 be any permutation matrix, and D + be a diagonal matrix with strictly positive elements, then we can write
[Equation]
as well.
Thus ReLU networks can only ever be uniquely given subject to these two isometries.
When describe unique recovery in the rest of this section, we mean modulo these two isometries.
In Rolnick & K¨ording (2020), the authors describe how all parameters of a ReLU network can be recovered uniquely (called reverse engineered in Rolnick & K ¨ ording (2020)), subject to the so called ‘linear 7 region assumption’, LRA.
The input space R n can be partitioned into a ﬁnite number of open {S i } n i
i =1 , where for each k , F ( x ) =
W k
i + b i , i.e. the network corresponds to an afﬁne polyhedron in the output space.
The algorithms (Rolnick & K¨ording, 2020, Alg.s 1 & 2) are roughly described below.
First, identify at least one point within each afﬁne polyhedra {H j } n j j =1 .
Then identify the boundaries between polyhedra.
The boundaries between sections are always one afﬁne ‘piece’ of piecewise hyperplanes {H j } n j j =1 .
These {H j } n j j =1 are the central objects which indicate the (de)activation of an element of a ReLU somewhere in the network.
If the H j are full hyperplanes, then the ReLU that is (de)activates occurs in the ﬁrst layer of the network.
If H j is not a full hyperplane, then it necessarily has a bend where it intersects another hyperplane H j (cid:48) .
Further, except for a Lebesgue measure 0 set, when H j intersects H j (cid:48)
the latter does not have a bend.
If this is the case, then
H j (cid:48) corresponds to a ReLU (de)activation in an earlier layer than H j .
In this way the activation functions of every layer can be deduced.
Once this is done, the normals of the hyperplanes can be used to infer the row-vectors of the various weight matrices, letting one recover the entire network.
The above algorithm recovers all of the weights exactly provided that the LRA is satisﬁed.
The
LRA is satisﬁed if for every distinct S i and S i (cid:48) , either W i (cid:54)
=
W i (cid:48) or b i (cid:54) =
b i (cid:48) .
That is, different sign patterns produce different afﬁne sections in the output space.
This is a natural assumption, as the algorithm as described above reconstruction works by ﬁrst detecting the boundaries between adjacent afﬁne polyhedra, which is only possible if the LRA holds.
Given the weights of a network there is currently no simple way to detect if the LRA is satisﬁed, to our knowledge.
Nevertheless the authors of Rolnick & K¨ording (2020) show that if it is satisﬁed, then unique recovery follows.
Nevertheless recovery of the range of the entire network is possible, but this recovery may not be unique.
In Bui Thi Mai & Lampert (2020) the authors also consider the problem of recovering weights of a
ReLU neural network, however the authors therein study the question of when there exist isometries beyond the two natural ones described above.
In particular the main result (Bui Thi Mai & Lampert, 2020, Theorem 1) shows the following.
Let E n 0 ,n L be a ReLU network that is L layers deep and non-increasing.
Suppose that E 1 , E 2 ∈ E n 0 ,n L , E 1 and E 2 are general 8 and for all x ∈ R n
E 1 ( x ) =
E 2 ( x ) , then E 1 is parametrically identical to E 2 subject to the two natural isometries.
This work provides the stronger result, however does not apply to the networks that we consider out of the box.
It does apply to our expressive elements (provided that they use ReLU activation functions, and are non-increasing), but not necessarily apply to the network on the whole.