U NIVERSAL J OINT A PPROXIMATION OF M ANIFOLDS AND D ENSITIES BY S IMPLE I NJECTIVE F LOWS
Anonymous authors
Paper under double-blind review
A BSTRACT
We analyze neural networks composed of bijective ﬂows and injective expansive elements.
We ﬁnd that such networks universally approximate a large class of manifolds simultaneously with densities supported on them.
Among others, our results apply to the well-known coupling and autoregressive ﬂows.
We build on the work of Teshima et al.
2020 on bijective ﬂows and study injective architectures proposed in Brehmer et al.
2020 and Kothari et al.
Our results leverage a new theoretical device called the embedding gap, which measures how far one continuous manifold is from embedding another.
We relate the embedding gap to a relaxation of universally we call the manifold embedding property, capturing the geometric part of universality.
Our proof also establishes that optimality of a network can be established “in reverse,” resolving a conjecture made in Brehmer et al.
2020 and opening the door for simple layer-wise training schemes.
Finally, we show that the studied networks admit an exact layer-wise projection result,
Bayesian uncertainty quantiﬁcation and black-box recovery of network weights.
I NTRODUCTION
In the past several years, invertible ﬂow networks emerged as powerful deep learning models to learn maps between distributions (Durkan et al., 2019a; Grathwohl et al., 2018; Huang et al., 2018;
Jaini et al., 2019; Kingma et al., 2016; Kingma & Dhariwal, 2018;
Kobyzev et al., 2020; Kruse et al., 2019; Papamakarios et al., 2019).
They generate excellent samples (Kingma & Dhariwal, 2018) and facilitate solving scientiﬁc inference problems (Brehmer & Cranmer, 2020; Kruse et al., 2021).
By design, invertible ﬂows are bijective and, hence, may not be a natural choice when the target distribution has low-dimensional support.
This problem can be overcome by combining bijective ﬂows with expansive, injective layers, which map to higher dimensions (Brehmer & Cranmer, 2020;
Cunningham et al., 2020; Kothari et al., 2021).
Despite their empirical success, the theoretical aspects of such globally injective architectures are not well understood.
In this work, we present approximation-theoretic properties of injective ﬂows, the architecture of which alternates between bijective ﬂows and expansive, injective layers, with an emphasis on approximating measures with low-dimensional support.
We state conditions under which these networks are universal approximators and describe how their design enables applications to inference and inverse problems.
1. P RIOR W ORK
The idea of combining ﬂow layers with expansive layers has been explored by Brehmer & Cranmer (2020) and Kothari et al.
In Brehmer & Cranmer (2020) the authors combine two ﬂow networks and a simple expansive element, and obtain a network they argue may be universal.
We discuss the connection between our works in more detail in Appendix B.1.
In Kothari et al.
the authors show that the ideas of Brehmer & Cranmer (2020) can be taken further by using more than two ﬂow layers and more than one expansive element.
The authors further observe that the resulting network is faster to train as compared to Brehmer & Cranmer (2020), and still produces high-quality output on a variety of standard machine learning and inverse problems tasks.
Unlike for the injective case, there is considerable work on universality of invertible ﬂows; see
Kobyzev et al.
(2020) for an overview.
Several works show that certain bijective ﬂow architectures are distributionally universal, e.g., autoregressive ﬂows with sigmoidal activations (Huang et al., 2018) or
sum-of-squares polynomial ﬂows (Jaini et al., 2019).
Teshima et al.
(2020) show that several ﬂow networks including those from Huang et al.
; Jaini et al.
(2019) are also universal approximators of diffeomorphisms.
The injective ﬂows considered here, have key applications in inference and inverse problems.
For a general overview on deep learning and data-driven inverse problems, we refer to Arridge et al.
Bora et al.
(2017) proposed the recovery in the range of (pre-trained) generative models in the context of compressed sensing.
Using particular injective ﬂows as generative models, we can ensure that we have a projection onto their ranges guarantee.
This projection enables the implementation reconstruction algorithms.
One may also express an inverse model as a conditional probability, approximate it by a tractable model, while taking advantage of the possibility to create an arbitrary amount of training data from a known forward model and a suitable prior.
Ardizzone et al.
(2018) encode the forward operator using a invertible neural network (INN).
In their work, INNs are employed for learning the forward operator, using additional latent output variables to capture the information otherwise lost.
Due to invertibility, a model of the corresponding inverse process is learned implicitly.
Given a speciﬁc measurement and the distribution of the latent variables, the inverse pass of the INN provides the full posterior over parameter space.
It is noted that any changes to the acquisition protocol or instrumentation may require retraining, however.
Sun & Bouman (2020) propose variational approximation to the posterior employing an untrained deep generative model.
They train a normalizing ﬂow which produces samples from the posterior, with the prior and the noise model given implicitly by the regularized misﬁt functional.
In Kothari et al.
this concept is generalized to injective ﬂows.
1. O UR C ONTRIBUTION
We derive new approximation results for neural networks composed of bijective ﬂows and injective expansive layers, including those introduced by Brehmer & Cranmer (2020) and Kothari et al.
We show that these networks universally jointly approximate a large class of manifolds and densities supported on them.
We build on the results of Teshima et al.
and develop a new theoretical device which we refer to as the embedding gap.
This gap is a measure of how nearly a mapping from R o
→ R m embeds an n -dimensional manifold in R m , where n ≤ o .
We ﬁnd a natural relationship between the embedding gap and the problem of approximating probability measures with low-dimensional support.
We then relate the embedding gap to a relaxation of universality we call manifold embedding property .
We show that this property captures the essential geometric aspects of universality and uncover important topological restrictions on the approximation power of these networks, to our knowledge, heretofore unknown in the literature.
We establish universality for manifolds with suitable topology.
Our proof shows that optimality of the approximating network can be established in reverse: optimality of a given layer can be established without optimality of preceding layers.
This settles a (generalization of a) conjecture posed for a two-layer case in (Brehmer & Cranmer, 2020).
Finally, we show that these universal architectures are also practical and admit exact layer-wise projections, as well as other properties discussed in Section 3.4.
D ESCRIPTION OF THE A RCHITECTURE
Let C ( X, Y ) denote the space of continuous functions
X → Y .
We study networks in F ⊂ C
( X, Y ) that are of the form:
[Equation]
where L ∈ N is the number of networks, n 0 = n , n L = m , and n (cid:96) ≥ n (cid:96) − 1 for (cid:96) = 1 , . .
, L .
We introduce a well-tuned shorthand notation and write H ◦ G :=
{ h ◦ g : h ∈ H , g ∈ G} throughout the paper.
We identify R with the expansive layers and T with the bijective ﬂows.
Loosely speaking, the purpose of the expansive layers is to allow the network to parameterize high-dimensional functions by low-dimensional coordinates in an injective way.
The ﬂow networks give the network the expressivity necessary for universal approximation of manifold-supported distributions.
2. E XPANSIVE L AYERS
The expansive elements transform an n -dimensional manifold M embedded in R n (cid:96) − 1 , and embed it in a higher dimensional space R n (cid:96) .
To preserve the topology of the manifold, this must be done injectively.
We thus make the following assumptions about the expansive elements:
Deﬁnition 1 (Expansive Element) .
Let (cid:96) = 1 , . . .
, L , and R n (cid:96) − 1 ,n (cid:96) (cid:96) be a family of functions from
[Equation]
is a family of expansive elements if every R ∈ R n (cid:96)
[Equation]
1 ,n
[Equation]
is injective and Lipschitz.
Examples of expansive elements include
(R1)
Zero padding: R ( x ) = (cid:2) x T , 0 ( m − n ) (cid:3) T where 0 ( m − n ) is the zero vector (Brehmer & Cranmer, 2020).
(R2) Multiplication by an arbitrary full-rank matrix, or one-by-one convolution: R ( x ) =
Wx, or R ( x ) = w (cid:63) x
2) where W ∈ R m × n and rank( W ) =
n (Cunningham et al., 2020), and w is a convolution kernel (cid:63) denotes convolution Kingma & Dhariwal (2018).
(R3)
Injective ReLU layers: R = ReLU( Wx ) , W = (cid:2) B T , − DB T , M T (cid:3) T , R ( x )
= ReLU (cid:0)(cid:2) w T , − w T (cid:3) (cid:63) x (cid:1) for matrix B ∈ GL n ( R ) , positive diagonal matrix D ∈ R n × n , and arbitrary matrix M ∈ R ( m − 2 n ) × n (Puthawala et al., 2020).
(R4) Injective ReLU networks (Puthawala et al., 2020, Theorem 15).
These are functions R : R n → R m of the form R ( x ) =
W L +1 ReLU( . . .
ReLU( W 1 x + b 1 ) . . . )
+ b L where W (cid:96) are n (cid:96) +1 × n (cid:96) matrices and b (cid:96) are the bias vectors in R n (cid:96) +1 .
The weight matrices W L satisfy the Directed Spanning Set (DSS) condition for (cid:96) ≤ L (that make all layers injective) and W L +1 is a generic matrix which makes the map R : R n → R m injective.
Note that the DSS condition requires that n (cid:96) ≥ 2 n (cid:96) − 1 + 1 for (cid:96) ≤ L and we have n 1 = n and n L +1 = m.
2. B IJECTIVE F LOW N ETWORKS
The bulk of our theoretical analysis is devoted to expressive elements.
The expressive elements bend the range of the expansive elements into the correct shape.
We make the following assumptions about the expressive elements:
Deﬁnition 2 (Bijective Flow Network) .
Let (cid:96) = 0 , . . .
, L be given and let T n (cid:96) (cid:96)
[Equation]
T n
[Equation]
is a family of bijective ﬂow networks if every T ∈ T n (cid:96) (cid:96) is Lipschitz continuous and bijective.
Examples of bijective ﬂow networks include
(T1)
Coupling ﬂows , introduced by Dinh et al.
2014) consider R ( x ) =
H k ◦ · · · ◦ H 1 ( x ) where
[Equation]
In Eqn. 3, h i : R d × R e → R d is invertible w.r.t.
the ﬁrst argument given the second, and g i : R n − d → R e is arbitrary.
Typically in practice the operation in Eqn. 3 is combined with additional invertible operations such as permutations, masking or convolutions Dinh et al.
(2014; 2016); Kingma & Dhariwal (2018).
(T2) Autoregressive ﬂows , introduced by Kingma et al.
(2016) are generalizations of triangular ﬂows
A : R n → R n where for i = 1 , . .
. , n the i ’th value of A is given by of the form
[Equation]
In Eqn. 4, h i : R × R m → R where again
h
i is invertible w.r.t.
the ﬁrst argument given the second, and g i : R i − 1 → R m is arbitrary except for g 1 = 0 .
In Huang et al.
, the authors choose h i ( x, y ) ,
where y ∈ R m , to be a multi-layer perceptron (MLP) if the form
[Equation]
where φ is a sigmoidal increasing non-linear activation function.
M AIN R ESULTS 3. E MBEDDING G AP
We call a function f an embedding and denote it by f ∈ emb( X, Y ) if f : X → Y is continuous, injective, and f − 1 : f ( X ) → X is continuous 1 . Also we denote emb k ( R n , R m )
= emb( R n , R m ) ∩ C k ( R n , R m ) . In order to set up our result concerning embedding of manifolds, we ﬁrst need a way to measure the degree to which a mapping g ∈ emb( R o , R m ) nearly embeds a manifold M = f ( K ) for compact K ⊂ R n and f ∈ emb( K, R m ) .
With this in mind we introduce embedding gap B K,W ( f, g ) , a non-symmetric notion of distance between f and g .
Later in the paper, f will be the function to be learned, and g a ﬂow-network to be trained.
Deﬁnition 3 (Embedding Gap) .
Let K ⊂ R n be compact and non-empty, W ⊂ R
o contain the closure of set U which is open in the subspace topology of some vector subspace V of dimension p , where n ≤ p ≤ o ≤ m , f ∈ emb( K, R m ) , and
g ∈ emb( W, R m ) .
Then we deﬁne the Embedding
Gap between f and g on sets K and W as
[Equation]
where I : f ( K ) → f ( K ) is the identity function and (cid:107) h (cid:107) L ∞ ( X )
=
ess sup x ∈ X (cid:107) h ( x ) (cid:107) 2 for h : X → Y .
We refer to the embedding gap between f and g without specifying K and W when it is clear from context.
Remark 1 .
As W ⊂ R o contains U , an open set in V , there is an afﬁne map A : R n → V such that
A ( K )
⊂ W .
Then, the map r 0 = g ◦ A ◦ f − 1 : f ( K ) → g ( W ) is an injective continuous map from a compact set to its range and
hence r 0 ∈ emb( f ( K ) , g ( W )) .
Thus, in the above inﬁmum the set emb( f ( K ) , g ( W )) is non-empty.
Before giving properties of B K,W ( f, g ) , we brieﬂy describe its interpretation and meaning.
We denote by P ( X ) the set of probability measures over X .
If the embedding gap between two functions is small, then g is nearly an embedding of the range of f into R o .
B K,W ( f, g ) is constructed expressly to serve as an upper bound
[Equation]
, g ) where µ n ∈ P ( K ) is given, and W 2 ( ν 1 , ν 2 ) denotes the Wasserstein-2 distance with (cid:96) 2 ground metric (Villani, 2008), as shown in Lemma 7 part 5.
The above result has a simple meaning in the context of machine learning.
Suppose we want to learn a generative model g to (approximately) sample from a probability measure ν with low-dimensional support, by applying g to samples from a base distribution µ o .
Suppose further that ν is a pushforward of some (known or unknown) distribution µ n via f .
The embedding gap B K,W ( f, g ) then upper bounds the 2-Wasserstein distance between ν and g # µ 0 for the best possible choice of µ o .
The embedding r can be interpreted as a candidate transport map from any measure pushed forward by f , that can be pulled back through g .
Loosely speaking, for µ (cid:48)
o = g − 1 ◦ r ◦ f # µ n , r is a valid Wasserstein transport map that transports f # µ n to g # µ (cid:48) o with cost no more than (cid:107) I − r (cid:107) L ∞ ( f ( K )) .
See Fig.
1 for a visualization of the embedding gap between two toy functions.
Further, the embedding gap satisﬁes inequalities useful for studying networks of the form of Eqn. 1, see Lemma 7.
3. M ANIFOLD E MBEDDING P ROPERTY
We now introduce a central concept, the manifold embedding property (MEP).
A family of networks satisﬁes the MEP if it nearly embeds a large class of manifolds of certain dimension and regularity, as measured by the embedding gap.
The MEP is a property of a family of functions
E o,m ⊂ emb( W, R m ) where W ⊂ R o .
For this manuscript, E o,m will always be formed by taking
E o,m :=
T m ◦ R o,m .
We note here that the MEP is closely related to the question of whether or not a given n -dimensional manifold M that is an image of an embedding f : K → R m , that is, M = f ( K ) , K ⊂ R n , can be approximated by the images of neural networks E : K
→ R m of a given type.
In particular, we will consider neural networks that are compositions where R : R o
→ R m are ‘simple’ non-universal functions, and ﬂow-maps T : R m → R m that are diffeomorphism.
This choice of applying nonuniversal expansive layers, and then diffromorphisms has some topological consequences, which we discuss below.
T OPOLOGICAL OBSTRUCTIONS TO MANIFOLD LEARNING WITH NEURAL NETWORKS
We illustrate the problem when n = 2 , m = 3 , and K = S 1 ⊂ R 2 is the circle.
We consider maps
E =
T ◦ R : R n → R m where R : R n → R m is, e.g., a linear map of rank n , and T : R m → R m is a coupling ﬂow which is a homeomorphism.
Let f ∈ emb
( K, R 3 ) be an embedding that maps K to a trefoil knot M =
f ( S 1 ) , see Fig.
Such a function f can not be written as a restriction T ◦ R on S 1 .
In Sec.
C.2.2 we prove this fact and leverage it to build a compactly supported measure µ ∈ P ( R 2 ) for which there are no E :=
T ◦ R for linear injective R and bijection T such that f # µ =
E # µ .
To sidestep this fundamental difﬁculty, we deﬁne the MEP property for a certain subclass of manifolds { f ( K ) : f ∈ F} .
Finally, when considering ﬂow networks which are universal approximators of C 2 diffeomorphisms, we restrict the class of manifolds to be approximated even further.
This is because manifolds that are homeomorphic are not necessarily diffeomorphic; for example exotic spheres.
These are topological structures that are homeomorphic, but not diffeomorphic, to the sphere Milnor (1956).
Moreover, it is known that general homeomorphisms F : R m → R m can not be approximated in C 0 -topology by C 2 -smooth diffeomorphisms T : R m → R m .
See M¨uller (2014) for a precise statement.
However, all C 1 -smooth diffeomorphims F : R m → R m can be approximated in the strong topology of C 1 by C 2 -smooth diffeomorphims ˜ F : R m → R m , (cid:96) ≥ k , [Figure] Hirsch (
Because of this, have to pay attention to the
For these reasons, when we refer to the MEP, we consider it with respect to a class of functions F ⊂ emb( R n , R m ) .
The MEP can be interpreted as a density statement, saying that our networks E o,m are dense in the ‘ B K,W distance’ in the set F ⊂ emb( R n , R m ) .
Two examples we are particularly interested in are the following.
First, when F = Φ ◦ A where A : R n → R m are linear maps of rank n and Φ : R m → R m are C k -smooth diffeomorphisms with k ≥ 1 .
Second, when F ∈ emb( R n , R m ) .
Deﬁnition 4.
Let E o,m ⊂ emb
( R o , R m ) and F n,m ⊂ emb
( R n , R m ) be two families of functions.
We say that E o,m has the m, n, o Manifold Embedding Property (MEP) w.r.t.
F n,m if for every compact set K ⊂ R n , f ∈ F n,m , and (cid:15) > 0 , there is an E ∈ E
o,m and a compact set W ⊂ R o such that the restriction of f in K and the restriction of E in W satisfy
[Equation]
When it is clear from the context, we abbreviate the m, n, o MEP w.r.t.
F n,m simply by the m, n, o
MEP, or simply the MEP.
We also note here that if a model from R o → R m is a uniform universal approximator of F o,m =
C 0 ( R n , R m ) on compact sets, such as those considered in Yarotsky (2017; 2018), then it has the m, n, o MEP w.r.t F n,m for any n ≤ o .
Thus, networks that are uniform universal approximators automatically possess the MEP, as shown in Lemma 1.
However, we have the following result.
Deﬁnition 5 (Uniform Universal Approximator) .
For a non-empty subset F n,m ⊂ C ( R n , R m ) , a family E n,m ⊂ C ( R n , R m ) is said to be a uniform universal approximator of F n,m if for every f ∈ F n,m , every non-empty compact K ⊂ R n , and each (cid:15) > 0 , there is an E ∈ E n,m satisfying:
[Equation]
Let I ( R n , R m ) be the set of functions F : R n → R m that are compositions F =
Φ ◦ R where
R : R n → R m is a linear map of rank n and Φ : R m → R m is a C 1 -smooth diffeomorphism.
We call I ( R n , R m ) be the set of extendable embeddings. Also we denote I k ( R n , R m )
=
I ( R n , R m ) ∩ C k ( R n , R m ) . Observe that I ( R n , R m ) ⊂ emb( R n , R m ) .
Lemma 1.
(i)
If R o,m is a universal approximator of C ( R n , R m ) and I ∈
T m where I is the identity map, then E o,m :=
T m ◦ R o,m has the MEP w.r.t.
emb( R n , R m ) .
(ii)
If R o,m is such that there is an injective R ∈ R o,m and open set U
⊂ R o such that R (cid:12)(cid:12)
U is linear, and T m is a sup universal approximator in the space of Diff 2 ( R m , R m ) , in the sense of Teshima et al. (2020), of the C 2 -smooth diffeomorphisms, then E o,m :=
T m ◦ R o,m has the MEP w.r.t.
F = I ( R n , R m ) .
Proof of Lemma 1 is in Appendix C.2.1.
Lemma 1 has the following implications for the architectures studied in Section 2.
Example 1.
Let E :=
T m ◦ R o,m , then
(i)
If T m is either (T1) or (T2) and R o,m is (R4), then E o,m has the m, n, o MEP w.r.t.
emb( R n , R m ) .
(ii)
If T m is (T2) with sigmoidal activations Huang et al.
(2018), then if R o,m is any of (R1), ..., (R4), then E o,m has the m, n, o MEP w.r.t.
I ( R n , R m ) .
Proof of Example 1 is in Appendix C.2.3.
We add a remark here showing that if m is sufﬁciently larger than n , and subject to some regularity assumptions, The two classes considered in Example 1 coincide. Lemma 2. When m ≥ 3 n + 1 and k ≥ 1 , for any C k -any embedding f ∈ emb k ( R n , R m ) and compact set K ⊂ R n , there is a map in the closures of the ﬂow type neural network E ∈ I k ( R n , R m ) such that E ( K ) =
f ( K ) .
Moreover,
[Equation]
This means that the topological structure of a data set M in R m is diffeomorphic to the set K ⊂ R n , and thus some ﬂow type network can approximate this data set M .
Proof of Lemma 2 in Appendix C.2.4.
We ﬁnally have two more lemmas to present before moving on to our discussion of universality as it ties to the MEP.
Lemma 3.
Let F n,o ⊂ emb( R n , R o ) and F o,m ⊂ emb( R 0 , R m ) E p,o⊂ emb( R p , R o ) have the o, n, p MEP w.r.t. F n,o ⊂ emb( R n , R o ) and E o,m⊂ emb( R o , R m ) have the m, o, o MEP w.r.t. F o,m ⊂ emb
( R o , R m ) .
If each E o,m∈ E o,mis locally Lipschitz, then E o,m◦ E p,ohas the m, n, p MEP w.r.t.
F o,m ◦ F n,o .
Proof of Lemma 3 is in Appendix C.2.5.
We note that in the case that the elements of E o,mare differentiable, local Lipschitzness is automatic, and needn’t be assumed, see e.g. (Tao, 2009, Ex. 10.2.6).
We also record a near-converse of Lemma 3 that shows that if E o,m◦ E p,ohas the m, n, p MEP, then E o,mhas the m, n, o MEP.
Lemma 4.
Let F ⊂ emb( R n , R m ) .
Let E p,o⊂ emb( R p , R o ) and E o,m⊂ emb( R o , R m ) be such that E o,m◦E p,ohas the m, n, p MEP with respect to family F .
Then E o,mhas the m, n, o MEP with respect to family F .
Proof of Lemma 4 is in Appendix C.2.6.
3. U NIVERSALITY
Here, we present our universal approximation result for networks given in Eqn. 1, and a decoupling property.
Theorem 1 (Qualitative Universality for Embeddings) .
Let n 0 = n, n L = m , µ ∈ P ( K ) be an absolutely continuous measure w.r.t.
Lebesgue measure.
Further let, for each (cid:96) = 1 , . . . , L , E [Equation]
where R n (cid:96)
[Equation]
1 ,n (cid:96) (cid:96) is a family of injective expansive elements that contains a linear map, and T n (cid:96) (cid:96) is a family of bijective family networks.
Finally let T n 0 be distributionally universal, i.e. for any absolutely continuous µ ∈ P ( R n ) and ν ∈ P ( R n ) , there is a { T i } ∞ i =1 such that T i # µ → ν in distribution. Suppose that one of the following two cases hold. (i) Let F ∈ F n L − 1 ,m L ◦· · ·◦F n,n 1and E n (cid:96) − 1 ,n (cid:96) (cid:96) have the the n (cid:96) , n, n (cid:96) − 1 MEP for (cid:96) = 1 , . . .
, L with respect to F n (cid:96) − 1 ,n (cid:96) (cid:96) .
(ii) Let F ∈ emb 1 ( R n , R m ) be a C 1 -smooth embedding, and assume that for (cid:96) = 1 , . . . , L it holds that n (cid:96) ≥ 3 n (cid:96) − 1 + 1 and the families T n (cid:96) (cid:96) are dense in the space of C 2 -diffeomorphism Diff 2 ( R n (cid:96) ) .
Then, there is a sequence of { E i } i =1 ,..., ∞ ⊂
E n L − 1 ,m L ◦ · · · ◦ E n 1 ,n◦ T n 0 such that [Equation]
[Equation]
Proof of Theorem 1 is in Appendix C.3.1.
As discussed in above and in Figure 2, there are topological obstructions for Theorem 1 with a general embedding F : R n → R m . When n = 2 , m = 3 , L = 1 , and µ is the uniform measure on an annulus K ⊂ R 2 target measure F # µ is the uniform measure on a knotted ribbon M = F ( K ) ⊂ R 3 . There are no injective linear maps
R : R 2 → R 3 and diffeomorphisms T : R 3 → R 3 such that E
= T ◦ R would satisfy M = E ( K ) and E
# µ = F # µ .
This happen even though the set of diffeomorphism T : R 3 → R 3 are universal distributional approximators.
In this case the condition m = n 1 ≥ 3 n + 1 is not valid.
We remark here that our networks are designed expressly to approximate manifolds, and hence injectivity is key.
This separates our results from, e.g. (Lee et al., 2017, Theorem 3.1) or (Lu & Lu, 2020, Theorem 2.1), where universality results of ReLU networks are also obtained.
The previous theorem shows that the entire network is universalif it can be broken into pieces that have the MEP.
The following lemma shows that if E n,m = H o,m ◦ G n,o , then H o,m must have the m, n, o MEP if E n,m is universal.
Lemma 5.
Suppose that E n,m =
H
o,m ◦ G n,o where E n,m ⊂ emb( R n , R m ) , H o,m ⊂ emb
( R o , R m ) , and G n,o ⊂ emb( R n , R o ) .
If H o,m does not have the m, n, o MEP w.r.t.
F , then there exists a f ∈ F , compact K ⊂ R n and (cid:15) > 0 such that for all E ∈ E n,m , and r ∈ emb( f ( K ) , E ( W ))
[Equation]
Proof of Lemma 5 is in Appendix C.3.2.
The proof of Theorem 1 also has the following implication which, loosely speaking, says that later layers decouple from earlier ones.
That is, given a sequence of functions that has the MEP on the last
L − ˜ (cid:96) layers, there are always functions in the ﬁrst ˜ (cid:96) layers so that the entire network is end-to-end optimal.
Corollary 1.
Let F n,o ⊂ emb( R n , R o ) , F o,m ⊂ emb
( R o , R m ) , and let E o,m ⊂ emb ( R o , R m ) have the m, n, o MEP w.r.t. F o,m ◦ F n,o .
For every F ∈ F o,m ◦ F n,o then there is a compact K ⊂ R n and { E i } ∞ i =1 ⊂ E o,m such that
[Equation]
Further, if E n,o ⊂ emb( W (cid:48) , R o ) has the o, n, n MEP w.r.t.
F n,o , and T n is a universal approximator for distributions, then for any µ ∈ P ( K ) where K ⊂ R n is compact, there is a sequence { E (cid:48) i } i =1 ,..., ∞ ⊂ E n,o and { T i } i =1 ,..., ∞ ⊂ T n so that
[Equation]
Proof of Cor.
1 is in Appendix C.3.3.
Typically results about neural network design are in terms of the network end-to-end.
Cor. 1 is notable as it shows that the layers of approximating networks can be built ‘one layer at a time.’
Cor.1 is related to an observation made in (Brehmer & Cranmer, 2020, Section B), concerning training strategies, speciﬁcally the section on separate manifold and density training.
In that section the authors remark that they ‘expect faster and more robust training of a network’ of the form in Eqn.
when L = 1 , that is F = T m 1 ◦ R n,m◦ T n 0 .
Cor.
1 shows that there exists a minimizing sequence in T m 1 that need only minimize Eqn. 12, and then the T n 0 layers can be minimized after.
Further Lemma 5 and Cor.
1 combine to further imply that not only can the network from Brehmer & Cranmer (2020) be minimized layerwise, but that any universal network can necessarily be minimized layerwise in reverse order, provided that it can be written as the product of two smaller layers.
3. L AYER - WISE INVERSION , UNCERTAINTY QUANTIFICATION AND RECOVERY OF WEIGHTS
In this subsection, we describe how our network can be augmented with more useful properties if the architecture satisﬁes a few more assumptions without affecting universal approximation.
We focus on a new layerwise projection result, with a further discussion of Bayesian uncertainty quantiﬁcation, and black-box recovery of our network’s weights in Appendices D.2 & D.3.
Given a point y ∈ R m that does not lie in the range of the network, projecting y onto the range of the network is a practical problem without an obvious answer.
The crux of the problem is inverting the injective (but non-invertible) R layers when R .
When R contains only full-rank matrices as in (R1) or (R2) then we can compute a least-squares solution.
If, however, R contains layers which are only piecewise linear, as in (R3), then the problem of computing a least squares solution is more difﬁcult, see Fig.
Nevertheless, we ﬁnd that if R is (R3) we can still compute a least-squares solution.
Assumption 1.
Let R be given by one of (R1) or (R2), or else (R3) when m = 2 n .
If R only contains linear operators, then the least-squares problem can be computed by solving the normal equations (see (Golub, 1996, Section 5.3).)
This includes cases (R1) or (R2).
For (R3) we have the following result.
Deﬁnition 6.
Let W = (cid:2) B t − DB t (cid:3) t ∈ R 2 n × n and y ∈ R 2 n be given, and let R ( x ) =
ReLU( Wx ) .
Then deﬁne
[Equation]
where the max in Eqn.
14 is taken element-wise.
Theorem 2.
Let y ∈ R 2 n .
If for i = 1 , . . . , n , [ y ] i (cid:54) =
[ y ] i + n then
[Equation] [Equation]
Further, if there is a i ∈ { 1 , . . . , n } such that [ y ] i =
[ y ] i + n , then there are multiple minimizers of (cid:107) y − R ( x ) (cid:107) 2 , one of which is R † ( y ) .
Proof of Theorem 2 is given in Appendix D.1.
Remark 2 .
We note that Theorem 2 is different from many of the existing work on inverting expansive layers, e.g. Aberdam et al.
Bora et al.
Lei et al.
(2019), our result gives a direct inversion algorithm that is provably the least-squares minimizer.
Further, if each expansive layer is any combination of (R1), (R2), or (R3) then the entire network can be inverted end-to-end by using either the above result or solving the normal equations directly.
C ONCLUSION
Bijective ﬂow networks are a powerful tool for learning a push-forward mappings in a space of ﬁxed dimension.
Increasingly, these ﬂow networks have been used in combination with networks that increase dimension in order to produce networks which are purportedly universal.
In this work, we have studied the theory underpinning these ﬂow and expansive networks by introducing two new notions, the embedding gap and the manifold embedding property.
We show that these notions are both necessary and sufﬁcient for proving universality, but require important topological and geometrical considerations which are, heretofore, under-explored in the literature.
We also ﬁnd that optimality of the studied networks can be established ‘in reverse,’ by minimizing the embedding gap, which we expect opens the door to convergence of layer-wise training schemes.
Without compromising universality, we can use speciﬁc expansive layers that some with a new layerwise projection result.
Moreover, we show that the studied networks provide Bayesian uncertainty quantiﬁcation and allow black-box recovery of their weights.
A PPENDIX A S UMMARY OF N OTATION
Throughout the paper we make heavy use of the following notation.
Unless otherwise stated, X and Y always refer to subsets of Euclidean space, and K and W always refer to compact subsets of Euclidian space.
f ∈ C ( X, Y ) means that f : X → Y is continuous.
For families of functions F and G where each F (cid:51) f : X → Y and G (cid:51) g :
Y → Z , then we deﬁne G ◦ F = { g ◦ f : X → Z : f ∈ F , g ∈ G} .
f ∈ emb( X, Y ) means that f ∈ C ( X, Y ) is continuous and injective on the range of f , i.e. an embedding, and furthermore that f − 1 : f ( X ) → X is continuous.
µ ∈ P ( X ) means that µ is a probability measure over X . 6.
W 2 ( µ, ν ) for µ, ν ∈ P ( X ) refers to the Wasserstein-2 distance, always with (cid:96) 2 ground metric.
(cid:107)·(cid:107)
L p ( X ) refers to the L p norm of functions, from X to R . 8.
For vector-valued f : X → Y , (cid:107) f (cid:107) L ∞ ( X ) =
ess sup x ∈X (cid:107) f (cid:107) 2 .
Note that Y is always ﬁnite dimensional, and so all discrete 1 ≤ q ≤ ∞ norms are equivalent.
Lip( g ) refers to the Lipschitz constant of f . 10.
For x ∈ R n , [ x ] i ∈ R is the i ’th component of x .
Similarly, for matrix A ∈ R m × n , [ A ] ij refers to the j ’th element in the i ’th column.
A PPENDIX B D ETAILED C OMPARISON TO P RIOR WORK B. C ONNECTION TO B REHMER & C RANMER (2020)
In Brehmer & Cranmer (2020), the authors introduce manifold-learning ﬂows as an invertible method for learning probability density supported on a low-dimensional manifold.
Their model can be written as
[Equation]
where T m 1 ⊂ C
( R m , R m ) , T m 0 ⊂ C
( R n , R n ) , and R = (cid:26)(cid:20)
I n × n 0 ( m − n ) × n (cid:21)(cid:27) is a zero-padding (R1).
They invert F ∈ F in two different ways.
For manifold-learning ﬂows ( M -ﬂows) they restrict T m 1 to be an invertible ﬂow, and for manifold-learning ﬂows with separate encoder (
M e -ﬂows) they place no such restrictions on
T m 1 and instead train a separate neural network e to invert elements of
T m 1 .
Our results apply out-of-the-box to the architectures used in Experiment A of Brehmer & Cranmer (2020).
The architecture described in Eqn.
18 is of the form of Eqn.
where L = 1 , and applies to the entire more broadly for rational quadratic ﬂows, which are not studied here.
The network used in (Brehmer & Cranmer, 2020, Experiment 4.A) uses coupling networks, (T1), where T m 1 and T n 0 are both 5 layers deep.
For (Brehmer & Cranmer, 2020, Experiments 4.B and 4.C) the authors choose expressive elements T that are rational quadratic ﬂows Durkan et al.
(2019b) for both T m 1 and T n 0 .
In Experiment 4.B they let T 1 and T 0 again be 5 layers deep, and in 4.C they again let T 1 by 20 layers deep and T 0 15 layers.
For the ﬁnal experiment, 4.D, the choose more complicated expressive elements that combine Glow Kingma & Dhariwal (2018) and Real
NVP Dinh et al.
(2016) architectures.
These elements include the actnorm, 1 × 1 convolutions and rational-quadratic coupling transformations along with a multi-scale transformation.
The authors mention universality of their network without our proof, but our universality results in
Theorem 1 apply to their networks from Experiment A wholesale.
Further in their work the authors describe how training can be split into a manifold phase and density phase, wherein the manifold phase T m 1 is trained to learn the manifold, and in the density phase T m 1 if ﬁxed and T n 0 is trained to learn the density thereupon.
This statement is made formal and proven by our Cor. 1.
B. C ONNECTION TO K OTHARI ET AL . (2021)
In Kothari et al.
(2021), the authors introduce the ‘Trumpet’ architecture, for it’s architecture, which has many alternating ﬂow networks & expansive layers with many ﬂow-networks in the lowdimensional early stages of the network, which gives the architecture a shape similar to the titular instrument.
The architecture studied in Kothari et al.
(2021) is precisely of the form of Eqn.
1, where the bijective ﬂow networks are revnets Gomez et al.
; Jacobsen et al.
(2018) architecture, and the expansive elements are 1 × 1 convolutions, as in (R2).
To out knowledge, there are no results that show that the revnets used are universal approximators, but if they revnets are substituted with either (T1) or (T2), then the, we could apply Theorem 1 to the resulting architecture.
The authors of Kothari et al. (2021) also remark that their entire network can be designed in order to facilitate uncertainty quantiﬁcation, a remark that we too can apply to our networks, as discussed further in Section D.2.
A PPENDIX C P ROOFS C. M AIN R ESULTS C.1. H ELPER L EMMA
Before presenting the proof of Lemma 7, we present the following three helper inequalities
Lemma 6.
For all of the following results, f ∈ emb( K, R m ) and g ∈ emb( W, R m ) and n ≤ o ≤ m .
[Equation]
Let X, Y ⊂ W , let g be Lipschitz on W , and
r ∈ emb( X, Y ) .
Then, there is a r (cid:48) ∈ emb( g ( X ) , g ( Y ))
such that g ◦ r = r (cid:48) ◦ g and (cid:107)
I − r (cid:48) (cid:107) L ∞ ( g ( X ))
≤ (cid:107)
I − r (cid:107) L ∞ ( X ) Lip( g ) .
[Equation]
Let K ⊂ R n , X ⊂ R p and W ⊂ R o be compact sets.
Also, let f ∈ emb
( K, W ) and h ∈ emb( X, W ) , and let g ∈ emb( W, R m ) be a Lipschitz map.
Then
[Equation]
Proof.
Let r ∈ C ( f ( K ) , g ( W )) , then
[Equation]
2 where
x o =
g −
◦ r ◦ f
( x n
[Equation]
inf
[Equation]
g [Equation] g is injective on X , hence we can deﬁne r (cid:48)
such that r (cid:48) =
g ◦ r ◦ g − 1 : g ( X ) → g ( r ( X ))
⊂ g ( Y ) such that r (cid:48) ∈ emb( g ( X ) , g ( Y )) and thus ∀ x ∈ X , [Equation]
[Equation]
where we have used (cid:107) r ( x
[Equation]
− x (cid:107) 2 ≤ (cid:107) I − r (cid:107)
[Equation]
X
[Equation]
For every x ∈ r ( K ) , we have a y ∈ K such that x
= r ( y ) , thus ∀ x ∈ r ( K ) ,
[Equation]
But r is clearly surjective onto it’s range, hence taking the supremum over all x ∈ X yields
[Equation]
As g ∈ emb( W, R m ) , the map g : W → g ( W ) is a homeomorphism and there is g − 1 ∈ emb( g ( W ) , W ) .
For a map r ∈ emb( g ◦ f ( K ) , g ◦ h ( X )) , we see that
ˆ r =
g − 1 ◦ r ◦ g ∈ emb( f ( K ) , h ( X )) .
Also, the opposite is valid as if ˆ r ∈ emb( f ( K ) , h ( X )) then r = g ◦ ˆ r ◦ g − 1 ∈ emb( g ◦ f ( K ) , g ◦ h ( X )) .
Thus
[Equation]
P ROOF OF L EMMA
Lemma 7.
Let f ∈ emb
( K, R m ) and g ∈ emb( W, R m ) and n ≤ o ≤ m .
Then the following hold:
B K,W ( f, g ) ≤ sup x ∈ K (cid:107) g ◦ h ( x ) − f ( x ) (cid:107) 2 where h ∈ emb( K, R o ) is a map satisfying h ( K )
⊂ W . 2.
For any X that is the closure of an open set , if h ∈ emb( X, W ) then
[Equation]
[Equation]
For any r ∈ emb( f ( K ) , R m ) ,
[Equation]
For any r ∈ emb( f ( K ) , g ( W )) and h ∈ emb( X, W ) where X ⊂ R p is the closure of a compact set where n ≤ p ≤ o we have that [Equation] where Lip( g ) denotes the Lipschitz constant of g . 5. For any µ n ∈ P ( K ) , there is a µ o ∈ P ( W ) such that
[Equation]
of Lemma
[Equation]
If we let r :=
g ◦ h ◦ f − 1 , then r ∈ emb( f ( K ) , g ( W )) , and
[Equation]
Given that g ◦ h ( X ) ⊂ g ( W ) , we have that emb( f ( K ) , g ◦ h ( X ))
⊂ emb( f ( K ) , g ( W )) , thus the inﬁmum in Eqn.
6 is taken over a smaller set, thus B K,W ( f, g ) ≤ B K,X ( f, g ◦ h ) .
Note that for any r (cid:48) ∈ emb( r ◦ f ( K ) , g ( W )) , r (cid:48) ◦ r ∈ emb( f ( K ) , g ( W )) , and so we have
[Equation]
where we have used that r is injective for the ﬁnal equality.
This holds for all possible r (cid:48) , hence we have the result.
Recall that f ∈ emb [Equation] [Equation] [Equation]
emb [Equation] [Equation] [Equation]
emb
[Equation] [Equation] and r ∈ emb( f ( K ) , g ( W )) . Then g − 1 ∈ emb( g ( W ) , W ) .
As r ◦ f ( K ) ⊂ g ( W ) , we see that
[Equation]
This, Lemma 7, point 3 and Lemma 6, point 4 yield that
[Equation]
) B K,X ( g − 1 ◦ r ◦ f, h ) , which proves the claim.
Let r ∈ emb
( f ( K ) , g ( W )) be such that (cid:107) I − r (cid:107) L ∞ (Range( f ))
≤ B K,W ( f, g ) + (cid:15) for some (cid:15) > 0 , then for every x ∈ K , there exists y ∈ W such that g ( y ) =
r ◦ f ( x ) .
From injectivity of g , we have that y
=
g − 1 ◦ r ◦ f ( x ) .
Note that g − 1 ◦ r ◦ f ∈ emb
K, W ) , hence
K (cid:48) :
= g − 1 ◦ r ◦ f
( K ) is compact.
Deﬁne µ (cid:48) :
= ( g − 1 ◦ r ◦ f ) # µ .
Clearly g # µ (cid:48)
=
r ◦ f # µ , and thus
[Equation] [Equation]
but W 2 ( g # µ (cid:48) , r ◦ f # µ ) =
and
[Equation]
This is true for every (cid:15) > 0 , hence taking the inﬁmum over all r ∈ emb( f ( K ) , g ( K )) establishes W 2 ( g # µ (cid:48) , f # µ ) ≤ B K,W ( f, g ) + (cid:15) for every (cid:15) > 0 , and thus we have that
[Equation]
C. M ANIFOLD E MBEDDING P ROPERTY C.2. P ROOF OF L EMMA
Proof of Lemma 1.
(i) Let us consider (cid:15) > 0 , a compact set K ⊂ R n and f ∈ emb( R n , R m ) .
Let W = K × 0 } o − n and F : R o
→ R m be the map given by F ( x, y )
=
f
( x ) , ( x, y ) ∈ R n × R o − n .
Because R o,m is a uniform universal approximator of C ( R n , R m ) , there is an R ∈ R o,m such that (cid:107) F − R (cid:107) L ∞ ( W ) < (cid:15) .
Then for the map E
= I ◦
R we have that B K,W ( f, E ) < (cid:15) .
This is true for every (cid:15) > 0 , and so E o,m has the MEP property w.r.t.
the family emb( R n , R m ) .
(ii) Recall that f :=
Φ 0 ◦ R 0 for Φ 0 ∈ Diff 1 ( R m , R m ) and linear R 0 : R n → R m , and that R ∈ R is such that R (cid:12)(cid:12) U is linear for open U .
We present the proof in the case when n = o , and we make the assumption that R (cid:12)(cid:12) K is linear.
In this case, we have the existence of an afﬁne map A : R m → R m so that R
=
A ◦ R so that ˜ K :=
R 0 ( K ) =
A ( R ( K )) .
Let (cid:15) > 0 be given.
By (Hirsch, 2012, Chapter 2, Theorem 2.7), the space Diff 2 ( R m , R m ) is dense in the space Diff 1 ( R m , R m ) , and so there is some Φ 1 ∈ Diff 2 ( R m , R m ) such that
[Equation]
Then, let T ∈ T m be such that (cid:107) T − Φ 1 ◦ A (cid:107) L ∞ ( R ( K );
R m ) < (cid:15) 2 .
Then we have that
[Equation]
Hence, if we let r =
T ◦ R ◦ f − 1 ∈ emb( f ( K ) , T ◦ R ( K )) then we obtain that B K,K
( f, T ◦ R ) < (cid:15) .
This holds for any (cid:15) , and hence we have that
T ◦ R has the MEP for I ( R n , R m ) .
The proof in the case that o ≥ n follows with minor modiﬁcation, and applying Lemmapoint 1.
C.2. S 1 CAN NOT BE MAPPED EXTENDABLY TO THE TREFOIL KNOT
We ﬁrst show that there are no maps E := T ◦ R where R : R 2 → R 3 such that T is a homeomorphism and E ( S 1 ) is a trefoil knot. We use the fact that the trivial knot S 1 and the trefoil knot
M = f ( S 1 ) are not equivalent.
Indeed, by (Murasugi, 2008, Section 3.2), the trefoil knot M and its mirror image are not equivalent, whereas the trivial knot S 1 and its mirror image are equivalent.
Hence, M and R ( S 1 ) are not equivalent knots in R 3 .
Thus by (Murasugi, 2008, Deﬁnition 1.3.
and Theorem 1.3.1), we see that there is no orientation preserving homeomorphism T : R 3 → Rsuch that T ( R 3 \ R ( S 1 ))
=
R 3 \ M.
As the orientation of the map T can be changed by composing T with the reﬂection J : R 3 → R 3 across the plane
Range( R ) that deﬁnes a homeomorphim
J : R 3 \ R ( S 1 ) → R 3 \ R ( S 1 ) , we see that there is no homeomorphism T : R 3 → R 3 such that
T ( R 3 \ R ( S 1 )) =
[Equation]
\ M.
This example shows that the composition E
= T ◦ R of a linear map R and a coupling ﬂow T cannot have the property that E ( S 1 ) =
f ( S 1 ) for this embedding f .
Moreover, the complement R\ E ( S 1 ) is never homeomorphic to R 3 \ f ( S 1 ) for any such map E .
We now construct another example, similar to Figure 2, where an annulus that is mapped to a knotted ribbon in R 3 .
To do this, replace the circle S 1 by an annulus K = { x ∈ R 2 : 1 / 2 ≤ | x | ≤ 3 / 2 } , that in the polar coordinates is { ( r, θ ) : 1 / 2 ≤ r ≤ 3 / 2 } and deﬁne a map
F : K → R 3 by deﬁning in the polar coordinates
[Equation]
where f : S 1 → Σ 1 ⊂ R 3 is an smooth embedding of S 1 to a trefoil knot Σ 1 and v ( θ ) ∈ R 3 is a unit vector normal to Σ 1 at the point f
( θ ) such that v ( θ ) is a smooth function of θ , and a > 0 is a small number.
In this case, M 1 = F ( K ) is a 2-dimensional submanifold of R 3 with boundary, which can visualizes M 1 as a knotted ribbon.
We now show that there are no maps E
= T ◦ R such that E ( K )
=
F ( K ) where T : R 3 → R 3 is an embedding, and R : R 2 → R 3 injective and linear.
The key insight is that if such a T existed, then this implies that the trefoil knot is equivalent to S 1 in R 3 , which is known to be false.
Let U ρ ( A ) denote the ρ -neighborhood of the set
A in R 3 .
It is easy to see that R 2 \ ( { 0 } × [ − 1 , 1]) is homeomorphic to R 2 \ B R 2 (0 , 1) , which is further homeomorphic to R 2 \ { 0 } .
Thus, using tubular coordinates near Σ 1 and a sufﬁciently small ρ > 0 , we see that R 3 \ M 1 is homeomorphic to R\ U
ρ (Σ 1 ) , which is further homeomorphic to R 3 \ Σ 1 .
Also, when R : R 2 → R 3 is an injective linear map, we see that M 2 =
R ( K ) is a un-knotted band in R 3 and R 3 \ M 2 is homeomorphic to R 3 \ Σ 2 .
If R 3 \ M 1 and R 3 \ M 2 would be homeomorphic, then also R 3 \ Σ 1 and R 3 \ Σ 2 would be homeomorphic that is not possible by knot theory, see (Murasugi, 2008, Deﬁnition 1.3.1 and Theorem 1.3.1).
This shows that there are no injective linear maps R : R 2 → R 3 and homeomorphisms Φ : R 3 → R 3 such that (Φ ◦ R )( K ) =
M 1 .
Similar examples can be obtained in a higher dimensional case by using a knotted torus S´equin (2011) 3 and their Cartesian products.
C.2. P ROOF OF E XAMPLE
Proof.
(i) From (Puthawala et al., 2020, Theorem 15)
we have that R o,m can approximate any continuous function f ∈ emb( R n , R m ) .
Further, clearly (T1) and (T2) both contain the identity map, thus Lemma 1 (i) applies.
(ii) Let T m be the family autoregressive ﬂows with sigmoidal activations deﬁned in (Huang et al., 2018).
By (Teshima et al., 2020, App. G, Theorem 1 and Proposition 7), T m are sup -universal approximators in the space Diff 2 ( R m , R m ) of C 2 -smooth diffeomorphisms Φ : R m → R m .
When R o,m is one of (R1) or (R2)
the network is always linear, hence the conditions are satisﬁed.
If R o,m is (R4), then R o,m contains linear mappings, and if (R3), then we can shift the origin, so that R ( x ) is linear on K .
In all cases, Lemma 1 part (ii) applies.
C.2. P ROOF OF L EMMA
Before we present he proof, give a brief outline.
Given an f ∈ emb ∞ ( K, R m ) we ﬁrst show that for m ≥ 2 n +1 there are always a diffeomorphisms Ψ:
R m → R m so that Ψ ◦ f : R n → { 0 } n × R m − n .
The existence of such a Ψ borrows ideas from
Witney’s embedding theorem (Hirsch, 2012, Theorems 3.4 & 3.5) and is constructed by iteratively constructing an injective projection. Next if m − n ≥ 2 n + 1 , then we can apply (Madsen et al., 1997, Lemma 7.6), a consequence of the Tietze extension theorem, to show that Ψ: M → { 0 } n × R m − n can be extended to a diffeomorphism on the entire space, h : R m → R m . Hence f ( x ) =
Ψ − 1 ◦ h ◦
R ( x ) for diffeomorphism Ψ − 1 ◦ h : R m → R m and zero-padding operator R : R n → R m , and thus f ∈ I ∞ ( K, R m ) .
This fact that for m sufﬁciently large compared to n such a diffeomorphism can always be extended is related to the fact that in 4-dimensions, all knots can be opened.
This can be contrasted with the case in Figure 2.
Now we present our proof.
Proof.
Let use next prove Eq.
when m ≥ 3 n + 1 .Let
[Equation]
[Equation]
be a C k -smooth map and M = f ( R n ) be an embedded submanifold of R m .
We have that m ≥ 3 n + 1 > 2 n + 1 .
Let S m − 1 be the unit sphere of R m and let
[Equation]
be the sphere bundle of R m that is a manifold of dimension 2 m − 1 .
By the proof’s of Whitney’s embedding theorem, by Hirsch, (Hirsch, 2012, Chapter 1, Theorems 3.4 and 3.5), there is a set of ‘problem points’ H 1 ⊂ S m − 1 of Hausdorff dimension 2 n such that for all w ∈ R m \ H 1 the orthogonal projection
[Equation]
has a restriction P w | M on M deﬁnes an injective map
[Equation]
Moreover, let T x M be the tangent space of manifold M at the point x and let us deﬁne another set of ‘problem points’ as
[Equation]
For w ∈ S m − 1 \ H 2 the map
[Equation]
is an immersion, that is, it has an injective differential.
The sphere tangent bundle S M of M has dimension 2 n − 1 , and the set H 2 has the Hausdorff dimension at most 2 n − 1 .
Thus H
= H 1 ∪ Hhas Hausdorff dimension at most 2 n < m − 1 and hence the set S m − 1 \ H is non-empty.
For w ∈ S m − 1 \ H the map P w | M : M → { w } ⊥ is a C k -smooth injective immersion and thus
[Equation]
is a C k -smooth submanifold.
Let Z : P w ( M ) → M be the C k -smooth function deﬁned by
[Equation]
, that is it is the inverse of P w | M : M
→ P w ( M ) , where P w ( M ) ⊂ { w } ⊥ .
Let g : ˜ N = P w ( M ) → R be the function
[Equation]
Then ˜ N is a n -dimensional C k -smooth submanifold of ( m − 1) -dimensional
Euclidean space H = { w } ⊥ and g is a C k -smooth function deﬁned on it.
By deﬁnition of a C k -smooth submanifold of
H , any point x ∈ ˜ N has a neighborhood U ⊂ H with local C k -smooth coordinates ψ : U → R m such that ψ ( ˜ N ∩ U ) =
( { 0 } m − 1 − n × R n ) ∩ ψ ( U ) .
Using these coordinates, we see that g can be extended to a C k -smooth function in U .
Using a suitable partition of unity, we see that there is a
C k -smooth map G : { w } ⊥ → R that a C k -smooth extension of g that is, G | ˜ N = g .
Then the map Φ
[Equation]
[Equation]
is a C k -smooth diffeomorphism of R m that maps M to m − 1 dimensional space { w } ⊥ , that is
[Equation]
In the case when m ≥ 3 n + 1 , we can repeat this construction n times.
This is possible as m − n ≥ 2 n + 1 .
Then we obtain C k -smooth diffeomorphisms Φ j : R m → R m , j = 1 , . . .
, n such that their composition Φ n ◦ · · · ◦ Φ 1 : R m → R m is a C k -diffeomorphism
such that which
[Equation]
where Y (cid:48)
⊂ R m is a m − n dimensional linear space.
By letting Ψ = Q ◦ Φ n ◦ · · · ◦ Φ 1 for rotation matrix Q ∈ R m × m , we have that Y :=
Q ( Y (cid:48) )
= { 0 } n × R m − n .
Also, letting φ : X = R n × { 0 } m − n → R m be the map
[Equation]
where f is the function given in Eq.
35, we have that φ : X → Y is a C k -diffeomorphism.
{+Observe that m − n ≥ 2 n + 
+}
and so we can apply (Madsen et al., 1997, Lemma 7.6) to extend φ to a
C k -diffeomorphism
[Equation]
such that h | X = φ .
Note that (Madsen et al., 1997, Lemma 7.6) concerns an extension of a homeomorphism, but as the extension h is given by an explicit formula which is locally a ﬁnite sum of
C k -smooth functions, the same proof gives a C k -diffeomorphic extension h to a diffeomorphism φ .
This technique is called the ‘clean trick’.
Finally, to obtain the claim, observe that when R : R n → R m , R ( x ) = ( x, 0) ∈ { 0 } n × R m − n is the zero padding operator, we have
[Equation]
As h | X = φ and R ( x ) ∈ X , this yields
[Equation]
that is,
[Equation]
where E =
Ψ − 1 ◦ h : R m → R m is a C k -smooth diffeomorphism.
Thus f ∈ I k
( R n , R m ) . This proves Eq. when m ≥ 3 n + 1 . C.2. P ROOF OF L EMMA
Proof of Lemma 3. Let f = F 2 ◦ F 1 where F 2 ∈ F o,m and F 1 ∈ F n,o and (cid:15) > 0 be given, and let E o,m . Clearly, B K,W ( f, E ) ≤ B K,W ( F 2 , E ) and so by the m, o, o MEP of E o,m with respect to F o,m , we have the existence of an r m ∈ emb( f ( K ) , E o,m ) such that (cid:107) I − r (cid:107) L ∞ ( f ( K )) < (cid:15) .
K o :=
( E o,m ) − 1 ◦ r ◦ f
( K ) is compact, hence E
o,m is Lipschitz on K o , so we can apply Lemmapoint 4, so
[Equation] o,m ◦ E p,o ) ≤ (cid:107) I − r (cid:107) L ∞ ( f ( K ))
+ Lip( E o,m ) B K,W (( E o,m ) − 1 ◦ r ◦ f, E p,o ) .
But, because f ∈ F o,m ◦F n,o , we can choose a E p,o ∈ E p,o 1 so that B K,W (( E o,m ) − 1 ◦ r ◦ f, E p,o ) ≤ (cid:15)
Lip( E o,m ) which, combined with Eqn. 36, proves the result.
C.2. P ROOF OF L EMMA
Proof of Lemma 4.
Recall that F ⊂ emb
( R n , R m ) .
Suppose that E o,mdoes not have the m, n, o
MEP with respect to F , then there are some (cid:15) > 0 and f ∈ F so that
[Equation]
From Lemma 7 point 2, we have that
[Equation]
for all E p,o∈ E p,oand for all compact sets W ⊂ R p that satisfy E p,o 1 ( W 1 ) ⊂ W .
Observe that if
W (cid:48)
⊂ R p is a compact set such that W (cid:48)
⊂ W , we have
[Equation]
Thus, inequality Eq.
38 holds for all E p,o∈ E p,oand for all compact sets W ⊂ R p .
Summarising, we have seen that there are f ∈ F and (cid:15) > 0 such that for all E p,o∈ E p,oand for all compact sets
W ⊂ R p we have (cid:15) ≤ B K,W ( f, E o,m◦ E p,o 1 ) .
Hence E o,mdoes not have the m, n, o MEP with respect to F , and we have obtained a contradiction, which proves the result.
C. U NIVERSALITY C.3. P ROOF OF T HEOREM
Proof of Theorem 1.
First we prove the claim under the assumptions (i).
First we prove the claim under assumption (i). Let W ⊂ R n be an open relatively compact set.
From Lemma 3 we have that
[Equation]
has the m, n, n MEP w.r.t.
F := F n L − 1 ,m L [Equation] n,n 1.
Thus for any (cid:15) 1 > 0 , we have an ˜ E ∈ E n,m s.t.
B K,W ( F, ˜ E ) < (cid:15) 1 .
From Lemma 7 point 5, we have the existence of a µ (cid:48)
∈ P ( W ) so that W 2 (cid:16) F # µ, ˜ E # µ (cid:48) (cid:17) < (cid:15) 1 .
By convolving µ (cid:48) with a suitable molliﬁer φ , we can obtain a measure µ (cid:48)(cid:48) = µ (cid:48) ∗ φ ∈ P ( W ) that is absolutely continuous with respect to the Lebesgue measure so that [Equation]
, see (Ambrosio et al., 2008, Lemma 7.1.10.), and so W 2 (cid:16) ˜ E # µ (cid:48) , ˜ E # µ (cid:48)(cid:48) (cid:17) < (cid:15) 1 .
Hence,
[Equation]
Next, from universality of T n 0 for any (cid:15) 2 > 0 , we have the existence of a T 0 ∈ T n 0 so that W
[Equation]
µ (cid:48)(cid:48) , T 0# µ ) < (cid:15) 2 .
From Lemma 7 points 3 and 4 we have that
[Equation] [Equation]
For a given (cid:15) > 0 , choosing (cid:15) 1 < (cid:15) 4 and (cid:15) 2 < (cid:15) 2(1+Lip( ˜ E )) yields that the map E =
˜ E ◦ T 0 ∈
E is such that W 2 ( F # µ, E # µ ) < (cid:15) . This yields the result.
Next we prove the claim under the assumptions (ii).
By our assumptions, in the weak topology of the space C 2 ( R n j , R n j ) , the closure of the set T n j ⊂ C 2 ( R n j , R n j ) contains the space of
Diff 2 ( R n j , R n j ) .
Moreover, by our assumptions R n j − 1 ,n j contains a linear map R .
Observe that as R n j − 1 ,n j is a space of expansive elements, the map R is injective. and hecne by Lemma 1, the family [Equation] has the MEP w.r.t.
F = I 1 ( R n , R m ) .
By Lemma 2, we have that I 1 ( R n , R m ) coincides with the space emb 1 ( R n , R m ) .
Finally, by the assumption that T n 0is dense in the space of C 2diffeomorphism Diff 2 ( R n (cid:96) ) implies that T n 0is a L p -universal approximator for the set of C ∞ - smooth triangular maps for all p < ∞ .
Hence by Lemma 3 in Appendix A of Teshima et al.
T n 0is a distributionally universal.
From these the claim in the case (ii) follows in the same way as the case (i) using the family F = emb 1 ( R n , R m ) .
C.3. P ROOF OF L EMMA
Proof of Lemma 5.
The proof follows from taking the logical negation of the MEP for F .
If the
MEP is not satisﬁed, then there is some f ∈ F so that B K,W ( f, E ) is never smaller than (cid:15) > 0 for all E ∈ E .
Applying the deﬁnition of B K,W ( f, E ) from Eqn.
6 yields the result.
C.3. P ROOF OF C OR .
Proof of Cor.
The proof of Eqn 12 follows from the deﬁnition of the MEP. The proof of Eqn. 13 follows from applying Lemma 7 point 4, and applying the same arguments as in the proof of Theorem 1 so that lim i →∞ 1 Lip( E i ) B K,W ( E − 1 i ◦ r ◦ f, E (cid:48) i ) → 0 , and so that
W 2 ( F # µ, E i ◦ E (cid:48) i # µ (cid:48) )
= (cid:15) i where (cid:15) i → 0 as i → ∞ ..
From the distributional universality of T n 0 , and continuity of E i ◦ E (cid:48)
i we have the existence of a T i ∈ T n 0 so that W 2 ( F # µ, E i ◦ E (cid:48) i ◦ T i # µ ) = 2 (cid:15) i . Choosing (cid:15) i so that lim i →∞ (cid:15) i yields the result.
A PPENDIX D A DDITIONAL P ROPERTIES D. L AYER - WISE P ROJECTION
Here we provide the details of our closed-form layerwise projection algorithm The ﬂow layers are injective, and are often implemented to be numerically easy to invert.
Thus, the crux of the algorithm comes from inverting the injective expansive layers, R .
The range of the ReLU layer is piece-wise afﬁne, hence the inversion follows a two-step program.
First, identify which afﬁne piece (described algebraically, onto which sign pattern) to project.
Second, project to this point using a standard least-squares solver.
The second step is always straight-forward to analyze, but the ﬁrst is more complicated.
The key step in our algorithm is the fact that for the speciﬁc choice of weight matrix W = (cid:20) B − DB (cid:21) , given any y ∈ R 2 n , we can always solve the least-squares inversion problem exactly.
We prove this result in several parts given below.
For any y ∈ R 2 n , M y W ∈ R n × n is full-rank.
If [ y ] i (cid:54) =
[ y ] i + n for each i = 1 , . .
. , n , then the argmin in Eqn.
17 is well deﬁned, i.e. that there is a unique minimizer.
Otherwise there are 2 I minimizers, where I is the number of distinct i such that [ y ] i
=
[ y ] i + n .
If ˜ M y = (cid:2) ∆ y
( I n × n − ∆ y ) (cid:3) , then
[Equation]
We verify Eqn. 17.
Proof of Theorem 2.
Using the deﬁnition of M y , we have,
[Equation]
[Equation] But, ( I n × n − ∆ y − ∆ y D ) is a full-rank diagonal matrix (with entries either 1 or [ D ] i,i ), and B is full rank by assumption, hence M y
[Equation]
is too.
Because B is square and full rank there exists a basis 4 (cid:110)
ˆ
b i (cid:111) i =1 ,...,n of R n such that
[Equation]
For an x ∈ R n , let α i
=
(cid:104)
x, b i (cid:105) for i = 1 , . . . , n be the expansion of x in the ˆ b i basis.
[Equation]
We now consider minizing Eqn. 46 by minimizing the basis expansion in terms of α i , n (cid:88) [Equation]
[Equation]
Eqn. 47 is clearly minimized by minizing each term in the sum, hence we search for a minimizer of the i ’th term
[Equation]
Noting f
( α i ) as the quantity inside the minimum of Eqn.
48, we consider the positive, negative and zero α i cases of Eqn.
48 separately and we get
[Equation]
If [ y ] i + n > [ y ] i , then the minimizer of Eqn.
48 is α i = −
[ y ] 2 i + n [ D ] ii < 0 .
Conversely if [ y ] i + n < [ y ] i then the minimzer of Eqn.
48 is α i =
[ y ] i > 0 .
This argument applies all i = 1 , . . . , n , and hence if [ y ] i (cid:54) =
[ y ] i +1 for all i = 1 , . . .
, n then the minimizing x is unique.
If [ y ] i =
[ y ] i +1 then there are exactly two minimizers of f ( α i ) , − [ y ] 2 i + n [ D ] ii
and [ y ] i , for both of which f
( α i ) =
[ y ] 2 i =
[ y ] 2 i + n .
If we suppose that [ y ] i + n − [ y ] i > 0 , then [ c ( y )] i =
and [ c ( y )] i + n > 0 , thus [∆ y ] ii = 1 , hence if we let x min be the minimizing x from part 1, then
[Equation]
If [ y ] i + n − [ y ] i ≤ 0 then we have
[Equation]
Thus combining Eqn.s 45, 46, 54 and 57 for each i = 1 , . .
. , n , we have that
[Equation] [Equation]
For the ﬁnal point, combining all of the above points we have
[Equation]
[Equation] Further we have from Point 1 that M y W is full rank, hence ( M y W ) − 1 M y y
= R †
( y ) is a minimizer of Eqn.
If [ y ] i (cid:54) =
[ y ] i + n for all i
= 1 , . .
. , n then Part 2 applies, and R † ( y ) is the unique minimizer of (cid:107) y − R ( x ) (cid:107) 22 .
In either case, we have that R † ( y ) is a minimizer.
D. B AYESIAN U NCERTAINTY Q UANTIFICATION
We now discuss the process for making the network amenable to Bayesian Uncertainty Quantiﬁcation (Bayesian UQ).
Assumption 2.
Let y 0 ∈ R n be given, let
y 1 = T 0 y 0 and for each (cid:96) = 2 , . . .
, L , then let y (cid:96) := R (cid:96) ( y (cid:96) − 1 / 2 ) and y (cid:96) − 1 / 2 :=
T (cid:96) − 1 ( y (cid:96) − 1 ) . We assume that for (cid:96) = 1 , . . . , L − 1 , T (cid:96) ( y (cid:96) ) , R (cid:96) ( y (cid:96) − 1 / 2 ) and T 0 ( y 0 ) are differentiable.
Lemma 8.
If a network F of the form of Eqn. 1 satisﬁes Assumption 2, then we have the following upper bound on the log-likelihood,
[Equation]
Proof.
The proof of Lemma 8 follows from computing the log of,
[Equation]
and then applying deﬁnition 1 and expanding term using the laws of logarithms we obtain.
[Equation]
for (cid:96) = 1 , . . .
, L , see (Kothari et al., 2021, Appendix B) for details details.
We note here that in Equation 60 we have an inequality rather than an equality.
The inequality comes from the expansive layers, as the ﬂow-layers are all endomorphisms, i.e. R n (cid:96) → R n (cid:96) .
We now remark on the ease of approximating, or at least bounding, T and R in Eqn. 62.
Values for R when R is (R1) or (R2) are straightforward to compute or approximate 5 .
When R n,m is (R3), R ∈ R are not differentiable, but are differentiable a.e..
Further, ∇ R (cid:96) ( x n (cid:96) ) can a.e. be computed by applying a simple mask 0 and 1 mask to the weight matrices for an exact computation of R .
Additionally, from Hutchinson (1989)
; Kothari et al.
we have the following formula that approximates R ,
[Equation]
where η ∈ R n (cid:96) .
Computation of the T term is often easier, comparatively.
This is because these networks are often designed to make this value as easy to compute as possible.
For example the NICE Dinh et al.
, Real-NVP Dinh et al.
(2016), and GLOW Kingma & Dhariwal (2018)
architectures are all designed so that T can be computed form a short scalar product.
D. B LACK - BOX RECOVERY
We now discuss assumptions that enable black-box recovery of the weights of our entire network post-training.
Assumption 3.
For each (cid:96) = 1 , . . .
, L if R (cid:96) is an afﬁne ReLU layer.
Each T (cid:96) and T 0 is constructed from a ﬁnite number of afﬁne ReLU layers.
Lemma 9.
If a network F of the form of Eqn. 1 satisﬁes Assumption 3, then given the range of the network, the range of the network can be recovered exactly. Further, if the linear region assumption from Rolnick & K¨ording (2020) is satisﬁed, then the exact weights are recovered, subject to two natural isometries discussed below.
Remark 3 .
The ReLU part of Assumption 3 is for all examples in Sec. 2.1.
Further it is also satisﬁed by both ﬂows considered in Sec. 2.2, provided that the various g i are given by layers of afﬁne ReLU ’s.
In Rolnick & K¨ording (2020), the authors show that, although ReLU networks depend on the value of their wight matrix in non-linear ways, it is still possible to recover the exact weights of a given
ReLU network in a black-box way, subject to natural isometrics.
The authors shows that this is possible not only in theory, but in numerical applications as well.
The works of Rolnick & K¨ording (2020); Bui Thi Mai & Lampert (2020) imply that provided the activation functions of the expressive elements are ReLU then the entire network can be recovered in a black-box way.
Further, provided that either the ‘linear region assumption’ from Rolnick & K¨ording (2020) or the generality assumption from Bui Thi Mai & Lampert (2020) is satisﬁed, than the entire network can be recovered uniquely modulo the natural isometries of rescaling and permutation of weight matrices.
First we describe the two natural isometries of scaling and permutation.
Consider the following function
[Equation]
64) where φ is coordinate-wise homogeneous degree 1 (such as ReLU ) and W 1 ∈ R n 1 × n 2 and W 2 ∈ R n 2 × n 3 .
If we let P ∈ R n 2 × n 2 be any permutation matrix, and D + be a diagonal matrix with strictly positive elements, then we can write
[Equation]
as well.
Thus ReLU networks can only ever be uniquely given subject to these two isometries.
When describe unique recovery in the rest of this section, we mean modulo these two isometries.
In Rolnick & K ¨ ording (2020), the authors describe how all parameters of a ReLU network can be recovered uniquely (called reverse engineered in Rolnick & K ¨ ording (2020)), subject to the so called ‘linear 6 region assumption’, LRA.
The input space R n can be partitioned into a ﬁnite number of open {S i } n i
i =1 , where for each k , F ( x ) =
W k
i + b i , i.e. the network corresponds to an afﬁne polyhedron in the output space.
The algorithms (Rolnick & K ¨ ording, 2020, Alg.s 1 & 2) are roughly described below.
First, identify at least one point within each afﬁne polyhedra {H j } n j j =1 .
Then identify the boundaries between polyhedra.
The boundaries between sections are always one afﬁne ‘piece’ of piecewise hyperplanes {H j } n j j =1 .
These {H j } n j j =1 are the central objects which indicate the (de)activation of an element of a ReLU somewhere in the network.
If the H j are full hyperplanes, then the ReLU that is (de)activates occurs in the ﬁrst layer of the network.
If H j is not a full hyperplane, then it necessarily has a bend where it intersects another hyperplane H j (cid:48) .
Further, except for a Lebesgue measure 0 set, when H j intersects H j (cid:48)
the latter does not have a bend.
If this is the case, then
H j (cid:48) corresponds to a ReLU (de)activation in an earlier layer than H j .
In this way the activation functions of every layer can be deduced.
Once this is done, the normals of the hyperplanes can be used to infer the row-vectors of the various weight matrices, letting one recover the entire network.
The above algorithm recovers all of the weights exactly provided that the LRA is satisﬁed.
The
LRA is satisﬁed if for every distinct S i and S i (cid:48) , either W i (cid:54)
=
W i (cid:48) or b i (cid:54) =
b i (cid:48) .
That is, different sign patterns produce different afﬁne sections in the output space.
This is a natural assumption, as the algorithm as described above reconstruction works by ﬁrst detecting the boundaries between adjacent afﬁne polyhedra, which is only possible if the LRA holds.
Given the weights of a network there is currently no simple way to detect if the LRA is satisﬁed, to our knowledge.
Nevertheless the authors of Rolnick & K¨ording (2020) show that if it is satisﬁed, then unique recovery follows.
Nevertheless recovery of the range of the entire network is possible, but this recovery may not be unique.
In Bui Thi Mai & Lampert (2020) the authors also consider the problem of recovering weights of a
ReLU neural network, however the authors therein study the question of when there exist isometries beyond the two natural ones described above.
In particular the main result (Bui Thi Mai & Lampert, 2020, Theorem 1) shows the following.
Let E n 0 ,n L be a ReLU network that is L layers deep and non-increasing.
Suppose that E 1 , E 2 ∈ E n 0 ,n L , E 1 and E 2 are general 7 and for all x ∈ R n
E 1 ( x ) =
E 2 ( x ) , then E 1 is parametrically identical to E 2 subject to the two natural isometries.
This work provides the stronger result, however does not apply to the networks that we consider out of the box.
It does apply to our expressive elements (provided that they use ReLU activation functions, and are non-increasing), but not necessarily apply to the network on the whole.