G LOBALLY I NJECTIVE R E LU N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Injectivity plays an important role in generative models where it enables inference; in inverse problems and compressed sensing with generative priors it is a precursor to well posedness.
We establish sharp characterizations of injectivity of fullyconnected and convolutional ReLU layers and networks.
First, through a layerwise analysis, we show that an expansivity factor of two is necessary and sufﬁcient for injectivity by constructing appropriate weight matrices.
We show that global injectivity with iid Gaussian matrices, a commonly used tractable model, requires larger expansivity between 3.4 and 5.7.
We also characterize the stability of inverting an injective network via worst-case Lipschitz constants of the inverse.
We then use arguments from differential topology to study injectivity of deep networks and prove that any Lipschitz map can be approximated by an injective
ReLU network.
Finally, using an argument based on random projections, we show that an end-to-end—rather than layerwise—doubling of the dimension sufﬁces for injectivity.
Our results establish a theoretical basis for the study of nonlinear inverse and inference problems using neural networks.
I NTRODUCTION
Many applications of deep neural networks require inverting them on their range.
Given a neural network N : Z → X , where X is often the Euclidean space R m and Z is a lower-dimensional space, the map N − 1 : N ( Z ) → Z is only well-deﬁned when N is injective.
The issue of injectivity is particularly salient in two applications: generative models and (nonlinear) inverse problems.
Generative networks model a complicated distribution p X over X as a pushforward of a simple distribution p Z through N .
Given an x in the range of N , inference requires computing p Z ( N − 1 ( x )) which is well-posed only when N is injective.
In the analysis of inverse problems (Arridge et al., 2019), uniqueness of a solution is a key concern; it is tantamount to injectivity of the forward operator.
Given a forward model that is known to yield uniqueness, a natural question is whether we can design a neural network that approximates it arbitrarily well while preserving uniqueness.
Similarly, in compressed sensing with a generative prior N and a possibly nonlinear forward operator A injective on the range of N , we seek a latent code z such that A ( N ( z )) is close to some measured y
=
A ( x ) .
This is again only well-posed when N can be inverted on its range (Balestriero et al., 2020).
Beyond these motivations, injectivity is a fundamental mathematical property with numerous implications.
We mention a notable example: certain injective generators can be trained with sample complexity that is polynomial in the image dimension (Bai et al., 2018).
1. O UR R ESULTS
In this paper we study injectivity of neural networks with ReLU activations.
Our contributions can be divided into layerwise results and multilayer results.
Layerwise results.
For a ReLU layer f : R n → R m we derive sufﬁcient and necessary conditions for invertibility on the range.
For the ﬁrst time, we construct deterministic injective ReLU layers with minimal expansivity m = 2 n .
We then derive specialized results for convolutional layers which are given in terms of ﬁlter kernels instead of weight matrices.
We also prove upper and lower bounds on minimal expansivity of globally injective layers with iid Gaussian weights.
This generalizes certain existing pointwise results (Theorem 2 and Appendix A.2).
We ﬁnally derive the worst-case inverse
Lipschitz constant for an injective ReLU layer which yields stability estimates in applications to inverse problems.
Multilayer results.
A natural question is whether injective models are sufﬁciently expressive.
Using techniques from differential topology we prove that injective networks are universal in the following sense: if a neural network N 1 : Z → R 2 n +1 models the data, Z ⊂ R n , then we can approximate N 1 by an injective neural network N 2 : Z → R 2 n +1 .
As N 2 is injective, the image set
N 2 ( Z ) is a Lipschitz manifold.
We then use an argument based on random projections to show that an end-to-end expansivity by a factor of ≈ 2 is enough for injectivity in ReLU networks, as opposed to layerwise 2-expansivity implied by the layerwise analysis.
We conclude with preliminary numerical experiments to show that imposing injectivity improves inference in GANs while preserving expressivity.
1. W HY G LOBAL I NJECTIVITY ?
locally
The attribute “global” relates to global injectivity of the map N : Z → R m on the lowdimensional latent space Z , but it does not imply global invertibility over R m , only on the range N ( Z )
⊂ R m .
If we train a GAN generator to map iid normal latent vectors to real images from a given distribution, we expect that any sampled latent vector generates a plausible image.
We thus desire that any N ( z ) be produced by a unique latent code z ∈ Z .
This is equivalent to global injectivity, or invertibility on the range.
Our results relate to the growing literature on using neural generative models for compressed sensing (Bora et al., 2017).
They parallel the related guarantees for sparse recovery where the role of the low-dimensional latent space is played by the set of all k -sparse vectors.
One then looks for matrices which map all k -sparse vectors to distinct measurements (Foucart & Rauhut, 2013).
As an example, in the illustration in Figure 1 images coresponding to latent codes in orange, brown, and violet wedges cannot be compressively sensed.
Finally, a neural network is often trained to directly reconstruct an image x from its (compressive) low-dimensional measurements
y =
A ( x ) without introducing any generative models.
In this case, whenever A is
Lipschitz, it is immediate that the learned inverse must be injective.
1. R ELATED W ORK
Closest to our work are the papers of Bruna et al.
(2013), Hand et al.
and Lei et al.
Bruna et al.
(2013) study injectivity of pooling motivated by the problem of signal recovery from feature representations.
They focus on (cid:96) p pooling layers; their Proposition 2.2 gives a criterion similar to the DSS (Deﬁnition 1) and bi-Lipschitz bounds for a ReLU layer (similar to our Theorem 3).
Unlike Theorems 1 and 3, their criterion and Lipschitz bound are in some cases not precisely aligned with injectivity; see Appendix E.1.
Compressed sensing with GAN priors requires inverting the generator on its range (Bora et al., 2017;
Shah & Hegde, 2018; Wu et al., 2019; Mardani et al., 2018;
Hand et al., 2018).
Lei et al.
(2019) replace the end-to-end inversion by the faster and more accurate layerwise inversion when each layer is injective.
They show that with high probability a ReLU layer with an iid normal weight matrix can be inverted about a ﬁxed point if the layer expands at least by a factor of 2.1.
This result is related to our Theorem 2 which gives conditions for global injectivity or layers with random matrices.
Hand & Voroninski (2017) show that when the weights of a ReLU network obey a certain weighted distribution condition, the loss function for the inversion has a strict descent direction everywhere except in a small set.
The condition is in particular satisﬁed by random matrices with expansivity n j = Ω( n j − 1 log n j − 1 ) , where n j is the output dimension of layer j .
A continuous analogy of our convolutional construction (Deﬁnition 3) was considered by Mallat et al.
They show that ReLU acts as a phase ﬁlter and that the layer is bi-Lipschitz and hence injective when the ﬁlters have a diverse phase and form a frame.
Discretizing their model gives a statement related to Corollary 2 and Theorem 4.
Injectivity is automatic in invertible neural networks such as normalizing ﬂows (Grover et al., 2018;
Kingma & Dhariwal, 2018; Grathwohl et al., 2018).
Specialized architectures with simple Jacobians give easy access to the likelihood (Dinh et al., 2014; 2016; Gomez et al., 2017), which facilitates application to inverse problems (Ardizzone et al., 2018).
Inference with GANs can be achieved by jointly training a generative network and its inverse (Donahue et al., 2016; Dumoulin et al., 2016), which is well-deﬁned when the generator is injective.
Relaxed injective probability ﬂows resemble
GAN generators but are trained via approximate maximum likelihood (Kumar et al., 2020).
Injectivity is promoted by keeping the smallest singular value of the Jacobian away from zero at the training examples, a necessary but not sufﬁcient condition.
In general, Jacobian conditioning improves GAN performance (Heusel et al., 2017; Odena et al., 2018).
Finally, lack of injectivity interferes with disentanglement (Chen et al., 2016; Lin et al., 2019).
In this context, injectivity seems to be a natural heuristic to increase latent space capacity without increasing its dimension (Brock et al., 2016).
Notation.
Given a matrix W ∈ R m × n we deﬁne the notation w ∈ W to mean that w ∈ R n is a row vector of W .
For a matrix W ∈ R m × n with rows { w j } mj =1 and x ∈ R n , we write
S ( x, W ) := { j ∈ [[ m ]]: (cid:104) w j , x (cid:105) ≥ 0 } (1) and S c ( x, W ) for its complement, with [[ m ]] = { 1 , . . .
, m } .
We let NN ( n, m, L, m ) be the family of functions
N θ : R n → R m of the form
[Equation]
Indices (cid:96) = 1 , . . .
, L index the network layers, b (cid:96) ∈ R n (cid:96) +1 are the bias vectors, W (cid:96)
∈
R
n (cid:96) +
× n (cid:96) are the weight matrices with n 1 = n , n L = m , and φ (cid:96) are the nonlinear activation functions.
We will denote ReLU( x ) = max( x, 0) .
We write n = ( n 1 , n 2 , . . .
n L − 1 ) and θ =
( W 1 , b 1 , . . .
, W L , b L ) for the parameters that determine the function N θ .
We also write
NN ( n, m, L ) =
(cid:83) n ∈ Z L − 1 NN ( n, m, L, n ) and NN ( n, m )
=
(cid:83) L ∈ Z NN ( n, m, L ) .
L AYERWISE I NJECTIVITY OF R E LU N ETWORKS
For a one-to-one activation function such as a leaky ReLU or a tanh , it is easy to see that injectivity of x (cid:55)→ W i x implies the injectivity of the layer.
We therefore focus on non-injective ReLU activations.
2. D IRECTED S PANNING S ET
Unlike in the case of a one-to-one activation, when φ ( x ) =
ReLU( x ) , x (cid:55)→ φ ( Wx ) cannot be injective for all x ∈ R n if W is a square matrix.
Noninjectivity of ReLU( Wx ) occurs in some simple situations, for example if Wx is not full rank or if Wx ≤ 0 element-wise for a nonzero x , but these two conditions on their own are not sufﬁcient to characterize injectivity.
Figure 2 gives an example of a weight matrix that yields a noninjective layer that passes these two tests.
In order to facilitate the full characterization of injectivity of ReLU layers, we deﬁne a useful theoretical device:
Deﬁnition 1 (Directed Spanning Set) .
Let W ∈ R n × m .
We say that W has a directed spanning set (DSS) of Ω ⊂ R n with respect to a vector x ∈ R n if there exists a ˆ W x such that each row vector of ˆ W x is a row vector of W ,
[Equation]
and Ω ⊂ span( ˆ W x ) .
When omitted Ω is understood to be R n .
For a particular x ∈ R n , it is not hard to verify if a given W = { w i } i =1 ,...,m has a DSS of Ω .
One can simply let ˆ W x = { w i ∈ R n : (cid:104) x, w i (cid:105) ≥ 0 } .
Then, W has a DSS for Ω with respect to x if and only if ˆ W x is a basis of Ω , which can be checked efﬁciently.
Note that having full rank is necessary for it to have a DSS for any x ∈ R n .
It is however not sufﬁcient.
For example, Id n ∈ R n × n , the identity matrix in R n , is clearly full rank, but it doesn’t have a DSS of R n w.r.t.
x = ( − 1 , 0 , . . .
, 0) ∈ R n .
To check whether W has a DSS for all x ∈ R n , note that W partitions R n into open wedges S k , R n = (cid:83) k S k , with constant sign patterns.
That is, for x 1 , x 2 ∈ S k , sign( Wx 1 ) =
sign( Wx 2 ) .
(See also proof of Theorem 2 in Appendix A.2.2.)
Checking whether W has a DSS for all x is equivalent to checking that for every wedge there are at least n vectors W k ⊂ W with positive sign, (cid:104) w, x (cid:105) >for x ∈ S k , and that W k spans R n .
Since the number of wedges can be exponential in m and n (Winder, 1966) this suggests an exponential time algorithm.
We also note the connection between
DSS and spark in compressed sensing (Foucart & Rauhut, 2013), deﬁned as the size of the smallest set of linearly dependent vectors in W .
If every wedge has n or more positive signs, then full spark n + 1 is sufﬁcient for W to have a DSS w.r.t all x ∈ R n .
Computing spark is known to be NP-hard (Tillmann & Pfetsch, 2013); whether one can do better for DSS remains an open question.
2. F ULLY C ONNECTED L AYER
The notion of a DSS immediately leads to our main result for fully connected layers.
Theorem 1 (Conditions for Injectivity of ReLU( Wx ) ) .
Let W ∈ R m × n where n > 1 be a matrix with row vectors { w j } mj =1 , and ReLU( y )
=
max( y, 0) .
The function ReLU( W ( · )): R n → R m is injective if and only if W has a DSS w.r.t every x ∈ R n .
The question of injectivity in the case when b = 0 and when b (cid:54) = 0 are very similar and, as Lemmashows, the latter question is equivalent to the former on a restricted weight matrix.
Lemma 1 (Injectivity of ReLU( Wx + b ) ) .
Let W ∈ R m × n and b ∈ R m .
The function ReLU( W ( · )+ b ): R n → R m is injective if and only if ReLU( W | b ≥ 0 · ) is injective,
where W | b ≥ 0 ∈ R m × n is rowwise the same as W where b i ≥ 0 , and is a row of zeroes
when b i < 0 .
Remark 1 (Injectivity of ReLU( Wx ) , Positive x ) .
Without normalization strategies, inputs of all but the ﬁrst layer are element-wise non-negative.
One can ask whether ReLU( Wx ) is injective when x is restricted to be element-wise non-negative.
Following the same argument as in Theorem 1, we ﬁnd that W must have a DSS w.r.t.
x ∈ R n for every x that is element-wise non-negative.
With normalizations strategies however (for example batch renormalization (Ioffe, 2017))
the full power of
Theorem 1 and Lemma 1 may be necessary.
In Appendix B we show that common normalization strategies such as batch, layer, or group normalization do not interfere with injectivity.
Remark 2 (Global vs Restricted Injectivity) .
Even when the conditions of Theorem 1 and Lemmaare not satisﬁed the network might be injective in some X ⊂ R n .
Indeed, N ( x ) = ReLU(Id n · ) is in general not injective, but it is injective in X = { x ∈ R n : x i > 0 for i = 1 , . . . , n } , a convex and open set.
Theorem 1 and Lemma 1 are the precise criteria for single-layer injectivity for all x ∈ R n .
The above layerwise results imply a sufﬁcient condition for injectivity of deep neural networks.
Corollary 1 (layerwise Injectivity Implies End-to-End Injectivity) .
Let N : R n → R m be a deep neural network of the form (2).
If each layer φ (cid:96) ( W (cid:96) · + b (cid:96) ): R n i → R n i +1 is injective, then so is N .
Note that Corollary 1 is sufﬁcient but not necessary for injectivity of the entire network.
Consider for example an injective layer ReLU( W ) , and N ( x ) =
ReLU(Id m · ReLU( W ( x ))) , where I is the identity matrix.
Clearly,
[Equation]
so N is injective, but it fails the criterion of Corollary 1 (at the ReLU( I · ) layer).
An important implication of Theorem 1 is the following result on minimal expansivity.
Corollary 2 (Minimal Expansivity) .
For any W ∈ R m × n , ReLU( W · ) is non-injective if m < 2 · n .
If W ∈ R 2 n × n satisﬁes Theorem 1 and Lemma 1, than up to row rearrangement W can be written as
[Equation]
where B, D ∈ R n × n and B is a basis and D a diagonal matrix with strictly positive diagonal entries.
where asymptotics of I ( m, n ) in Theorem 2 are valid.
While m < 2 n immediately precludes injectivity, Corollary 2 gives a simple recipe for the construction of minimally-expansive injective layers with m = 2 n .
To the best of our knowledge this is the ﬁrst such result.
Further, we can also use Corollary 2 to build weight matrices for which m > 2 n .
For example, the matrix
[Equation]
where B ∈ R n × n has full rank, D ∈ R n × n is a positive diagonal matrix, and M ∈ R m − 2 n × n is arbitrary, has a DSS w.r.t.
every x ∈ R n .
This method can be used in practice to generate injective layers by using standard spectral regularizers to ensure that B is a basis (Cisse et al., 2017).
In Section 3 we will show that, in fact, an end-to-end rather than layerwise doubling of dimension is sufﬁcient for injectivity.
Previous work that sought injectivity uses random-weight models.
Lei et al.
(2019) show that a layer is invertible about a point in the range provided that m ≥ 2 .
1 n and W is iid Gaussian.
In Appendix A.3 we show that m = 2 .
1 n is not enought to guarantee injectivity.
In fact, we show that with high probability an iid Gaussian weight matrix W yields a globally invertible ReLU layer, when W is sufﬁciently expansive, and conversely does not satisfy Theorem 1 if it is not expansive by at least a factor of 3.4:
Theorem 2 (Injectivity for Gaussian Weights) .
Let I ( m, n ) =
P { x (cid:55)→ ReLU
( Wx ) is injective } with the entries of W iid standard normal and c = m/n ﬁxed.
Then as n → ∞ ,
I ( m, n ) → 1 if c (cid:39) 5 . 7 and I ( m, n ) → 0 if c (cid:47) 3 . 4 .
The parameter regions deﬁned in Theorem 2 are illustrated in Figure 3.
We note that the study of neural networks with random weights is an important recent research direction (Pennington & Worah, 2017; Benigni & Péché, 2019).
Pennington & Worah (2017) articulate this clearly, arguing that large complex systems, of which deep networks are certainly an example, can be proﬁtably studied by approximating their constituent parts as random variables.
2. I NVERSE L IPSCHITZ C ONSTANT FOR N ETWORKS
Because (cid:107) ReLU( Wx ) − ReLU( Wy ) (cid:107) ≤ (cid:107) W (cid:107) (cid:107) x − y (cid:107) , it is clear that ReLU( W · ) is Lipschitz with constant (cid:107) W (cid:107) ; whether the inverse is Lipschitz, and if so, with what Lipschitz constant, is less obvious.
We can prove the following result (see Appendix C.1):
Theorem 3 (Global Inverse Lipschitz Constant) .
Let W ∈ R m × n have a DSS w.r.t.
every x ∈ R n .
There exists a C ( W ) > 0 such that for any x 0 , x 1 ∈ R n ,
[Equation]
where C ( W ) = 1 √ 2 m min x ∈ R n σ ( W | S ( x,W ) ) , and σ denotes the smallest singular value.
This result immediately yields stability estimates when solving inverse problems by inverting (injective) generative networks.
We note that the (2 m ) − 1 / 2 factor is essential; the naive “smallest singular value” estimate is too optimistic.
2. C ONVOLUTIONAL L AYER
[Figure]
Since convolution is a linear operator we could simply apply Theorem 1 and Lemma 1 to the convolution matrix.
It turns out, however, that there exists a result specialized to convolution that is much simpler to verify.
In this section, capital letters such as I, J and K denote multi-indexes; the same holds for their element-wise limits such as N and
M .
N j refers to the j th index in the multi-index N .
We use the shortcuts c ∈ R N for c ∈ R N 1 × · · · × R N p , and (cid:80) NI
[Equation]
=
[Equation]
N
[Equation]
=
[Equation] [Equation]
, for I =
( I 1 , . .
, I p ) .
The symbol 1 can refer to number 1 or a p -tuple (1 , . .
Further, the notation I ≤ J means that
I k ≤ J k for all k = 1 , . .
p and likewise for ≥ , < and > .
If I (cid:54)≤ J , then there is at least one k such that
I k > J k .
Deﬁnition 2 (Convolution Operator) .
Let c ∈ R O .
We say that c is a convolution kernel of width O .
Given c and x ∈ R N , we deﬁne the convolution operator C ∈ R N × N with stride 1 as,
[Equation]
When 1 (cid:54)≤ K or K (cid:54)≤ O we will set c K = 0 .
We do not specify the boundary condition on x (zero-padded, periodic, or otherwise), as our results hold generally.
Deﬁnition 3 (Convolutional Layer) .
We say that a matrix W ∈ R M × N is a convolution operator, with n Q convolutions, provided that W can be written (up to row rearrangement) in the form
W = (cid:2) C T
C T· · ·
C T n Q (cid:3) T where each C k is a convolution operator for k = 1 , . .
. , n Q .
A neural network layer for which W is a convolution operator is called a convolution layer.
Deﬁnitions 2 and 3 automatically model the standard multi-channel convolution used in practice.
For 2D images of size 512 × 512 , n c input channels, and 3 × 3 convolutions, we simply let x ∈ R N with
N = (512 , 512 , n c ) and c ∈ R O with O = (3 , 3 , n c ) .
To state our main result for convolutions we also need to deﬁne the set of zero-padded kernels for c ∈ R O .
Think of a multi-index as a box (or a hyperrectangle).
Let P be a multi-index such that O “ﬁts” in P .
Then we deﬁne
[Equation]
c within the box
[Equation]
Theorem 4 (Sufﬁcient Condition for Injectivity of Convolutional Layer) .
Suppose that W ∈ R M × N is a convolution layer with convolutions { C k } q k =1 , and corresponding kernels { c k } q k =1 .
If for any P ,
[Equation]
has a DSS for R P with respect to all x ∈ R P , then ReLU( W · ) satisﬁes Theorem 1.
Theorem 4 applies to multi-channel convolution of width ( O, n c )
=
( O 1 , . . .
, O p , n c ) provided that we choose a ( P, n c ) such that O ≤ P .
This theorem shows that if a convolution operator W has a DSS w.r.t.
vectors with support in the pink region in Figure 4, then it has a DSS w.r.t.
vectors supported on the green region as well.
U NIVERSALITY AND E XPANSIVITY OF D EEP I NJECTIVE N ETWORKS
We now consider general properties of deep injective networks.
Note that a neural network N θ ∈
NN ( n, m ) is Lipschitz smooth: there is L 0 > 0 such that | N θ ( x ) − N θ ( y ) | ≤ L 0 | x − y | for all x, y ∈ R n .
If N θ : R n → R m is also injective, results of basic topology imply that for any bounded and closed set B ⊂ R n the map N θ : B → N θ ( B ) has a continuous inverse N − 1 θ : N θ ( B )
→ B and the sets B and its image
N θ ( B ) are homeomorphic.
Thus, for example, if Z is a random variable supported on the cube [0 , 1] n ⊂ R n , the sphere S n − 1 ⊂ R n , or the torus T n − 1 ⊂ R n , we see that
N θ ( Z ) is a random variable supported on a set in R m that is homeomorphic to an n -dimensional cube or an n − 1 dimensional sphere or a torus, respectively.
This means that injective neural networks can model random distributions on surfaces with prescribed topology.
Moreover, if N θ : R n → R m is a decoder, all objects x ∈ N θ ( R n ) correspond to the unique code z such that x = N θ
( z ) .
A fundamental property of neural networks is that they can uniformly approximate any continuous function f : Z → R m deﬁned in a bounded set Z ⊂ R n .
For general dimensions n and m , the injective neural networks do not have an analogous property.
For instance, if f : [ − π, π ] → R is the trigonometric function f ( x ) = sin( x ) , it is easy to see that there is no injective neural network (or any other injective function)
N θ : [ − π, π ] → R such that | f ( x ) − N θ ( x ) | < 1 .
Consider, however, the following trick: add two dimensions in the output vector and consider the map F : [ − π, π ] → R 3 given by F ( x ) =
(0 , 0 , sin( x ))
∈ R 3 .
When f is approximated by a one-dimensional, non-injective ReLU network f θ : [ − π, π ] → R and α > 0 is small, then the neural network N θ ( x ) =
( αφ ( x ) , αφ ( − x ) , f θ ( x )) is an injective map that approximates F .
In general, as ReLU networks are piecewise afﬁne maps, it follows in the case m = n that if a ReLU network N θ : R n → R n is injective, it has to be surjective (Scholtes, 1996, Thm. 2.1).
This is a limitation of injective neural networks when m = n .
Building on these ideas we show that when the dimension of the range space is sufﬁciently large, m ≥ 2 n + 1 , injective neural networks become sufﬁciently expressive to universally model arbitrary continuous maps.
Theorem 5 (Universal Approximation with Injective Neural Networks) .
Let f : R n → R m be a continuous function, where m ≥ 2 n + 1 , and L ≥ 1 .
Then for any ε > 0 and compact subset
Z ⊂ R n there exists a neural network N θ ∈ NN ( n, m ) of depth L such that N θ : R n → R m is injective and
[Equation]
Before describing the proof, we note that this result also holds for leaky ReLU networks with no modiﬁcation.
The key fact used in the proof is that the network is a piecewise afﬁne function.
To prove this result, we combine the approximation results for neural networks (e.g., Pinkus’s density result for shallow networks (Pinkus, 1999, Theorem 3.1) or Yarotsky’s result for deep neural networks (Yarotsky, 2017)), with the Lipschitz-smooth version of the generic projector technique.
This technique from differential topology is used for example to prove the easy version of the Whitney’s embedding theorem (Hirsch, 2012, Chapter 2, Theorem 3.5).
Related random projection techniques have been used earlier in machine learning and compressed sensing (Broomhead & Kirby, 2000; 2001; Baraniuk & Wakin, 2009; Hegde et al., 2008; Iwen & Maggioni, 2013).
The proof of Theorem 5 is based on applying the above classical results to locally approximate the function f : R n → R m by some (possibly non-injective) ReLU-based neural network F θ : R n → R m , and augment it by adding additional variables, so that H θ ( x ) =
( x, F θ ( x )) is an injective map
H θ : R n → R n + m .
The image of this map, M = H
θ ( R n ) , is an n -dimensional, Lipschitz-smooth submanifold of R n + m .
The dimension of m + n is larger than 2 n + 1 , which implies that for a randomly chosen projector P 1 that maps R n + m to a subspace of dimension m + n − 1 , the restriction of P 1 on the submanifold M , that is, P 1 : M → P 1 ( M ) , is injective.
By applying n random projectors, P 1 , . . .
, P n , we have that N θ =
P n ◦ P n − 1 ◦ · · · ◦ P 1 ◦ H θ is an injective map whose image is in a m dimensional linear space.
These projectors can be multiplied together, but are generated sequentially.
By choosing the projectors P j in a suitable way, the obtained neural network
N θ approximates the map f .
We point out that in the proof of Theorem 5 it is crucial that we ﬁrst approximate function f by a neural network and then apply to it a generic projection to make the neural network injective.
Indeed, doing this in the opposite order may fail as an arbitrarily small deformation (in C ( R n ) ) of an injective map may produce a map that is non-injective.
The proof of Theorem 5 implies the following corollary on cascaded neural networks where the dimensions of the hidden layers can both increase and decrease (see Figure 5):
Corollary 3.
Let n, m, d j ∈ Z + , j = 0 , 1 , . .
, 2 k be such that d 0 = n , d 2 k = m ≥ 2 n + 1 and d j ≥ 2 n + 1 for even indexes j ≥ 2 .
Let
[Equation]
where f ( j ) θ : R d 2 j − 2 → R d 2 j − 1 are injective neural networks and B j ∈ R d 2 j − 1 → R d 2 j are random matrices whose joint distribution is absolutely continuous with respect to the Lebesgue measure of (cid:81) kj
=
( R d 2 j × d 2 j − 1 ) .
Then the neural network
F k : R n → R m is injective almost surely.
Observe that in Corollary 3 the weight matrices B j and B j (cid:48) may not be independent.
Moreover, the distribution of the matrices B j may be supported in an arbitrarily small set, that is, a small random perturbation of the weight matrices make the function F k injective.
Corollary 3 characterizes deep globally injective networks, while avoiding the exponential growth in dimension as indicated by Corollary 2.
N UMERICAL E XPERIMENT : I NJECTIVE GAN S I MPROVE I NFERENCE
We devise a simple experiment to demonstrate that 1) one can construct injective networks using
Corollary 2 that 2) perform as well as the corresponding non-injective networks, while 3) being better at inference problems.
One way to do inference with GANs is to train a so-called inference network
I net : X → Z jointly with the generator G : Z → X so that I ( G ( z ))
≈
z (Donahue et al., 2016;
Dumoulin et al., 2016); I net is trained to invert G on its range.
This is used to evaluate the likelihood of an image x in the range of G as p Z
( I net ( x )) .
However, for p Z ( I net ( G ( z ))
=
p Z ( z ) to hold, G must be injective—otherwise the distribution of I net ( G ( z )) will be different from that of z .
One would thus hope that the distribution of I net ( G ( z )) will be closer to that of z if we enforce injectivity.
If that is the case and the sample quality is comparable, then we have a practical tool to improve inference.
We use the DCGAN (Radford et al., 2015) architecture with the same hyperparameters, layer sizes and normalization strategies for the regular GAN and the injective GAN; see Appendix F. Note that the original DCGAN uses several ReLU conv layers with an expansivity of 2 per layer.
By
Corollary 2 these layers are non-injective. Therefore, the ﬁlters of the DCGAN are modiﬁed to be of the form [ C ; − s 2 C ] in order to build an injective GAN.
We train the networks to generate 64 × 64 ×images and draw the latent code z ∈ R 256 iid from a standard normal distribution.
We test on CelebA (Liu et al., 2015) and FFHQ (Karras et al., 2019) datasets.
To get a performance metric, we ﬁt Gaussian distributions N ( µ, Σ) and N ( µ inj , Σ inj ) to G ( z ) and G inj ( z ) .
We then compute the Wasserstein-2 distance W 2 between the distribution of z ∼ N (0 , I 256 ) and the two ﬁtted Gaussians using the closed-form expression for W 2 between Gaussians, W 22 ( N ( µ 1 , Σ 1 ) , N ( µ 2 , Σ 2 ))
= (cid:107) µ 1 − µ 2 (cid:107) 2 + Tr(Σ 1 + Σ 2 −
2(Σ 1 Σ 2 ) 1 / 2 ) .
We summarize the results in Table 1.
Despite the restrictions on the weights of injective generators, their performance on popular GAN metrics—Fréchet inception distance (FID) (Heusel et al., 2017) and inception score (IS) (Salimans et al., 2016)—is comparable to the standard GAN while inference improves.
That the generated samples are indeed comparable to the standard GAN can be also gleaned from the from the ﬁgure in Appendix F.
[Table]
C ONCLUSION
We derived and explored conditions for injectivity of ReLU neural networks.
In contrast to prior work which looks at random weight matrices, our characterizations are deterministic and derived from ﬁrst principles.
They are also sharp in that they give sufﬁcient and necessary conditions for layerwise injectivity.
Our results apply to any network of the form (2)—they only involve weight matrices but make no assumptions about the architecture.
We included explicit constructions for minimally expansive networks that are injective; interestingly, this simple criterion already improves inference in our preliminary experiments.
The results on universality of injective neural networks further justify their use in applications; they also implicitly justify the various Jacobian conditioning strategies when learning to generate real-world data.
Further, injective neural networks are topology-preserving homeomorphisms which opens applications in computational topology and establishes connections to tools such as self-organizing maps.
Analysis of deep neural networks has an analogue in the analysis of inverse problems where one studies uniqueness, stability and reconstruction.
Uniqueness coincides with injectivity, quantitative stability with the Lipschitz constant of the inverse, and, following Lei et al.
(2019), from a linear program we get a reconstruction in the range.
A. P ROOF OF T HEOREMA.2. U PPER B OUND ON M INIMAL E XPANSIVITY
Let (Ω , F , P ) be a probability space and let w i : Ω → R n , 1 ≤ i ≤ m be m iid Gaussian vectors on Ω stacked in a matrix W : Ω → R m × n .
We aim to ﬁnd conditions for a ReLU layer with the matrix W to be injective.
Injectivity fails at p the half-space { x : (cid:104) x, p (cid:105) ≥ 0 } contains fewer than n vectors w i .
Equivalently, injectivity fails if there is a half-space which contains more than m − n vectors w i (which implies that its opposite half-space has fewer than n ).
Let k = m − n + 1 and denote by A ∈ R k × n some k × n submatrix of W (for example, the ﬁrst k rows).
Our strategy is to ﬁrst bound the probability that for a ﬁxed subset of k rows of W , there exists an x having positive inner products will all k rows (which signals non-injectivity per above discussion).
Second, since there are (cid:0) mk (cid:1) subsets of k rows, we use the union bound to get an upper bound on the probability of non-injectivity.
For the ﬁrst part we follow the proof of Bürgisser & Cucker (2013, Theorem 13.6), parts of which we reproduce for the reader’s convenience.
For a sign pattern σ ∈ {− 1 , 0 , 1 } k , we denote by R A ( σ ) the set of all x ∈ R n which produce the sign pattern σ (they belong to the σ -“wedge”, possibly empty),
[Equation] [Equation]
For σ ∈ {− 1 , +1 } k = Σ , we deﬁne the event E σ as
[Equation]
We are interested in σ 0 = (1 , . .
. , 1) , meaning that all the inner products are positive.
Note that the probability of E σ is the same for all σ due to the symmetry of the Gaussian measure.
Further, note that (cid:80)
[Equation]
| is the number of wedges deﬁned by A .
Then
[Equation]
by Windner’s bound (Winder, 1966).
(The hyperplane arrangement is generic almost surely so we use equality.)
We now have the probability that for a subset of k vectors w i , there exists an x ∈ R n which has positive inner products with all k vectors.
We are interested in the following event which implies non-injectivity,
[Equation]
Conversely, ω ∈ E c NI implies almost sure injectivity.
Since there are (cid:0) mk (cid:1) =
(cid:0) m m − n +1 (cid:1) =
(cid:0) mn − 1 (cid:1) different subsets of k rows, we can bound the probability of E NI as
[Equation]
with (cid:15) =
( c − 1) − 1 .
, where H ( (cid:15) ) = − (cid:15) log 2 ( (cid:15) ) − (1 − (cid:15) )
H (1 − (cid:15) ) is the binary entropy function and we used a related bound on the sums of binomial coefﬁcients.
A.2. L OWER B OUND ON M INIMAL E XPANSIVITY OF L AYERS WITH G AUSSIAN W EIGHTS
In this appendix we prove that large random Gaussian weight matrices W ∈ R m × n yield noninjective ReLU layers with high probability if m < c ∗ n , where c ∗ ≈ 3 . 4 .
Note that injectivity fails if there exists a half-space in R n which contains less than n rows of W .
Equivalently, injectivity fails if there is a half-space with more than m − n rows of W , since then the opposite halfspace has less than n .
The core idea of the proof is to make an educated guess for a half-space which has many vectors, and compute the probability that it has more than m − n .
A good such guess is to take the halfspace deﬁned by the average direction of a row of W .
Equivalently, we study of the size of S ( w, W ) where w = 1 m (cid:80) mk
=1 w k , the row average of a matrix W .
If | S ( w, W ) | > m − n , then | S ( − w, W ) | < n , and W cannot have a DSS w.r.t.
− w .
As the Figure 6 shows, when m < c ∗ n , W does not contain a
DSS w.r.t.
− w with high probability.
Let W ∈ R m × n be a matrix such that the rows of W are i.i.d.
random vectors distributed as N (0 , I n ) .
Deﬁne the event E i as
[Equation]
Lemma A. Let m, n → ∞ so that mn → c .
Then P ( E i ) → 12 erfc (cid:16) − 1 √ 2 c (cid:17) .
Proof.
We have
[Equation]
where Y ∼ N (0 , m − 1)
We now use the fact that (cid:107) w i (cid:107) 2 concentrates around n .
Deﬁne the event
[Equation] [Equation]
[Table]
One can show using standard concentration arguments
that P ( D i ) ≥ 1 − 2 exp( − (cid:15) 2 n/ 4) , and so by the law of total probability,
[Equation]
Choosing (cid:15) =
(cid:15) ( n ) = n − 1 / 4 yields
[Equation]
Finally, substituting Z =
Y √ m − 1 ∼ N (0 , 1) yields
[Equation]
Both sandwiching probabilities converge to P (cid:110)
Z ≥ − 1 √ c (cid:111) =
12 erfc (cid:16) − 1 √ 2 c (cid:17) .
Noting that
P { D i } → 1 and P { D ci } → 0 we ﬁnally have from (27) that
[Equation]
Lemma B. Under the same conditions as Lemma A when i (cid:54) =
j ,
[Equation]
Proof.
The proof is similar to that of Lemma A.
In addition to concentration of norm of iid Gaussian vectors, we use the fact that (cid:104) w i , w j (cid:105) /m is of order n − 1 / 2 and that if P { D } → 1 as n → ∞ , then for a ﬁxed c and some event A , P { A | D } − P { A } → 0 .
Here D is the event that the various quantities are close to the value they concentrate about, and A asymptotically has the same probability as
E i ∩ E j .
Theorem 6.
Given a Gaussian weight matrix W ∈ R m × n , the layer ReLU( Wx ) with m/n → c is not injective with probability → 1 as n → ∞ when c < c ∗ , where c ∗ is the unique positive real solution to
[Equation]
The numerical value of c ∗ is ≈ 3 . 4 .
Proof.
Let X i = 1 (cid:104) w i , 1 m (cid:80) mj
=1 w j (cid:105) ≥ 0 , the indicator function of the event E i .
The expected number of w i with a positive inner product with (cid:80) mk =1 w i is by the linearity of expectation equal to
E (cid:32) m (cid:88) i =
[Equation]
X i (cid:33) = m · P { E i } =: mp.
By Chebyshev’s inequality
[Equation]
where σ 2 = V (cid:0) 1 m (cid:80) mi =1 X i (cid:1) is the variance of the sum.
We compute m 2 σ 2 = V (cid:32) m (cid:88) i =
X i (cid:33) =
E (cid:32) m (cid:88) i =
X i (cid:33)− (cid:32)
E m (cid:88) i =
X i (cid:33)= mp + (cid:88) i (cid:54) =
j
P ( E i ∩ E j ) − ( mp ) 2 .
Since P ( E i ∩ E j ) → p 2 by Lemmas A and B, we have for any i and j (cid:54)
=
i
[Equation]
Thus indeed for any t >
[Equation]
Finally,
[Equation]
Combining with Lemma A we get that
[Equation]
Thus
P
[Equation]
A. E XPERIMENTS TESTING THE MINIMAL EXPANSIVITY THRESHOLD
Figure 6 suggests that an expansivity factor of 2 . 1 as suggested by Lei et al.
(2019) is not enough to ensure global injectivity with Gaussian weights.
This in turn leads to failure cases when inverting the network in applications to inverse problems, as we show in this section.
We use a single ReLU layer, f ( x ) =
ReLU( Wx ) , and choose x to be an MNIST digit for ease of visualization of the “latent” space.
We choose W ∈ R cn × n with iid Gaussian entries and set c = m/n = 2 . 1 .
Given y =
f ( x ) , our goal is to reconstruct x using, for example, gradient descent.
This requires at least n of the cn entries of f ( x ) to be non-zero since x ∈ R n .
As shown in Figure 6, x 0 = − 1 m (cid:80) j w j / (cid:107) w j (cid:107) violates this condition with high probability when c
= 2 . 1 .
We choose 2 data samples, x 1 and x 2 to be 2 “latent” codes.
Note that W has a DSS with respect to x 1 .
This ensures that given f ( x 1 ) one can reconstruct x 1 .
Now, in order to show that injectivity around a few points is insufﬁcient for global injectivity, we linearly interpolate between x 0 and xand x 0 and x 2 in a 2 D grid.
We show the intermediate x s as images in Figure 7 .
The x s shown in red cannot be uniquely recovered given f ( x ) , i.e., there exist inﬁnitely many samples close to the red samples that all map to the same output.
To further illustrate this, we also run a simple experiment where we have y = f
( x ) + η , with a 10 dB signal-to-noise ratio.
We invert f to estimate x for 3 different samples in the domain of f .
We can easily see that non-injective points of the domain give poorer reconstructions due to the lower number of positive inner products with W .
In order to avoid problems of non-convexity in optimization we report the best reconstruction out of 10 in 4 different trials with different random seeds, similar to experiments of Bora et al.
While these results show reconstructions with single layer, the issues are only exacerbated with multiple layers.
A. P ROOF OF T HEOREM
We divide the proof into parts for ease of understanding.
Lemma 2 (Inverse Lipschitz Constant: Face Adjacent Wedges) .
Let W ∈ R m × n have a DSS w.r.t.
every x ∈ R n and x 0 , x 1 ∈ R n .Deﬁne
[Equation]
[Figure] [Figure]
and suppose that x 0 and x 1 are such that there is a t (cid:48) ∈ (0 , 1) such that for all t ∈ [0 , 1]
[Equation]
Suppose further that x 0 , x 1 are such that there is some δ > 0 such that for all δx ∈ R n , if (cid:107) δx (cid:107) 2 < δ then there is a t (cid:48)
+ δt such that
[Equation]
Then
[Equation]
where σ ( M ) is the smallest singular value of the matrix M .
Remark 3 .
The conditions of Lemma 2 on x 0 and x 1 may look very odd, but they have a very natural geometric meaning.
The DSS can be thought of as slicing R n into wedges by a series of hyperplanes that have the rows of W as normals.
The condition in (31) is interpreted as that the line segment that connects x 0 to x 1 passes from x 0 ’s wedge into x 1 ’s wedge without passing through any wedges in between (see Figure 8a, as opposed to
Figure 8b).
The implies that x 0 and x 1 must be in wedges that share a boundary.
The condition in (32) requires that the wedges of x 0 and x 1 share a face, and not just a corner.
Proof.
We denote
[Equation]
First we will show that if w i , w i (cid:48) ∈ W 0 \ 1 , then w i and w i (cid:48) must be parallel.
From equation (31) and the continuity of ReLU( W(cid:96) x 0 ,x 1 ( t )) w.r.t.
t , we have that there is a t (cid:48) ∈ (0 , 1) such that
[Equation]
If w i and w i (cid:48) are not parallel, then let δx be some vector 0 < (cid:107) δx (cid:107) <
δ that is perpendicular to w i but not w i (cid:48) then
[Equation]
[Equation]
37) which contradicts (32).
W.l.o.g.
the same argument applies to W 1 \ 0 and also it is straight forward to see that all elements of W 0 \ 1 must be anti-parallel to all elements of W 1 \ 0 .
From W 1 \ 0 and W 0 \parallelism, there is a c ≥ 0 such that for all x ∈ R n ,
[Equation]
Assume that c ≥
[Equation]
then
[Equation] [Equation]
The antepenultimate inequality comes as from the deﬁnition of W 0 \ 1 , we have that W 0 \ 1 x 0 and − W 0 \ 1 x 1 are the same sign, thus (98) applies.
In the case that c < 1 , then the rolls of W 0 \ 1 and
W 1 \ 0 can be switched, and the same result (with σ ( W 1 ) in place of σ ( W 0 ) ) is obtained.
In either case,
[Equation] [Equation]
Lemma 3 (Inverse Lipschitz Constant: Connected through Faces) .
Let W ∈ R m × n have a DSS w.r.t.
every x ∈ R n .
Let x 0 , x 1 be such that the line connecting passes through n t wedges, and through their faces (in the sense of Lemma 3).
Then
[Equation]
Proof.
Let t 1 = 0 , t n t = 1 , and let T = { t k } n t k =1 such that t k < t k +1 for k < n t and x k :=
(cid:96) x 0 ,x 1 ( t k ) are each in different wedges.
Let c = 1 √ 2 min t ∈ [0 , 1] σ ( W | S ( (cid:96) x 0 ,x 1 ( t ) ,W ) ) , then
[Equation]
(47) then by Lemma 2,
[Equation]
and by Lemma 10 we have n
[Equation]
Combining (47) - (49) yields
[Equation]
Proof of Theorem 3.
Let x 0 and x 1 be given.
If the line connecting x 0 and x 1 does not pass through any wedge corners
[Equation]
then we can apply Lemma 3 directly
[Equation]
and get that
[Equation]
We now argue that the number of wedges that (cid:96) x 0 ,x 1 ( t ) passes through (and so n t ) is at most m .
For each j ∈ [[ m ]] ,
[Equation]
52) is monotone increasing or decreasing.
This implies that each w j ∈ W can enter or exit the DSS of W w.r.t.
(cid:96) x 0 ,x 1 ( t ) at most once, therefore the total number of unique DSS w.r.t.
(cid:96) x 0 ,x 1 ( t ) (i.e. wedges (cid:96) x 0 ,x 1 ( t ) pass through) is at most m .
Hence, n t ≤ m .
Clearly
[Equation] [Equation]
[Equation]
hence we have
[Equation] [Equation]
Now we show that if Lemma 3 does not apply to two points (namely, the line (cid:96) x 0 ,x 1 ( t ) passes through a corner in the sense of Remark 3), then the two points can be perturbed an arbitrarily small amount, so that the perturbed points do satisfy Lemma 3.
The corners (again, as in Remark 3) describe the points which are orthogonal to at least two w j 1 , w j 2 ∈ W .
w j 1 and w j 2 must not be parallel to each other (otherwise Lemma 2 would apply), thus the set of points orthogonal to both w j 1 and w jconstitute a n − 2 dimensional linear space in R n .
Let x 0 and x 1 be such that (cid:96) x 0 ,x 1 ( t ) intersects one of these corners.
By considering ˜ x 0 = δx + x 0 , ˜ x 1 = δx + x 1 where δx is perpendicular to x 1 − x 0 , we can obtained a line (cid:96) ˜ x 0 , ˜ x 1 ( t ) so that (cid:96) ˜ x 0 , ˜ x 1 ( t ) and (cid:96) x 0 ,x 1 ( t ) do not intersect.
The choice of δx is n − 1 dimensional, thus for every δ >and w j 1 and w j 2 (that are non-perpendicular) there is a δx so that (cid:107) δx (cid:107)
<
δ and (cid:96) ˜ x 0 , ˜ x 1 ( t ) does not intersect the corner of w j 1 and w j 2 .
Consider a sequence of ˜ x
( i ) 0 , ˜ x
( i ) 1 , i = 1 , . . .
such that lim i →∞ (˜ x ( i ) 0 , x ( i ) 1 ) =
( x 0 , x 1 ) and (cid:96) ˜
x ( i ) 0 ,x ( i ) 1 ( t ) does not pass through a corner for any i .
Given that (cid:107)·(cid:107) 2 and ReLU( W ( · )) are continuous and so by
Lemma
[Equation]
thus
[Equation]
and so
[Equation]
Remark 4 (Factor of 1 √ m in Theorem 3.) .
Throughout this paper we deﬁne the discrete norm of y ∈ R d as
[Equation] [Equation]
This is to be contrasted with the norm that arise from the discretization of the L 2 function norm on a ﬁnite domain.
For example
[Equation]
if we instead thought of y as a discrete sampling of a continuous function ˜
y ∈
L 2 ([0 , 1]) , such that ∀ j = 1 , . . .
, d
[Equation]
(60) then we could approximate the L 2 ([0 , 1]) norm of ˜ y by
[Equation]
If we express Theorem 3 in terms of (cid:107)·(cid:107) l 2 ([0 , 1]) , then it would become
[Equation]
A. T HEOREM
Example 1 (Applying Theorem 4, One Channel) .
Consider a layer of the form Reshape (ReLU( Wx ))
where W =
[ C T 1 , · · · , C Tq ] T and each C k is a convolution operator with kernel c k .
Suppose further that W ∈ R 4 × 4 × 1024 × 100 =
R 16384 × 100 (as in Radford et al.
The reshaping operator takes the 16384 single-channel output of W and transforms it into a multi-channel signal.
This is necessary for subsequent convolutions, but plays no role in injectivity.
Let q = 8 , and the 2 × 2 convolution kernels be given as
[Equation]
Directly proving that a 16384 × 100 dimension operator has a DSS w.r.t.
every x ∈ R 100 is daunting.
However, since each layer of the operator is given by one of only 8 simple convolutions, we can leverage Theorem 4 to signiﬁcantly simplify the problem.
Choosing P =
(2 , 2) , implies that
Z (2 , 2) ( c k ) =
c k , and so W | Z (2 , 2) =
(cid:83) 8 k =1 c k .
Further, it is easy to see, that { c 1 , c 2 , c 3 , c 4 } is a basis for R 2 × 2 , so Corollary 2 applies, W | Z P has a DSS for R 4 , and by Theorem 4, ReLU( Wx ) is injective.
An example when a layer is injective but P in Theorem 4 must be greater than O is when W is a convolution of 4 kernels of width
[Equation]
If we choose P = (3) , then W | Z (3) =
(cid:83) 4 k
=
{ c k } .
only has four elements, and so cannot have a DSS w.r.t.
every x R 3 by Corollary 2.
If we however choose P = (4) , then Z (4) ( c k ) =
{ [ c k 0] , [0 c k ] } and W | Z (4) has a DSS of R 4 w.r.t.
all x ∈ R 4 (from Corollary 2), so W has a DSS w.r.t.
all x ∈ R N .
This example suggests that to apply Theorem 4 to a convolution layer with q kernels of width O , we must choose P so that W | Z P has at least the minimal number 2 (cid:81) pj =1 P j of vectors to have a DSS of a vector space of dimension | P | = (cid:81) pj =1 P j .
Some algebra gives that q ≥ 2 (cid:81) pj
= 1 1 − O j /P
j and P j ≥ O j 1 − (
[Equation]
/q ) 1 /p , where the last inequality holds only when P j O j is independent of j .
Before we can commence with the proof of Theorem 4, we prove the following results.
Lemma 4 (Domain Decomposition) .
Suppose that R n = span { Ω 1 , . . .
, Ω K } 3 where each Ω k is a subspace and for each k = 1 , . . .
, K we have a
[Equation]
and
[Equation]
such that w k,(cid:96) ∈ Ω k and W k has a DSS of Ω k w.r.t.
every x ∈ Ω k .
Then W has a DSS of R n w.r.t.
every x ∈ R n .
Proof.
For every k = 1 , . . .
, K W k | S ( x,W k ) has a DSS of Ω k with respect to P Ω k ( x ) where P Ω k is the orthogonal projection of R n onto Ω k .
For every w k,(cid:96) ,
[Equation]
thus S ( x, W k ) ⊂
S ( x, W ) .
From Ω k ⊂ span( W k | S ( x,W k ) ) for each k we have a set spanning Ω k that lie in W | S ( x,W ) , hence
[Equation]
contains a DSS of R n w.r.t.
x .
The set (cid:83) Kk
=1 { w k,(cid:96) } N k
(cid:96)
=1 has no dependence on x , thus it is true for all x ∈ R n .
Lemma 5.
Given a convolution operator C ∈ R N × N .
Let 0 ≤ V be such that V + O ≤ N .
For each x ∈ C ,
[Equation]
where
[Equation]
Note that (aug V : V + O ( x ))
J restricted to the indices 1 + V through V + P is exactly x
Deﬁnition 4 (Zero-Padded Kernel) .
Let c be a convolution kernel of width O , and let P be another multi-index.
We deﬁne the set of zero-padded kernels 4 of c as the set
Z P ( c
[Equation]
Note that the above notation implies that Z P ( c ) = {} if O (cid:54)≤ P .
The above is always well deﬁned, as c Q − T + O +1 is well deﬁned iff 1 ≤ Q − T + O + 1 ≤ O (recall that the kernel c is of width O ), and so
[Equation]
Now we proceed with the proof of Theorem 4.
Proof of Theorem 4.
The strategy for this proof will be to use Lemma 4 by decomposing R n into some number of different domains { Ω v } n v v =1 , each of which are a restriction of R n to P non-zero components.
For each Ω v , the elements of W that lie in Ω v can be identiﬁed as the elements in
W | Z P .
If for one v the components of W | Z P form a DSS of Ω v w.r.t.
every x ∈ Ω v , then it has a
DSS for every such Ω k , so we can apply Lemma 4 and get that W has a DSS of R n w.r.t.
all x ∈ R n .
Given an offset V (cid:48) ≥ 0 such that V (cid:48)
+ P ≤ N , deﬁne Ω V (cid:48) as the subspace of all vectors x ∈ R N such that
[Equation]
From Lemma 5, for any k = 1 , . . . , n v , if x k,P ∈ Z P ( c k ) , C k is a submatrix of W and
[Equation]
Further, for any such x k,P , aug
[Equation]
x k
[Equation]
(cid:48) .
[Equation]
If W | Z P contains a DSS for R P w.r.t.
all x ∈ R P , then aug V (cid:48)
[Equation]
P ( W | Z P ) contains a DSS for Ω V (cid:48)
w
[Equation]
x
[Equation] [Equation]
This follows from Lemma 4.
From Lemma 5 for any V (cid:48) such that
[Equation]
if W | Z P has a DSS of R P w.r.t.
all x ∈ R P , then W contains a DSS of Ω V (cid:48) for all 0 ≤ V (cid:48) ≤ N − P .
Finally, note that span( { Ω V (cid:48) } N − P V (cid:48)
=0 ) , and so using (75),
(76) we can apply Lemma 4 and ﬁnd that W contains a DSS of R N w.r.t.
all x ∈ R N .
For a multi-channel input (with n c channels) x ∈ R N × R N × · · · × R N (cid:124) (cid:123)(cid:122) (cid:125) n c times , a multi-channel convolution C on x is given by Cx = (cid:80) n c q =1 C q x q where C o is a convolution on R N (deﬁned by
Deﬁnition 2) and x o ∈ R N is the restriction of x to the o ’th channel.
Because of the additive structure of multi-channel convolutions a n c over R N = R N 1 × · · · × R N p dimensional domain of width O =
( O 1 , . . .
, O p ) with kernels c 1 , . .
, c
n c is equivalent to a single convolution of width ( O, n c ) =
( O 1 , . . .
, O p , n c ) over R ( N,p ) =
R N 1 × · · · × R N p × R n c .
This follows from
[Equation]
A PPENDIX B R OBUSTNESS TO B ATCH , W EIGHT , AND S PECTRAL N ORMALIZATION
Normalization strategies such as batch (Ioffe & Szegedy, 2015), layer (Ba et al., 2016), instance (Ulyanov et al., 2016), group (Wu & He, 2018), weight (Salimans & Kingma, 2016) and spectral (Miyato et al., 2018) normalization promote convergence during training and encourage low generalization error.
Normalization is a many-to-one operation.
In this section we show that batch, weight, and spectral normalization do not interfere with injectivity provided, of course, that the network is injective without normalization.
We ask if injectivity is compatible with normalization in a trained network.
Let { x i } i =1 ,...,m represent the inputs to a given layer over a mini-batch.
The batch normalization adds two learnable parameters ( γ, β ) and transforms x i to y i as
[Equation]
Although for a given γ, β the relationship between x i and y i is not injective , batch normalization is usually present during training but not at test time.
Provided that the learned weights satisfy Theorem 1 and Lemma 1, batch normalization does not spoil injectivity.
In batch renormalization (Ioffe, 2017) it is still desirable to whiten the input into each layer.
During run time there may be no mini-batch, so running averages of the σ i and µ i (denoted ˆ σ, ˆ µ ) computed during training are used.
Batch renormalization at test time is then ˆ x = x − ˆ µ √ ˆ σ 2 + (cid:15) , y
= γ ˆ x + β .
Since, importantly, ˆ σ and ˆ µ are not functions of x , the mapping between x and y is one-to-one (when γ (cid:54) = 0 ), thus such normalization in an injective network does not spoil the injectivity.
In weight normalization (Salimans & Kingma, 2016) the coefﬁcients of the weight matrices are normalized to have a given magnitude.
This normalization is not a function of the input or output signals, but rather of the weight matrices themselves.
This plays no role in injectivity.
B. L AYER , I NSTANCE AND G ROUP N ORMALIZATION
layer, instance and group normalization, unlike batch normalization, take place during both training and execution and, unlike weight and spectral normalization, the normalization is done on the input/outputs of layers instead of on the weight matrices.
For these normalizations (2) is modiﬁed so that it becomes
[Equation]
where M (cid:96) : R n i +1 → R n i +1 are normalization functions that are many-to-one.
In general φ (cid:96) ( W (cid:96) M (cid:96) ( · ) + b (cid:96) ) will not be injective for any φ (cid:96) , W (cid:96) , b (cid:96) on account of M (cid:96) , but for all of the mentioned normalization techniques we can get near injectivity.
Before we descend into the particular we make the following observation about normalization methods that obey a certain structure.
Deﬁnition 5 (Scalar-Augmented Injective Normalization) .
Let M (cid:96) ( x ): R n → R n be a normalization function that is understood to be many-to-one.
We say that M (cid:96) ( x ) is scalar-augmented injective if there exists a function m (cid:96) ( x ): R n → R k where k (cid:28) n and ˜ M (cid:96) : R n × R k
→ R n such that
[Equation] [Equation]
and ˜ M (cid:96) ( x ; m (cid:96) ( x )) is injective on x given m (cid:96) ( x ) .
An example of a normalization function that is scalar-augmented injective is
[Equation]
For this choice of M (cid:96) , k = 1 , and
[Equation]
With this deﬁnition, we can prove the following trivial but useful result
Lemma 6 (Restricted Injectivity of Scalar-Augmented Normalized Networks) .
Let N be a deep network of the form in (78) and let each φ (cid:96) ( W (cid:96) · ) be layer-wise injective.
Let the normalization functions { M (cid:96) } (cid:96)
=1 ,...,L each be scalar-augmented injective.
Then given { m (cid:96) ( x ) } (cid:96) =1 ,...,L , the network ˜ N ( z ; m 1 , . . .
, m (cid:96) ) =
φ L ( W L ˜ M L ( · · · φ 2 ( W 2 ˜ M 2 ( φ 1 ( W 1 z + b 1 ); m 2 ) + b 2 ) · · · + b L ; m L ))
(82) is injective.
Proof.
The proof of Lemma 6 follows from a straightforward application of induction, combined with Deﬁnition 5.
Remark 5 .
Note that Lemma 6 implies that for a ﬁxed { m (cid:96) ( x ) } (cid:96) =1 ,...,L , there is at most one value of z such that
[Equation]
where N ( z ) is given by (78), and that the z ’s on both sides of (83) are the same.
It is still entirely possible that there are is another choice of z (cid:48) , { m (cid:96) ( x ) } (cid:96) =1 ,...,L such
[Equation]
An example of this would be if M (cid:96) is of the form in (80), then
[Equation]
In other words, Lemma 6 implies that the deep network is injective (in z ) for a ﬁxed { m (cid:96) ( x ) } (cid:96) =1 ,...,L , but it may still not be injective for all z and { m (cid:96) ( x ) } (cid:96) =1 ,...,L .
With Lemma 6 in tow, we can show that layer, instance, and group normalization are all scalaraugmented injective normalizations, so Lemma 6 applies and yields a kind of injectivity.
Layer, instance and group normalization are all related insofar as they can all be expressed in the same abstract form.
For a given input x , all three break x up into K parts denoted { x | S k } k =1 ,...,K such that for each k = 1 , . . .
, K
[Equation]
The differences between the three normalization are how { S k } k =1 ,...,K is chosen.
For layer normalization K = 1 and the normalization is applied to the entire input signal.
For instance normalization, there is one S k for each channel, and the x | S k restricts x to just one channel of inputs, that is the normalization is done channel-wise.
Group normalization is part way between these two, where k is less than the number of channels, and channels are batched together.
In any case, for any of these normalization methods, they are all scalar-augmented injective normalization where
[Equation]
Thus, by Lemma 6 their corresponding deep networks are all injective, provided that for each (cid:96) , { σ k,(cid:96) , µ k,(cid:96) } k =1 ,...,K is saved.
B. P OOLING O PERATIONS
Although pooling may have a different aim than typical normalization, we consider it in this section, as it is mathematically similar to (78).
Pooling is similar to layer, instance and group normalization in the sense that they partition the input space into K disjoint pieces, and then output a weighted average upon each piece.
Speciﬁcally, if M p ( x ): R n → R K where for k = 1 , . .
, K ,
[Equation]
where (cid:107)·(cid:107) p is the discrete p norm of x restricted to the set S k .
For p = 1 this is the mean of the absolute value, for p =
this is the Euclidean mean and for p = ∞
it is the maximum of the absolute value.
The injectivity of this operation in the cases where p = 1 , 2 , ∞ is considered in the work
Bruna et al.
A PPENDIX C P ROOFS FROM S ECTIONC. P ROOF OF T HEOREM
To prove Theorem 5 we combine the approximation results for neural networks and the low regularity version of the generic orthogonal projector technique used to prove the easy version of the Whitney’s embedding theorem (Hirsch, 2012, Chapter 2, Theorem 3.5) that shows that a C 2 -smooth manifold of dimension n can be embedded in R 2 n +1 with an injective, C 2 -smooth map.
To prove the result, we ﬁrst approximate f ( x ) by a ReLU-type neural network that is only Lipschitz-smooth, so the graph of the map F θ is only a Lipschitz-manifold.
We note that limited regularity often causes signiﬁcant difﬁculties for embedding results, as for example for the Lipschitz-smooth manifolds it is presently known only that a n -dimensional manifold can be embedded (without preserving distances) in the
Euclidean space R N of dimension N = ( n + 1) 2 , and the classical Whitney problem, whether a n -dimensional manifold can be embedded in R 2 n +1 , is still open Luukkainen & Väisälä (1977);
Cobza¸s et al.
Due to this lack of smoothness, we recall the details how this generic projector technique works.
Proof of Theorem 5.
Let ε > 0 and Z ⊂ R n be a compact set.
As a continuous function f : R n → R n can be uniformly approximated in a compact set by a C ∞ -smooth function (see e.g. (Adams, 1975, Thm. 2.29)), we can without loss of generality assume that f smooth and therefore a locally
Lipschitz function.
By classical results of approximation theory for shallow neural networks, see Hornik (1991); Leshno et al.
; Pinkus (1999), for any L ≥ 1 there are m and a neural network F θ ∈ NN ( n, m, L, m ) such that
[Equation]
We note that by using recent results for deep neural networks, e.g. by Yarotsky (2017), one can obtain efﬁcient estimates on how a given accuracy ε can be obtained using sufﬁciently large L and m .
Our aim is the perturb F θ : R n → R m so that it becomes injective.
We assume that Z ⊂ B n (0 , r 1 ) , where B n (0 , r 1 ) ⊂ R n is an open ball having centre 0 and radius r 1 > 0 .
We denote the closure of this ball by B n (0 , r 1 ) .
Let D = m + n , α > 0 , and deﬁne a map H θ : R n → R m + n ,
[Equation]
Observe that the map H θ : R n → R D is injective.
Let V ( k, D ) denote the set of k -tuples ( v 1 , v 2 , . .
, v k ) where v j are orthonormal vectors in R D .
Such vectors span a k -dimensional linear space.
Furthermore, let G ( k, D ) denote the set of k -dimensional linear subspaces of R D , and for V ∈ G ( k, D ) , V = span ( v 1 , v 2 , . .
. , v k ) , let
P V
=
P
v 1 ,v 2 ,...,v k ) :
R D → R D be an orthogonal projection which image is the space V .
As the dimension of the orthogonal group O ( k ) is k ( k − 1) / 2 and by Milnor & Stasheff (1974), the set G ( k, D ) , called the
Grassmannian, is a smooth algebraic variety, of dimension k
( D − k ) and the dimension of V ( k, D ) is k
( D − k )
+ k ( k − 1) /
2 =
k (2 D − k − 1) / 2 .
To prove Theorem 5, we need the following lemma.
Lemma 7.
Let H θ ∈ NN ( n, D ) , D > d ≥ 2 n + 1 be a neural network such that H θ : R n → R D is injective.
Let X θ
=
{ V ∈ G ( d, D ) : P V ◦ H θ : R n → R D is injective } .
Then the set X θ is an intersection of countably many open and dense subsets of G ( d, D ) , that is, elements of X θ are generic.
Moreover, the d ( D − d ) dimensional Hausdorff measure of the complement of X θ in G ( d, D ) is zero.
Note that for V ∈ X θ , we have P V ◦ H θ ∈ NN ( n, D ) .
Proof.
We use that fact that H θ : R n → R D is injective and locally Lipschitz-smooth.
Recall that f : R n → R m is locally Lipschitz if for any compact set Z ⊂ R n there is L Z > 0 such that | f ( x ) − f
( y )
| ≤ L Z | x − y | for all x, y ∈ Z .
Let v ∈ R m be a unit vector and let Q v : R D → R D be the projection Q v ( z ) =
z − ( z · v ) v .
Let
[Equation]
As H θ : R n → R D is an injection, we can deﬁne the map
[Equation]
As observed in the proof of the Whitney’s embedding theorem (Hirsch, 2012, Chapter 2, Theorem 3.5), the map Q w ◦ H θ : R n → R D is an injection when w ∈ S D − 1 .
To see this, assume that w ∈ S D − 1 satisﬁes w (cid:54)∈ s θ ( A ) , that is, w is not in the image of s θ .
Then, if there are x, y ∈ R n , x (cid:54) =
y such that Q w H θ ( x ) =
Q w H θ ( y ) , we see that there is t ∈ R such that H θ ( x ) − H θ ( y ) =
tw.
As w is a unit vector, this yields that t = | H θ ( x ) − H θ ( y )
| (cid:54) = 0 and w =
( H θ ( x ) − H θ ( y )) /t
=
s θ ( x, y ) , which is in contradiction with the assumption that w (cid:54)∈ s θ ( A ) .
Hence, w (cid:54)∈ s θ ( A ) yields that
Q w ◦ H θ : R n → R D is an injection.
We consider the image s θ ( A ) ⊂ S D − 1 .
For h > 0 , let
[Equation]
As H θ : B n (0 , h − 1 ) → R m is Lipschitz-smooth with some Lipschitz constant L h , we see that the map s θ : A h → S D − 1 is Lipschitz-smooth.
Since the set A has the Hausdorff dimension 2 n and the map s θ : A h → S D − 1 is Lipschitz-smooth, the Hausdorff dimension of the set s θ ( A h ) is at most 2 n , see e.g. Morgan (2016, p. 26).
The set A is the union of all sets A h j , where h j = 1 /j and j ∈ Z .
By
Mattila (1995, p. 59) the Hausdorff dimension of a countable union of sets S j is the supremum of the Hausdorff dimension of the sets S j .
Hence s θ ( A ) =
(cid:83) ∞
j =1 s θ ( A h j ) has the Hausdorff dimension less or equal 2 n .
Since the dimension D − 1 of S D − 1 is strictly larger than 2 n , we see that the set s θ ( A h j ) is closed, its complement is an open and dense set, and thus the set Y 1 ( θ ) :=
S D − 1 \ s θ ( A ) is an intersection of countably many open and dense sets.
Observe that as Q w 1 is a linear map, the map Q w 1 ◦ H θ is also a neural network that belongs in NN ( n, D ) , and we can denote Q w 1 ◦ H θ
= H θ 1 with some parameters θ 1 .
Thus we can repeat the above arguments using the map Q w 1 ◦ H θ : R n → span ( w 1 ) ⊥ ≡ R D − 1 instead of
H θ : R n → R D .
Repeating the above arguments D − d times, can choose orthonormal vectors w j ∈ S D − 1 , j = 1 , 2 , . . .
, D − d , and sets Y j ( θ, w 1 , . . .
, w j − 1 )
⊂
S D − 1 ∩ ( w 1 , . .
. , w j − 1 ) ⊥ , which D − j dimensional Hausdorff measures vanish and which complements are intersections of countably many open and dense sets.
Let B θ to be the set of all n-tuples ( w 1 , . . .
, w D − d ) where w 1 ∈ Y 1 ( θ ) and w j ∈ Y 2 ( θ, w 1 , . . . , w j − 1 ) for all j = 2 , . . .
, D − d .
Note that all such vectors w j , j = 1 , 2 , . . .
, D − d are orthogonal vectors spanning a D − d dimensional vector space V , and the map
[Equation]
where P
[Equation]
is injective.
By the above, construction the ( D − ( d + 1) / 2) d dimensional Hausdorff measure of the complement of B θ in V ( D, d ) is zero and B θ is generic set.
As the dimension of the set of the orthogonal basis in a d -dimensional vector space is ( d − 1) d/ 2 , we obtain the claim.
Now we continue with the proof of Theorem 5.
Let V
= { (0 , 0 , . .
× R m ⊂ R D .
By applying
Lemma 7 with d = m we see that any neighborhood of V 0 in G ( d, D ) contain a m
-dimensional vector space
V ∈ X θ .
Then P V ◦ H θ is injective.
We can choose V ∈ X θ to be so close to V 0 that there is a rotation R V ∈
O ( D ) of the space R D , that maps the subspace V to the subspace V 0 , such that
[Equation]
where Id is the identity function.
Let π 0 : R D
→ R m be the map π 0 ( y
[Equation]
, y
[Equation]
) =
y (cid:48)(cid:48) be the projection to the last m coordinates.
Then
[Equation]
This and (90) imply that the claim of Theorem 5 holds for the injective neural network N θ
=
π 0 ◦ R V ◦ P V ◦ H θ : R n → R m .
Lemma 7 used in the above proof yields also Corollary 3.
Proof of Corollary 3.
Observe that a measure µ that is absolutely continuous with respect to the
Lebesgue (cid:81) kj =1 R d 2 j × d 2 j − 1 is absolutely continuous also with respect to the normalized Gaussian distribution, and that if the set S = { F k is not injective } has measure zero with respect to the normalized Gaussian distribution, then its µ -measure is also zero.
Thus we can assume without loss of generality that the elements of matrices B j are independent and have normalized Gaussian distributions.
Assume next that we have shown that F j − 1 : R n → R d 2 j − 2 is injective almost surely.
Then, f ( j ) θ ◦ F j − 1 : R n → R d 2 j − 1 is injective almost surely.
If d 2 j > d 2 j − 1 , the matrix B j is almost surely injective and so is
F j = B
j ◦ f ( j ) θ ◦ F j − 1 : R n → R d 2 j .
Thus, it is enough to consider the case when d 2 j − 1 ≥ d 2 j ≥ 2 n + 1 .
Then the matrix B j : R d 2 j − 1 → R d 2 j has almost surely rank d 2 j .
By using the singular value decomposition, we can write B j = R 1 j D j R 2 j where R 1 j ∈ O
( d 2 j ) , R 2 j ∈ O ( d 2 j − 1 ) ,
D j ∈ R
d 2 j × d 2 j − 1 is matrix which principal diagonal elements are almost surely strictly positive and the other elements are zeros.
Let V j
=
( R 2 j ) − 1 ( R d 2 j × { (0 , 0 , . . . , 0) } )
⊂
R d 2 j − 1 .
Let P V
j be an orthogonal projector in R d 2 j − 1 onto the space V j of dimension d 2 j .
As the distribution of the matrix
B j is invariant in rotations of the space, so is the distribution of linear space V j in G ( d 2 j − 1 , d 2 j ) .
By Lemma 7, we see that the map P V j ◦ f ( j ) θ ◦ F j − 1 : R n → R d 2 j − 1 is injective almost surely.
This implies that the map B j ◦ f ( j ) θ ◦ F j − 1 : R n → R d 2 j is injective almost surely, that is, the map
F j : R n → R d 2 j is injective almost surely.
The claim follows by induction.
In Baraniuk & Wakin (2009), see also Hegde et al.
; Iwen & Maggioni (2013), Broomhead &
Kirby (2001; 2000), the authors study manifold learning using random projectors.
These results are related to the proof of Theorem 5 above.
Let H θ be given by (91), a ReLU-based neural network whose graph M ⊂ R d , d = 2 n + m .
When P V is a random projector in R d onto a m -dimensional linear subspace V , the injectivity of the neural network P V ◦ H θ is closely related to the property that P V ( M ) is an n -dimensional submanifold with a large probability.
In Broomhead & Kirby (2001; 2000) the authors use Whitney embedding results for C 2 -smooth manifold for dimension reduction of data.
Our proof applies similar techniques for Lipschitz-smooth maps.
In Baraniuk &
Wakin (2009); Hegde et al. (2008); Iwen & Maggioni (2013), the authors apply the result
that when
M ⊂ R D is a submanifold and D is large enough, a random m -dimensional projector P V satisﬁes on M the restricted isometry property with a large probability.
In this case, P V ◦ H θ is not only an injection but its inverse map is also a local Lipschitz map.
In this sense, the techniques in Baraniuk & Wakin (2009); Hegde et al.
(2008); Iwen & Maggioni (2013) would give improved results to the generic projection technique used in this paper.
The results in Baraniuk & Wakin (2009); Hegde et al.
; Iwen & Maggioni (2013), however, require that the dimension m of image space of the map f : R n → R m satisﬁes m ≥ C log( ε − 1 ) , where ε is the precision parameter in the inequality (10).
Our result, Theorem 5 requires only that m ≥ 2 n + 1 .
A PPENDIX D M ISCELLANEOUS L EMMAS
Lemma 8.
Let x, y, z ∈ R and z ≥ 0 .
Then
[Equation]
Lemma 9 (Useful Inequalities) .
The following geometric inequalities are useful and have straight forward proofs.
Suppose that a, b, c ∈ R n , then 1.
For i = 1 , . . .
, k , let a i ∈ R m .
If
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)  , (97) then if for each j = 1 , . . .
, m , a i | j · a i (cid:48) | j ≥ 0 for each pair i, i (cid:48) ∈ [[ m ]] , then k (cid:88) i
= (cid:107)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) k (cid:88)
i = a i (cid:13)(cid:13)(cid:13)(cid:13) (cid:13).
If a, b ∈ R n and (cid:104) a, b (cid:105) ≥ 0 , then
[Equation]
Lemma 10 (Co-linear Additivity of ReLU( W · ) ) .
Let W ∈ R m × n , x 1 , x 2 ∈ R n , (cid:96) x 1 ,x 2 ( t ) =
(1 − t ) x 1 + tx 2 .
Let there be
[Equation]
then
[Equation]
and
[Equation]
Proof.
Let x t = (cid:96) x 1 ,x 2 ( t ) , then as a function of t , the j ’th component of ReLU( Wx t ) is either increasing (if (cid:104) w j , x 2 − x 1 (cid:105) ≥ 0 ) or decreasing (if (cid:104) w j , x 2 − x 1 (cid:105) ≤ 0 ).
In either case, it is clear that for each j = 1 , . . .
, m
[Equation]
)) | j ≥(103) hence clearly (cid:104) (ReLU( Wx 1 ) − ReLU( Wx 2 )) , (ReLU( Wx 2 ) − ReLU( Wx 3 )) (cid:105) ≥(104) thus we can apply (99) and we obtain (101).
Applying (98) then yields (102).
A PPENDIX E D ETAILED C OMPARISON TO P RIOR W ORK E. C OMPARISON TO B RUNA et al.
In Bruna et al.
(2013, Proposition 2.2.)
the authors give a result invoking a condition similar to our
DSS condition (Deﬁnition 1).
It also concerns injectivity of a ReLU layer in terms of the injectivity of the weight matrix restricted to certain rows.
The authors also compute a bi-Lipschitz bound for a layer (similar to our Theorem 3), though as we show in the following examples their analysis is in some cases not precisely aligned with injectivity.
Their criterion is given in two parts.
For a weight matrix, they ﬁrst deﬁne a notion of admissible set which indicates the points where the weight matrix’s injectivity must be tested.
Injectivity follows provided that the weight matrix is non-singular when restricted to each admissible set.
Given a weight matrix W ∈ R M × N and bias b , the authors say that Ω ⊂ { 1 , . . .
, M } is admissible if
[Equation]
is not empty.
For our analysis we focus on the case when b ≡ 0 .
In this case Ω is admissible if and only if
[Equation]
such that
[Equation]
Note that the inequality in (106) is strict, unlike (2).
If, for example, W has a column that is the zero vector, then there are no admissible Ω .
The authors use the notation Ω to denote all admissible sets for a given weight matrix.
In their notation F is the transpose of our weight matrix W , F Ω are the Ω rows of the weight matrix, F Ω | V Ω is the subspace generated by the Ω rows of W .
The authors also call the ReLU function the half-rectiﬁcation function.
λ − ( F ) and λ + ( F ) denote the lower and upper frame bounds of F respectively.
The injectivity criterion from Bruna et al.
(2013) is
Proposition 1.
Let A
= min Ω ∈ Ω λ − ( F Ω | V Ω ) .
Then the half-rectiﬁcation operator M b ( x ) =
ReLU( F T x + b ) is injective if and only if A 0 > 0 .
Moreover, it satisﬁes
[Equation]
with B 0 = max Ω ∈ Ω λ + ( F Ω ) ≤ λ + ( F ) .
We now show that Proposition 1 does not precisely align with injectivity of ReLU( W ( · )) .
We construct a weight matrix for which A 0 > 0 , but does not yield an injective ReLU( W ( · )) .
If
[Equation]
then clearly ReLU( Wx ) is not injective (for all α <
, ReLU( W (cid:20)
[Equation]
α (cid:21) )
= 0 ).
The only admissible sets are ¯Ω = {{ 1 } , { 3 } , { 1 , 2 } , { 2 , 3 }} (notably { 2 } is not admissible).
W is full rank on all Ω ∈ ¯Ω , so
A 0 > 0 so Proposition 1 implies that ReLU( W ( · )) is injective.
Now consider the case when
[Equation]
where B is a basis of R n , D is a strictly positive diagonal matrix, and 0 is the zero row vector.
From
Corollary 2, W satisﬁes Theorem 1, and so ReLU( W ( · )) is injective.
On account of the zero row vector in (109), ∀ x ∈ R n , (cid:104) x, 0 (cid:105) =
so there are no Ω that are admissible Ω according to (105).
Thus A 0 is undeﬁned.
Now we construct an example of a W and x, x (cid:48) ∈
R n for which A 0 (cid:107) x − x (cid:48) (cid:107) >
(cid:107) ReLU( Wx ) − ReLU( Wx (cid:48) )
(cid:107) .
Let
W =
[Equation]
Clearly on every admissible set λ − ( W Ω | V Ω ) = 1 , so
[Equation] [Equation]
    −
[Equation]

 
[Equation]

[Equation]


[Equation]
(111) hence, (cid:107) ReLU( Wx ) − ReLU( Wx (cid:48) )
(cid:107) = 1 √
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) −  (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =(112) and (cid:107) x − x (cid:48) (cid:107) = 1 √ 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:20) 20 (cid:21)(cid:13)(cid:13)(cid:13)(cid:13) = √
[Equation] [Equation]
From this we have √
[Equation] [Equation]
On the other hand, substituting this into (6) yields
[Equation]
which does hold, suggesting that the lower bound in Proposition 1 is not pessimistic enough.
E. R ELATIONSHIP TO M ALLAT ET AL . (2018)
In Mallat et al.
the authors consider a construction analogous to our convolutional construction (in Deﬁnition 3) deﬁned on a continuum (i.e. inﬁnite-dimensional function deﬁned on an interval) rather than on a vector (i.e. discrete ﬁnite dimensional function) deﬁned on a subset of R n .
The authors posit that CNNs ﬁrst learn a layer of ﬁlters localized in frequency varied in phase.
The authors also show that a a ReLU activation function acts as a ﬁlter on the phase of the convolution of the ﬁlters against the input signal and that, provided that the ﬁlters are sufﬁciently different in phase and satisfy a frame condition then the layer is bi-Lipschitz, and hence is injective.
Their analysis a particularized version of ours, and can be straight forwardly subsumed by our work.
The frame condition is given by Proposition 2.6 in Mallat et al.
(2018) that the weight matrix must satisfy in order ensure that W is invertible and stable.
In the notation of Mallat et al.
the ﬁlters ˆ ψ λ are analogous to the Fourier transform of the kernels in Deﬁnition 2, and the condition in (2.25) in Mallat et al.
(2018) is one natural way to generalize the notion of a basis to a continuous signal.
Hence, Proposition 2.6 in Mallat et al.
(2018) can be loosely interpreted as a statement that the kernels in a given layer of width P form a basis of R P .
The second condition is that kernels are given in terms of belonging to a family, and members of this family are related to each other in the sense that members of the same family are centered in the same
Fourier domain, and act as phase offsets of differing phase.
Equation 2.14 of Mallat et al.
(2018) describes that a phase ﬁlter
H : C × [0 , 2 , π ] → C is deﬁned by
[Equation]
[Equation]
where | z | standard modulus of a complex number, ϕ ( z ) is the complex phase, and h ( α ) =
ReLU(cos( α )) .
If we consider just the value of Hz (0) and Hz ( π ) , then we ﬁnd
[Equation]
where R z is the real part of z .
If (as in Mallat et al.
z is given by z = x (cid:63) c k , where x is a real signal and (cid:63)c k denotes the convolution against a kernel c k , then (117) and (118) imply that
[Equation] [Equation]
[Equation]
that is, that for every kernel c k in a layer, the kernel − c k is also in that layer.-
Combining the two logical conditions above implies that the kernels of width c k form a basis of R P and that for every kernel c k there is also a kernel − c k .
Together these two (by Corollary 2) that the c k form a DSS of R P , and thus by Theorem 4 the entire layer is injective.
A PPENDIX F A RCHITECTURE DETAILS FOR EXPERIMENTS
Generator network : We train a generator with 5 convolutional layers.
The input latent code is 256-dimensional which is treated by the network as a 1 × 1 × 256 size tensor.
The ﬁrst layer is a transposed convolution with a kernel size of 4 × 4 with stride 1 and 1024 output channels.
This is followed by a leaky ReLU.
We follow this up by 3 conv layers each of which halve the number of channels and double the image size (i.e. we go from N/ 2 × N/ 2 × C to N × N × C/ 2 tensor) giving an expansivity of two, the minimum required for injectivity of ReLU networks.
Each of these 3 convolution layers has kernel size 3 , stride 2 and is followed by the ReLU activation.
These layers are made injective by having half the ﬁlters as w and the other half as − s 2 w .
Here, w and s are trainable parameters.
The biases in these layers are kept at zero.
We do not employ any normalization schemes.
Lastly, we have a convolution layer at the end to get to 3 channels and required image size.
This layer is followed by the sigmoidal activation.
We compare this to a regular GAN which has all the same architectural components including nonlinearities except the ﬁlters are not chosen as w and − s 2 w
and we also allow biases (see Figure 9 for a qualitative comparison).
[Figure]
Critic network :
The discriminator has 5 convolution layers with 128 , 256 , 512 , 1024 and 1 channels per layer.
Each convolution layer has 4 × 4 kernels with stride 2 .
Each layer is followed by the leaky-ReLU activation function.
The last layer of the network is followed by identity.
Inference network : The inference network has the same architecture as the ﬁrst 4 convolution layers of the discriminator.
This is followed by 3 fully-connected layers of size 512 , 256 and 256 .
The ﬁrst 2 fully-connected layers have a Leaky ReLU activation while the last layer has identity activation function.
The inference net is trained in tandem with the GAN.
We use the Wasserstein loss with gradient penalty (Gulrajani et al., 2017) to train our networks.
We train for 40 epochs on a data set of size 80000 samples.
We use a batch size of 64 and Adam optimizer for training with learning rate of 10 − 4 .
We report FID (Heusel et al., 2017) and Inception score (Salimans et al., 2016) using 10000 generated samples.
The standard deviation was calculated using 5 sets of 10000 generated samples.
In order to calculate the mean and covariance of generated distributions, we sample 50000 codes.