T HE E FFECTS OF I NVERTIBILITY ON THE R EPRESENTATIONAL C OMPLEXITY OF E NCODERS IN VAE S
Anonymous authors
Paper under double-blind review
A BSTRACT
Training and using modern neural-network based latent-variable generative models (like Variational Autoencoders) often require simultaneously training a generative direction along with an inferential (encoding) direction, which approximates the posterior distribution over the latent variables.
Thus, the question arises: how complex does the inferential model need to be, in order to be able to accurately model the posterior distribution of a given generative model?
In this paper, we identify an important property of the generative map impacting the required size of the encoder.
We show that if the generative map is “strongly invertible" (in a sense we suitably formalize), the inferential model need not be much more complex.
Conversely, we prove that there exist non-invertible generative maps, for which the encoding direction needs to be exponentially larger (under standard assumptions in computational complexity).
Importantly, we do not require the generative model to be layerwise invertible, which a lot of the related literature assumes and isn’t satisﬁed by many architectures used in practice (e.g. convolution and pooling based networks).
As a non-invertible generator results in a distribution supported on a low-dimensional manifold, this provides theoretical support for the empirical wisdom that learning deep generative models is harder when data lies on a sufﬁciently complex low-dimensional manifold.
I NTRODUCTION
Many modern generative models of choice (e.g. Generative Adversarial Networks (Goodfellow et al., 2014), Variational Autoencoders (Kingma & Welling, 2013)) are modeled as non-linear, possibly stochastic transformations of a simple latent distribution (e.g. a standard Gaussian).
A particularly common task is modeling the inferential (encoder) direction : that is, modeling the posterior distribution on the latents z given an observable sample x .
Such a task is useful both at train time and at test time .
At train time, ﬁtting generative models like variational autoencoders via maximum likelihood often relies on variational methods, which require the joint training of a generative model (i.e. generator/decoder) , as well as an inference model (i.e. encoder) which models the posterior distribution of the latent given the observables.
At test time, the posterior distribution very often has some practical use, e.g. useful, potentially interpretable feature embeddings for data (Salimans et al., 2016; Berthelot et al., 2018), “intervening” on the latent space to change the sample in some targeted manner (Shen et al., 2020), etc.
As such, the question of the “complexity" of the inference model (i.e. number of parameters to represent it using a neural network-based encoder) as a function of the “complexity" of the forward model is of paramount importance, so that training is computationally tractable:
Question: How complex does the inference (encoder) model need to be relative to the complexity of the generative (decoder) model ?
In this paper we identify an important property of the generative direction governing the complexity of the inference direction for variational autoencoders : bijectivity/invertibility of the mean of the generative direction.
We prove that when mean of the generative direction is invertible, the complexity of the inference direction is not much greater than the complexity of the generative direction.
Conversely, when the mean of the generative direction is not invertible, modulo standard computational complexity conjectures from cryptography, we can exhibit instances where the inference direction has to be much more complex.
On the mathematical level, our techniques involve a neural simulation of a Langevin random walk to sample from the posterior of the latent variables and uncover novel connections between Langevin diffusions and (hierarchical) deep latent Gaussians.
On the lower bound side, we provide a reduction from the existence of one-way Boolean permutations in computational complexity: that is, permutations that are easy to calculate, but hard to invert.
We show that the existence of a small encoder for non-invertible generators would allow us to design an invertor for any Boolean permutation, thus violating the existence a one-way permutation.
This is the ﬁrst time such ideas have been applied to generative models.
Note that a non-invertible generator results in a distribution supported on a lower dimensional manifold.
Thus, our results show that learning deep generative models is harder when data lies on a sufﬁciently complex low-dimensional manifold, corroborating similarly ﬂavored empirical observations (Dai & Wipf, 2019; Arjovsky et al., 2017).
O UR R ESULTS
The Variational Autoencoder (VAE) (Kingma & Welling, 2013) is one of the most commonly used paradigms in generative models.
It’s trained by ﬁtting a generator which maps latent variables z to observables x , denoted by p θ ( x | z ) , as well as an encoder which maps the observables to the latent space, denoted by q φ ( z | x ) .
Here φ and θ are the encoder parameters and generator parameters respectively.
Given n training samples { x ( i ) } ni =1 , the VAE objective is given by
[Equation]
where p ( z ) is typically chosen to be a standard Gaussian.
This loss can be viewed as a variational relaxation of the maximum likelihood objective, where the encoder q φ , in the limit of inﬁnite representational power, is intended to model the posterior distribution p θ ( z | x ( i ) ) .
Setup: We will consider a setting in which the data distribution itself is given by some ground-truth generator G : R d l → R d o , and ask how complex (in terms of number of parameters) the encoder needs to be (as a function of the number of parameters of G ), s.t.
it approximates the posterior distribution p ( z | x ) of the generator.
We will consider two standard probabilistic models for the generator/encoder respectively.
Deﬁnition 1 (Latent Gaussian) .
A latent Gaussian is the conditional distribution given by a stochastic pushforward of a Gaussian distribution.
That is, for latent variable z ∈ R d l and observable x ∈ R
d
o , for a neural network G : R d l → R d o and noise parameter β 2 , we have p ( x | z ) =
N ( G ( z ) , β 2 I d o ) and p ( z ) =
N (0 , I d l ) .
In other words, a sample from this distribution can be generated by sampling independently z ∼ N (0 , I d l ) and ξ ∼ N (0 ,
β 2 I d o ) and outputting x = G ( z ) + ξ .
This is a standard neural parametrization of a generator with (scaled) identity covariance matrix, a fairly common choice in practical implementations of VAEs (Kingma & Welling, 2013; Dai & Wipf, 2019).
We will also deﬁne a probabilistic model which is a composition of latent Gaussians (i.e. consists of multiple stochastic layers), which is also common, particularly when modeling encoders in VAEs, as they can model potentially non-Gaussian posteriors (Burda et al., 2015; Rezende et al., 2014):
Deﬁnition 2 (Deep Latent Gaussian) .
A deep latent Gaussian is the conditional distribution given by a sequence of stochastic pushforwards of a Gaussian distribution.
That is, for observable z 0 ∈ R d 0 and latent variables { z i ∈ R d i } Li =1 , for neural networks { G i : R d i − 1 → R d i } Li = and noise parameters { β 2 i } Li =1 , the conditional distribution p ( z L | z 0 ) is a deep latent Gaussian when p ( z i
| z i − 1 )
=
N ( G i ( z i − 1 ) , β 2 i I d i ) , ∀ i ∈
[ L ] and p ( z 0 ) =
N (0 , I d 0 ) .
In other words, a deep latent Gaussian is a distribution, which can be sampled by ancestral sampling, one layer at a time.
Note that this class of distributions is convenient as a choice for an encoder in a VAE, since compositions are amenable to the reparametrization trick of Kingma & Welling (2013)—the randomness for each of the layers can be “presampled” and appropriately transformed (Burda et al., 2015; Rezende et al., 2014).
Then, we ask the following:
Question: If a VAE generator is modeled as a latent Gaussian (that is, p ( x | z ) ≡ N ( G ( z ) , β 2 I ) ), s.t.
the corresponding G has at most N parameters, and we wish to approximate the posterior p ( z | x ) by a deep latent Gaussian s.t.
the total size of the networks in it have at most N (cid:48) parameters, how large must N (cid:48) be as function of N ?
We will work in the setting d l = d o = d , and prove a dichotomy based on the invertibility of G : namely, if G : R d → R d is bijective, and β ≤ O (cid:18) 1 d 1 . 5 √ log d/(cid:15) (cid:19) , the posterior p ( z | x ) can be (cid:15) -approximated in total variation distance by a deep latent Gaussian of size
N (cid:48)
=
O ( N · poly ( d, 1 /β, 1 /(cid:15) )) .
Thus, if the neural network G is invertible, and for a ﬁxed (cid:15) and a small-enough variance term β 2 , we can approximate the posterior with a deep latent Gaussian polynomially larger than G .
On the other hand, if G is not bijective, if one-way-functions exist (a widely believed computational complexity conjecture), we will show there exists a VAE generator
G : R d → R d of size polynomial in d , for which the posterior p ( z | x ) cannot be approximated in total variation distance for even an inverse polynomial fraction of inputs x , unless the inferential network is of size exponential in d .
Remark 1: Note that it is in fact sufﬁcient to investigate only the case where d l = d o = d .
Let us elaborate why:
• For Theorem 1, a function G : R d l → R d o can be bijective, Lipschitz and strongly invertible (i.e. satisﬁes Assumptions 1 and 2) only if d l = d o , so Theorem 1 (upper bound) would not make any sense unless the dimensions are equal.
(Note: there are bijective maps between, say, R and R 2 , e.g. space-ﬁlling curves, but these cannot be Lipschitz, as Lipschitz maps preserve the Hausdorff dimension, and the Hausdorff dimensions of R and R 2 are 1 and 2 respectively).
• Theorem 2 is a lower bound – meaning we aim to exhibit an example of a hard instance when G is not bijective.
Having exhibited a hard instance for the d l = d o = d case, it is trivially easy to construct a hard instance in the theorem where the output dimension is larger—which is the more common setting in practice.
To do so, consider a circuit ˜ C : {± 1 } d l → {± 1 } d o , which equals to C , the one-way-circuit from Conjecture 1 on the ﬁrst d l coordinates, and is equal to {+
 (i e.+} is constant) on the last d o − d l coordinates.
Then ˜ C is one-way too—since the last d o − d l values are just ﬁxed to 1 , if we can invert it, we can invert C too.
The reduction in the proof of Theorem 2 then can just be performed with ˜ C instead, giving a lower bound instance for the d l < d o case.
2. U PPER BOUNDS FOR BIJECTIVE GENERATORS
We ﬁrst lay out the assumptions on the map G .
The ﬁrst is a quantitative characterization of bijectivity; and the second requires upper bounds on the derivatives of G upto order 3 .
We also have a centering assumption.
We state these below.
Assumption 1 (Strong invertibility) .
We will assume that the latent and observable spaces have the same dimension (denoted d ), and G : R d → R d is bijective.
Moreover, we will assume there exists a positive constant m > 0 such that ∀
z 1 , z 2 ∈ R d , (cid:107) G ( z 1 ) − G ( z 2 ) (cid:107) ≥ m · (cid:107) z
1 − z 2 (cid:107) .
Remark 2: This is a stronger quantitative version of invertibility.
Furthermore, the inﬁnitesimal version of this condition (i.e. (cid:107) z 1 − z 2 (cid:107) → 0 ) implies that the smallest magnitude of the singular values of the Jacobian at any point is lower bounded by m , that is ∀ z ∈ R d ,
min i ∈
[ d ] | σ i ( J G ( z ))
| ≥ m > 0 .
Since m is strictly positive, this in particular means that the Jacobian is full rank everywhere.
Remark 3:
Note, we do not require that G is layerwise invertible (i.e. that the each map from one layer to the next is invertible) – if that is the case, at least in the limit β → 0 , the existence of an inference decoder of comparable size to G is rather obvious: we simply invert each layer one at a time.
This is important, as many architectures based on convolutions perform operations which increase the dimension (i.e. map from a lower to a higher dimensional space), followed by pooling (which decrease the dimension).
Nevertheless, it has been observed that these architectures are invertible in practice—Lipton & Tripathi (2017) manage to get almost 100% success at inverting an off-the-shelf trained model—thus justifying this assumption.
Assumption 2 (Smoothness) .
There exists a ﬁnite positive constant M > 0 such that :
[Equation]
Moreover, we will assume that G has continuous partial derivatives up to order 3 at every z ∈ R d and the derivatives are bounded by ﬁnite positive constants M 2 and M 3 as
[Equation]
Remark 4: This is a mild assumption, stating that the map G is smooth to third order.
The inﬁnitesimal version of this means that the largest magnitude of the singular values of the Jacobian at any point is upper bounded by M , that is ∀ z ∈ R d ,
max i ∈
[ d ] | σ i ( J G ( z )) |
= (cid:107) J G ( z ) (cid:107) op ≤ M < ∞ .
Remark 5: A neural network with activation function σ will satisfy this assumption when σ : R → R is Lipschitz, and max a | σ (cid:48) ( a ) | & max a | σ (cid:48)(cid:48) ( a ) | are ﬁnite.
Assumption 3 (Centering) .
The map G : R d → R d satisﬁes G (0) = 0 .
Remark 6: This assumption is for convenience of stating the bounds—we effectively need the “range” of majority of the samples x under the distribution of the generator.
All the results can be easily restated by including a dependence on (cid:107) G (0) (cid:107) .
Our main result is then stated below.
Throughout, the O ( . ) notation hides dependence on the map constants, namely m, M, M 2 , M 3 .
We will denote by d TV ( p, q ) the total variation distance between the distributions p, q .
Theorem 1 (Main, invertible generator) .
Consider a VAE generator given by a latent Gaussian with µ = 0 , Σ
= I , noise parameter β 2 and generator G : R d → R d satisfying Assumptions 1 and 2, which has N parameters and a differentiable activation function σ .
Then, for
[Equation]
there exists a deep latent Gaussian with N (cid:48)
=
O (cid:16) N · poly ( d, 1 β , 1 (cid:15) ) (cid:17) parameters and activation functions { σ, σ (cid:48) , ρ } and a neural network φ with O ( N (cid:48) ) parameters and activation functions { σ, σ (cid:48) , ρ } , where ρ ( x ) = x 2 , such that with probability 1 − exp( −O ( d )) over a sample x from the VAE generator, φ ( x ) produces values for the parameters of the deep latent Gaussian such that the distribution q ( z | x ) it speciﬁes satisﬁes d TV ( q ( z | x ) , p ( z | x )) ≤ (cid:15) .
Remark 7: Having an encoder q ( z | x ) with parameters produced by a neural network taking x as input is fairly standard in practice—it is known as amortized variational inference.(Mnih & Gregor, 2014; Kingma & Welling, 2013; Rezende et al., 2014)
Remark 8: The addition of ρ in the activation functions is for convenience of stating the bound.
Using usual techniques in universal approximation it can be simulated using any other smooth activation, see Appendix G.
2. L OWER BOUNDS FOR NON - BIJECTIVE G ENERATORS
We now discuss the case when the generative map G is not bijective, showing an instance such that no small encoder corresponding to the posterior exists.
The lower bound will be based on a reduction from the existence of one-way functions—a standard complexity assumption in theoretical computer science (more concretely, cryptography).
Precisely, we will start with the following form of the one-way-function conjecture:
Conjecture 1 (Existence of one-way permutations (Katz & Lindell, 2020)) .
There exists a bijection f : {− 1 , 1 } d → {− 1 , 1 } d computable by a Boolean circuit C : {− 1 , 1 } d → {− 1 , 1 } d of size poly ( d ) , but for every T ( d ) = poly ( d ) and (cid:15) ( d ) = 1 poly ( d ) and circuit C (cid:48) : {− 1 , 1 } d → {− 1 , 1 } d of size T ( d ) it holds Pr z ∼{± 1 } d [ C (cid:48)
( C ( z ))
= z ] ≤ (cid:15) ( d ) .
In other words, there is a circuit of size polynomial in the input, s.t.
for every polynomially sized invertor circuit (the two polynomials need not be the same—the invertor can be much larger, so long as it’s polynomial), the invertor circuit succeeds on at most an inverse polynomial fraction of the inputs.
Assuming this Conjecture, we show that there exist generators that do not have small encoders that accurately represent the posterior for most points x .
Namely:
Theorem 2 (Main, non-invertible generator) .
If Conjecture 1 holds, there exists a VAE generator G :
R d → R d with size poly ( d ) and activation functions { sgn , min , max } , s.t., for every β
= o (1 / √ d ) , every T ( d ) =
poly ( d ) and every (cid:15) ( d ) = 1 / poly ( d ) , any encoder E that can be represented by a deep latent Gaussian with networks that have total number of parameters bounded by T ( d ) , weights bounded by W , activation functions that are L -Lipschitz, and node outputs bounded by M with probability 1 − exp( − d ) over a sample x from G and L, M, W =
o (exp( poly ( d ))) , we have:
[Equation]
Thus, we show the existence of a generator for which no encoder of polynomial size reasonably approximates the posterior for even an inverse-polynomial fraction of the samples x (under the distribution of the generator).
Remark 9: The generator G , though mapping from R d → R d will be highly non-invertible .
Perhaps counterintuitively, Conjecture 1 applies to bijections—though, the point will be that G will be simulating a Boolean circuit, and in the process will give the same output on many inputs (more precisely, it will only depend on the sign of the inputs, rather than their values).
Remark 10: The choice of activation functions { sgn , min , max } is for convenience of stating the theorem.
Using standard universal approximation results, similar results can be stated with other activation functions, see Appendix G.
Remark 11: The restrictions on the Lipschitzness of the activations, bounds of the weights and node outputs of E are extremely mild—as they are allowed to be potentially exponential in d —considering that even writing down a natural number in binary requires logarithmic number of digits.
R ELATED W ORK
On the empirical side, the impact of impoverished variational posteriors in VAEs (in particular, modeling the encoder as a Gaussian) has long been conjectured as one of the (several) reasons for the fuzzy nature of samples in trained VAEs.
Zhao et al.
(2017) provide recent evidence towards this conjecture.
Invertibility of generative models in general (VAEs, GANs and normalizing ﬂows), both as it relates to the hardness of ﬁtting the model, and as it relates to the usefulness of having an invertible model, has been studied quite a bit: Lipton & Tripathi (2017) show that for off-theshelf trained GANs, they can invert them with near-100% success rate, despite the model not being encouraged to be invertible during training; Dai & Wipf (2019) propose an alternate training algorithm for VAEs that tries to remedy algorithmic problems during training VAEs when data lies on a lowerdimensional manifold; Behrmann et al.
(2020) show that trained normalizing ﬂows, while being by design invertible, are just barely so — the learned models are extremely close to being singular.
On the theoretical side, the most closely relevant work is Lei et al.
They provide an algorithm for inverting GAN generators with random weights and expanding layers .
Their algorithm is layerwise — that is to say, each of the layers in their networks is invertible, and they invert the layers one at a time.
This is distinctly not satisﬁed by architectures used in practice, which expand and shrink — a typical example are convolutional architectures based on convolutions and pooling.
The same paper also shows NP-hardness of inverting a general GAN, but crucially they assume the network
G is part of the input (their proof does not work otherwise).
Our lower bound can be viewed as a “non-uniform complexity” (i.e. circuit complexity) analogue of this, since we are looking for a small neural network E , as opposed to an efﬁcient algorithm; crucially, however G is not part of the input (i.e. G can be preprocessed for an unlimited amount of time).
Hand & Voroninski (2018) provide similar guarantees for inverting GAN generators with random weights that satisfy layerwise invertibility, albeit via non-convex optimization of a certain objective.
P ROOF O VERVIEW
Overview of Theorem 1 Recall, given a generator G and noise level β 2 , we wish to provide a deep latent Gaussian (Deﬁnition 2) that approximates a sample from the distribution p ( z | x ) , with high probability over the draws of x from the underlying density.
To show Theorem 1, there are three main ingredients:
We show that gradient descent on the function f ( z ) = 12 (cid:107) G ( z ) − x (cid:107) 2 , run for O ( d/β 2 ) number of steps, recovers a point z init close to argmax z log p ( z | x ) – i.e. the mode of the posterior.
We show that Langevin diffusion, a Markov process that has p ( z | x ) as it’s stationary distribution, started close to the mode of the posterior via z init computed by GD above, run for O (log( d/(cid:15) ))
amount of time; returns a sample from a distribution close to p ( z | x ) in total variation distance.
We show that the above two steps can be “simulated" by a deep latent Gaussian of size at most poly ( d, 1 β , 1 (cid:15) ) larger than the original network, by inductively showing that each next iterate in the gradient descent and Markov chain can be represented by a neural network of size comparable to the generator G .
Part 1 is formalized in Lemma 1.
The main observation is that the gradient of the loss will have the form ∇ f ( z ) =
J G ( z ) T ( G ( z ) − x ) .
By Strong Invertibility (Assumption 1), J G ( z ) will be an invertible matrix—therefore, any stationary points will necessarily be the (unique) point z s.t.
G ( z ) = x . This in turn, for small β , will be close to argmax z log p ( z | x ) .
Part 2 is formalized in Lemma 2, and is technically most involved.
As β → 0 , the distribution p ( z | x ) concentrates more around it’s mode, and approaches a Gaussian centered at it.
More precisely, we show that starting at z init , there is a region D such that the probability of Langevin diffusion leaving this region is small (Lemma 13) and the distribution restricted to D is log-concave (Lemma 4).
As a consequence of the latter claim, reﬂected Langevin diffusion (a “restricted” version of the standard
Langevin diffusion) converges rapidly to p ( z | x ) restricted to D .
As a consequence of the former claim, the reﬂected and standard Langevin diffusion are close in the total variation sense.
Finally, Part 3 is based on the observation that we can approximately simulate gradient descent, as well as the Langevin dynamics using a deep latent Gaussian (Deﬁnition 2).
Proceeding to gradient descent, the key observation is that if we give a neural network z as the input, the point z − ηJ G ( z ) T ( G ( z ) − x ) can be calculated by a network of size comparable to the size of a neural network representing J G ; the Jacobian of a neural network.
And in turn, the Jacobian network can be represented by a neural network at most a factor O ( d ) bigger than the neural network representing
G —this is by noting that each partial derivative of G can be represented by a neural network of size comparable to G , essentially by the backpropagation algorithm (Lemma 5).
The idea to simulate Langevin dynamics is similar: the Euler-Mariyama discretization of the corresponding stochastic differential equation (with precision h ) is a discrete-time Markov chain, s.t.
the updates take the form
[Equation]
where L the unnormalized log-likelihood of the posterior p ( z | x ) (equation 3 below).
Same as is the case for gradient descent, a sample following the distribution of these updates can be implemented by a latent Gaussian, with a neural net of size at most O ( d ) bigger than the size of G —the randomness being handled by Gaussian noise that the latent Gaussian has its disposal.
Overview of Theorem 2 The proof of Theorem 2 proceeds by a reduction to Conjecture 1.
We consider G ( z ) =
C ( sgn ( z )) , where C is a circuit satisfying the assumptions of Conjecture 1.
We show that we can use an encoder q ( z | x ) that works with good probability over x (wrt to the distribution of G ) to construct a circuit C (cid:48) that violates Conjecture 1.
The main idea is as follows: the samples x will, with high probability, lie near points on the hypercube {± 1 } d .
For such points, the posterior p ( z | x ) will concentrate around C − 1 (˜ x ) , so taking the sign of a sample from q ( z | x ) should, with good probability, produce an inverse of sgn ( x ) .
Simulating this construction using a Boolean circuit, we can get a “randomized” circuit that inverts C .
Producing a deterministic circuit follows ideas from complexity theory, where we show that given a randomized circuit, there exists a single string of randomness that works for all inputs, if the success probability of the algorithm is high enough.
M AIN INGREDIENTS : T HEOREM
First, we introduce a few pieces of notation that will be used in multiple places in the proofs.
We will also omit subscripting quantities with x to ease notation whenever x is clear from context.
For notational convenience, we will denote the unnormalized log-likelihood of p ( z | x ) as L : R d → R , s.t.
[Equation]
Note, by Bayes rule, p ( z | x ) ∝ p ( x | z ) p ( z ) , so p ( z | x ) ∝ e − L ( z ) .
Note this is a well-deﬁned probability distribution since ∀ z ∈ R d , L ( z ) ≥ 12 (cid:107)
z (cid:107) 2 , therefore the partition function is ﬁnite.
Further, we will denote the inverse of x as ˆ z , namely:
[Equation]
Note since G is invertible, ˆ z is uniquely deﬁned, and it’s such that G (ˆ z )
= x .
Crucially due to this, the function L ( z ) is strongly convex in some region around the point ˆ z – which can be seen by evaluating ∇ 2 L (ˆ z ) from equation 3.
These properties are convenient – that is why we base our analysis on this inverse point.
5. P ART 1: C ONVERGENCE OF GRADIENT DESCENT
First, we show that gradient descent on the function f ( z ) = 12 (cid:107) G ( z ) − x (cid:107) 2 , started at the origin, converges to a point close to ˆ z in a bounded number of steps.
Precisely:
Lemma 1 (Convergence of Gradient Descent to the Inverse) .
When G satisﬁes Assumptions 1 and 2, with probability 1 − exp( −O ( d )) over the choice of x , running gradient descent on f ( z ) =
12 (cid:107) G ( z ) − x (cid:107) 2 with the starting point z (0) :
= 0 and learning rate η := 1 Q where Q = O ( d ) , for
S =
O ( d 2 δ 2 ) steps converges to a point z ( S ) , such that (cid:107) z
( S ) − ˆ z (cid:107) ≤ δ .
As mentioned in Section 4, the idea is that while f is non-convex, its unique stationary point is ˆ z , as ∇ f ( z ) =
J G ( z ) T ( G ( z ) − x ) and J G ( z ) is an invertible matrix ∀ z ∈ R d from Assumption 1.
The proof essentially proceeds by a slight tweak on the usual descent lemma (Nesterov, 2003), due to the fact that the Hessian’s norm is not bounded in all of R d and is given in Appendix B.
5. P ART 2: F AST MIXING OF L ANGEVIN DIFFUSION
We ﬁrst review some basic deﬁnitions and notation.
Recall, Langevin diffusion is described by a stochastic differential equation (SDE):
Deﬁnition 3 (Langevin Diffusion) .
Langevin Diffusion is a continuous-time Markov process { z t : t ∈ R + } deﬁned by the stochastic differential equation
[Equation]
where g : R d → R , and { B t : t ∈ R + } is a Brownian motion in R d with covariance matrix I d .
Under mild regularity conditions, the stationary distribution of this process is P : R d → R + such that P ( z ) ∝ e − g ( z ) .
For the reader unfamiliar with SDEs, the above process can be interpreted as the limit as h → 0 of the Markov chain
[Equation] [Equation]
The convergence time of the chain to stationarity is closely tied to the convexity properties of L , through functional-analytic tools like Poincaré inequalities and Log-Sobolev inequalities (Bakry & Émery, 1985).
We show that the distribution p ( z | x ) can be sampled (cid:15) -approximately in total variation distance by running Langevin dynamics for O (log( d/(cid:15) ))
number of steps.
Namely:
Lemma 2 (Sampling Complexity for general G ) .
Let β
= O (cid:18) 1 d 1 . 5 √ log d(cid:15) (cid:19) .
Then, with probability 1 − exp( −O ( d )) over the choice of x , if we initialize Langevin diffusion equation 6 at z 0 which satisﬁes (cid:107) z 0 − ˆ z (cid:107) ≤ Ω (cid:18) β · (cid:113) d log d(cid:15) (cid:19) , for some T
=
O
( 1 d 2 log 1 (cid:15) )
we have d TV ( P T , p ( z | x )) ≤ (cid:15)/where P t is the density corresponding to z t .
Note, the lemma requires the variance of the latent Gaussian β to be sufﬁciently small.
This intuitively ensures the posterior p ( z | x ) is somewhat concentrated around its mode — and we will show that when this happens, in some region around the mode, the posterior is close to a Gaussian.
The proof of this has two main ingredients: (1) z t stays in a region D around ˆ z with high probability; (2)
The distribution p ( z | x ) , when restricted to D is log-concave, so a “projected” version of Langevin diffusion (formally called reﬂected Langevin diffusion , see Deﬁnition 4) mixes fast to this restricted version of p ( z | x ) .
The ﬁrst part is formalized as Lemma 13 in Appendix C.
The main mathematical tool is a stochastic differential equation characterizing the change of the distance from ˆ z .
Namely, we show:
Lemma 3 (Change of distance from ˆ z ) .
Let η ( z ) :
= 12 (cid:107)
z − ˆ z (cid:107) 2 .
If z follows the Langevin SDE corresponding to z t , then for z ∈ D as deﬁned in Lemma 13 and β ≤ β 0 as deﬁned above, it holds that
[Equation]
This SDE is analyzed through the theory of Cox-Ingersoll-Ross (Cox et al., 2005) processes, which are SDEs of the type dM t = ( a − bM t ) dt + c √ M t dB t , where a, b, c are positive constants.
Intuitively, this SDE includes an ‘attraction term’ − bM t dt which becomes stronger as we increase M t (in our case, as we move away from the center of the region ˆ z ), and a diffusion term c √ M t dB t due to the injection of noise.
The term ( (cid:107) ˆ z (cid:107) 2 + d ) dt comes from the heuristic Itó calculus equality dB 2 t = dt .
The “attraction term” − m 2 β 2 η ( z ) dt comes from the fact that L ( z ) is approximately strongly convex near ˆ z .
(Intuitively, this means the Langevin process is attracted towards ˆ z .)
To handle the second part, we show the distribution p ( z | x ) restricted to D is log-concave:
Lemma 4 (Strong convexity of L over D ) .
For all z ∈ D , ∇ 2 L ( z ) (cid:23) I .
Since the set D is a l 2 ball (which is of course convex), we can show an alternate Markov process which always remains inside D , called the reﬂected Langevin diffusion mixes fast.
Deﬁnition 4 (Reﬂected Langevin diffusion, (Lions & Sznitman, 1984; Saisho, 1987)) .
For a sufﬁciently regular region D ⊆ R d , there exists a measure U supported on ∂ D , such that the continuoustime diffusion process { y t : t ∈ R + } starting with y 0 ∈ D deﬁned by the stochastic differential equation:
[Equation]
(7) has as stationary measure ˜ P :
D → R + such that ˜ P ( y ) ∝ e − g ( y ) .
Here g : R d → R , { B t : t ∈ R + } is a Brownian motion in R d with covariance matrix I d , and ν t is an outer normal unit vector to D .
For readers unfamiliar with SDEs, the above process can be interpreted as the limit as h → 0 of
[Equation]
where Π D is the projection onto the set D .
For total variationa distance, the mixing time in total variation distance of this diffusion process was characterized by Bubeck et al.
Theorem 3 (Mixing for log-concave distributions over convex set, Bubeck et al.
(2018): Proposition 2) .
Suppose a measure p : R d → R + of the form p ( x ) ∝ e − g ( x ) is supported over S ⊆ R d of diameter R which is convex, and ∀ x ∈ S, ∇ 2 g (cid:38) 0 .
Furthermore, let x 0 ∈ S .
Then, if p t is the distribution after running equation 7 for time t starting with a Dirac distribution at x 0 , we have
[Equation]
Since the set D in Lemma 13 is an l 2 ball (which is of course convex), it sufﬁces to show that L is convex on D —which is a straightforward calculation.
The details of this are again in Appendix C.
5. P ART 3: S IMULATION BY NEURAL NETWORKS
Finally, we show that the operations needed for gradient descent as well as Langevin sampling can be implemented by a deep latent Gaussian.
Turning the gradient descent ﬁrst, we need to “simulate” by a neural net the operation z ← z − ηJ G ( z ) T ( G ( z ) − x ) roughly O ( d/β 2 ) number of times.
We prove that there is a neural network that given an input z , outputs z − ηJ G ( z ) T ( G ( z ) − x ) .
The main ideas for this are that: (1) If G is a neural network with N parameters, each coordinate of J G ( z ) can be represented as a neural network with O ( N ) parameters.
(2) If f, g are two neural networks with N and M parameters respectively, f
+ g and fg can be written as neural networks with O ( M + N ) parameters.
The ﬁrst part essentially follows by implementing backpropagation with a neural network:
Lemma 5 (Backpropagation Lemma, Rumelhart et al.
Given a neural network G : R d → R of depth l and size N , there is a neural network of size O ( N + l ) which calculates the gradient ∂G/∂z i for i ∈
[ d ] .
For the second part, addition is trivially implemented by “joining” the two neural networks and adding one extra node to calculate the sum of their output.
For multiplication, the claim follows by noting that xy = 14 ( x + y ) 2 − 14 ( x − y ) 2 .
Using the square activation function, this can be easily computed by a 2 -layer network.
Proceeding to the part about simulating Langevin diffusion, we ﬁrst proceed to create a discrete-time random process that closely tracks the diffusion.
The intuition for this was already mentioned in
Section 4— the SDE describing Langevin can be viewed as the limit h → 0 of
[Equation]
It can be readily seen that the drift part, that is 1 β 2 J G ( z ) T ( G ( z ) − x ) can be calculated by a neural net via the same token as in the case of the gradient updates.
Thus, the only difference is in the injection of Gaussian noise.
However, each layer G i in a deep latent Gaussian exactly takes a
Gaussian sample ξ and outputs G i ( z ) + γξ – so each step of the above discretization of Langevin can be exactly implement by a layer in a deep latent Gaussian.
The details are in Appendix D.
M AIN INGREDIENTS : T HEOREM
We prove Theorem 2 proceeds by a reduction to Conjecture 1.
Namely, if C is a Boolean circuit satisfying the one-way-function assumptions in Conjecture 1, we consider G ( z ) =
C ( sgn ( z )) .
Suppose there is a deep latent Gaussian of size T ( n ) with corresponding distribution q ( z | x ) , s.t.
with probability at least (cid:15) ( d ) over the choice of x (wrt to the distribution of G ), it satisﬁes d TV ( p ( z | x ) , q ( z | x )) ≤ 1 / 10 .
We will how to construct from the encoder a circuit C (cid:48) that violates Conjecture 1.
Namely, consider a ˜ x constructed by sampling ˜
z ∼ {± 1 } d uniformly at random, and outputting ˜ x
= C (˜ z ) .
To violate
Conjecture 1, we’d like to ﬁnd C − 1 (˜ x ) , using the encoder E .
We ﬁrst note that x = ˜ x + ξ, ξ ∼ N (0 ,
β 2 I d ) is distributed according to the distribution of the generator G .
Moreover, we can show that with probability at least (cid:15) ( d ) / 2 , it will be the case that x is s.t.
both d TV ( p ( z | x ) , q ( z | x )) ≤ 1 / 10 and x is very close to a point on the discrete hypercube {± 1 } d .
The reason for this is that the samples x themselves for small enough β with high probability are close to a point on the hypercube.
For such points, it will be the case that q ( z | x ) is highly concentrated around C − 1 (˜ x ) , thus returning the sign of a sample will with high probability invert ˜ x .
By simulating this process using a Boolean circuit, we get a “randomized” circuit (that is, a circuit using randomness) that succeeds at inverting C with inverse polynomial success.
To ﬁnally produce a deterministic Boolean circuit that violates Conjecture 1, we use a standard trick in complexity theory: boosting the success probability by generating multiple samples together with the assumption on the weights and Lipschitzness of E .
For details, refer to Appendix E.
C ONCLUSION
In this paper we initiated the ﬁrst formal study of the representational complexity of the good encoders in VAEs.
We proved the following dichotomy: generators with invertible means give rise to posteriors that can be approximated by an encoders not much larger than the generator.
On the other hand, for non-invertible generators, the corresponding encoder may need to be exponentially larger.
We hope our work will stimulate research of other data distribution properties that render inference tractable.
A T ECHNICAL D EFINITIONS
In this section, we state several deﬁnitions and standard lemmas, to cover some technical prerequisites for the proof.
We ﬁrst recall a few deﬁnitions and lemmas about two notions of distance between probabilities we will use extensively, total variation and χ 2 distance.
Deﬁnition 5 (Total variation distance) .
The total variation distance between two distributions P, Q is deﬁned as
[Equation]
Deﬁnition 6 ( χ 2 distance) .
The χ 2 distance between two distributions P, Q , s.t.
P is absolutely continuous with respect to Q , is deﬁned as
[Equation]
The following two lemmas about the total variation and χ 2 distance are standard:
Lemma 6 (Coupling Lemma) .
Let P, Q : Ω → R be two distributions, and c : Ω ⊗ 2 → R be any coupling of P, Q .
Then, if ( X, X (cid:48) ) are random variables following the distribution c , we have:
[Equation]
Lemma 7 (Inequality between TV and χ 2 ) .
Let P, Q be probability measures,
s.t. P is absolutely continuous with respect to Q .
We then have:
[Equation]
We will be heavily using notions from continuous-time Markov Processes.
In particular:
Deﬁnition 7 (Markov semigroup) .
We say that a family of functions { P t ( x, y ) } t ≥ 0 on a state space Ω is a Markov semigroup if P t ( x, · ) is a distribution on Ω and
[Equation]
for all x, y ∈ Ω and s, t ≥ 0 .
Deﬁnition 8 (Continuous time Markov processes) .
A continuous time Markov process ( X t ) t ≥on state space Ω is deﬁned by a Markov semigroup { P t ( x, y ) } t ≥ 0 as follows.
For any measurable
A ⊆ Ω
[Equation]
Moreover P t can be thought of as acting on a function g as
[Equation]
Finally we say that p ( x ) is a stationary distribution if X 0 ∼ p implies that X t ∼ p for all t .
With this in place, we can ﬁnally deﬁne the Langevin diffusion Markov process:
Deﬁnition 9 (Langevin Diffusion, Deﬁnition 3 restated) .
Langevin Diffusion is a continuous-time
Markov process { z t : t ∈ R + } deﬁned by the stochastic differential equation
[Equation]
where g : R d → R , and { B t : t ∈ R + } is a Brownian motion in R d with covariance matrix I d .
Under mild regularity conditions, the stationary distribution of this process is P : R d → R + such that P ( z ) ∝ e − g ( z ) .
It will also be useful to have a deﬁnition of a diffusion process that only stays within some support D , and thus has a stationary distribution supported on D .
Speciﬁcally:
Deﬁnition 10 (Reﬂected Langevin diffusion, (Lions & Sznitman, 1984; Saisho, 1987), Deﬁnition 4 restated) .
For a sufﬁciently regular region D ⊆ R d , there exists a measure U supported on ∂ D , such that the continuous-time diffusion process { y t : t ∈ R + } starting with y 0 ∈ D deﬁned by the stochastic differential equation:
[Equation]
has as stationary measure ˜ P :
D
→ R + such that ˜ P ( y )
∝ e − g ( y ) .
Here g : R d → R , { B t : t ∈ R + } is a Brownian motion in R d with covariance matrix I d , and ν t is an outer normal unit vector to D .
Towards introducing quantities governing the mixing time of a continuous-time Markov process, we introduce
Deﬁnition 11.
The generator L of the Markov Process is deﬁned (for appropriately restricted functionals g ) as
[Equation]
Moreover if p is the unique stationary distribution, the corresponding Dirichlet form and variance are
E M
[Equation]
g,
[Equation] [Equation]
p
[Equation]
g, L h (cid:105) and Var p ( g ) =
E p ( g − E p g )respectively.
We will use the shorthand E ( g ) :
=
E ( g, g ) .
A quantity of particular interest when analyzing the mixing time of Markov processes is the Poincaré constant:
Deﬁnition 12 (Poincaré Inequality) .
A continuous Markov process satisﬁes a Poincaré inequality with constant C if for all functions g such that E M ( g ) is deﬁned (ﬁnite), it holds that:
[Equation]
where P (in the subscript) is the stationary distribution of the Markov process.
We will abuse notation, and for such a Markov process with stationary distribution P , denote by C pc ( P ) the smallest C such that above Poincaré inequality is satisﬁed; and call it the Poincaré constant of P .
The relationship between the Poincaré constant and the mixing time of a random walk (in particular, the one in Deﬁnition 4) is given by the following (standard) lemma:
Lemma 8 (Mixing in χ 2 from Poincaré) .
Let ˜ z t follow the stochastic differential equation of reﬂected
Langevin diffusion (Deﬁnition 4) with stationary measure ˜ P .
Let ˜ P t be the density of ˜ z t and let ˜ P 0 be absolutely continuous with respect to the Lebesgue measure.
Then:
(1) If ˜ P 0 is supported on D , ˜ P t is supported on D , ∀ t >(2) χ 2 ( ˜ P t , ˜ P )
≤ e − t/ C pc χ 2 ( ˜
[Equation] [Equation]
P )
Finally, we will also need a discretized version of the continuous-time Langevin diffusion.
Deﬁnition 13 (Euler-Mariyama Discretization for the Langevin Diffusion) .
For a potential g : R d → R , the Euler-Mariyama discretization of the continuous Langevin diffusion with discretization level h is the discrete-time random walk
[Equation]
We will also associate with this discrete-time process a stochastic differential equation
[Equation]
(12) which has the property that ˆ z kh and z k follow the same distribution for all k ≥ 0 , if z 0 and ˆ z 0 follow the same distribution.
Note the equivalence between equation 11 and equation 12 follows by noting that the drift in the interval [ th, ( t + 1) h ] in equation 12 is constant – so we only need to integrate the Brownian motion for time h .
We can also consider the discretized version of the reﬂected Langevin diffusion, which looks like projected gradient descent with Gaussian noise Bubeck et al.
Deﬁnition 14 (Discretization for the Reﬂected Langevin Diffusion) .
For a potential g : R d → R , the
Euler-Mariyama discretization of the reﬂected Langevin diffusion with discretization level h is the discrete-time random walk
[Equation]
where Π D is the projection onto the set D .
We can also associate with this discrete-time process a stochastic differential equation
[Equation]
for a measure U supported on ∂ D , which has the property that ˆ y kh and y k follow the same distribution for all k ≥ 0 , if y 0 and ˆ y 0 follow the same distribution.
B P ROOF OF L EMMA 1: C ONVERGENCE OF G RADIENT D ESCENT TO ˆ z
First, we state the tweak on the usual descent lemma (Nesterov, 2003), due to the fact that the
Hessian’s norm is not bounded in all of R d .
We only need to identify a region A containing the starting point z (0) such that the function’s value at points outside A are guaranteed to be more than the function value at z (0) .
Namely:
Lemma 9 (Adjusted Descent Lemma) .
Let f : R d
→ R be a twice differentiable function.
For a region A , let it hold that ∀ z ∈ A , (cid:107)∇ 2 f ( z ) (cid:107) op ≤ Q < ∞ .
Further, let ∀ z ∈ R d \A , f ( z ) ≥ f
( z (0) ) for a known point z
(0) ∈ A .
Then the gradient descent iterates starting with z (0) with the learning rate η := 1 Q satisfy
[Equation]
We ﬁrst state the well-known descent lemma.
Lemma 10 (Descent Lemma, Eq 1.2.13 in (Nesterov, 2003)) .
Let f be a twice differentiable function, and at a point z , the hessian satisﬁes (cid:107)∇ 2 f ( z ) (cid:107) op ≤ Q < ∞ .
Then with the learning rate η := 1 Q , the next gradient descent iterate z + satisﬁes
[Equation]
Using this, we prove the Adjusted Descent Lemma.
Proof of Lemma 9.
We will prove the Lemma by induction.
To that end, let us call by I ( s ) , ∀ s ≥the proposition in the Lemma statement, i.e.
[Equation]
We ﬁrst write two more propositions J ( s ) and K ( s ) :
[Equation]
We will use induction to show that I ( s ) ∧ J ( s ) ∧ K ( s ) hold true ∀ s ≥ 0 .
J (0) holds true, since z (
) ∈
A is given in the statement.
K (0) is also given in the statement of the lemma.
Since ∀ z ∈ A , (cid:107)∇ 2 f ( z ) (cid:107) op ≤ Q ; from Lemma 10, we conclude that
[Equation]
Hence I (0) also holds true.
So the base case of the induction ( s = 0 ) is done, i.e. I (0) ∧ J (0) ∧ K (0) .
Since I ( s ) implies descent, and K ( s ) implies that all points outside region A have larger function value, we also have
[Equation]
Combining implications equation 15 and equation 16 and Lemma 10, we get
[Equation]
Hence the induction step also goes through.
We can now use this to prove convergence of gradient descent.
Proof of Lemma 1.
To use the adjusted descent lemma (Lemma 9)
[Equation]
let us deﬁne the appropriate region A .
[Equation]
Obviously z (0) = 0 ∈ A .
We also have ∀ z ∈ R d \ A , the following holds
[Equation]
C + =
[Equation]
m
[Equation]
z − ˆ z
[Equation]
+ M
[Equation]
ˆ z (cid:107) ) denotes a non-negative quantity, which does not affect the sign of these inequalities.
In the above calculation, (0) follows from Assumptions 1 and 2, (1) uses the deﬁnition of A from equation equation 18, and (2) uses Lemma 23.
In A , the norm of the Hessian can be bounded as below.
The expression of ∇ 2 f can be inferred from the expression of ∇ 2 L given in equation equation 56.
We then get
[Equation]
where (0) follows from Assumption 2, (1) follows since ∀ i ∈
[ d ] , (cid:13)(cid:13) ∇ 2 G i ( z ) (cid:13)(cid:13) op ≤ (cid:107)∇ 2 G ( z ) (cid:107) op ; which follows from the sup -deﬁnition of the (cid:96) 2 norm (Lim, 2005).
In (1) , we also used the fact that (cid:107) v (cid:107) 1 ≤ (cid:112) dim ( v ) · (cid:107) v (cid:107) 2 .
And in (2) we used the diameter of A of equation equation 18.
With A , Q deﬁned through equations equation 18 and equation 19, the conditions of Lemma 9 are satisﬁed, so we can use the descent equation.
We can then use the standard argument below
[Equation]
Since f is a non-negative function
[Equation]
it is bounded below by zero.
This means that
[Equation]
We note that
[Equation]
By Lemma 9 we then have
[Equation]
Furthermore, by Assumption 1 we have
[Equation]
S
Hence, setting δ = 1 m 2 · (cid:113) 2 Qf ( z
[Equation]
S gives us the required number of gradient descent steps.
We will ﬁnally simplify the expression for S .
Since f ( z (0) )
=
f (0) =
12 (cid:107) G (0) − x (cid:107) 2 = 12 (cid:107) x (cid:107) 2 (from
Assumption 3), we get
[Equation]
Plugging in the expression for Q from equation 19, and using (cid:107) x (cid:107)
=
O ( √ d ) from Lemma 24, we get S
= O
( d 2 δ 2 ) as desired.
C P ROOF OF L EMMA 2: F AST M IXING OF L ANGEVIN D IFFUSION
Notation: We ﬁrst set up some notation.
We denote by z t the time t iterate of Langevin diffusion (Deﬁnition 3) with g :=
L , having as stationary distribution P ( z ) ≡ p ( z | x ) .
Further, we denote by
P t the density of z t .
Similarly, we denote by ˜
z t the iterates of the reﬂected Langevin chain (Deﬁnition 4) with g :=
L , with stationary distribution ˜ P ( z ) ∝ e − L ( z ) supported over z ∈ D .
We denote by ˜ P t the density of ˜ z t .
Lemma 11.
rad ( D ) deﬁned in Lemma 13 along with the restriction on β from the same lemma satisﬁes
[Equation]
This in particular means that rad ( D ) ≤
O (1 /d ) .
Proof.
Follows by straightforward substitution.
Organization:
This section will be organized as follows.
In Subsection C.1, we prove a lemma quantifying the difference between the distributions of the regular and reﬂected Langevin Chains (Lemma 12).
To use this lemma, we require a property of the Langevin diffusion for our setting (Lemma 13) — namely, that the chain with probability 1 − (cid:15) remains in the region D .
We tackle this in Subsection C.2.
In subsection C.3, we prove Lemma 4, namely that p ( z | x ) restricted to D is log-concave.
Finally, using the above three pieces, we will prove prove the main lemma of this section (Lemma 2) in subsection C.4.
We note that some of the techniques here have overlap with techniques used in (Moitra & Risteski, 2020), but we provide all the proofs in detail for completeness.
C. R ELATING R ESTRICTED AND U NRESTRICTED L ANGEVIN C HAINS
The main lemma of this section is as follows:
Lemma 12 (Comparing with reﬂected Langevin diffusion) .
Let ˜ z t follow the reﬂected Langevin
SDE (Deﬁnition 4) with g := L ), with stationary measure ˜ P and with domain D .
Let z t follow the
Langevin diffusion SDE (Deﬁnition 3) with g := L , with stationary measure P .
Let ˜ P t be the density of ˜ z t and let P t be the density of z t and let ˜ P 0 be absolutely continuous with respect to the Lebesgue measure.
Then if ∀ t ≥ 0 , Pr [ ∃ a ∈ [0 , t ] , z a / ∈ D ] ≤ α , it holds that:
[Equation]
Proof.
Consider the total variation distance between P t and P .
Using triangle inequality, we have
[Equation]
We ﬁrst bound the second term using Theorem 3.
Since D is convex and L is convex over D by
Lemma 4, we have
[Equation]
For the ﬁrst term, consider the coupling of z t and ˜
z t such that the Brownian motion B t is the same.
We then have for any t ≥
[Equation]
where the ﬁrst inequality follows by Lemma 6, and the second inequality follows from a rewrite that uses the fact that if the unrestricted chain z t stays inside the region D , then the two diffusions won’t differ at all due to the coupling of the Brownian motion.
The term α bounds the probability of the unrestricted chain z t leaving the region D .
For the third term, we need to bound d TV ( ˜ P, P ) .
Note that P : R d → R + , P ( z ) ∝ e − L ( z ) and ˜ P : D → R + , ˜ P
( z ) ∝ e − L ( z ) .
This means that ∀ z ∈ D , ˜ P ( z ) > P ( z ) and ∀ z ∈ R d \ D , P ( z ) > ˜ P
( z ) .
So we can write
[Equation]
where z ∞ is the follows the stationary distribution of the Langevin diffusion P .
From the drift bound in the lemma statement, we can take the limit of t → ∞ to get
[Equation]
[Equation]
Using equation equation 24 and equation 25, we get
[Equation]
[Equation]
The lemma statement follows from equations equation 23, equation 22 and equation 26.
C. E FFECTIVE REGION FOR L ANGEVIN DIFFUSION
In this section, we prove Langevin mostly stays in an effective region D , namely:
Lemma 13 (Effective region for Langevin diffusion) .
Let D :=
{ z : (cid:107)
z − ˆ z (cid:107) ≤ rad
( D ) } with rad
[Equation]
Let z 0 satisfy (cid:107) z 0 − ˆ z (cid:107) ≤ 12 rad ( D ) .
Then with β ≤ β 0 , we have that ∀ T > 0 , Pr
[ ∃ t ∈ [0 , T ] , z t / ∈ D ] ≤ (cid:15)/ 4 , where
[Equation]
Towards proving Lemma 13, we will ﬁrst prove Lemma 3.
Proof of Lemma 3.
Using dz = −∇
L ( z ) dt + √ 2 dB t along with Itó’s Lemma, we can compute
[Equation]
We will bound each term on the right hand side.
First note that ∇ η ( z ) =
z − ˆ z , and so (cid:107)∇ η ( z ) (cid:107) =
(cid:112) 2 η ( z ) for all z ∈ R d .
We proceed to the second and third term since they’re straightforward.
For the third term, we have (cid:104)∇ η ( z ) , dB t (cid:105) =
(cid:107) η ( z ) (cid:107) dB t
=
(cid:112) 2 η ( z ) dB t , using a slight abuse of notation to denote the 1-dimensional Brownian motion (in the RHS) as also B t .
For the second term, note that ∆ η ( z ) =
d since ∇ 2 η ( z ) = I .
So:
[Equation]
For the ﬁrst term, from Lemma 26 we get (cid:104)∇
[Equation] [Equation]
First, we analyze (cid:104)
z − ˆ z, ˆ z (cid:105) .
Using Cauchy-Schwartz we can write (cid:104)
z − ˆ z, ˆ z (cid:105) ≥ −(cid:107)
z − ˆ z (cid:107)(cid:107) ˆ z (cid:107) .
Case 1.
(cid:107)
z − ˆ z (cid:107) ≤ (cid:107) ˆ z (cid:107) .
We can write that (cid:104)
z − ˆ z, ˆ z (cid:105) ≥ −(cid:107)
ˆ z (cid:107) 2 .
Case 2.
(cid:107)
z − ˆ z (cid:107) > (cid:107) ˆ z (cid:107) .
We can write that (cid:104)
z − ˆ z, ˆ z (cid:105) ≥ −(cid:107)
z − ˆ z (cid:107) 2 .
In either case, it holds that
[Equation]
Second, by Assumption 1, we can say that
[Equation]
Third, by the expression of the norm of the remainder from Lemma 26, we get
[Equation]
For z ∈ D we have (cid:107) z − ˆ z (cid:107) ≤ rad ( D ) so (cid:104)
z − ˆ z, R
[Equation]
The inequality on rad ( D ) from Lemma 11 implies that 3 dMM 2 · rad ( D ) ≤ m 2 2 and √ dMM 3 · rad ( D ) 2 ≤ m 2 2 , and so
[Equation]
Plugging in equation equation 30, equation 31, equation 32 in equation equation 29, we get (cid:104)∇
[Equation]
Using this in equation equation 28, and substituting the deﬁnition of η ( z ) , i.e. (cid:107) z − ˆ z (cid:107) 2 = 2 η ( z ) , ﬁnishes the proof.
Next, we prove high probability deviation bounds for Cox-Ingersoll-Ross processes.
Precisely, we show:
Lemma 14 (Concentration Bounds on Cox-Ingersoll-Ross Process) .
Consider the SDE
[Equation]
for ˜ N ∈ N and w > 0 .
Then for any (cid:15) >
[Equation]
Proof.
The SDE describes a Cox-Ingersoll-Ross process of dimension ˜ N , which equals in distribution to the random variable Y t as deﬁned below, and with matching initial condition Y 0 :
= X 0 .
[Equation]
Here V ( i ) t follow the Ornstein-Uhlenbeck equation dV ( i ) t =
[Equation]
V
[Equation]
+ dB
[Equation]
and V ( i )
[Equation]
for all i ∈ [ ˜ N ] .
dB ( i ) t are (1-d) Brownian motions, and we have referenced them separately because they are independent for each i ∈
[ ˜ N ] .
Indeed, applying Itó’s Lemma, we recover the same SDE as:
[Equation]
Here in the RHS of the ﬁrst expression, the term 12 · 2 is the term 12 · ∂
[Equation]
from Itó’s Lemma.
And in the third expression, notice that we have replaced a sum of brownian motions with a single one of matching variance.
Now we can write an explicit solution for the SDE.
Since we know the solution of an OrnsteinUhlenbeck process, each V ( i ) t can be solved as (now we drop the indexing i since the solution is same for each i ∈
[ ˜ N ] )
[Equation] [Equation]
Applying the reﬂection principle to F ( t ) , we get
[Equation]
[Equation]
Further, writing the distribution of F ( t ) , we have
[Equation]
From standard Gaussian tail bounds, we have
[Equation]
From equation 35, substituting a := r (cid:113) 2 w (1 − e − wT ) we have
[Equation]
Since the quantity F ( t ) has a symmetric density about zero for all t
[Equation]
we can write the two sided bound as
[Equation]
Choosing r such that (cid:15)/ ˜ N = 4 e − r 2 in equation
[Equation]
and using a union bound over the indices
i ∈
[ ˜ N ]
[Equation]
we have
[Equation]
Plugging in V ( i )
[Equation]
N for all i ∈
[ ˜ N ] , we get ∀ T >w.p.
[Equation]
1 −
[Equation] [Equation]
Finally, using ( a + b ) 2 ≤ 2( a 2 + b 2 ) and summing over all indices i ∈
[ ˜ N ] , we get
[Equation]
Replacing the sum by Y t using equation 33 and replacing X t , X 0 in place of Y t , Y 0 gives the result.
Next, since all of our SDEs so far have been actually inequalities, we will need a standard comparison theorem for SDEs:
Lemma 15 (Comparison theorem, (Ikeda & Watanabe, 1977)) .
Let U t , X t be two stochastic processes following the SDEs
[Equation]
driven by the same Brownian motion B t .
Let at least one of these SDEs have a pathwise unique solution.
If ∀ x ∈ R d , f ( x ) ≤ g ( x ) , and if U 0 = X 0 , then with probability 1 ,
U t ≤ X t ∀ t ≥
Lemma 15 shows domination of stochastic variables based on an inequality in their respective SDEs.
This is almost what we need, since we showed an inequality in SDEs in Lemma 3 and showed that the upper bound is concentrated in Lemma 14.
We need a slight adjustment on top of this though.
Note that the inequality between SDEs in Lemma 3 holds only for z ∈ D , however the inequality that the above Lemma requires is on all points α , f ( α ) ≤ g ( α ) .
We can work through this by simply noting that conditioned on X t (the upper bound SDE) being concentrated in a region, U t will also be concentrated in the region.
We state and show this below.
Lemma 16.
Let U t , X t be two stochastic processes whose SDEs follow
[Equation]
driven by the same Brownian motion B t .
Let at least one of these SDEs have a pathwise unique solution.
If f ( α ) ≤ g ( α ) for all α ≤ α 0 , and if U 0 = X 0 ≤ α 0 ; then ∀
[Equation]
> 0 , ∀ ¯ X ≤
[Equation]
0 Pr (cid:2)
[Equation]
Proof.
Consider any T > 0 and ¯ X ≤ α 0 .
Pr (cid:2) ∀ t ∈ [0 , T ] , U t ≤ ¯ X (cid:3) ≥ Pr (cid:2) ∀ t ∈ [0 , T ] , U t ≤ X t and ∀ t ∈ [0 , T ] , X t ≤ ¯ X (cid:3) ≥ Pr (cid:2) ∀
[Equation]
[0 , T ] , X
[Equation]
≤ ¯ X (cid:3) · Pr (cid:2) ∀ t ∈ [0 , T ] , U t ≤ X
[Equation]
| ∀ t ∈ [0 , T ] , X t ≤ ¯ X (cid:3) (cid:124) (cid:123)(cid:122)
[Equation]
Where the last probability is one, because both U t and X t are continuous-time processes, and so
U t > X t given X t ≤ ¯ X ≤ α 0 is not possible because U t cannot cross over X t in the f ≤ g region (using comparison Lemma 15).
Given the above preliminaries, we can ﬁnally complete the proof of Lemma 13.
Proof of Lemma 13.
Instantiate Lemma 16 with U t :
= η ( z t ) and instantiate another SDE on X t with the same Brownian motion as U t , as below
[Equation]
such that X 0 :
= U
= η ( z 0 )
= 12 (cid:107) z 0 − ˆ z (cid:107) 2 .
From Lemma 14 (plugging in ˜ N := (cid:100)(cid:107) ˆ z (cid:107) 2 + d (cid:101) and w := m 2 β 2 ), we can conclude that for any (cid:15) >
[Equation]
Using the Lemma 16 on U t and X t , with ¯ X := 2 X 0 + 4 ˜ Nw log (cid:16) 4 ˜ N(cid:15) (cid:17) , and with α 0 :
= 12 rad ( D )(since the inequality between SDEs of U t and X t holds in the region U t ≤ 12 rad ( D ) 2 from lemma 3), we get
[Equation]
where the last inequality follows from equation 38.
Note that U t ≤ α 0 ≡ (cid:107) z t − ˆ z (cid:107) ≤ rad
( D ) ≡ z t ∈ D .
This is exactly the event whose probability we need to lower bound.
We now check that the condition for the above is met, namely:
[Equation]
Where X 0 = 12 (cid:107) z 0 − ˆ z (cid:107) 2 ≤ 12 · (cid:0) 12 rad ( D ) (cid:1) 2 from the lemma statement.
This then becomes
[Equation]
Putting in the values ˜ N, w from above, and using Lemma 23 to bound (cid:107) ˆ z (cid:107) (since the RHS above is an increasing function of ˜ N ), and bounding the ceiling trivially by (cid:100) κ (cid:101) ≤ κ + d (since d ≥ 1 ) means it sufﬁces that
[Equation]
Using the bound on β the above inequality is indeed satisﬁed for our choice of rad ( D ) .
C. P ROOF OF L EMMA
Proof.
Using the expression of ∇ 2 L from equation equation 56, we get that
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
And we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
[Equation]
In (0) , we used the fact that ∀ i ∈ [ d ] , (cid:13)(cid:13) ∇ 2 G i ( z ) (cid:13)(cid:13) op ≤ (cid:107)∇ 2 G ( z ) (cid:107) op ; which can be derived from the sup -deﬁnition of the tensor (cid:96) 2 norm from (Lim, 2005).
In (1) , we’ve used Assumption 2.
Using the expression of rad ( D ) from Lemma 11, we get
[Equation]
op ≤ m6 √ d z ∈ D ≤ mz ∈ D ( since d ≥ 1)
Using this in equation 39, we get ∇ 2 L ( z ) (cid:23) I for z ∈ D .
C. P ROOF OF L EMMA
We can now put together the proof of Lemma 2.
Proof of Lemma 2.
Using Lemma 12 with D deﬁned as in Lemma 13, with α :
=
(cid:15)/ 4 in Lemmaand setting T =
rad 2 ( D ) log 1 (cid:15) we get e −
T/ 2 rad 2 ( D ) ≤ (cid:15) 2 .
Using rad ( D )
=
O (1 /d ) from lemma 11, the result follows.
D P ROOF OF T HEOREM 1: E ULER -M ARIYAMA D ISCRETIZATION AND N EURAL S IMULATION
In this section, we use the mixing time bounds in the previous sections, along with a discretization error analysis when running a disretized (in the simplest manner, via a Euler-Mariyama scheme) version of the Langevin diffusion chain.
This is done in Subsection D.1.
Subsequently, we show that each of these steps can be simulated by a “layer” of a neural network.
This is done in Subsection D.2.
Finally, we put everything together to prove Theorem 1 in Subsection D.3.
D. B OUNDING THE ERROR IN THE E ULER -M ARIYAMA DISCRETIZATION
We ﬁnally wish to show that with a polynomially small discretization level (i.e. h ), the EulerMariyama scheme approximates appropriately for our purposes the Langevin diffusion process.
We recall from equation 2 that this Markov process is described by
[Equation]
There is a rich ﬁeld in the literature on discretizing Langevin diffusion (Dalalyan, 2016; Raginsky et al., 2017; Dalalyan, 2017) primarily used for designing efﬁcient sampling algorithms — a large fraction of them analyzing the above discretization.
We will mostly rely on tools from Bubeck et al.
We show:
Lemma 17 (Euler-Mariyama discretization) .
Let z t follow the Langevin diffusion dz t = −∇ L ( z t )
dt + √ 2 dB t , and let P t denote the distribution of z t .
Let ˆ z t follow the continuous-time Markov process determined by the Euler-Mariyama discretization scheme with discretization level h given by equation 12, namely
[Equation]
Let us denote by ˆ P t the distribution of ˆ z t .
Finally, let z
= ˆ z
0 ∈ D satisfying (cid:107) z 0 − ˆ z (cid:107) ≤ 12 rad ( D ) , with D deﬁned as in Lemma 13.
Then, if T = (2 rad ( D ))
2 log(1 /(cid:15) ) and h = poly ( (cid:15), 1 /d, 1 /M, m, β ) , we have d TV ( ˆ P T , P T ) ≤ (cid:15)
Proof.
Let y t follow the Reﬂected Langevin SDE (Deﬁnition 4), with g :
=
L and region D , and with the initialization y
= z 0 .
Let ˆ y t follow the continuous-time extension of
the discretized reﬂected
Langevin SDE given by equation 14 with g :=
L and region D ), with the initial point ˆ y
= z 0 .
Let
Q t and ˆ Q t denote the respective distributions.
Let Q denote the stationary distribution of y t .
From triangle inequality, we can decompose our error as
[Equation]
We will bound each of the terms in turn.
For the ﬁrst term, consider a coupling of y t and z t that uses the same Brownian motion for both processes, so long as neither has left D , and chooses an arbitrary coupling henceforth.
By Lemma 6, we have
[Equation]
where the last inequality follows by Lemma 13.
For the second term, we can further break it up, again by using the triangle inequality as
[Equation]
We will use a result about the discretization of reﬂected Langevin diffusion from (Bubeck et al., 2018).
Towards that, let us denote
[Equation] [Equation]
Using Theorem 1 from (Bubeck et al., 2018), if T = (2 rad ( D ))
2 log(1 /(cid:15) ) and h = poly ( (cid:15), 1 /d, 1 /L 1 D , 1 /L 2 D ) , both terms in equation 43 are upper bounded by (cid:15)/ 4 .
On the other hand, by equations equation 55 and equation 56, we have
[Equation]
Using Lemma 23 and the ﬁrst term of the RHS from Lemma
[Equation]
we get
[Equation]
Which follows using Lemma 24.
For the second order bound, a similar calculation as the proof of
Lemma 4 we get
[Equation]
Plugging the bounds above, we get that if T =
[Equation]
rad
[Equation]
D )) 2 log
[Equation]
and h
[Equation]
poly ( (cid:15), 1 /d, 1 /M, m, β ) , then d TV (cid:16) ˆ Q T , Q T (cid:17) ≤ (cid:15)/ 2 .
Finally, for the third term, we will show that
[Equation]
Towards that, by Lemma 6, coupling the Gaussian noise to be the same for { ˆ P t } t ∈ [ T ] and { ˆ Q t } t ∈
[ T ] , we have
[Equation]
By the deﬁnition of total variation distance, and Lemma 6 again, we have
[Equation] [Equation]
z
[Equation]
By Lemma 13 and Lemma 6, we have Pr [ ∃ t ∈
[ T ] , z t / ∈ D ] ≤ (cid:15)/ 4 and d TV (cid:0) { ˆ z t } t ∈ [ T ] , { z t } t ∈ [ T ] (cid:1) ≤ (cid:15)/ 4 which proves equation 46.
D. N EURAL S IMULATION OF A RITHMETIC O PERATIONS
The deep latent Gaussian will be an overall map from observables x ∈ R d to latents z ∈ R d , such that z is an approximate sample from the posterior p ( z | x ) .
In this section, we prove “simulation” lemmas that show how to combine neural networks for two functions f, g to create a neural network for some arithmetic operation applied to f, g (e.g. f + g , f × g , etc.).
Recall, ρ : R → R denotes the square activation function, i.e. ρ ( x )
Lemma 18.
If f : R d → R is a neural network with M parameters and g : R d → R is a neural network with N parameters, both with set of activation functions Σ , then w 1 f + w 2 g can be represented by a neural network with O ( M + N ) parameters and set of activation functions Σ for any w 1 , w 2 ∈ R .
Proof.
We simply note that we can make a network with one more layer than the maximum of the depth of f and g that adds up the scaled outputs of f , g to produce w 1 f + w 2 g .
No additional nonlinearity is needed.
Lemma 19.
If f : R d → R is a neural network with M parameters and g : R d → R is a neural network with N parameters, both with set of activation functions Σ , then for any w ∈ R , v = wfg can be represented by a neural network with O ( M + N ) parameters with set of activations Σ ∪ { ρ } .
Proof.
Noting xy = 14 (cid:0) ( x + y ) 2 − ( x − y ) 2 (cid:1) for all x, y ∈ R , we can implement the multiplication of scalars using a two-layer neural network with O (1) parameters using the ρ activation function.
We then proceed same as for addition, adding one layer to multiply the outputs of f, g .
Lemma 20.
If G : R d → R d is represented by a neural network with N parameters and differentiable activation function set Σ , given an x ∈ R d and z ∈ R d , one can compute for any c 1 , c 2 ∈ R :
[Equation]
G
[Equation]
T
[Equation]
G
[Equation]
) via a neural network with O ( Nd 2 ) parameters and activation functions
Σ ∪ Σ (cid:48) ∪ { ρ } , where Σ (cid:48) contains the derivatives of the activations in Σ .
Proof.
Each coordinate of J G : R d → R d × d can be written as a neural network with O ( N ) parameters by Lemma 5, with activation functions Σ ∪ Σ (cid:48) .
Each coordinate of G ( z ) − x can be computed in O ( N ) parameters using lemma 18.
Performing the multiplications needed for J G ( z ) T ( G ( z ) − x ) via Lemma 19 results in a network of size O ( Nd 2 ) due to the d 2 size of the Jacobian, and together with the addition of c 1 z (via O ( d ) parameters) results in the ﬁnal bound of O ( Nd 2 ) on the parameters.
D. P ROOF OF T HEOREM
Finally, we can put all the ingredients together to prove Theorem 1.
Proof of Theorem 1.
The deep latent Gaussian we construct will have all latent dimensions d as well as the output dimension d .
The input will be x ∈ R d , the observable on which we want to carry out the inference.
We will ﬁrst create a part for the GD simulation, and then one for the Langevin chain.
Using Lemma 1 with δ := 1 4 rad ( D ) , if we run Gradient Descent for S
= O (
[Equation]
steps (using the bound rad ( D ) =
Ω( β √ d ) from Lemma 13), we can get a z init with (cid:107) z init − ˆ z (cid:107) ≤ 14 rad ( D ) .
Using Lemma 20 , we can construct a network D one with O (cid:0) Nd 2 (cid:1) parameters that executes a single step of gradient descent.
In the deep latent Gaussian we construct, we repeat D one with zero variance
O ( dβ 2 ) times to simulate all the steps of gradient descent and construct z init .
The overall number of parameters of the deep latent Gaussian so far is then O (cid:16) poly ( d, 1 β ) · N (cid:17) .
Note, though the ﬁrst variable z 0 has unit variance, we can easily transform it to a zero variance Gaussian to simulate the ﬁrst step by multiplying by 0.
Starting with the random point z 0 , if we run the discretized Langevin dynamics given by equationfor T/h steps with T and h as given by Lemma 17, we get by the error bounds of Lemma 17 and
Lemma 2 and the triangle inequality:
[Equation]
which is the required overall bound.
We further note that each step of the discretized Langevin dynamics can be simulated by a layer of the deep latent Gaussian: by Lemma 20 we can construct a network S one with O (cid:0) Nd 2 (cid:1) parameters that computes the next Langevin iterate.
Choosing the variance to be 2 h , then, accomplishes exactly the distribution of a single step of the discretized dynamics.
Given the choice of T and h , the number of steps we need to run is O ( poly (1 /(cid:15), d, 1 /β )) .
Hence, the overall number of parameters required for simulating the discretized dynamics is O ( poly (1 /(cid:15), d, 1 /β ) · N ) .
Finally, it’s clear that the dependence on x in the layers comes through the J G ( z ) T ( G ( z ) − x ) terms in the gradient and
Langevin updates—so the weights for the deep latent Gaussian can be produced by a network of size
O ( poly (1 /(cid:15), d, 1 /β ) · N ) as well.
The multiplication by x can be implemented via the square activation ρ .
Thus, the proof is ﬁnished
E P ROOF OF T HEOREM 2: L OWER BOUND
First, we observe the following proposition, which shows that the samples x with high probability lie near a vertex of the hypercube {± 1 } d
Lemma 21 (Closeness to hypercube) .
A sample x from the VAE with G ( z ) =
C ( sgn ( z )) and variance β 2 I satisﬁes, for all
c >
[Equation]
Proof.
Since x − G ( z ) is a Gaussian with variance β 2 I , the claim follows via standard concentration of the norm of a Gaussian.
Furthermore, we also have the following lemma, that shows that when x is near a vertex of the hypercube, the posterior p ( z | x ) concentrates near the C − 1 ( sgn ( x )) , that is, the pre-image of the nearest point to x on the hypercube.
Lemma 22 (Concentration near pre-image) .
Let x be such that (cid:107) x − sgn
( x ) (cid:107) ≤ 6 β √ d .
Then,
[Equation]
Proof.
By Bayes rule, since p ( sgn ( z )) is uniform, we have
[Equation]
Consider then any b ∈ {± 1 } d , b (cid:54) = sgn
( z ) .
We have:
[Equation]
where the last inequality follows for β
= o (1 / √ d ) and a small enough constant in the o ( · ) notation.
Hence,
[Equation]
Proof of Theorem 2.
The proof will proceed by a reduction to Conjecture 1.
Namely, C be a Boolean circuit satisfying the one-way-function assumptions in Conjecture 1.
Let G be deﬁned as G ( z ) =
C ( sgn ( z ))
Suppose that there is an encoder E of size T ( n ) , s.t. with probability at least (cid:15) ( d ) over the choice of x (wrt to the distribution of G ), we have TV ( p ( z | x ) , q ( z | x )) ≤ 1 / 10 , where q ( z | x ) is the distribution of the encoder when x is supplied as input.
We will ﬁrst show how to construct a neural sequential Gaussian C (cid:48)(cid:48) which violates Conjecture 1 in an “average sense”: namely, for 1 / poly ( d ) fraction of inputs ˜ x ∈ {± 1 } d , with probability 1 − exp( − d ) it outputs the ˜ z , s.t.
˜ x
= C (˜ z ) .
Subsequently we will remove the randomness — i.e. we will produce from this a deterministic and a neural network, rather than a circuit – subsequently we will show how to remove both the randomness, and use simulate the neural networks using a Boolean circuit.
This deep latent Gaussian C (cid:48)(cid:48) receives a ˜ x ∈ {± 1 } d and constructs (randomly) a multiple samples x ∈ R d as
[Equation]
Subsequently, it samples (multiple) z from E ( z | x ) and checks whether ˜ x
= C ( sgn ( z )) .
If yes, it outputs sgn
( z ) .
If for none of the samples x and z , sgn ( z ) =
˜ x , it outputs (1 , 1 , . . . , 1) .
We will show that for 1 / poly ( d ) fraction of inputs ˜ x ∈ {± 1 } d , with probability 1 − exp( − d ) ,
˜ C
(cid:48)(cid:48)
outputs the ˜ z , s.t.
˜ x
= C (˜ z ) .
Note that x is distributed according to the distribution of the generator G .
(Since ˜ x is uniformly distributed on the hypercube.)
Let E 1 be the event that d TV ( p ( z | x ) , E ( z )) ≤ 110 , E 2 the event that (cid:107) x − sgn
( x ) (cid:107) ≤ 6 β √ d .
By Assumption, Pr [ ¯ E 1 ] ≤ 1 − (cid:15) ( d ) ; by Lemma 21, for a large enough d ,
Pr [ ¯ E 2 ] ≤ (cid:15) ( d ) 2 .
Hence, Pr [ ¯ E 1 ∨ ¯ E 2 ] ≤ 1 − (cid:15) ( d ) 2 . , i.e. Pr [ E 1 ∧ E 2 ] ≥ (cid:15) ( d ) 2 .
Consider an x for which both events E 1 and E 2 attain.
From the the deﬁnition of TV distance and
Lemma 22, we have
[Equation]
Hence, we have
[Equation]
By the tower rule of probability, we can rewrite this as
[Equation]
Let us denote the random variable
[Equation]
))] ≤ 1 / 5 (cid:1)(cid:3) such that we have
[Equation]
By the reverse Markov inequality, it follows that
[Equation]
Taking a = (cid:15) ( d ) / 2 we get Pr [ A ( z ) ≤ (cid:15) ( d ) / 2] ≤ 1 − (cid:15) 2 ( d ) / 4 .
Hence, with probability at least (cid:15) 2 ( d ) / 4 over the choice of ˆ z , we have
[Equation]
As we indicated, C (cid:48)(cid:48) will repeat sampling x and z | x multiple times.
Let us denote by ξ 1 the Gaussian noise sampled in equation 47 and { ξ i 2 } Li =
the Gaussian samples used in sampling z ∼ q ( z | x ) .
Consider sampling { ξ i 1 , i ∈ [1 , M ] } , where M = Ω (cid:16)
[Equation]
(cid:17) and { ξ i
[Equation]
[ M (cid:48)
[Equation]
where M (cid:48)
= Ω(log d ) , and for each i ∈
[ M ] , j ∈ [ M (cid:48) ] outputs x
= ˆ x + ξ i 1 , and samples from q ( z | x ) using the randomness in { ξ i (cid:48) ,j 2 } Li (cid:48) =1 .
By Chernoff bounds, with probability 1 − exp( − Ω( d )) , at least one pair of indices ( i, j ) is such that the corresponding choice of random vectors results in a choice of z , such that C ( sgn ( z ))
=
ˆ x .
Since the total number of possible strings ˆ x is 2 d , taking the constants in M, M (cid:48) sufﬁciently large, by union bound, there is a pair ( i, j ) , s.t.
the corresponding vectors ξ i 1 , { ξ i (cid:48) ,j 2 } Li (cid:48)
=1 works for all ˆ x .
Hard-coding these into the weights of a neural network, we get a deterministic neural network of size O ( |C| + T ( d )) , that with probability (cid:15) 2 ( d ) / 4 inverts C .
The only leftover issue is that E is a neural network (hence, a continuous function), whereas we are trying to produce a Boolean circuit.
Conversions between neural networks applied to Boolean inputs and Boolean circuits are fairly standard (see Siegelmann (2012), Chapter 4, Theorem 6), but we include a proof sketch for completeness nevertheless.
First, we show that having logarithmic in L, W, M precision for the weights and activations in E sufﬁces.
For notational convenience, let us denote by N the maximum degree of any node in E .
The proof is essentially the same as Lemma 4.2.1 in Siegelmann (2012) – the only difference is that our inputs are close to binary, but not exactly binary.
Namely, for some constants δ w , δ a to be chosen, consider a network ˆ E : R d → R d , which has the same architecture as E , and additionally:
• The edge weights { ˜ w i,j } are produced by truncating the weights w i,j to log( δ w ) signiﬁcant bits, s.t. ∀
i, j : | w i,j − ˜ w
i,j | ≤ δ w .
• If a node in E has activation σ , the corresponding node in E (cid:48) has activation σ (cid:48) that truncates the value of σ to log( δ a ) signiﬁcant bits, s.t.
∀ x, | σ ( x ) − σ (cid:48) ( x ) | ≤ δ a .
Then, we claim that for nodes at depth t from the input, denoting v ( x ) and ˆ v ( x ) the function calculated by the node in E and ˆ E respectively, we will show by induction:
[Equation]
Let us denote by (cid:15) t the maximum of | v ( x ) − ˆ v ( x ) | for any nodes v, ˆ v at depth t .
Suppose the claim is true for nodes at depth t and consider a node at depth t + 1 .
Denoting by v 1 , v 2 , . .
. , v n the inputs to v , we have
[Equation]
where equation 50 follows by Lipschitzness of σ and equation 52 follows from the fact that the weights are bounded by W and the outputs v, ˆ v are bounded by M .
Unfolding the recursion, we get
[Equation]
Given this estimate, we can choose δ w , δ a , s.t.
the values at each output coordinate of E and ˆ E match.
It sufﬁces to have ( LNW ) d ( LNMδ w + δ a ) < 1 , which obtains when δ w , δ
a = O (cid:0) 1 M ( LNW ) − d (cid:1) – i.e. it sufﬁces to only keep O ( d log( LNW
+ log( M ))
signiﬁcant bits.
Next, we show how to simulate ˆ E by a Boolean circuit – this is essentially part of the proof of Lemma 4.2.2 in Siegelmann (2012).
Namely, each pre-activation can be computed by a subcircuit of size
O (cid:16) N ( d log( LNW ) + log( M )) 2 (cid:17) by the classic carry-lookahead algorithm for addition.
Since we only require log( δ a ) bits of accuracy for the activation function, and each activation is at most M (so we need O (log M ) bits for the integer part), we need O (log M + log( δ a )) nodes for the activation function.
Hence, we can simulate ˆ E by a Boolean circuit which is at most a factor of poly ( N, log( L ) , log( W ) , log( M )) .
Since L, W, M are o (exp( poly ( d ))) , the size of the size of the resulting Boolean circuit is still poly ( d ) .
Thus, we’ve constructed a polynomial-sized circuit which inverts the one-way-permutation, thus violating Conjecture 1.
F T ECHNICAL L EMMAS
This section contains several equations and calculational lemmas that we repeatedly use across the proofs of Theorem 1.
F. B OUND ON (cid:107) ˆ z (cid:107)
Recall that we deﬁned the inverse point ˆ z for a given x ∈ R d in equation equation 4.
As mentioned earlier, due to bijectivity, it is unique and satisﬁed G
(ˆ z )
= x .
Further, we can easily prove a bound on the norm of ˆ z using our assumptions, which will be useful in various places in the proof.
Lemma 23.
ˆ z deﬁned as in equation 4 with G that satisﬁes Assumption 1, is such that
[Equation]
Proof.
Using strong invertibility from Assumption 1 and centering from Assumption 3, we can write
[Equation]
[Equation]
F. H IGH P ROBABILITY B OUND ON (cid:107) x (cid:107)
Lemma 24 (High-probability bound on (cid:107) x (cid:107) ) .
With x following the density from Deﬁnition 1 (with d l = d o = d ), s.t. G satisﬁes Assumption 2, we have that
[Equation]
Proof.
The density of x is deﬁned by the latent Gaussian.
Let’s denote the error term as e ∼
N (0 , β 2 I ) .
We have
[Equation]
(Where we used assumption 2).
Now z and e follow zero-mean Gaussian densities, with variances I and β 2 I respectively.
Using standard gaussian tail bounds (similar to lemma 21 with c = 2 ), we get the required claim.
Remark:
Since we work under the setting β ≤ O (1) from equation 1, we can use (cid:107) x (cid:107) ≤ O ( √ d ) with probability 1 − exp(Ω( d )) .
F. C HARACTERIZING LOSS NEAR ˆ z
We need to understand how L and ∇ L behaves near ˆ z for several claims in the proofs.
We will use a
Taylor expansion, bounding the contribution of higher order terms.
F.3. E XPRESSIONS FOR D ERIVATIVES
Using equation equation 3, we ﬁrst calculate the expressions of the derivatives as
[Equation]
At the inverse point ˆ z , since G (ˆ z ) =
x , we have:
[Equation]
Since ∇ 2 L (ˆ z ) (cid:23) I , the function L ( z ) is strongly convex in the neighbourhood of ˆ z .
F.3. B OUND ON 3 RD DERIVATIVE OF L
To show the behaviour of ∇ L around ˆ z , we will ﬁrst bound the norm of the 3rd order derivatives of L below.
Lemma 25 (Bounding the 3rd derivative of L ) .
For all z ∈ R d , it holds that
[Equation]
Proof.
An elementary calculation shows that
[Equation]
[Equation]
Here, the tensor product ⊗ T multiplies elements from R d and R d × d to give elements in R d × d × d .
We subscript with T to avoid confusion with the kronecker product
[Equation]
Then, we have:
[Equation]
op + (cid:107) G ( z ) − x (cid:107)
[Table]
max i ∈
[ d ] (cid:13)(cid:13) ∇ 3 G i ( z ) (cid:13)(cid:13) op 
 ≤ (1)β(cid:16) 3 · √ d · (cid:107) J G ( z ) (cid:107) F · (cid:13)(cid:13) ∇ 2 G ( z ) (cid:13)(cid:13) op + √ d · (cid:107) G ( z ) − x (cid:107) · (cid:13)(cid:13) ∇ 3 G ( z ) (cid:13)(cid:13) op (cid:17) ≤ (2)β(cid:16)
3 d · MM 2 + √ d
[Table]
MM 3 · (cid:107) z − ˆ z (cid:107) (cid:17)
In (1) we use (cid:80) i ∈
[ k ] | α i | ≤ √ k · (cid:113)(cid:80) i ∈
[ k ] α 2 i on the ﬁrst part of both the terms; and the fact that ∀ i ∈ [ d ] , (cid:107) T ( i ) (cid:107) op ≤ (cid:107) T (cid:107) op for any tensor T (where T ( i ) denotes a sub-tensor) on the second part of both terms.
This fact follows from the sup -deﬁnition of (cid:96) 2 norm on tensors from (Lim, 2005).
In (2) , we use Assumption 2, as well as (cid:107) A (cid:107) F ≤ √ α · (cid:107) A (cid:107) op for A being an α × α matrix.
F.3. P ERTURBATION BOUND OF ∇ L
Using the above
[Equation]
we show the Taylor expansion result for the gradient of L :
Lemma 26 (Perturbation of ∇ L ) .
For all z ∈ R d , it holds that
[Equation]
where (cid:107) R
[Equation]
Proof.
We Taylor expand ∇ L around ˆ z .
Using equation equation 57 we get :
[Equation]
To bound the norm of the remainder term
[Equation]
we note that
[Equation]
For z mid = t mid z
+ (1 − t mid ) ˆ z for some t mid ∈ [0 , 1] .
And for the third derivative
[Equation]
we use lemma 25 directly to get
[Equation]
This ﬁnishes the proof.
G R EMARK ON ACTIVATION FUNCTIONS
The choice of activation functions in our results is largely a matter of convenience.
Using standard techniques from approximation theory, e.g. Hornik (1991); Yarotsky (2017), one can approximate a neural network with one choice of nonlinearity via a (comparably sized) neural network with another choice of nonlinearity, under very mild conditions on the nonlinearities.
Crucially, this simulation only increases the size by a dimension-independent factor.
This result frees us (for purposes of deriving an expressibility result) to work with activation functions chosen for mathematical convenience and produce results that hold without loss of generality.
For instance, if we wished to use ReLU activations, we can use the following lemma.
We note that this proof is almost verbatim the same as the proof of Lemma 1.3 in Telgarsky (2017) and proofs for other activations like sigmoid or tanh can be written completely analogously.
Lemma 27.
Let Ω ⊆
[ − M, M ] d and let G 1 : [ − M, M ] d → R be a neural network with at most l layers and n parameters, s.t., the weights W ( i ) for each layer i and node j in G 1 are bounded as, i.e., ∀ i, j : (cid:80) k | W ( i ) j,k | ≤ B .
Furthermore, assume that the activation functions used in G 1 belong to the set Ξ , s.t.
all functions σ : R → R satisfy sup x ∈ [ − B · M,B · M ] σ < M .
Then there exists a neural network G 2 with ReLU activation and O ( n ( LB ) l lBM(cid:15) log( ( LB ) l lBM(cid:15) ))
parameters, such that for all sup x ∈ [ − M,M ] d | G 1 ( x ) − G 2 ( x ) | ≤ (cid:15) .
Proof.
For any σ ∈ Ξ , from Theorem 1 in Yarotsky (2017) it follows that there exists a neural network
R with ReLU activations and O ( LBM(cid:15) (cid:48) log( LBM(cid:15) (cid:48) ))
parameters such that sup x ∈ [ − B · M,B · M ] | σ ( x ) −
R ( x ) | ≤ (cid:15) (cid:48) .
We will construct the network G 2 by replacing each activation in G 1 with the corresponding network R as given by the result above with (cid:15) (cid:48) = (cid:15)/l .
Note, this network is at most a factor of
O ( ( LB ) l lBM(cid:15) log( ( LB ) l lBM(cid:15) )) bigger than G 1 , as the lemma requires.
We will prove the claim of the lemma by induction on l .
More precisely, we will show (by induction) that for each node at layer i , the network G 2 calculates a function that is ( LB ) i i(cid:15) (cid:48) away in l ∞ norm from the corresponding node in G 1 , and the inputs to the node are in [ − BM, BM ] .
For the base case i = 1 , since the input x ∈ [ − M, M ] d , the result follows by Theorem 1 in [Yarotsky (2017)].
We proceed to the inductive claim.
Let H ( x ) denote the vector valued mapping computed by the nodes at layer i , and let H R ( x ) be the corresponding vector in G 2 .
As inductive hypothesis, we assume that (cid:107) H ( x ) − H R ( x ) (cid:107) ∞ ≤ ( LB ) i i(cid:15) (cid:48) for all x ∈ [ − M, M ] d and (cid:107) H ( x ) (cid:107) ∞ ≤ M as well as (cid:107) H R ( x ) (cid:107) ∞ ≤ M .
Therefore, for the j th node in layer ( i + 1) in network G 1 we have | W Tj H ( x ) | ≤ (cid:107) W j (cid:107) 1 (cid:107) H ( x ) (cid:107) ∞ ≤ BM and σ is bounded by M on this interval, so we have (cid:107) σ 1 ( W Tj H ( x )) (cid:107) ∞ ≤ M .
Along with the bound on the activations, the part of the inductive hypothesis about the size of the input is proven.
To prove the error bound, we have: |
[Equation]
This ﬁnishes the proof of the inductive step, and thus the lemma.