D EMYSTIFYING OVERCOMPLETE NONLINEAR AUTOENCODERS : FAST SGD CONVERGENCE TOWARDS SPARSE REPRESENTATION FROM RANDOM INITIALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Auto-encoders are commonly used for unsupervised representation learning and for pre-training deeper neural networks.
When its activation function is linear and the encoding dimension ( width of hidden layer) is smaller than the input dimension, it is well known that auto-encoder is optimized to learn the principal components of the data distribution
Oja However
, when the activation is nonlinear and when the width is larger than the input dimension (overcomplete), auto-encoder behaves differently from
PCA and in fact is known to perform well empirically for sparse coding problems
We provide a theoretical explanation for this empirically observed phenomenon, when rectiﬁed-linear unit (ReLu) is adopted as the activation function and the hidden-layer width is set to be large.
In this case, we show that, with signiﬁcant probability, initializing the weight matrix of an auto-encoder by sampling from a spherical Gaussian distribution followed by stochastic gradient descent (SGD) training converges towards the ground-truth representation for a class of sparse dictionary learning models.
In addition, we can show that, conditioning on convergence, the expected convergence rate is O ( 1 t ) , where t is the number of updates.
Our analysis quantiﬁes how increasing hidden layer width helps the training performance when random initialization is used, and how the norm of network weights inﬂuence the speed of SGD convergence.
I NTRODUCTION
Let x denote a vector in R d .
An auto-encoder can be decomposed into two parts, encoder and decoder.
The encoder can be viewed as a composition function s e ◦ a e : R d → R n ; function a e : R d → R n is deﬁned as
[Equation]
W e and b e are the network weights and bias associated with the encoder.
s e is a coordinate-wise activation function deﬁned as
[Equation]
s
[Equation]
y j ) where s : R → R is typically a nonlinear function
The decoder takes the output of encoder and maps it back to R d .
Let x e := s
e ( a e ( x )) .
The decoding function, which we denote as ˆ x , is deﬁned as
[Equation]
where ( W d , b d ) and s d are the network parameters and the activation function associated with the decoder respectively.
Suppose the activation functions are ﬁxed before training.
One can view ˆ x as a reconstruction of the original signal/data using the hidden representation parameterized by ( W e , b e ) and ( W d , b d ) .
The goal of training an auto-encoder is to learn the “right” network parameters, ( W e , b e , W d , b d ) , so that ˆ x has low reconstruction error.
Weight tying A folklore knowledge when training auto-encoders is that, it usually works better if one sets W d = W Te .
This trick is called “weight tying”, which is viewed as a trick of regularization, since it reduces the total number of free parameters.
With tied weights, the classical auto-encoder is simpliﬁed as
[Equation]
In the rest of the manuscript, we focus on weight-tied auto-encoder with the following speciﬁc architecture:
[Equation]
Here we abuse notation to use ˆ x W,b to denote the encoder-decoder function parametrized by weights
W and bias b .
In the deep learning community, s ReLu is commonly referred to as the rectiﬁed-linear (ReLu) activation.
Reconstruction error
A classic measure of reconstruction error used by auto-encoders is the expected squared loss.
Assuming that the data fed to the auto-encoder is i.i.d distributed according to an unknown distribution, i.e., x ∼ p ( x ) , the population expected squared loss is deﬁned as
[Equation]
Learning a “good representation” thus translates to adjusting the parameters ( W, b ) to minimize the squared loss function.
The implicit hope is that the squared loss will provide information about what is a good representation.
In other words, we have a certain level of belief that the squared loss characterizes what kind of network parameters are close to the parameters of the latent distribution p ( x ) .
This unwarranted belief leads to two natural questions that motivated our theoretical investigation:
• Does the global minimum (or any of global minima, if more than one) of L ( W, b ) correspond to the latent model parameters of distribution p ( x ) ?
• From an optimization perspective, since L ( W, b ) is non-convex in W and is shown to have exponentially many local minima Safran & Shamir (2016), one would expect a local algorithm like stochastic gradient descent, which is the go-to algorithm in practice for optimizing L ( W, b ) , to be stuck in local minima and only ﬁnd sub-optimal solutions.
Then how should we explain the practical observation that auto-encoders trained with SGD often yield good representation?
Stochastic-gradient based training Stochastic gradient descent (SGD) is a scalable variant of gradient descent commonly used in deep learning.
At every time step t , the algorithm evaluates a stochastic gradient g ( · ) of the population loss function with respect to the network parameters using back propagation by sampling one or a mini-batch of data points.
The weight and bias update has the following generic form
[Equation]
where η tw and η tb are the learning rates for updating W and b respectively, typically set to be a small number or a decaying function of time t .
The unbiased gradient estimate g ( W t ) and g ( b t ) can be obtained by differentiating the empirical loss function deﬁned on a single or a mini-batch of size m , denoted by (cid:96) m :
[Equation]
i =
[Equation]
, b
[Equation]
x i ) , where (cid:96) (
[Equation] [Equation]
x
[Equation]
Then the stochastic or mini-batch gradient descent update can be written as
( b
[Table]
Max-norm regularization A common trick called “max-norm regularization” Srivastava et al.
(2014) or “weight clipping” is used in training deep neural networks.
1 In particular, after each step of stochastic gradient descent, the updated weights is forced to satisfy
[Table]
for some constant c .
This means the row norm of the weights can never exceed the preﬁxed constant c .
In practice, whenever (cid:107) W i,(cid:63) (cid:107) 2 > c , the max-norm constraint is enforced by projecting the weights back to a ball of radius c .
P RELIMINARIES
In this section, we start by deﬁning notations.
Then we introduce a norm-controlled variant of SGD algorithm that operates on the auto-encoder architecture formalized in (1).
Finally, we introduce assumptions on the data generating model.
General principle of notations We use the same notation for network parameters W, b , and for activation a ( · ) , as in Section 1.
We use s ( · ) as a shorthand for the ReLu activation function s ReLu ( · ) .
We use capital letters, such as W or F , either to denote a matrix or an event; we use lower case letters, such as x , for vectors.
W T denotes the transpose of W .
We use W s,(cid:63) to denote the s -th row of W .
When a matrix W is modiﬁed through time, we let W t denote the state of the matrix at time t , and W ts,(cid:63) for the state of the corresponding row.
We use (cid:107) · (cid:107) for l 2 -norm of vectors and | · | for absolute value of real numbers.
Matrix-vector multiplication between W and x (assuming their dimensions match) is denoted by Wx .
Inner product of vectors x
and y is denoted by (cid:104) x, y (cid:105) .
Organization of notations Throughout the manuscript, we introduce notations that can be divided into “model”, “algorithm”, and “analysis” categories according to their utility.
They are organized in Table 1 to help readers interpreting our results.
For example, If a reader is interested in knowing how to apply our result to parameter tuning in training auto-encoders, then she might ignore the auxiliary notations and only refer to algorithmic parameters and model parameters in Table 1, and examine how does the setting of the former is inﬂuenced by the latter in Theorem 1.
2. N ORM - CONTROLLED SGD TRAINING
We assume that the algorithm has access to i.i.d.
samples from an unknown distribution p ( x ) .
This means the algorithm can access stochastic gradients of the population squared-loss objective in (2) via random samples from p ( x ) .
The norm-controlled SGD variant we analyze is presented in
Algorithm 1 (it can be easily extended to the mini-batch SGD version, where for each update we sample more than one data points).
It is almost the same as what is commonly used in practice: it random initializes the weight matrix by sampling unit spherical Gaussian, and at every step the algorithm moves towards the direction of the negative stochastic gradient with a linearly decaying learning rate.
However, there are two differences between Algorithm 1 and original SGD: ﬁrst, we impose that the norm of the rows of W t be controlled; this is akin to the practical trick of “max-norm regularization” as explained in Section 1; second the update of bias is chosen differently than what is usually done in practice, which deserves additional explanation.
Comment on the setting of bias in Algorithm 1 The stochastic gradient of bias b with respect to squared loss in (2) can be evaluated by sampling a single data point and differentiate against the empirical loss in (3), can be derived as
[Equation]
(derivation can be found in (6) of the Appendix)
Since the gradient is noisy, the generic form of SGD suggests modifying b tj using the update
[Equation]
for a small learning rate η tb to mitigate noise.
This amounts to stepping towards the negative gradient direction and move a little.
On the other hand, we can directly ﬁnd the next update b t +1 j as the point that sets the gradient to zero, that is, we ﬁnd b ∗ j such that
[Equation]
The closed form solution to this is to choose
[Equation]
This strategy, which is essentially Newton’s algorithm, should perform better than gradient descent if we have an accurate estimate of the true gradient, so it would likely beneﬁt from evaluating the gradient using a mini-batch of data.
If, on the other hand, the gradient is very noisy, then this method will likely not work as well as the original SGD update.
Analyzing the evolvement of both
W t and b t , which has dependent stochastic dynamic if we follow the original SGD update, would be a daunting task.
Thus, to simplify our analysis, we assume in our analysis that we have access to
[Equation]
The substitute of W t +1 j(cid:63) for W tj(cid:63) is to further simplify our analysis.
In practice, this update can be implemented by ﬁrst updating W tj(cid:63) to W t +1 j(cid:63) , and then updating b tj using W t +1 j(cid:63) .
2. A SIMPLE SPARSE DICTIONARY LEARNING MODEL
We assume that the data x we sample follows the dictionary learning model
[Equation]
where W ∗ ∈ R k × d .
Here k is the size of the dictionary, which we assume to be at least two
(otherwise, the model becomes degenerate), and the true value of k is unknown to the algorithm .
The rows of W ∗ are the dictionary items; W ∗ j(cid:63) satisﬁes
[Equation]
Let the incoherence between dictionary items be deﬁned as λ := max j,i (cid:54) =
j,i,j ∈ [ k ] |(cid:104) W ∗ j(cid:63) , W ∗ i(cid:63) (
cid:105)| , we assume that λ ≤ 18 k .
In our simpliﬁed model, the coefﬁcient vector s ∈ { 0 , 1 } k is assumed to be 1 -sparse, with
[Equation]
E(cid:15) = 0 and E [ (cid:15)(cid:15) T ] = σ 2 I with σ 2 ≤
[Equation]
Finally, we assume that the noise has bounded norm 2 : max (cid:107) (cid:15) [Equation]
[Equation]
Input: width parameter n ; norm parameter c ; learning rate parameters c (cid:48) , t o , δ ; total number of iterations, t max .
Initialization of W o :
For all s ∈ [ n ] ,
W os(cid:63)
← c z (cid:107)
z (cid:107) , where z ∈ R d , z
i ∼ N (0 , 1)
Initialization of b o : Sample x ∼ p ( x ) ; for all s ∈ [ n ] ,
Find b ∗ s such that ∂(cid:96)
( x ; W o ,(cid:126) 0) ∂b ( b ∗ s )
=
b os ← b ∗ s (version used in analysis: b os ← E x (cid:104) x, W os(cid:63) (cid:105) ( 1 c 2 − 1) ) while t ≤ t max do
W t +1 ← W t − η t ∂(cid:96) ( x,W t ,b t ) ∂W , where x ∼ p ( x )
W t +1 j(cid:63) ← W t +1 j(cid:63) (cid:107) W t +1 j(cid:63) (cid:107)
Draw a fresh sample x (cid:48) ∼ p ; for all s ∈ [ n ] ,
Find b ∗ s such that ∂(cid:96)
( x (cid:48) ; W t ,b t ) ∂b ( b ∗ s ) =
b t +1 s ← b ∗ s , or equivalently, b t +1 s ← (cid:104) x (cid:48)
1 { a t ( x (cid:48) )
> 0 } , W t +1 s(cid:63) (cid:105) ( 1 c 2 − 1) ( version used in analysis b t +1 s ← E x (cid:104) x (cid:48) 1 { a t ( x (cid:48) )
> 0 } , W t +1 s(cid:63) (cid:105) ( 1 c 2 − 1) ) end while
Output: W t max , b t max
While auto-encoders are often related to PCA, the latter cannot reveal any information about the true dictionary under this model even in the complete case, where d = k , due to the isotropic property of the underlying distribution.
The data generating model can be equivalently viewed as a mixture model: for example, when s j = 1 , it means x is of the form W ∗ j(cid:63)
+ (cid:15) .
When (cid:15) is Gaussian, the model coincides with mixture of
Gaussians model, with the dictionary items being the latent locations of individual Gaussians.
Thus, we adopt the concept from mixture models, and use x ∼ C j to indicate that x is generated from the j -th component of the distribution.
M AIN RESULTS
To formally study the convergence property of Algorithm 1, we need a measure to gauge the distance between the learned representation at time t , W t , and the ground-truth representation, W ∗ , which may have different number of rows.
There are potentially different ways to go about this.
The distance measure we use is
[Equation]
Note that ∆( W ts(cid:63) , W ∗ j(cid:63) ) is the squared sine of the angle between the two vectors, which decreases monotonically as their angle decreases, and equals zero if and only if the vectors align.
Thus, min s ∈ [ n ] ∆( W ts(cid:63) , W ∗ j(cid:63) ) can be viewed as the angular distance from the best approximation in the learned hidden representations of the network, to the ground-truth dictionary item W ∗ j(cid:63) .
And Θ( · , · ) measures this distance averaged over all dictionary items.
Our main result provides recovery and speed guarantee of Algorithm 1 under our data model.
Theorem 1.
Suppose we have access to i.i.d.
samples x ∼ p ( x ) , where the distribution p ( x ) satisﬁes our model assumption in Section 2.2.
Fix any δ ∈ (0 , ne ) .
If we train auto-encoder with norm-controlled SGD as described in Algorithm 1, with the following parameter setting
• The row norm of weights set to be (cid:107) W ts(cid:63) (cid:107) =
c ( ∀ s ∈ [ n ] , ∀ t ) such that 32 ≤ c ≤ √ 6 k • If the bias update at t is chosen such that
[Equation]
• The learning rate of SGD is set to be η t := c (cid:48) t + t o , with c (cid:48) >
2 kc and t o ≥ 192( c [Equation]
[Equation]
nδ
[Equation]
Then Algorithm 1 has the following guarantees
• When random initialization with i.i
d. samples from N (0 , 1) is used, the algorithm will be initialized successfully (see deﬁnition of successful initialization in Deﬁnition 1) with probability at least 1 − k exp {− n ( λ √ 2 ) d − 3 } . • When random initialization with i.i.d. samples x ∼ p ( x ) is used, the algorithm will be initialized successfully with probability at least
[Equation] • Conditioning on successful initialization, let Ω denote the sample space of all realizations of the algorithm’s stochastic output, ( W 1 , W 2 , . . . , ) .
Then at any time t , there exists a large subset of the sample space, F t ⊂ Ω , with Pr ( F t ) ≥ 1 − δ , such that
[Equation]
Interpretation The ﬁrst statement of the theorem suggests that the probability of successful initialization increases as the width of hidden layer increases.
In particular, when Gaussian initialization is used, in order to ensure a signiﬁcantly large probability of successful initialization, the analysis suggests that the number of neurons required must scale as Ω( λ − d )
= Ω( k d ) , which is exponential in the ambient dimension.
When the neurons are initialized with samples from the unknown distribution, the analysis suggests that the number of neurons required scale as Ω( kλ 2 ) = Ω( k 3 ) , which is polynomial in the number of dictionary size. Hence, our analysis suggests that, at least under our speciﬁc model, initializing with data is perhaps a better option than Gaussian initialization.
The second statement suggests that conditioning on a successful initialization, the algorithm will have expected convergence towards W ∗ , measured by Θ( · , · ) , of order O ( 1 t ) .
If we examine of form of bound on the convergence rate, we see that the rate will be dominated by the second term, whose constant is heavily inﬂuenced by the choice of learning rate parameter c (cid:48) .
Explaining distributed sparse representation via gradient-based training The main advantage of gradient-based training of auto-encoders, as revealed by our analysis, is that it simultaneously updates all its neurons in parallel, in an independent fashion.
During training, a subset of neurons will specialize at learning a single dictionary item: some of them will be successful while others may fail to converge to a ground-truth representation.
However, since the update of each neuron is independent (in an algorithmic sense), when larger number of neurons are used (widening the hidden layer), it becomes more likely that each ground-truth dictionary will be learned by some neuron, even from random initialization.
R ELATED WORKS
Despite the simplicity of auto-encoders in comparison to other deep architectures, we still have a very limited theoretical understanding of them.
For linear auto-encoders whose width n is less than than its input dimension d , the seminal work of Oja (1982) revealed their connection to online stochastic PCA.
For non-linear auto-encoders, recent work Arpit et al.
(2016) analyzed sufﬁcient conditions on the activation functions and the regularization term (which is added to the loss function) under which the auto-encoder learns a sparse representation.
Another work Rangamani et al.
(2017) showed that under a class of sparse dictionary learning model (which is more general than ours) the ground-truth dictionary is a critical point (that is, either a saddle point or a local miminum) of the squared loss function, when ReLu activation is used.
We are not aware of previous work providing global convergence guarantee of SGD for non-linear auto-encoders, but our analysis techniques are closely related to recent works Balsubramani et al.
; Ge et al.
; Tang &
Monteleoni (2017) that are at the intersection of stochastic (non-convex) optimization and unsupervised learning.
PCA, k -means, and sparse coding The work of Balsubramani et al.
(2013) provided the ﬁrst convergence rate analysis of Oja’s and Krasulina’s update rule for online learning the principal component (stochastic 1-PCA) of a data distribution.
The neural network corresponding to 1-PCA has a single node in the hidden layer without activation function.
We argue that a ReLu activated width n auto-encoder can be viewed as a generalized, multi-modal version of 1-PCA.
This is supported by our analysis: the expected improvement of each neuron, W ts , bears a striking similarity to that obtained in Balsubramani et al.
The training of auto-encoders also has a similar ﬂavor to online/stochastic k -means algorithm Tang & Monteleoni (2017): we may view each neuron as trying to learn a hidden dictionary item, or cluster center in k -means terminology.
However, there is a key difference between k -means and auto-encoders: the performance of k -means is highly sensitive to the number of clusters.
If we specify the number of clusters, which corresponds to the network width n in our notation, to be larger than the true k , then running n -means will over-partition data from each component, and each learned center will not converge to the true component center (because they converge to the mean of the sub-component).
For auto-encoders, however, even when n is much larger than k , the individual neurons can still converge to the true cluster center (dictionary item) thanks to the independent update of neurons.
SGD training of auto-encoders is perhaps closest to a family of sparse coding algorithms Schnass (2015); Arora et al. (2015).
For the latter, however, a critical hyper-parameter to tune is the threshold at which the algorithm decides to cut off insigniﬁcant signals.
Existing guarantees for sparse coding algorithms therefore depend on knowing this threshold.
For ReLu activated auto-encoders, the threshold is adaptively set for each neuron s at every iteration as − b ts via gradient descent.
Thus, they can be viewed as a sparse coding algorithm that self-tunes its threshold parameter.
A NALYSIS
In our analysis, we deﬁne an auxiliary variable
[Equation]
Note that φ ( · , · ) is the squared cosine of the angle between W ts(cid:63) and W ∗ j , which increases as their angle decreases.
Thus, φ can be thought as as measuring the angular “closeness” between two vectors; it is always bounded between zero and one and equals one if and only if the two vectors align.
Our analysis can be divided into three steps.
We ﬁrst deﬁne what kind of initialization enables
SGD to converge quickly to the correct solution, and show that when the number of nodes in the hidden layer is large, random initialization will satisfy this sufﬁcient condition.
Then we derive expected the per-iteration improvement of SGD, conditioning on the algorithm’s iterates staying in a local neighborhood (Deﬁnition 4).
Finally, we use martingale analysis to show that the local neighborhood condition will be satisﬁed with high probability.
Piecing these elements together will lead us to the proof of Theorem 1, which is in the Appendix.
5. P ART I: P ERFORMANCE GUARANTEE OF INITIALIZATION
Covering guarantee from random initialization Intuitively, for each ground-truth dictionary item, we only require that at least one neuron is initialized to be not too far from it.
Deﬁnition 1.
If the rows of W o have ﬁxed norm c > 0 .
Then we deﬁne the event of successful initialization as
[Equation]
Lemma 1 (Random initialization with Gaussian variables) .
Suppose W o ∈ R n × d is constructed by drawing z i,j ∼ N (0 , 1) for all i ∈ [ n ] , j ∈ [ d ] , and setting W o i,(cid:63) = c z i(cid:63) (cid:107) z i(cid:63) (cid:107) . Then [Equation]
Lemma 2 (Random initialization with data points) . Suppose W o ∈ R n × d is constructed by drawing X 1 , . . . , X n from the data distribution p ( x ) , and setting W o i,j = c
X i (cid:107) X
i (cid:107) , for all i ∈ [ n ] .
If σ 2 ≤ λ 2 √ 2 d ,
W o 1 (cid:63)
W o 2 (cid:63)
W o 3 (cid:63)
W o 4 (cid:63)
W o 5 (cid:63) g
W ∗ 1 (cid:63)
W ∗ 2 (cid:63) then
[Equation]
Deﬁnition 2.
Conditioning on F o , we can map the rows of W o to an dictionary item W ∗ j(cid:63) [Equation] j ∈ [ k ] , according to the following ﬁring map
[Equation]
Figure 1 provides an illustration of ﬁring map g ( · ) .
Note that some rows in W o may not be mapped to any dictionary item, in which case we let g ( s ) = 0 .
This means such neurons are not close (in angular distance) to any ground-truth after random initialization.
Also note that for some rows W os(cid:63) , there might exist multiple j ∈ [ k ] such that g ( s ) =
j according to our criterion in the deﬁnition.
But when λ ≤ 12 , which is always the case by our model assumption on incoherence, Lemma 3 shows that the assignment must be unique, in which case the mapping is well deﬁned.
Lemma 3 (Uniqueness of ﬁring) .
Suppose during training, the weight matrix has a ﬁxed norm c .
At time t , for any row of weight matrix W ts(cid:63) , we denote by τ s, 1 := max j (cid:104) W ts(cid:63) c , W ∗ j(cid:63) (cid:105) , and we denote by τ s, 2 := max j ∈
k ] ,j (cid:54) =1 |(cid:104) W ts(cid:63)
c , W ∗ j(cid:63) (cid:105)| .
Then for any λ ≤ 12 , τ s, 1 ≥ √ 1 − λ 2 = ⇒ { τ s, 2 < τ s, 1 } .
Thus, for any s ∈ [ n ] with g ( s ) > 0 , the uniqueness of ﬁring condition holds and the mapping g is deﬁned unambiguously.
So we simplify notations on measure of distance and closeness as
[Equation]
5. P ART II: THE EVOLVEMENT OF WEIGHTS AND BIAS DURING SGD TRAINING
This section lower bounds the expected increase of φ ts after each SGD update, conditioning on F t .
We ﬁrst show that conditioning on F t , the ﬁring of a neuron s with g ( s ) =
j , will indicate that the data indeed comes from the j -th component, which is characterized by event E t .
Deﬁnition 3.
At step t , we denote the event of correct ﬁring of W t as
[Equation]
0 {+} ∩{∀ 
+} ≤ i ≤ t, ∀ s s.t.
g ( s ) > 0 and x ∼ C j , j (cid:54) =
g ( s ) , (cid:104) W is(cid:63) , x (cid:105) + b is < 0 }
Deﬁnition 4.
At step t , we denote the event of satisfying local condition of W t as
[Equation]
Lemma 4 (Correctness of ﬁring) .
If at t ≥ 0 , if ∀ j ∈ [ k ] , the network parameters ( W ts(cid:63) , b ts ) is chosen that satisﬁes
[Equation]
with (cid:107) W ts(cid:63) (cid:107) =
c s.t.
such that 32 ≤ c ≤ √ 6 k .
Then for any t > 0 , F t = ⇒ E t .
Then we proceed to characterize the expected change of φ ts , conditioning on E t .
Theorem 2.
Suppose E t holds, then after one step of stochastic gradient descent update on W t ,
W t +1 satisﬁes version
[Equation]
versionφ t +1 s ≥ φ ts { 1 + 2 η t k (cid:107) W ts(cid:63) (cid:107) (1 − φ ts ) } − η t Z − ( η t ) 2 B with E
[ Z | E t ] = 0 and | Z | ≤ B for some constant B > 0 where B is a constant depending on the model parameter (cid:107) (cid:15) (cid:107) and the norm of rows of weight matrix.
5. P ART III: CONVERGENCE OF MARTINGALES
By Theorem 2, the sequence φ os , φ 1 s , . . .
, φ ts , . .
. is a sub-martingale.
One caveat of is that the expected increase of the cosine of angle between W ts(cid:63) and W ∗ g ( s ) (cid:63) is conditional on E t , the correct ﬁring condition.
So showing that the correct ﬁring event indeed holds is crucial to our overall convergence analysis.
Since by Lemma 4, F t = ⇒ E t , it sufﬁces to show that F t holds.
To this end, note that F t ’s form a nested sequence
[Equation]
We denote the limit of this sequence as
[Equation]
So F ∞ is the event that
[Equation]
Theorem 3 shows that Pr ( F ∞ ) is in fact arbitrarily close to one, conditioning on F o .
We note that there is a line of recent work that analyze the convergence of SGD on non-convex functions
Balsubramani et al.
; Ge et al.
; Tang & Monteleoni (2017), where similar technical difﬁculty arise: to show local improvement of the algorithm on a non-convex functions, one usually needs to lower bound the probability of the algorithm entering a “bad” region, which can be saddle points Ge et al.
; Balsubramani et al.
(2013) or the part of solution space outside of a local neighborhood Tang & Monteleoni (2017).
Some variant of martingale concentration is usually used for obtaining such result.
Here, since event F t can be equivalently interpreted as W ts(cid:63) remains within the local neighborhood of W ∗ g ( s ) (cid:63) deﬁned as { y : (cid:104) y, W ∗ j(cid:63) (cid:105) ≥ c (1 − δ o ) } , we employ a technique similar to that in Tang & Monteleoni (2017) to show that F t holds with high probability for all t .
Theorem 3.
Fix any δ ∈ (0 , n e ) .
Suppose we choose η t = c (cid:48) t + t o such that
[Equation]
Then conditioning on F o , we have
[Equation]
O PEN PROBLEMS
There are several interesting questions that are not addressed here.
First, as noted in our discussion in
Section 2, the update of bias as analyzed in our algorithm is not exactly what is used in original SGD.
It would be interesting (and difﬁcult) to explore whether the algorithm has fast convergence when b t is updated by SGD with a decaying learning rate.
Second, our model assumption is rather strong, and it would be interesting to see whether similar results hold on a relaxed model, for example, where one may relax to 1 -sparse constraint to m -sparse, or one may relax the ﬁnite bound requirement on the noise structure.
Third, our performance guarantee of random initialization depends on a lower bound on the surface area of spherical caps.
Improving this bound can improve the tightness of our initialization guarantee.
Finally, it would be very interesting to examine whether similar result holds for activation functions other than ReLu, such as sigmoid function.
A PPENDIX
Derivation of stochastic gradients Upon receiving a data point x , the stochastic gradient with respect to W is a jacobian matrix whose ( j ∗ , i ∗ ) -th entry reads ∂(cid:96) ( W [Equation] x ) ∂w j
[Equation]
∗ = ∂ ∂w [Equation]
∗ [Equation] ∗ d (cid:88) i = [Equation]
x [Equation] − n (cid:88) j = w ji s ( d (cid:88) l = w jl x l + b j )]=
d (cid:88) i = { x i − n (cid:88)
j = w [Equation]
(cid:88) [Equation]
=
[Equation]
(cid:88)
[Equation]
w
[Equation]
d
[Equation]
=
[Equation]
w
[Equation]
=
[Equation]
d (cid:88) i =
[Equation]
n (cid:88)
[Equation]
[Equation] (cid:88) [Equation] For i (cid:54) = i ∗ , the derivative of the second term can be written using the chain rule as
[Equation]
where we let a j ∗ := w j ∗ l x l + b j ∗ , which is the activation of the j ∗ -th neuron upon receiving x in the hidden layer before going through the ReLu unit.
For i = i ∗ , the derivative of the second term can be written using product rule and chain rule as w j ∗ i ∗
[Equation]
Let r ∈ R
d be the residual vector with r i := x i − (cid:80) nj =1 w ji s ( (cid:80) dl
=1 w jl x l + b j ) .
Then we have ∂(cid:96) ( W, b ; x ) ∂w j ∗ i ∗ = −{ d (cid:88) i = r
[Equation]
In vector notation
[Equation]
the stochastic gradient of loss with respect to the j -th row of W can be written as
[Equation]
Similarly
[Equation]
we can obtain the stochastic gradient with respect to the j -th entry of the bias term as
[Equation]
Now let us examine the terms ∂s ( a j ) ∂a
j and (cid:104) r, W j(cid:63) (cid:105) .
By property of ReLu function,
[Equation]
Mathematically speaking, the derivative of ReLu at zero does not exist.
Here we follow the convention used in practice by setting the derivative of ReLu at zero to be 0 .
In effect, the event { a j = 0 } has zero probability, so what derivative to use at zero does not affect our analysis (as long as the derivative is ﬁnite).
Proof of main theorem .
Consider any time t > 0 .
By Lemma 1 and 2, the probability of successfully initializing the network can be lower bounded by Pr
[Equation]
F o
[Equation]
) ≥
[Equation]
− k exp {− n
[Equation]
λ
[Equation]
[Equation]
d − 3 } if the network is initialized with Gaussian variables or
Pr ( F o ∗ ) ≥ (1 −
[Equation]
[Equation]
[Equation]
[Equation]
100 k 2 } ) if initialized with data
Conditioning on F o ∗ and applying Theorem 3, we get that for all t ≥ 0 ,
[Equation]
Since F t = ⇒ E t by Lemma 4, we can apply version 1 of Theorem 2 to get the expected increase in φ ts for any s such that g ( s ) > 0 as:
[Equation]
Since by F o , ∀ j ∈ [ k ] , there exists s ∈ [ n ] such that g ( s ) =
j .
Let s ( j ) be any s ∈ [ n ] such that g ( s )
=
j .
Then, the inequality above translates to
[Equation]
where
[Equation]
by our choice of c (cid:48) and by our assumption on the initial value ∆ os ( j ) .
Taking total expectation up to time t , conditioning on F t , and letting β denote a lower bound on β t , we get
[Equation]
where the last inequality is by the same argument as that in Lemma 8.
This has the exact same form as in Lemma D.1 of Balsubramani et al.
Applying it with u t := E [∆ ts ( j ) | F t ] , a = β , and
b = ( c (cid:48) ) 2 B (note our t + t o matches their notion of t ), we get
[Equation]
By the upper bound on β t , we can choose β as small as 2 c (cid:48) kc .
So we can get an upper expressed in algorithmic and model parameters as
[Equation]
The second inequality holds because by F o ,
[Equation]
Finally,
[Equation]
where the last inequality is by our requirement that c (cid:48) > 2 kc .
7. P ART I: PERFORMANCE GUARANTEE OF INITIALIZATION
Proof of Lemma 1 .
Let u
= z (cid:107) z (cid:107) , where z ∈ R d with z
i ∼ N (0 , 1) .
We know that u is a random vector on S d − 1 , the d -dimensional unit sphere.
For any ﬁxed vector v ∈ S d − 1 , we have
[Equation]
where S cap ( v, h ) is the surface of the spherical cap centered at v with height h .
By property of spherical Gaussian, we know that u is uniformly distributed on S d − 1 .
So we can directly calculate the probability above as
[Equation]
where µ measures the area of surface.
The latter ratio can be lower bounded (see Lemma 5 in the
Appendix) as a function of d and h :
[Equation]
Since for all row i ∈ [ n ] , their entries are W oi(cid:63) =
c z (cid:107) z (cid:107) , then for any ground-truth dictionary item
W ∗ j(cid:63) , j ∈ [ k ] , we have
[Equation]
By union bound, this implies that
[Equation]
Now by our choice of the form of lower bound on the inner product, we have
[Equation]
substituting this into the function f ( d, h ) , we get a nice form
[Equation]
Substituting this into the previous inequality written in terms of h and letting ρ = λ 2 2 completes the proof. [Figure] θ O
W [Figure] j(cid:63) (cid:107) (cid:15) (cid:107) Lemma 5 (Lower bound on surface area of spherical cap) .
Let S d − 1 denote the d -dimensional hypersphere, and let S cap ( ν, h ) ⊂
S d − 1 denote the surface of a hyper spherical cap centered at ν with height h .
Let µ ( · ) denote measure of area of surface in d -dimensional Euclidean space.
Let f ( h, d ) :=
[Equation]
√ 2 h − h 2 )
d − 1 h 2 d , then
[Equation]
Proof.
Let B d denote the d -dimensional unit ball in Euclidean space, and let B cap ( ν, h ) denote the spherical cap centered at ν with height h .
Then applying Lemma 4.1 of Micciancio & Voulgaris (2010), we get
[Equation]
where V ol ( · ) denotes measure of volume in R d .
So this lower bounds the ratio between volumes between spherical cap and the unit ball.
We show that we can use this to lower bound the ratio between surface areas between spherical cap and the unit ball.
Since by ?
, we know that the ratio between their area can be expressed exactly as
[Equation]
and the ratio between their volume
[Equation]
2) where I x ( a, b ) is the regularized incomplete beta function.
By property of I x ( a, b ) ,
I x ( a + 1 , b ) <
I x ( a, b )
So we have
[Equation]
Proof of Lemma 2.
For any X i ∼ C j , for any j ∈ [ k ] . We ﬁrst claim that [Equation] Proof of claim.
Let us consider the two-dimensional plane H determined by X i and W ∗ j(cid:63) .
Clearly, (cid:15) i = W ∗ j(cid:63)
− X i also lies in H .
Let θ := ∠
( X i , W ∗ j(cid:63) ) denote the angle between X i and W ∗ j(cid:63) .
Note that cos θ = (cid:104) X
i (cid:107) X
i (cid:107) , W ∗ j(cid:63) (cid:105) .
Fix the norm of noise (cid:107) (cid:15) i (cid:107) .
It is clear from elementary geometric insight that θ is maximized (and hence cos θ is minimized) when the line of X i is tangent to the ball centered at W ∗ j(cid:63) with radius (cid:107) (cid:15) i (cid:107) .
We can directly calculate the value of cos θ at this point (see Figure 2) as cos θ = (cid:112) 1 − (cid:107) (cid:15) i (cid:107) 2 , which ﬁnishes the proof of claim.
Now, we denote two events
[Equation]
and
[Equation]
The probability of event A can be lower bounded by concentration inequality for multinomial distribution Devroye (1983): Let n j := (cid:80) i ∈ [ n ] 1 { X
i ∈ C j } .
We get
Pr
[Equation]
[Equation]
where the second inequality is by Lemma 3 of Devroye (1983).
[Equation]
where (cid:107) (cid:15) (cid:107) 2 is the empirical mean of (cid:107) (cid:15) i (cid:107) 2 for all X i , i ∈ [ n ] belonging to the same component C j for some j ∈ [ k ] . Conditioning on A , we know that the average is taken over at least n 2 k samples for each component C j . By one-sided Hoeffding’s inequality, [Equation]
(note, we abuse notation B in the exponent as an upper bound on (cid:107) (cid:15) (cid:107) 2 , as in other parts of the analysis).
Thus,
[Equation]
Proof of Lemma 3. To simplify notation, we use τ 1 ( τ 2 ) as a shorthand for τ s, 1 ( τ s, 2 ). Let j = arg max i ∈
[ k ] (cid:104) W ts(cid:63) , W ∗ i(cid:63) (cid:105) .
Then (cid:104) W ts(cid:63) , W ∗ j(cid:63) (cid:105) =
cτ 1 .
If W ts(cid:63) and W ∗ j(cid:63) align, then |(cid:104) W t s(cid:63) , W ∗ j(cid:63) (cid:105)| [Equation] | [Equation]
[Equation]
|
[Equation]
where the second inequality is by relation of λ and τ s, 1 .
If W ts(cid:63) and W ∗ j(cid:63) do not align, then they determine a two-dimensional plane H .
Let i ∈
[ k ] and i (cid:54) =
j .
We can decompose the unit vector W ∗ i(cid:63) as
[Equation]
[Figure] where proj H ( W ∗ i(cid:63) ) is the orthogonal projection of W ∗ i(cid:63) onto H .
First, note that by orthogonality, [Equation] Since proj H ( W ∗ i(cid:63) ) , W ts(cid:63) [Equation]
and W ∗ j(cid:63) lie on the same plane [Equation] we can easily see via geometric insight (see Figure 3) that,
[Equation]
Therefore,
[Equation]
Suppose τ 21 ≥ 1+ λ 2 .
It can be veriﬁed that this implies
[Equation]
Observing that τ 21 ≥ 1+ λ 2 indeed holds provided λ ≤ 12 and τ 1 ≥ √ 1 − λ 2 completes the proof.
P ART II: THE EVOLVEMENT OF WEIGHTS AND BIAS DURING SGD TRAINING
Proof of Lemma 4 .
Conditioning on F t , we show that E t holds by induction on 0 ≤ i ≤ t .
We start by showing that E o holds.
Base case In this case
[Equation]
Note that by our model assumption,
[Equation]
Since F o holds [Equation] we know that the ﬁring of neuron s is unique.
Let τ s, 1 and τ s, 2 as deﬁned in Lemma 3.
Let x ∼ C j .
Consider the case g ( s ) =
j .
In this case [Equation] Lemma 3 implies that
[Equation]
We will repeatedly use the following relation, as proven by Lemma 6,
[Equation]
|
We have
[Equation]
Now, observe that 1 c 2 − 1 < 0 and
[Equation]
So we get,
[Equation]
Since by our assumption, λ ≤ 18 k , (1 − 1 c 2 )( 1 k + 2 λ ) ≤ (1 − 1 c 2 ) 54 k ≤ 54 k .
Furthermore, since by our assumption, (cid:107) (cid:15) (cid:107) ≤ √ 1 − λ 2 6 k
[Equation]
[Equation] [Equation] [Equation]
[Equation] [Equation]
1 we get W os(cid:63)
[Equation]
os ≥ 0 .
Consider the case g ( s ) =
j (cid:48) , j (cid:48) (cid:54) =
j .
We ﬁrst upper bound b
o in this case.
Since 1 c 2 − 1 < 0 , we would like to lower bound
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
On the other hand,
[Equation]
So
[Equation]
where the last inequality is by assumptions c > 3 2 and (cid:107) (cid:15) (cid:107) ≤ √ 1 − λ 2 6 k .
Case 0 < i ≤ t Suppose E i holds, we show that E i +1 holds for i ≤ t − 1 .
Let x ∼ C j for any j ∈ [ k ] .
Since E i holds, we know that
[Equation]
[Equation] So for neurons W i +1 s(cid:63) with g ( s ) > 0 and g ( s ) (cid:54) =
j , we know that these neurons are not updated, that is,
[Equation]
is(cid:63) and b i
+
[Equation] = [Equation] This means the two conditions in E i +1 on neurons g ( s ) =
j (cid:48) (cid:54) =
j .
For neurons W i +1 s(cid:63) with g ( s ) =
j , b i +1 s is set such that [Equation] −
[Equation]
Consider the case x (cid:48) ∼ C j , we have
[Equation]
where the last inequality is by assumption τ 2 s, 1 ≥ 1 − λ 2 .
It follows that ( W i +1 s(cid:63) )
T x (cid:48)
+ b
i +1 s >holds by our assumptions that c ≤ √ 6 k , and that (cid:107) (cid:15) (cid:107) ≤ √ 1 − λ 2 6 k .
Finally, consider the case x (cid:48) ∼ C j (cid:48) for j (cid:48) (cid:54) =
j, j (cid:48) ∈
[ k ] .
Again, since F i +1 holds,
[Equation]
we can apply Lemma 3 to get
[Equation]
where the last inequality holds similarly as in the base case.
Lemma 6.
Let τ s, 1 , τ s, 2 be as deﬁned in Lemma 3, and let λ be the incoherence parameter.
If τ 2 s, 1 ≥ 1 − λ 2 , then τ s, 2 ≤ cλτ s, 1 .
Proof.
Using the same argument as the proof of Lemma 3, we get
[Equation]
where the last inequality is by the relation between τ 1 and λ .
Proof of Theorem 2 .
We start by proving version 1.
The proof of version 2 follows directly.
Let x ∼ C j for any j ∈ [ k ] .
We consider and any neuron s s.t.
g ( s ) =
j .
Again, since E t holds,
[Equation]
We now examine E [ φ t +1 s | x ∼ C j , g ( s )
=
j, E t ] .
To ease notation
[Equation]
we let w := W ts(cid:63) , w ∗ := W ∗ j(cid:63) , η :=
η tw , r :=
r t , a := a ts
[Equation]
and b :=
b ts in the proof.
[Equation]
The denominator reads
[Equation]
[-)( x T r ) + a 2 (cid:107) r (cid:107) 2 ] = (cid:107) w (cid:107) 2 (1 + (cid:107) w (cid:107) 2 
 η [( r T w )( x T w ) + a ( r T w )] + (cid:107) w (cid:107) 2 η 2 [( r T w ) 2 (cid:107) x (cid:107) 2 + 2 a ( r T w )( x T r ) + a 2 (cid:107)-] r (cid:107) 2 ])
Let A ( w ) :
= ( r T w )( x T w ) + a ( r T w ) and B ( w ) :=
( r T w ) 2 (cid:107) x (cid:107) 2 + 2 a ( r T w )( x T r ) + a 2 (cid:107) r (cid:107) 2 .
(cid:107) w + η [ (cid:104) r, w (cid:105) x
+ ar ] (cid:107)
= (cid:107) w (cid:107) 2 (1 + 1 (cid:107) w (cid:107) 2 2 ηA ( w ) + 1 (cid:107) w (cid:107)
η 2 B ( w ))
= 1 − ( 1 (cid:107) w (cid:107) 2 2 ηA ( w ) + 1 (cid:107) w (cid:107)
η 2 B ( w ))
(cid:107) w (cid:107) 2 { 1 − ( 1 (cid:107) w (cid:107) 2 2 ηA ( w ) + 1 (cid:107) w (cid:107)
η 2 B ( w )) 2 } ≥ 1 − ( 1 (cid:107) w (cid:107) 2 2 ηA ( w ) + 1 (cid:107) w (cid:107) 2 η 2 B ( w )) (cid:107) w (cid:107)
Let C ( w ∗ ) :
=
( r T w )( x T w ∗ )
+ a ( r T w ∗ ) .
The numerator reads ( w T w ∗ + η [( r T w )( x T w ∗ )
+ a ( r T w ∗ )]) 2 = ( w T w ∗ ) 2 + 2 η ( w T w ∗ ) C ( w ∗ ) + η 2 C ( w ∗ )
Putting these together,
[Equation]
Bounding major direction of improvement Now let us focus on line (7), which can be viewed as the major direction of change guided by the stochastic gradient, while the rest terms can be viewed as noise.
It equals
[Equation]
Note that both terms C ( w ∗ ) and A ( w ) are stochastic, since their value is determined by data we sampled from the model distribution.
We continue with the key step of our analysis: bounding
EC ( w ∗ ) and EA ( w ) .
Since
[Equation]
and
[Equation]
So
[Equation]
Since b is chosen such that b = ( w T w ∗ )( 1 (cid:107) w (cid:107) 2 − 1) conditioning on E t , after some calculation we get
[Equation]
On the other hand, note that
[Equation]
where the second inequality is by the condition that x is drawn from the j -th component
[Equation]
and fourth inequality is due to our model assumption on the covariance structure of (cid:15) .
Similarly,
[Equation]
Therefore, we can calculate the other part as
[Equation]
Combining, we get
[Equation]
Thus,
[Equation]
Now [Equation] we turn to the term A ( w ) , which equals
[Equation]
So
[Equation]
This implies that
[Equation]
Finally, combining the derivation for expected values relevant to C ( w ∗ ) and A ( w ) , we get E
[Equation]
So the expectation of line (7) is
[Equation]
Also, note that since the terms w, w ∗ , x are all bounded, the noise term must be bounded by some constant B 1 .
Therefore, we can get a lower bound on increase of conditional expectation
[Equation]
Since Pr ( x ∼ C j ) = 1 k , we get
[Equation]
where the last inequality holds because a ts ( x ) ≤ 0 for x ∼ C j (cid:48) and g ( s )
=
j ( j (cid:54) =
j (cid:48) ) by E t , and for neuron s such that a ts ( x ) ≤ 0 , the gradient evaluates to zero so those entries are not updated after t .
Thus, since E t holds, we know that
[Equation]
x ∼ C j version 2 Let Y
[Equation]
Then, from the proof of version 1, we know that
[Equation]
t +1 s
[Equation]
So letting Z :
=
Y − E
[ Y | E t ] , we know that E [ Z | E t ] =
Again, since the terms w, w ∗ , x are all bounded, there must exists B 2 ≥ | Z | .
Letting B := max { B 1 , B 2 } ﬁnishes the proof.
7. P ART III: CONVERGENCE OF MARTINGALES
Proof outline of Theorem 3 .
Conditioning on F o , note that F ∞ can be written as the union of events
[Equation]
We denote these individual events regarding neuron s by F ∞ s , that is,
[Equation]
The proof strategy is to show that the individual probability can be lower bounded by δn , and taking the union bound over all neurons.
Since ( F ∞ s ) c can be partitioned into a union of disjoint subsets ∪ ∞ t =
( F ts \ F t +1 s ) , where
[Equation]
We can deﬁne E ts similarly [Equation] and note we can easily show that
[Equation]
by going through Lemma 4 exactly the same.
To lower bound Pr ( F ∞ s ) , we upper bound the individual error probability Pr ( F ts \ F t +1 s ) , and show that their summation vanishes for t → ∞ .
The same approach is taken in Proposition 2 Tang & Monteleoni (2017) [Equation] and our main idea is to adapt the analysis there to our case.
Thanks to the special form of inequality obtained in Theorem 2, we can neatly re-write the statement of its version 2 in terms of ∆ ts as
[Equation]
Essentially, the sufﬁcient condition for the analysis of Proposition 2 Tang & Monteleoni (2017) to work are
• The expected decrease on the objective function ∆ ts is of the form
[Equation]
• β t ≥ 2 , ∀ t ≥• The noise terms Z, B are bounded, and E [ Z | F t ] = 0 .
By our choice of η t , the relation in (8) satisﬁes the special form in (9).
Since conditioning on F t , we have
[Equation]
and since conditioning on F o , φ os ≥
[Equation]
+ [Equation] for g ( s ) > 0 and our choice of δ o in F o .
Since we also set
[Equation]
we get that β t ≥ 2 always hold.
On the other hand, here the noise terms are obviously bounded by our model assumption.
Therefore, we only need to slightly adapt Proposition 2 Tang & Monteleoni (2017) for our purpose.
The exact proof is included for completeness.
Complete proof of Theorem 3 .
We consider each F ts individually.
Since the proof for each s is exactly the same, we abuse the notation F t to let it denote F ts for any ﬁxed s .
Similarly, we let ∆ t denote ∆ t s .
Conditioning on F t , we know that ∃ β > 2 s.t. β ≤ β t .
By Lemma 7, for any λ > 0 , and any 0 ≤ i ≤ t − 1 , we have
[Equation]
[Equation]
where λ (1) = λ (1 − β t o + i ) , and the second inequality is by Lemma 8.
For k ≥ 1 , we deﬁne [Equation]
and
[Equation]
We can similarly get, for k = 0 , . .
, i ,
[Equation]
Since ∀ β > 0 , k ≥ 1 ,
[Equation]
Since the bound is shrinking as β increases and β ≥ 2 , λ
[Equation]
Recursively applying the relation until we get to the term
[Equation]
Combining all these recursive inequalities with the bound on λ ( k ) , we get E
[Equation]
e λ ∆ i [Equation] i
[Equation]
Let τ o :=
λ 2 .
Then we can apply the conditional Markov’s inequality, for any λ i > 0 ,
[Equation]
Since event F o = { φ o ≥ 1 − λ 2 2 } implies ∆ o = 1 − φ o ≤ λ 2
= τ o 2 , we have
[Equation]
Combining this with the upper bound on E [ e λ i ∆ i +1 | F i ] , we get
[Equation]
We choose λ i
= 1∆ ln
( i +1) 2 δ with ∆ =
τ o 4 , and show that τ o 2 − ( B
+ λ i B 2 2 ) 4( c (cid:48) )
2 i ( t o + i ) 2 is lower bounded by ∆ .
Case 1: B > λ i B 2 2 .
Since t o ≥ 32( c (cid:48) ) 2 B τ o , we get
[Equation]
Case 2: B ≤ λ i B 2 2 .
We get
[Equation]
Now we show
[Equation]
Since
[Equation]
ln 1 δ ≥ 1 , and obviously, 16( c (cid:48)
[Equation]
B
[Equation]
≥ 13 , we can apply Lemma ??
with b = 2 , C :=[-
-]
[Equation]
τ [Equation] ) 2 , t
[Equation]
= t o + i ≥
[Equation]
3 C b − 1 ln 1 δ ) 2 b − 1 , and get
[Equation]
Or equivalently, 1∆ ln ( t o + i ) 2 δ
[Equation]
≤ ∆ .
Thus, for both cases,
[Equation]
This implies
[Equation]
Finally, we have
[Equation]
Now recall that this holds for each s , that is, ∀ s ∈ [ n ] ,
[Equation]
substituting δ s =
δk for δ in the proof above, and taking the union bound completes the proof.
Lemma 7 (Inequality of moment generating function) .
Suppose the conditions of Theorem 2 hold.
Then conditioning on E t ( F t ), we can upper bound the moment generating function of ∆ t +1 as
[Equation]
Proof.
We apply the result of Theorem 2.
Rewriting version 2 of Theorem 2 using ∆ t := 1 − φ t s , for any s ∈ [ n ] , we get
[Equation]
We let β t := 2 c (cid:48) (1 − ∆ t ) k (cid:107) W ts(cid:63) (cid:107) .
Conditioning on F t , the moment generating function of the ∆ t +1 is upper bounded by
[Equation]
By Theorem 2,
[Equation]
Applying Hoeffding’s lemma for bounded random variable
[Equation]
we can bound its m
[Equation]
g.f. by
[Equation]
So, ﬁnally
[Equation]
Lemma 8 (Lemma from Tang & Monteleoni (2017)) .
For any λ > 0 ,
[Equation]
[Equation]
λ
[Equation]
7. T ECHNICAL LEMMAS
Lemma 9 (Tang & Monteleoni (2017)) .
For any ﬁxed b ∈ (1 , 2] .
If C ≥ b − 13 , δ ≤ 1 e , and t ≥ ( 3 C b − 1 ln 1 δ ) 2 b − 1 , then t b − 1 − 2 C ln t − C ln 1 δ > 0 .
Proof.
Let f ( t ) :
= t b − 1 − 2 C ln t − C ln 1 δ .
Taking derivative, we get f (cid:48) ( t ) =
( b − 1) t b − 2 − 2 Ct ≥when t ≥ ( 2 C b − 1 ) 1 b − 1 .
Since ln 1 δ 3 C b − 1 ≥ 3 C b − 1 ≥ 1 , (ln 1 δ 3 C b − 1 ) 2 b − 1 ≥ ( 2 C b − 1 ) 1 b − 1 , it sufﬁces to show f ((ln 1 δ 3 C b − 1 ) 2 b − 1 ) > 0 for our statement to hold.
f ((ln 1 δ 3 C b − 1 ) 2 b − 1 ) =
(ln 1 δ 3 C b − 1 ) 2 − 2 C ln { (ln 1 δ 3 C b − 1 ) 2 b − 1 } − C ln 1 δ
=
(ln 1 δ ) 2 9 C 2 ( b − 1) 2 − 4 C b − 1 ln(ln 1 δ 3 C b − 1 ) − C ln
δ = 4 C b − 1 [ 32 C b − 1 ln 1 δ − ln( 3 C b − 1 ln 1 δ )]
+ C ln 1 δ [ 3 C ( b − 1) 2 − 1] > 0 , where the ﬁrst term is greater than zero because x − ln(2 x ) > 0 for x > 0 , and the second term is greater than zero by our assumption on C .
Lemma 10.
Suppose our model assumptions on parameters (cid:15), c, α hold, and that our assumptions on the algorithmic parameter c in Theorem 1 holds, then
[Equation]
Proof.
[Equation]
where the last term is greater than zero because
[Equation]
So the term
[Equation]