I MPROVED L EARNING OF O NE - HIDDEN - LAYER C ONVOLUTIONAL N EURAL N ETWORKS WITH O VERLAPS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a new algorithm to learn a one-hidden-layer convolutional neural network where both the convolutional weights and the outputs weights are parameters to be learned.
Our algorithm works for a general class of (potentially overlapping) patches, including commonly used structures for computer vision tasks.
Our algorithm draws ideas from (1) isotonic regression for learning neural networks and (2) landscape analysis of non-convex matrix factorization problems.
We believe these ﬁndings may inspire further development in designing provable algorithms for learning neural networks and other complex models.
While our focus is theoretical, we also present experiments that illustrate our theoretical ﬁndings
I NTRODUCTION
Giving provably efﬁcient algorithms for learning neural networks is a core challenge in machine learning theory.
The case of convolutional architectures has recently attracted much interest due to their many practical applications.
Recently Brutzkus & Globerson (2017) showed that distributionfree learning of one simple non-overlapping convolutional ﬁlter is NP-hard.
A natural open question is whether we can design provably efﬁcient algorithms to learn convolutional neural networks under mild assumptions.
We consider a convolutional neural network of the form
[Equation]
where w P R r is a shared convolutional ﬁlter, a P R k is the second linear layer and
[Equation]
selects the pp j ´ 1 q s ` 1 q -th to pp j ´ 1 q s ` r q -th coordinates of x with stride s and σ p¨q is the activation function.
Note here that both w and a are unknown vectors to be learned and there may be overlapping patches because the stride size s may be smaller than the ﬁlter size r .
Our Contributions
We give the ﬁrst efﬁcient algorithm that can provably learn a convolutional neural network with two unknown layers with commonly used overlapping patches.
Our main result is the following theorem.
Theorem 1.1 (Main Theorem (Informal)) .
Suppose s ě t r 2 u ` 1 and the marginal distribution is symmetric and isotropic.
Then the convolutional neural network deﬁned in equation 1 with piecewise linear activation functions is learnable in polynomial time.
We refer readers to Theorem 3.1 for the precise statement.
Technical Insights Our algorithm is a novel combination of the algorithm for isotonic regression and the landscape analysis of non-convex problems.
First, inspired by recent work on isotonic regression, we extend the idea in Goel et al.
(2018) to reduce learning a CNN with piecewise linear activation to learning a convolutional neural network with linear activation (c.f. Section 4).
Second, we show learning a linear convolutional ﬁlter can be reduced to a non-convex matrix factorization problem which admits a provably efﬁcient algorithm based on non-convex geometry (Ge et al., 2017a).
Third, in analyzing our algorithm, we present a robust analysis of Convotron algorithm proposed by Goel et al.
(2018), in which we draw connections to the spectral properties of Toeplitz matrices.
We believe these ideas may inspire further development in designing provable learning algorithms for neural networks and other complex models.
Related Work From the point of view of learning theory, it is well known that training is computational infeasible in the worst case (Goel et al., 2016; Brutzkus & Globerson, 2017).
Thus distributional assumptions are needed for efﬁcient learning.
A line of research has focused on analyzing the dynamics of gradient descent conditioned on the input distribution being standard Gaussian (Tian, 2017; Soltanolkotabi, 2017; Li & Yuan, 2017; Zhong et al., 2017b; Brutzkus & Globerson, 2017;
Zhong et al., 2017a; Du et al., 2017).
Speciﬁcally for convolutional nets, existing analyses heavily relied on the analytical formulas which can only be derived if the input is Gaussian and patches are non-overlapping.
Recent work has tried to relax the Gaussian input assumption and the non-overlapping structure for learning convolutional ﬁlters.
Du et al.
(2017) showed if the patches are sufﬁciently close to each other then stochastic gradient descent can recover the true ﬁlter.
Goel et al.
(2018) proposed a modiﬁed iterative algorithm inspired from isotonic regression that gives the ﬁrst recovery guarantees for learning a ﬁlter for commonly used overlapping patches under much weaker assumptions on the distribution.
However, these two analyses only work for learning one unknown convoutional ﬁlter.
Moving away from gradient descent, various works have shown positive results for learning general simple fully connected neural networks in polynomial time and sample complexity under certain assumptions using techniques such as kernel methods (Goel et al., 2016; Zhang et al., 2015; Goel &
Klivans, 2017a;b) and tensor decomposition (Sedghi & Anandkumar, 2014; Janzamin et al., 2015).
The main drawbacks include the shift to improper learning for kernel methods and the knowledge of the probability density function for tensor methods.
In contrast to this, our algorithm is proper and does not assume that the input distribution is known.
Learning a neural network is often formulated as a non-convex problem.
If the objective function satisﬁes (1) all saddle points and local maxima are strict (i.e., there exists a direction with negative curvature), and (2) all local minima are global (no spurious local minmum), then noise-injected (stochastic) gradient descent (Ge et al., 2015; Jin et al., 2017) ﬁnds a global minimum in polynomial time.
Recent work has studied these properties for the landscape of neural networks (Kawaguchi, 2016; Choromanska et al., 2015; Hardt & Ma, 2016; Haeffele & Vidal, 2015; Mei et al., 2016;
Freeman & Bruna, 2016; Safran & Shamir, 2016; Zhou & Feng, 2017; Nguyen & Hein, 2017a;b;
Ge et al., 2017b; Zhou & Feng, 2017; Safran & Shamir, 2017; Du & Lee, 2018).
A crucial step in our algorithm is reducing the convolutional neural network learning problem to matrix factorization and using the geometric properties of matrix factorization.
P RELIMINARIES
We use bold-faced letters for vectors and matrices.
We use }¨} 2 to denote the Euclidean norm of a ﬁnite-dimensional vector.
For a matrix A , we use λ max p
A q to denote its eigenvalue and λ min p
A q its smallest singular value.
Let O p¨q and Ω p¨q denote standard Big-O and Big-Omega notations, only hiding absolute constants.
In our setting, we have n data points t x i , y i u ni “ 1 where x i P R d and y P R .
We assume the label is generated by a two-layer convolutional neural network with ﬁlter size r , stride s and k hidden neurons.
Compactly we can write the formula in the following form: y i “ f p x i , w ˚ , a ˚ q , x i „ Z where the prediction function f is deﬁned in equation 1.
To obtain a proper scaling, we let } w ˚ } 2 } a ˚ } 2 “ σ 1 .
We also deﬁne the induced patch matrix as
[Equation]
which will play an important role in our algorithm design.
Our goal is to properly learn this convolutional neural network, i.e., design a polynomial time algorithm which outputs a pair p w , a q that satisﬁes E x „ Z
[Equation]
ı
[Equation]
M AIN R ESULT
In this section we describe our main result.
We ﬁrst list our main assumptions, followed by the detailed description of our algorithm.
Lastly we state the main theorem which gives the convergence guarantees of our algorithm.
3. A SSUMPTIONS
Our ﬁrst assumption is on the input distribution Z .
We assume the input distribution is symmetric, bounded and has identity covariance.
The symmetry assumption is used in Goel et al.
(2018) and many learning theory papers Baum (1990).
The identity covariance assumption is true if the data whitened.
Further, in many architectures, the input of certain layers is assumed having these properties because of the use of batch normalization (Ioffe & Szegedy, 2015) or other techniques.
Lastly, the boundedness is a standard regularity assumption to exclude pathological input distributions.
We remark that this assumption considerably weaker than the standard Gaussian input distribution assumption used in Tian (2017); Zhong et al.
(2017a)
; Du et al.
(2017), which has the rotational invariant property.
Assumption 3.1 (Input Distribution Assumptions) .
We assume the input distribution satisﬁes the following conditions.
• Symmetry: P p x q “ P p´ x q . • Identity covariance: E x „ Z “ xx T [Equation]
“ I . • Boundedness: @ x „ Z , } x } 2 ď B almost surely for some B ą 0 .
Our second assumption is on the patch structure.
In this paper we assume the stride is larger than half of the ﬁlter size.
This is indeed true for a wide range of convolutional neural network used in computer vision.
For example some architecture has convolutional ﬁlter of size 3 and stride 2 and some use non-overlapping architectures (He et al., 2016).
Assumption 3.2 (Large Stride) .
s ě t r 2 u ` 1 .
Next we assume the activation function is piecewise linear.
Commonly used activation functions like rectiﬁed linear unit (ReLU), Leaky ReLU and linear activation all belong to this class.
Assumption 3.3 (Piece-wise Linear Activation) .
[Equation]
[Equation]
A LGORITHM
Now we are ready to describe our algorithm (see Algorithm 1).
The algorithm has three stages, ﬁrst we learn the outer layer weights upto sign, second we use these ﬁxed outer weights to recover the ﬁlter weight and last we choose the best weight combination thus recovered.
Stage 1: Learning the Non-overlapping Part of the Convolutional Filter and Linear Weights
Our ﬁrst observation is even if there may be overlapping patches, as long as there exists some nonoverlapping part, we can learn this part and the second layer jointly.
To be speciﬁc, with ﬁlter size being r and stride being s , if s ě t r 2 u ` 1 , for j “ 1 . . .
, k we deﬁne the selection matrix for the non-overlapping part of each patch
[Equation]
Note that for any j 1 ‰ j 2 , there is no overlapping between the selected coordinates by P nonjand
P nonj 2 .
Therefore, for a ﬁlter w , there is a segment r w r ´ s ` 1 , . . .
, w s s with length p 2 s ´ r q which acts on the non-overlapping part of each patches.
We denote w non “ r w r ´ s ` 1 , . . .
, w s s and our goal in this stage is to learn w ˚ non and a ˚ jointly.
In this stage, our algorithm proceeds as follows.
Given w non , a and a sample p x , y q , we deﬁne g p w non , a , x , y q “
[Equation]
w
[Equation]
a
[Equation]
x
[Equation]
y
[Equation]
where ˆ f p w non , a , x q “ ř kj “ 1 a j σ ` w J non P nonj x ˘ is the prediction function only using w non .
As will be apparent in Section 4, g and h are unbiased estimates of the gradient for the loss function of learning a linear CNN.
The term
[Equation]
and
[Equation]
¯ a are is the gradient induced by the regularization´ } w non
[Equation] [Equation]
a
[Equation]
¯, which is used to balance the magnitude between w non and a and make the algorithm more stable.
With some initialization w p 0 q non and a p 0 q , we use the following iterative updates inspired by isotonic regression (Goel et al., 2018), for t “ 0 , . .
, T 1 ´
[Equation] [Equation]
where η 1 ą 0 is the step size parameter, ξ p t q w non and ξ p t q a are uniformly sampled a unit sphere and at iteration we use a fresh sample ` x p t q , y p t q ˘ .
Here we add isotropic noise ξ p t q w non and ξ p t q a because the objective function for learning a linear CNN is non-convex and there may exist saddle points.
Adding noise can help escape from these saddle points.
We refer readers to Ge et al.
(2015) for more technical details regarding this.
As will be apparent in Section 4, after sufﬁcient iterations, we obtain a pair ` w p T 1 q , a p T 1 q ˘ such that either it is close to the truth p w ˚ non , a ˚ q or close to the negative of the truth p´ w ˚ non , ´ a ˚ q .
Remark 3.1 (Non-overlapping Patches) .
If there is no overlap between patches, we can skip Stage 2 because after Stage 1 we have already learned a and w non “ w .
Stage 2: Convotron with ﬁxed Linear Layer
In Stage 1 we have learned a good approximation to the second layer (either a p T 1 q
or ´ a p T 1 q ).
Therefore, the problem reduces to learning a convolutional ﬁlter.
We run Convotron (Algorithm 3) proposed in Goel et al.
(2018) using a p T 1 q and ´ a p T 1 q to obtain corresponding weight vectors w p`q and w p´q .
We show that the Convotron analysis can be extended to handle approximately known outer layer weight vectors.
Stage 3: Validation In stage 2 we have obtained two possible solutions ` w p`q , a p T q ˘ and ` w p´q , ´ a p T q ˘ .
We know at least one of them is close to the ground truth.
Closeness in ground truth implies small squared loss (c.f. Lemma A.1).
In the last stage we use a validation set to choose
Algorithm
Initialize w p 0 q non P R 2 s ´ r and a p 0 q P R k randomly for t “ 1 to T do
Draw p x p t q , y p t q q „ Z
Compute g ` w non , a , x p t q , y p t q ˘ and h ` w non , a , x p t q , y p t q ˘ according to equation 2 and equation 3.
Set w p
[Equation]
Set a p t ` 1 q “ a p t q ´ η 1 h ´ w p t q non , a p t q , x p t q , y p t q ¯ ` η 1 ξ p t q a
Return a p T ` 1 q
Initialize w 1 : “ 0 P R r .
for t “ 1 to T do
Draw p x p t q , y p t q q „ Z
Let g p t q “ p y p t q ´ f p w p t q , a , x p t
q
qq
´
ř k i “ 1 a i P i x
p t q ¯
Set w p t ` 1 q “ w p t q ` η g p t q
Return w T `the right one.
To do this, we simply use T 3 “ poly ` k, B, 1 (cid:15) ˘ fresh samples and output the solution which gives lower squared error.
[Equation]
Since we draw many samples, the empirical estimates will be close to the true loss using standard concentration bounds and choosing the minimum will give us the correct solution.
3. M AIN T HEOREM
The following theorem shows that Algorithm 1 is guaranteed to learn the target convolutinoal neural network in polynomial time.
To our knowledge, this is the ﬁrst polynomial time proper learning algorithm for convolutional neural network with overlapping patches.
Theorem 3.1 (Theorem 1.1 (Formal)) .
Under Assumptions 3.1-3.3, if we set T 1 , T 2 , T 3 “ Ω ` poly ` k, B, 1 (cid:15) ˘˘ and η 1 , η 2 “ O ` poly `k , 1 B , (cid:15) ˘˘ then with high probability, Algorithm 1 returns a pair p w , a q which satisﬁes
[Equation]
P ROOFS AND T ECHNICAL I NSIGHTS
In this section we list our key ideas used for designing the Algorithm 1 and proving its correctness.
We discuss the analysis stage-wise for ease of understnading.
4. A NALYSIS OF S TAGE
Learning a non-overlapping CNN with linear activation.
We ﬁrst consider the problem of learning a convolutional neural network with linear activation function and non-overlapping patches.
For this setting, we can write the prediction function in a compact form:
[Equation]
The label also admits this form y “ x P p x q , w ˚ p a ˚ q J y .
A natural way to learn w ˚ and a ˚ is to consider solving a square loss minimization problem:
[Equation]
Now, taking expectation with respect to x , we have
[Equation]
where the last step we used our assumptions that patches are non-overlapping and the covariance of x is the identity.
From equation 7, it is now apparent that the population L 2 loss is just the standard loss for rank- 1 matrix factorization problem.
Recent advances in non-convex optimization shows the following regularized loss function
[Equation]
satisﬁes all local minima are global and all saddles points and local maxima has a negative curvature Ge et al.
(2017a)
and thus allows simple local search algorithm to ﬁnd a global minimum.
Though the objective function in equation 8 is a population risk, we can obtain its stochastic gradient by our samples if we use fresh sample at each iteration.
We deﬁne
[Equation]
where ` x p t q , y p t q ˘ is the sample we use in the t -th iteration.
In expectation this is the standard gradient descent algorithm for solving equation 8:
[Equation]
With this stochastic gradient oracle at hand, we can implement the noise-injected stochastic gradient descent proposed in Ge et al.
[Equation]
where ξ p t q w and ξ p t q a are sampled from a unit sphere.
Theorem 6 in Ge et al.
(2015) implies after polynomial iterations, this iterative procedure returns an (cid:15) -optimal solution of the objective function equation 8 with high probability.
Learning non-overlapping part of a CNN with piece-wise linear activation function Now we consider piece-wise linear activation function.
Our main observation is that we can still obtain a stochastic gradient oracle for the linear convolutional neural network using equation 2 and equation 3.
Formally, we have the following theorem.
Lemma 4.1 (Properties of Stochastic Gradient for Linear CNN) .
Deﬁne
[Equation]
Under Assumption 3.1, we have
[Equation]
where g p w non , a , x , y q and h p w non , a , x , y q are deﬁned in equation 2 and equation 3, respectively.
Further, if } w non } 2 “ O p poly p σ 1 qq , } a } 2 “ O p poly p σ 1 qq , then the differences are also bounded ››› ›
[Equation]
[Equation]
for some D “ O p poly p B, k, σ 1 qq .
Here the expectation of g and h are equal to the gradient of the objective function for linear CNN because we assume the input distribution is symmetric and the activation function is piece-wise linear.
This observation has been stated in Goel et al.
and based on this property, Goel et al.
(2018) proposed Convotron algorithm
(Algorithm 3), which we use in our stage 2.
Lemma 4.1 is a natural extension of Lemma 2 of Goel et al.
(2018) that we show even for one-hidden-layer CNN, we can still obtain an unbiased estimate of the gradient descent for linear CNN.
Now with Lemma 4.1 at hand, we can use the theory from non-convex matrix factorization.
Ge et al.
(2017a) has shown if η 1 “ O ` poly `k , 1 B , (cid:15) ˘˘ then for all iterates, with high probability, ››› w p t q non ››› 2 “ O p poly p σ 1 qq , } a t } 2 “ O p poly p σ 1 qq .
Therefore, we can apply the algorithmic result in
Ge et al.
and obtain the following convergence theorem.
Theorem 4.1 (Convergence of Stage 1) .
If w p 0 q “ O ` ? σ 1 ˘ , a p 0 q “ O ` ?
σ 1 ˘ , and η 1 “
O ` poly `k , 1 B , (cid:15) ˘˘ then after T 1 “ O ` poly ` r, k, B, 1 (cid:15) ˘˘ we have ››››› a p T 1 q ›› a p T 1 q ›› 2 ´ a ˚ } a ˚ }››››› ď (cid:15).
or ››››› a p T 1 q ›› a p T 1 q ›› 2 ` a ˚ } a ˚ }››››› ď (cid:15).
4. A NALYSIS OF S TAGE
After Stage 1, we have approximately recovered the outer layer weights.
We use these as ﬁxed weights and run Convotron to obtain the ﬁlter weights.
The analysis of Convotron inherently handles average pooling as the outer layer.
Here we extend the analysis of Convotron to handle any ﬁxed outer layer weights and also handle noise in these outer layer weights.
Formally, we obtain the following theorem:
Theorem 4.2.
(Learning the Convolutional Filter)
Suppose } a
´ a ˚ } 2 ď (cid:15) for (cid:15) ď 1 k 3 } w ˚ } 2 and without loss of generality 1 let } a } 2 “ } a ˚ } 2 “ 1 .
For suitably chosen η “ O ` poly `k , 1 B ˘˘ ,
Convotron (modiﬁed) returns w such that with a constant probability, } w ´ w ˚ } 2 ď
O p k 3 } w ˚ } (cid:15) q in poly p k, } w ˚ } , B, log p 1 { (cid:15) qq iterations.
Note that we present the theorem and proof for covariance being identity and no noise in the label but it can be easily extended to handle non-identity convariance with good condition number and bounded (in expectation) probabilistic concept noise.
Our analysis closely follows that from Goel et al.
However, in contrast to the known second layer setting considered in Goel et al.
(2018), we only know an approximation to the second layer and a robust analysis is needed.
Another difﬁculty arises from the fact that the convergence rate depends on the least eigenvalue of P a : “ ř 1 ď i,j ď k a i a j P i P Tj .
By simple algebra, we can show that the matrix has the following form:
[Equation]
Using property of Toeplitz matrices, we show the least eigenvalue of P a is lower bounded by 1 ´ cos [Equation] c
[Equation]
f [Equation] Theorem A.2) for all a with norm 1.
4. A NALYSIS OF S TAGE
Here we show how we can pick the correct hypothesis from the two possible hypothesis.
Under our assumptions, the individual loss p y p i q ´ f p w , a , x p i q qq 2 is bounded.
sThus, a direct application of Hoeffding inequality gives the following guarantee.
Theorem 4.3.
Suppose T 3 “ Ω ` poly ` r, k, B, 1 (cid:15) ˘˘ and let p w , a q .
If either ` w p`q , a T 1 ˘ or ` w p´q , ´ a T 1 ˘ has population risk smaller than 2 (cid:15) , then let p w , a q be the output according to equation 6, then with high probability
[Equation]
[Figure]
A
[Figure]
l e be t w een a and a *
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] Loga
[Figure]
i t h m o f
T e s t
Lo ss s
[Figure]
[Figure]
Stage
[Figure]
[Figure]
[Figure]
be t
w een a and
[Figure]
* s
[Figure] s=12s= (c) U: Stage     10 Iterations 10 4 - - - - - - Loga r i t h m o f T e s t Lo ss s=9s=12s= (
[Figure]
Stage
Now we put our analyses for Stage 1-3 together and prove Theorem 3.1. By Theorem 4.1, we know we have a p T 1 q such that ›› a T 1 ´ a ˚ ›› ď O ´ (cid:15)[-σ 
 r 1 { 2 k 3 { 2 or ›› w p´q-]
[Equation]
σ¯ (without loss of generality, we assume a and a ˚ are normalized) with η 1 “ O ´ poly ´ 1 k , 1 B , 1 σ 1 , (cid:15) ¯¯ and | S 1 | “ poly ` r, k, B, σ 1 , 1 (cid:15) ˘ .
Now with Theorem 4.2, we know with η “ O ` poly `k , 1 B ˘˘ and | S 2 | “ O ` poly ` k, σ 1 , log 1 (cid:15) ˘˘ we have either ›› w p`q ´ w ˚ ›› 2 ď (cid:15) σ 1 r 1 { 2 k 3 { 2 or ›› w p´q ´
w ˚ ›› 2 ď (cid:15) σ 1 r 1 { 2 k 3 { 2 .
Lastly, the following lemma bounds the loss of each instance in terms of the closeness of parameters.
Lemma 4.2.
For any a and w , we have
[Equation]
Therefore, we know either ` w p`q , a p T 1 q ˘ or ` w p´q , ´ a p
[Equation]
1 q ˘ achieves (cid:15) prediction error.
Now combining Theorem 4.3 and Lemma A.1 we obtain the desired result.
E XPERIMENTS
In this section we use simulations to verify the effectiveness of our proposed method.
We ﬁx input dimension d “ 160 and ﬁlter size r “ 16 for all experiments and vary the stride size s “ 9 , 12 , 16 .
For all experiments, we generate w ˚ and a ˚ from a standard Gaussian distribution and use 10 ,samples to calculate the test error.
Note in Stage 2 of Algorithm we need to test a “ a p T 1 q and ´ a p T 1 q .
Here we only report the one with better performance in the Stage 2 because in Stage 3 we can decide which one is better.
To measure the performance of Stage 1, we use the angle between a t an a ˚ (in radians).
We ﬁrst test Gaussian input distribution x „ N p 0 , I q .
Figure 1a shows the convergence in Stage 1 of Algorithm 1 with T 1 “ 10000 and η 1 “ 0 . 0001 .
Figure 1b shows the convergence in Stage 2 of Algorithm 1 with T 2 “ 10000 and η 2 “ 0 . 0001 .
We then test uniform input distribution x „ Unif r´?
3 s d (this distribution has identity covariance).
Figure 1c shows the convergence in Stage 1 of Algorithm 1 with T 1 “ 40000 and η 1 “ 0 . 0001 .
Figure 1d shows the convergence in Stage 2 of Algorithm 1 with T 2 “ 100000 and η 2 “ 0 . 00001 .
Note for both input distributions and all choices of stride size, our algorithm achieves low test error..
C ONCLUSION AND F UTURE W ORK
In this paper, we propose the ﬁrst efﬁcient algorithm for learning a one-hidden-layer convolutional neural network with possibly overlapping patches.
Our algorithm draws ideas from isotonic regression, landscape analysis of non-convex problem and spectral analysis of Toeplitz matrices.
These ﬁndings can inspire further development in this ﬁeld.
Our next step is extend our ideas to design provable algorithms that can learn complicated models consisting of multiple ﬁlters.
To solve this problem, we believe the recent progress on landscape design (Ge et al., 2017b) may be useful.
A U SEFUL L EMMAS /T HEOREMS
In this section we present a few lemmas/theorems that are useful for our analysis.
Proof of Lemma 4.2.
Observe that,
[Equation]
since p a ` b q 2 ď 2 ` a 2 ` b 2 ˘ for all a, b P R .
The ﬁrst term can be bounded as follows,
[Equation]
Here the ﬁrst inequality follows from observing that σ p a q ď | a | and the last follows from } v } 1 ď ?
k } v }
for all v P R k .
Similarly, the other term can be bounded as follows,
[Equation]
Here we use the Lipschitz property of σ to get the ﬁrst inequality.
The lemma follows from combining the above two.
The following lemma extends this to the overall loss.
Lemma A.1.
For any a and w ,
[Equation]
The following lemma from Goel et al.
(2018) is key to our analysis.
Lemma A.2 (Lemma 1 of Goel et al.
For all a , b P R n , if Z is symmetric then,
[Equation]
The following well-known theorem is useful for bounding eigenvalues of matrices.
Theorem A.1 (Gershgorin Circle Theorem Weisstein (2003)) .
For a n ˆ n matrix A , deﬁne R i : “ ř nj “ 1 ,j ‰ i | A
i,j | .
Each eigenvalue of A must lie in at least one of the disks t z : | z
´ A i,i | ď R i u .
The following lemma bounds the eigenvalue of the weighted patch matrices.
Theorem A.2.
For all a P S k ´ 1 ,
[Equation]
Proof.
Since s ě t r 2 u ` 1 , only adjacent patches overlap, and it is easy to verify that the matrix P a has the following structure:
[Equation]
Using the Gershgorin Circle Theorem, stated below, we can bound the eigenvalues, λ min p P a q ě
[Equation]
To bound the eigenvalues, we will bound ˇˇˇř k ´
i “ 1 a i a i ` 1 ˇˇˇ by maximizing it over all a such that } a } 2 “ 1 .
We have max !
ˇ ˇˇř
[Equation]
since the maximum can be achieved by setting all a i to be non-negative.
This can alternatively be viewed as max } a } 2 “ 1 a T Ma “ λ max p M q where M is a tridiagonal symmetric Toeplitz matrix as follows:
[Equation]
It is well known that the eigenvalues of this matrix are of the form cos
[Equation]
for i “ 1 ,
[Equation] [Equation]
k (c
[Equation]
f.
Boe´ottcher & Grudsky (2005)).
The maximum eigenvalue is thus cos ´ πk `¯ .
This gives us the result.
B O MITTED P ROOFS B. P ROOF OF L EMMA 4.
First by deﬁnition, we have
E
[Equation]
¯ w
[Equation]
ﬀ .
Because the input distribution is symmetric and the covariance is identity, by Lemma A.2, we have
E x
[Equation]
Similarly, we have
E x
[Equation]
y k
[Equation]
P
[Equation]
˚
[Equation]
˚ .
Also recall
[Equation]
Thus
[Equation]
The proof for h p w non , a , x , y q is similar.
To obtain a bound of the gradient
[Equation]
note that
[Equation]
Similar argument applies to y ř kj “ 1 a j P nonj x .
B. P ROOF OF T HEOREM 4.
We follow the Convotron analysis and include the changes.
Deﬁne S t “ tp x 1 , y 1 q , . . .
, p x t , y t qu .
The modiﬁed gradient update is as follows,
[Equation]
The dynamics of Convotron can then be expressed as follows:
[Equation]
We have,
[Equation]
 “ E x t ,y t «´ w ˚ ´ w p t q ¯ T ´ y t ´ f ´ w p t q , a , x t ¯¯ ˜
k ÿ i “a i P i x t ¸ˇˇˇˇˇ S t ´ﬀ “ E x t «´ w ˚ ´ w p t q ¯ T ´ f p w ˚ , a ˚ , x
t q ´ f ´ w p t q , a , x t ¯¯ ˜
k ÿ i “a i P i x t ¸ˇˇˇˇˇ S t ´ﬀ “ ÿ 1 ď i,j ď k
E x t r ˆ a ˚ i σ ´ p w ˚ q T P i x t ¯ ´ a i σ ˆ´ w p t q ¯ T
P i x t ˙˙
ˆ a j p w ˚ q T ´ a j ´ w p t q ¯ T ˙
P j x t | S t ´ 1 s “ 1 ` α ÿ 1 ď i,j ď k
E x t „ˆˆ a ˚ i p w ˚ q T ´ a
i ´ w p t q ¯ T ˙
P i x t ˙
ˆ a j p w ˚ q T ´ a j ´ w p t q ¯ T ˙
P j x t | S t ´ (11) “ 1 ` α ÿ 1 ď i,j ď k ˆ a ˚ i p w ˚ q T ´ a i ´ w p t q ¯ T ˙
P i E x t r x t x
Tt s P
Tj ´ a j w ˚ ´ a j w p t q ¯ (12) “ 1 ` α ÿ 1 ď i,j ď k ˆ a ˚ i p w ˚ q T ´ a i p w ˚ q T ` a i p w ˚ q T ´ a
i ´ w p t q ¯ T ˙
P i P
Tj ´ a j w ˚ ´ a j w p t q ¯ “ 1 ` α ˜´ p w ˚ q T ´ p w ˚ q T ¯
P a p w ˚ ´ w
[Equation]
[Equation]
(11) follows from using Lemma A.2, (13) follows from deﬁning P a : “ ř 1 ď i,j ď k a i a j P i P Tj , (12) follows from setting the covariance matrix to be identity and (14) follows from observing that P a is symmetric, thus @ x , x T P a x ě λ min p P a q } x } 22 as well as lower bounding the second term in terms of the norms of the corresponding parts.
Now we bound the variance of g p t q .
[Equation]
[Equation]
˙ (17) (15) follows from observing that ˇˇˇˇˇˇř ki “ 1 a i P i x ˇˇˇˇˇˇ 2 ď λ max p P a q } x } 22 for all x
and (16) follows from
Lemma 4.2.
Combining the above equations and taking expectation over S t ´ 1 , we get
[Equation]
for β “ 1 ` α 2 λ min p P a q , γ “ 2 λ max p P a q kB , δ “ 1 ` α 2 k } w ˚ } 2 and χ “ 2 λ max p P a q kB } w ˚ } 22 .
From Theorem A.2, we have that λ min p P a q “ 1 ´ cos
´ πk `¯ “ Ω ` 1 { k 2 ˘ (by Taylor expansion) implying β “ ω ` 1 { k 2 ˘ and γ “ O p kB q , χ “ O ´ kB } w ˚ }¯ .
We set η “ β min ´ 1 γ , 1 χ ¯ .
First we show that E S t ´ 1 r ›› w p t q ´ w ˚ ›› 2
s ď 1 for all iterations t .
We prove this inductively.
For t “ 1 , since w 1 “ 0 , this is satisﬁed.
Let us assume it holds for iteration t , then we have that,
[Equation]
The last inequality follows from (cid:15) ď 1 k 3 } w ˚ } 2 ď βδ ` β .
Thus we have that for each iteration,
E S t ´ 1 r ›› w p t q ´ w ˚ ›› 2 2 s ď 1 .
Substituting this in the recurrence and solving the recurrence gives us,
[Equation]
Thus for T “ O ´
[Equation]
˘¯ , we have,
[Equation]
Now using Markov’s inequality, we know that the above holds for some constant probability.
B
P ROOF OF T HEOREM 4.
For i “ T 2 ` 1 , . . .
, T 3 , deﬁne z p
i q “ ` y p i q ´ f ` w , a , x p i q ˘˘ 2 .
Using our assumptions, we know z p i q ď O p poly p r, k, B qq almost surely.
Now applying Hoeffding inequality we obtain our desired result.