H OW DOES UNLABELED DATA IMPROVE GENERALIZATION IN SELF - TRAINING ? A ONE - HIDDEN - LAYER THEORETICAL ANALYSIS
A BSTRACT
Self-training, a semi supervised learning algorithm leverages a large amount of unlabeled data to improve learning when the labeled data are limited.
Despite em - pirical successes its theoretical characterization remains elusive.
To the best of our knowledge this work establishes the ﬁrst theoretical analysis for the known iterative self-training paradigm and proves the beneﬁts of unlabeled data in both training convergence and generalization ability.
To make our theoretical analysis feasible we focus on the case of one-hidden-layer neural networks.
However, theoretical understanding of iterative self-training is non-trivial even for a shallow neural network.
One of the key challenges is that existing neural network landscape analysis built upon supervised learning no longer holds in the (semisupervised) self-training paradigm.
We address this challenge and prove that iterative self-training converges linearly with both convergence rate and generalization accuracy improved in the order of 1 / √ M , where M is the number of unlabeled samples.
Experiments from shallow neural networks to deep neural networks are also provided to justify the correctness of our established theoretical insights on self-training
I NTRODUCTION
Self-training (Scudder, 1965; Yarowsky, 1995; Lee et al., 2013; Han et al., 2019), one of the most powerful semi-supervised learning (SemiSL) algorithms, augments a limited number of labeled data with unlabeled data so as to achieve improved generalization performance on test data, compared with the model trained by supervised learning using the labeled data only.
Self-training has shown empirical success in diversiﬁed applications such as few-shot image classiﬁcation (Su et al., 2020;
Xie et al., 2020; Chen et al., 2020a; Yalniz et al., 2019; Zoph et al., 2020), objective detection (Rosenberg et al., 2005), robustness-aware model training against adversarial attacks (Carmon et al., 2019), continual lifelong learning (Lee et al., 2019), and natural language processing (He et al., 2019; Kahn et al., 2020).
The terminology “self-training” has been used to describe various SemiSL algorithms in the literature, while this paper is centered on the commonly-used iterative self-training method in particular.
In this setup, an initial teacher model (learned from the labeled data) is applied to the unlabeled data to generate pseudo labels.
One then trains a student model by minimizing the weighted empirical risk of both the labeled and unlabeled data.
The student model is then used as the new teacher to update the pseudo labels of the unlabeled data.
This process is repeated multiple times to improve the eventual student model.
We refer readers to Section 2 for algorithmic details.
Despite the empirical achievement of self-training methods with neural networks, the theoretical justiﬁcation of such success is very limited, even in the ﬁeld of SemiSL.
The majority of the theoretical results on general SemiSL are limited to linear networks (Chen et al., 2020b; Raghunathan et al., 2020; Oymak & Gulcu, 2020; Oneto et al., 2011).
The authors in (Balcan & Blum, 2010) show that unlabeled data can improve the generalization bound if the unlabeled data distribution and target model are compatible.
For instance, the unlabeled data need to be well-chosen such that the target function for labeled data can separate the unlabeled data clusters, which, however, may not be able to be veriﬁed ahead.
Moreover, (Rigollet, 2007; Singh et al., 2008) proves that unlabeled data can improve the convergence rate and generalization error under a similar clustering assumption, where the data contains clusters that have homogeneous labels.
A recent work by Wei et al.
(2020) analyzes SemiSL on nonlinear neural networks and proves that an inﬁnite number of unlabeled data can improve the generalization compared with training with labeled data only.
However, Wei et al.
(2020) considers single shot rather than iterative SemiSL, and the training problem aims to minimize the consistency regularization rather than the risk function in the conventional self-training method (Lee et al., 2013).
Moreover, Wei et al.
(2020) directly analyzes the global optimum of the nonconvex training problem without any discussion about how to achieve the global optimum.
To the best of our knowledge, there exists no analytical characterization of how the unlabeled data affect the generalization of the learned model by iterative self-training on nonlinear neural networks.
100K 200K 300K 400K 500K
Number of unlabeled data 5. T e s t acc u r ac y I m p r ov e m e n t ( % )
Contributions.
This paper provides the ﬁrst theoretical study of iterative self-training on nonlinear neural networks.
Focusing on one-hidden-layer neural networks, this paper provides a quantitative analysis of the generalization performance of iterative self-training as a function of the number of labeled and unlabeled samples.
Speciﬁcally, our contributions include 1.
Quantitative justiﬁcation of generalization improvement by unlabeled data.
Assuming the existence of a ground-truth model with weights W ∗ that maps the features to the corresponding labels, we prove that the learned model via iterative self-training moves closer to W ∗ as the number M of unlabeled data increases, indicating a better testing performance.
Specifically, we prove that the Frobenius distance to W ∗ , which is approximately linear in the generalization error, decreases in the order of 1 / √ M .
As an example, Figure 1 shows that the proposed theoretical bound matches the empirical self-training performance versus the number of unlabeled data for image classiﬁcation; see details in Section 4.2.
Analytical justiﬁcation of iterative self-training over single shot alternative.
We prove that the student models returned by the iterative self-training method converges linearly to a model close to W ∗ , with the rate improvement in the order of 1 / √ M . 3.
Sample complexity analysis of labeled and unlabeled data for learning a proper model.
We quantify the impact of labeled and unlabeled data on the generalization of the learned model.
In particular, we prove that the sample complexity of labeled data can be reduced compared with supervised learning.
1. R ELATED WORKS
Semi-supervised learning.
Besides self-training, many recent SemiSL algorithms exploit either consistency regularization or entropy minimization.
Consistency regularization is based on the as- sumption that the learned model will return same or similar output when the input is perturbed (Laine & Aila, 2016; Bachman et al., 2014; Sajjadi et al., 2016; Tarvainen & Valpola, 2017; Reed et al., 2015).
(Grandvalet & Bengio, 2005) claims that the unlabeled data are more informative if the pseudo labels of the unlabeled data have lower entropy.
Therefore, a line of works (Grandvalet & Bengio, 2005; Miyato et al., 2018) adds a regularization term that minimizes the entropy of the outputs of the unlabeled data.
In addition, hybrid algorithms that unify both the above regularizations have been developed like (Berthelot et al., 2019a;b; Sohn et al., 2020).
Domain adaptation.
Domain adaptation exploits abundant data in the source domain to learn a model for the target domain, where only limited training data are available (Liebelt & Schmid, 2010;
Vazquez et al., 2013; Zhang et al., 2013; Long et al., 2015; Tzeng et al., 2014).
Source and target domain are related but different.
Unsupervised domain adaptation (Ganin & Lempitsky, 2015;
Ganin et al., 2016; Gong et al., 2013; Bousmalis et al., 2016), where training data in target domain are unlabeled, is similar to SemiSL, and self-training methods have been used for analysis (Zou et al., 2018; Tang et al., 2012; French et al., 2018).
However, self-training and unsupervised domain adaptation are fundamentally different.
The former learns a model for the domain where there is limited labeled data, with the help of a large number of unlabeled data from a different domain.
The latter learns a model for the domain where the training data are unlabeled, with the help of sufﬁcient labeled data from a different domain.
Generalization analysis of supervised learning.
In theory, the testing error is upper bounded by the training error plus the generalization gap between training and testing.
These two quantities are often analyzed separately and cannot be proved to be small simultaneously for deep neural networks.
For example, neural tangent kernel (NTK) method (Jacot et al., 2018;
Du et al., 2018; Lee et al., 2018) shows the training error can be zero, and the Rademacher complexity in (Bartlett & Mendelson, 2002) bounds the generalization gap (Arora et al., 2019a).
For one-hidden-layer neural networks (Safran & Shamir, 2018), the testing error can be proved to be zero under mild conditions.
One common assumption is that the input data belongs to the Gaussian distribution (Zhong et al., 2017;
Ge et al., 2018; Kalai et al., 2008; Bakshi et al., 2019; Zhang et al., 2016; Brutzkus & Globerson, 2017; Li & Yuan, 2017; Soltanolkotabi et al., 2018).
Another line of approaches (Brutzkus et al., 2018; Li & Liang, 2018; Wang et al., 2019) consider linearly separable data.
The rest of this paper is organized as follows.
Section 2 introduces the problem formulation and self-training algorithm.
Major results are summarized in Section 3, and empirical evaluations are presented in Section 4.
Section 5 concludes the whole paper.
All the proofs are in the Appendix.
F ORMALIZING S ELF -T RAINING : N OTATION , F ORMULATION , AND A LGORITHM
Problem formulation.
Given N labeled data sampled from distribution P l , denoted by D = { x n , y n } N n =1 , and M unlabeled data drawn from distribution P u , denoted by (cid:101)
D
= { (cid:101) x m } Mm =1 .
The aim is to ﬁnd a neural network model g ( W ) , where W denotes the trainable weights, that minimizes the testing error on data sampled from P l .
(S1) Initialize iteration (cid:96)
{+= 
+}
and obtain a model W ( (cid:96) ) as the teacher using labeled data D only; (S2) Use the teacher model to obtain pseudo labels (cid:101) y m of unlabeled data in (cid:101) D ; (S3) Train the neural network by minimizing (1) via T -step mini-batch gradient descent method using disjoint subsets {D t } T − 1 t =0 and { (cid:101) D t } T − 1 t =0 of (cid:101) D .
Let W ( (cid:96) +1) denote the obtained student model; (S4)
Use W ( (cid:96) +1) as the current teacher model.
Let (cid:96) ← (cid:96) + 1 and go back to step (S2);
Iterative self-training.
In
each iteration, given the current teacher predictor g ( W ( (cid:96) ) ) , the pseudolabels for the unlabeled data in (cid:101) D are computed as ˜ y m
= g ( W ( (cid:96) ) ; (cid:101) x m ) .
The method then minimizes the weighted empirical risk ˆ f D , (cid:101) D ( W ) of both labeled and unlabeled data through stochastic gradient descent, where
[Equation]
and λ + (cid:101) λ
= 1 .
The learned student model g ( W ( (cid:96) +1) ) is used as the teacher model in the next iteration.
The initial model g ( W (0) ) is learned from labeled data.
The formal algorithm is summarized as in Table 1.
[-= 
-]
Model and assumptions.
This paper considers regression 1 , where g is a one-hidden-layer fully connected neural network equipped with K neurons.
Namely, given the input x ∈ R d and weights W = [ w 1 , w 2 , · · · , w K ] ∈ R d × K , we have
[Equation]
where φ is the ReLU activation function 2 , and φ ( z ) =
max { z, 0 } for any input z ∈ R .
Here, we ﬁx the top layer weights as 1 for simplicity, and the equivalence of such a simpliﬁcation is discussed in Appendix K.
Moreover, we assume an unknown ground-truth model with weights W ∗ that maps all the features to the corresponding labels drawn from P l , i.e., y = g ( W ∗ ; x ) , where ( x , y ) ∼ P l .
The generalization function (GF) with respect to g ( W ) is deﬁned as
[Equation]
By deﬁnition I (cid:0) g ( W ∗ ) (cid:1) is zero.
Clearly, W ∗ is not unique because any column permutation of W ∗ , which corresponds to permuting neurons, represents the same function as W ∗ and minimizes GF in (3) too.
To simplify the representation, we follow the convention and abuse the notation that the distance from W to W ∗ , denoted by (cid:107) W − W ∗ (cid:107) F , means the smallest distance from W to any permutation of W ∗ .
Additionally, some important notations are summarized in Table 2.
We assume the inputs of both the labeled and unlabeled data belong to the zero mean Gaussian distribution, i.e., x ∼ N (0 ,
δ 2 I d ) , and (cid:101) x ∼ N (0 , ˜
δ 2 I d ) .
The Gaussian assumption is motivated by the data whitening (LeCun et al., 2012) and batch normalization techniques (Ioffe & Szegedy, 2015) that are commonly used in practice to improve learning performance.
Moreover, training onehidden-layer neural network with multiple neurons is NP-Complete (Blum & Rivest, 1992) without any assumption.
The focus of this paper.
This paper will analyze three aspects about self-training: (1) the generalization performance of W ( L ) , the returned model by self-training after L iterations, measured by (cid:107) W ( L ) − W ∗ (cid:107) F 3 ; (2) the inﬂuence of parameter λ in (1) on the training performance; and (3) the impact of unlabeled data on the training and generalization performance.
[Table]
; W
returned by self-
iterations
; W
[Equation]
T HEORETICAL RESULTS
Beyond supervised learning: Challenge of self-training.
The existing theoretical works such as (Zhong et al., 2017) verify that for one-hidden-layer neural networks, if only labeled data are available, and x are drawn from the standard Gaussian distribution, then supervised learning by minimizing (1) with λ
= 1 can return a model with ground-truth weights W ∗ (up to column permutation), as long as the number of labeled data N is at least N ∗ , which depends on κ, K and d .
In contrast, this paper focuses on the low labeled-data regime when N is less than N ∗ .
Speciﬁcally,
[Figure]
[Equation]
Intuitively, if N < N ∗ , the landscape of the empirical risk of the labeled data becomes highly nonconvex, even in a neighborhood of W ∗ , thus, the existing analyses for supervised learning do not hold in this region.
With additional unlabeled data, the landscape of the weighted empirical risk becomes smoother near W ∗ .
Moreover, as M increases, and starting from a nearby initialization, the returned model W ( L ) by iterative self-training can converge to a local minimum that is closer to W ∗ (see illustration in Figure 2).
Compared with supervised learning, the formal analyses of self-training need to handle new technical challenges from two aspects.
First, the existing analyses of supervised learning exploit the fact that the GF and the empirical risk have the same minimizer, i.e., W ∗ .
This property does not hold for self-training as W ∗ no longer minimizes the weighted empirical risk in (1).
Second, the iterative manner of self-training complicates the analyses.
Speciﬁcally, the empirical risk in each iteration is different and depends on the model trained in the previous iteration through the pseudo labels.
In what follows, we provide theoretical insights and the formal theorems.
Some important quantities ˆ λ and µ are deﬁned below
[Equation]
and
[Equation]
where ρ is a positive function deﬁned in (73).
ˆ λ is an increasing function of λ .
Also, from Lemma 11 (in Appendix), ρ ( δ ) is in the order of δ 2 when δ ≤ 1 for ReLU activation functions.
Thus, µ is a ﬁxed constant, denoted by µ ∗ , for all δ, ˜ δ ≤ 1 .
When δ and ˜ δ are large, µ increases as they increase.
The formal deﬁnition of N ∗ in (4) is c ( κ ) µ ∗ 2 K 3 d log q , where c ( κ ) is some polynomial function of κ and can be viewed as constant.
3. I NFORMAL KEY THEORETICAL FINDINGS
[Equation]
To the best of our knowledge, Theorems 1 and 2 provide the ﬁrst theoretical characterization of iterative self-training on nonlinear neural networks.
Before formally presenting them, we summarize the highlights as follows.
1. Linear convergence of the learned models.
The learned models converge linearly to a model close to W ∗ .
Thus, the iterative approach returns a model with better generalization than that by the single-shot method.
Moreover, the convergence rate is a constant term plus a term in the order of 1 / √ M (see ∆ 1 in Figure 3), indicating a faster convergence with more unlabeled data.
Returning a model with guaranteed generalization in the low labeled-data regime.
Even when the number of labeled data is much less than the required sample complexity to obtain W ∗ in supervised learning, we prove that with the help of unlabeled data, the iterative self-training can return a model in the neighborhood of W [ˆ λ ] , where W [ˆ λ ] is in the line segment of W (0) ( ˆ λ = 0 ) and ground truth W ∗ ( ˆ λ = 1 ).
Moreover, ˆ λ is upper bounded by (cid:112) N/N ∗ .
Thus W ( L ) moves closer to W ∗ as N increases ( E 0 in Figure 3), indicating a better generalization performance with more labeled data.
Guaranteed generalization improvement by unlabeled data.
The distance between W ( L ) and W [ˆ λ ] ( E 1 in Figure 3) scales in the order of 1 / √ M .
With a larger number of unlabeled data M , W ( L ) moves closer to W [ˆ λ ] and thus W ∗ , indicating an improved generalization performance (Theorem 1).
When N is close to N ∗ but still smaller as deﬁned in (12), both W ( L ) and W [ˆ λ ] converge to W ∗ , and thus the learned model achieves zero generalization error (Theorem 2).
3. F ORMAL THEORY IN LOW LABELED - DATA REGIME
Takeaways of Theorem 1: Theorem 1 characterizes the convergence rate of the proposed algorithm and the accuracy of the learned model W ( L ) in a low labeled-data regime.
Speciﬁcally, the iterates converge linearly, and the learned model is close to W
[ˆ λ ] and guaranteed to outperform the initial model W (0) .
Theorem 1.
Suppose the initialization W (0) and the number of labeled data satisfy
[Equation]
with p ∈ (cid:0)
[Equation]
[Equation]
(cid:3) [Equation] 6) and
[Equation]
If the value of ˆ λ in (5) and unlabeled data amount M satisfy
[Equation]
and
[Equation]
Then, when the number T of SGD iterations is large enough in each loop (cid:96) [Equation] with probability at least 1 − q − d [Equation]
the iterates { W ( (cid:96) ) } L(cid:96)
=
converge to W [ˆ λ ] as
[Equation]
[Equation]
where W [ˆ λ ]
= ˆ λ W ∗
+ (1 − ˆ λ ) W (0) .
Typically, when the iteration number L is sufﬁcient large [Equation] we have
[Equation]
The accuracy of the learned model W ( L ) with respect to W ∗ is characterized as (10), and the learning model is better than initial model as in (11) if the following conditions hold.
First, the weights λ in (1) are properly chosen as in (8).
Second, the number of unlabeled data is sufﬁciently large as in (9).
Selection of λ in self-training algorithms .
When ˆ λ increases, the required number of unlabeled data is reduced from (9), and the convergence point W ( L ) becomes closer to W ∗ from (11), which indicates a smaller generalization error.
Thus, a large ˆ λ within its feasible range (8) is desirable.
When the initial model W (0) is closer to W ∗ (corresponding to a larger p ), and the number of labeled data N increases, the upper bound in (8) increases, and thus, one can select a larger ˆ λ .
The initial model W (0) .
The tensor initialization from (Zhong et al., 2017) can return a W (0) that satisﬁes (6) when the number of labeled data is N = p 2 N ∗ (see Lemma 3 in Appendix).
Combining with the requirement in (7), Theorem 1 applies to the case that N is at least N ∗ / 4 .
3. F ORMAL THEORY OF ACHIEVING ZERO GENERALIZATION ERROR
Takeaways of Theorem 2: Theorem 2 indicates the model returned by the proposed algorithm converges linearly to the ground truth W ∗ .
Thus the distance between the learned model and the ground truth can be arbitrarily small with the ability to achieve zero generalization error.
The required sample complexity is reduced by a constant factor compared with supervised learning.
Theorem 2.
Consider the number of unlabeled data satisﬁes
[Equation]
we choose ˆ λ such that
[Equation]
Suppose the initial model W (0) and the number of unlabeled data M satisfy
[Equation]
and M ≥ c
[Equation]
κ ) µ
[Equation]
1 −
[Equation] ) 2 K 3 d log q, [Equation] 14) the iterates { W ( (cid:96) ) } L(cid:96)
=
converge to the ground truth W ∗ as follows,
[Equation]
The models W ( (cid:96) ) ’s converge linearly to the ground truth W ∗ as (15) when the number of labeled data satisﬁes (12).
In contrast, supervised learning requires at least N ∗ labeled samples to estimate W ∗ accurately without unlabeled data, which suggests self-training at least saves a constant fraction of labeled data.
3. T HE MAIN PROOF IDEA
Our proof builds upon and extends one recent line of works on supervised learning such as (Zhong et al., 2017; Zhang et al., 2020a;b; 2021).
The standard framework of these works is ﬁrst to show that the generalization function I ( g ( W )) in (3) is locally convex near W ∗ , which is its global minimizer.
Then, when M = 0 and N is sufﬁciently large, the empirical risk function using labeled data only can approximate I ( g ( W ))
well in the neighborhood of W ∗ .
Thus, if initialized in this local convex region, the iterations, returned by applying gradient descent approach on the empirical risk function, converge to W ∗ linearly.
The technical challenge here is that in self-training, when unlabeled data are paired with pseudo labels, W ∗ is no longer a global minimizer of the empirical risk ˆ f D , (cid:101) D in (1), and ˆ f D , (cid:101)
D does not approach I ( g ( W )) even when M and N increase to inﬁnity.
Our new idea is to design a population risk function f
( W ; ˆ λ ) in (17) (see appendix), which is a lower bound of ˆ f D , (cid:101) D when M and N are inﬁnity.
f ( W ; ˆ λ ) is locally convex around its minimizer W [ˆ λ ] , and W [ˆ λ ] approaches W ∗ as ˆ λ increases.
Then we show the iterates generated by ˆ f D , (cid:101)
D stay close to f ( W ; ˆ λ ) , and the returned model W ( L ) is close to W [ˆ λ ] .
New technical tools are developed to bound the distance between the functions ˆ f D , (cid:101) D and f ( W ; ˆ λ ) .
E MPIRICAL RESULTS 4. S YNTHETIC DATA EXPERIMENTS
We generate a ground-truth neural network with the width K = 10 .
Each entry of W ∗ is uniformly selected from [ − 2 . 5 , 2 . 5] .
The input of labeled data x n are generated from Gaussian distribution N (0 , I d ) independently, and the corresponding label y n is generated through (2) using W ∗ .
The unlabeled data (cid:101) x m are generated from N (0 , (cid:101)
δ 2 I d ) independently with (cid:101) δ
= 1 except in Figure 7.
d is set as 50 except in Figure 9.
The value of λ is selected as (cid:112) N/ (2 Kd ) except in Figure 8.
We consider one-hidden-layer except in Figure 4.
The initial teacher model W (0) in self-training is randomly selected from { W |(cid:107) W − W ∗ (cid:107) F / (cid:107) W ∗ (cid:107) F ≤ 0 . 5 } to reduce the computation.
In each iteration, the maximum number of SGD steps T is 10 .
Self-training terminates if (cid:107) W
( (cid:96) +1)
− W ( (cid:96) )
(cid:107) F / (cid:107) W ( (cid:96) )
(cid:107) F ≤ 10 − 4 or reaching 1000 iterations.
In Figures 5 to 8, all the points on the curves are averaged over 1000 independent trials, and the regions in lower transparency indicate the corresponding one-standard-deviation error bars.
Our empirical observations are summarized below.
(a) GF (testing performance) proportional to (cid:107) W − W ∗ (cid:107) F .
Figure 4 illustrates the GF in (3) against the distance to the ground truth W ∗ .
To visualize results for different networks together, GF is normalized in [0 , 1] , divided by its largest value for each network architecture.
All the results are averaged over 100 independent choice of W .
One can see that for one-hidden-layer neural networks, in a large region near W ∗ , GF is almost linear in (cid:107) W − W ∗ (cid:107) F .
When the number of hidden layers increases, this region decreases, but the linear dependence still holds locally.
This is an empirical justiﬁcation of using (cid:107) W − W ∗ (cid:107) F to evaluate the GF and, thus, the testing error in Theorems 1 and 2.
(b) (cid:107) W ( L ) − W ∗ (cid:107) F as a linear function of 1 / √ M .
Figure 5 shows the relative error (cid:107) W ( L
) − W ∗ (cid:107) F / (cid:107) W ∗ (cid:107) F when the number of unlabeled data and labeled data changes One can
see that the relative error decreases when either M or N increases
Additionally the dash-dotted lines represent the best ﬁtting of the linear functions of / √ M using the least square method
Therefore, the relative error is indeed a linear function of 1 / √ M , as predicted by our results in (11) and (15).
[Figure]
[Figure] [Figure]
(c) Convergence rate as a linear function of 1 / √ M .
Figure 6 illustrates the convergence rate when M and N change.
We can see that the convergence rate is a linear function of 1 / √ M , as predicted by our results (11) and (15).
When M increases, the convergence rate is improved, and the method converges faster.
(d) Increase of (cid:101) δ slows down convergence.
Figure 7 shows that the convergence rate becomes worse when the variance of the unlabeled data (cid:101) δ increases from 1 .
When (cid:101) δ is less than 1 , the convergence rate almost remains the same, which is consistent with our characterization in (10) that the convergence rate is linear in µ .
From the discussion after (5), µ increases as (cid:101) δ increases fromand stays constant when (cid:101) δ is less than 1 .
(e) (cid:107) W ( L ) − W ∗ (cid:107) F / (cid:107) W ∗ (cid:107) F is improved as a linear function of ˆ λ .
Figure 8 shows that the relative errors of W ( L ) with respect to W ∗ decrease almost linearly when ˆ λ increases, which is consistent with the theoretical result in (11).
Moreover, when λ exceeds a certain threshold positively correlated with N , the relative error increases rather than decreases.
That is consistent with the analysis in (8) that ˆ λ has an upper limit, and such a limit increases as N increases.
(f) Unlabeled data reduce the sample complexity to learn W ∗ .
Figure 9 depicts the phase transition of returning W ( L ) .
For every pair of d and N , we construct 100 independent trials, and each trial is said to be successful if (cid:107) W ( L ) − W ∗ (cid:107) F / (cid:107) W ∗ (cid:107) F ≤ 10 − 2 .
The white blocks correspond to the successful trials, while the block in black indicates all failures.
When d increases, the required number of labeled data to learn W ∗ is linear in d .
Thus, the sample complexity bound in (12) is order-wise optimal for d .
Moreover, the phase transition line when M = 1000 is below the one when M = 0 .
Therefore, with unlabeled data, the required sample complexity of N is reduced.
[Figure] [Figure] [Figure]
C onv e r g e n ce r a t e
[Figure]
[Figure]
[Figure]
[Figure] N [Figure] [Figure]
[Figure]
4. I MAGE CLASSIFICATION ON AUGMENTED CIFAR-10 DATASET
We evaluate self-training on the augmented CIFAR-10 dataset, which has 50K labeled data.
The unlabeled data are mined from 80 Million Tiny Images following the setup in (Carmon et al., 2019) 4 , and additional 50K images are selected for each class, which is a total of 500K images, to form the unlabeled data.
The self-training method is the same implementation as that in (Carmon et al., 2019).
λ and (cid:101) λ is selected as N/ ( M + N ) and M/ ( N + M ) , respectively, and the algorithm stops after 200 epochs.
In Figure 10, the dash lines stand for the best ﬁtting of the linear functions of 1 / √ M via the least square method.
One can see that the test accuracy is improved by up to 7% using unlabeled data, and the empirical evaluations match the theoretical predictions.
Figure 11 shows the convergence rate calculated based on the ﬁrst 50 epochs, and the convergence rate is almost a linear function of 1 / √ M , as predicted by (10).
[Figure] [Figure]
C ONCLUSION
This paper provides new theoretical insights into understanding the inﬂuence of unlabeled data in the iterative self-training algorithm.
We show that the improved generalization error and convergence rate is a linear function of 1 / √ M , where M is the number of unlabeled data.
Moreover, compared with supervised learning, using unlabeled data reduces the required sample complexity of labeled data for achieving zero generalization error.
Future directions include generalizing the analysis to multi-layer neural networks and other semi-supervised learning problems such as domain adaptation. A CKNOWLEDGEMENT
This work was supported by AFOSR FA9550-20-1-0122, ARO W911NF-21-1-0255, NSFand the Rensselaer-IBM AI Research Collaboration (http://airc.rpi.edu), part of the IBM AI Horizons Network (http://ibm.biz/AIHorizons).
Appendix A O VERVIEW OF THE PROOF TECHNIQUES
We ﬁrst provide an overview of the techniques used in proving Theorems 1 and 2.
Characterization of a proper population risk function.
To characterize the performance of the iterative self-training algorithm via the stochastic gradient descent method, we need ﬁrst to deﬁne a population risk function such that the following two properties
hold. First, the landscape of the population risk function should be analyzable near { W ( (cid:96) ) } L(cid:96)
=0 .
Second, the distance between the empirical risk function in (1) and the population risk function should be bounded near { W ( (cid:96) ) } L(cid:96)
=0 .
The generalization function deﬁned in (3), which is widely used in the supervised learning problem with a sufﬁcient number of samples, failed the second requirement.
To this end, we turn to ﬁnd a new population risk function deﬁned in (17), and the illustrations of the population risk function and objection function are included in Figure 12.
Local convex region of the population risk function.
The purpose is to characterize the iterations via the stochastic gradient descent method in the population risk function.
To obtain the local convex region of the population risk function, we ﬁrst bound the Hessian of the population risk function at its global optimal.
Then, we utilize Lemma 12 in Appendix H.1 to obtain the Hessian of the population risk function near the global optimal.
The local convex region of the population risk function is summarized in Lemma 1, and the proof of Lemma 1 is included in Appendix H.1.
Bound between the population risk and empirical risk functions.
After the characterization of the iterations via the stochastic gradient descent method in the population risk function, we need to bound the distance between the population risk function and empirical risk function.
Therefore, the behaviors of the iterations via the stochastic gradient descent method in the empirical risk function can be described by the ones in the population risk function and the distance between these two.
The key lemma is summarized in Lemma 2 (see Appendix H.2), and the proof is included in Appendix H.2.
𝑾𝑾
Local convex region of ̂𝑓𝑓 𝒟𝒟 near 𝑾𝑾 ( 0 , 0 ) 𝑾𝑾 ∗ 𝑾𝑾 ( 0 , 2 ) 𝑾𝑾 ( 0 , 1 ) 𝑾𝑾 ( 0 , 𝑇𝑇 ) 𝛽𝛽 ( 𝑾𝑾 0 , 1 −𝑾𝑾 ( 0 , 0 ) )
𝜂𝜂𝜂𝜂 ̂𝑓𝑓 𝒟𝒟 . . .
Population risk function: 𝑓𝑓 ( 𝑾𝑾 ; 𝑝𝑝 )
Objective function: ̂𝑓𝑓 𝒟𝒟 ( 𝑾𝑾 ) 𝑾𝑾 [ 𝑝𝑝 ]
In the following contexts, the details of the iterative self-training algorithm are included in Appendix B. We then ﬁrst provide the proof of Theorem 2 in Appendix E, which can be viewed as a special case of Theorem 1.
Then, with the preliminary knowledge from proving Theorem 2, we turn to present the full proof of a more general statement summarized in Theorem 3 (see Appendix F), which is related to Theorem 1.
The deﬁnition and relative proofs of µ and ρ are all included in Appendix G.
The proofs of preliminary lemmas are included in Appendix H.
B I TERATIVE SELF - TRAINING ALGORITHM
In this section, we implement the details of the mini-batch stochastic gradient descent used in each stage of the iterative self-training algorithm.
After t number of iterations via mini-batch stochastic gradient descent at (cid:96) -th stage of self-training algorithm, the learned model is denoted as W ( (cid:96),t ) .
One can easily check that W ( (cid:96) ) in the main context is denoted as W ( (cid:96), 0) in this section and the following proofs.
Last, the pseudo-code of the iterative self-training algorithm is summarized in Algorithm 1. for
= L − 1 do y m = g ( W ( (cid:96), 0) ; (cid:101) x m ) m =
· · · , M for T − 1 do W ( (cid:96),t +1) =
W ( (cid:96),t ) − η · ∇ ˆ f D t , (cid:101) D t ( W ( (cid:96),t ) ) + β · (cid:0) W ( (cid:96),t ) − W ( (cid:96),t − 1) (cid:1)
= W ( end
C N OTATIONS
In this section, we ﬁrst introduce some important notations that will be used in the following proofs, and the notations are summarized in Table 1.
As shown in Algorithm 1, W ( (cid:96),t ) denotes the learned model after t number of iterations via minibatch stochastic gradient descent at (cid:96) -th stage of the iterative self-training algorithm.
Given a student model (cid:102) W , the pseudo label for (cid:101) x ∈ (cid:101) D is generated as
[Equation]
Further, let W [ p ] =
p W ∗
+ (1 − p ) W
(0 , 0) , we then deﬁne the population risk function as
[Equation] [Equation]
where y ∗ ( p ) =
g ( W [ p ] ; x ) with x ∼ N (0 , δ 2 I ) and (cid:101)
y ∗ ( p ) =
g ( W [ p ] ; (cid:101) x ) with (cid:101) x ∼ N (0 , ˜
δ 2 I ) .
When p = 1 , we have W [ p ] =
W ∗ and y ∗ ( p ) =
y for data in D .
Moreover, we use σ i to denote the i -th largest singular value of W ∗ .
Then, κ is deﬁned as σ 1 /σ K , and γ = (cid:81)
Ki =
1 σ i /σ K .
Additionally, to avoid high dimensional tensors, the ﬁrst order derivative of the empirical risk function is deﬁned in the form of vectorized W as
[Equation]
18) with W = [ w 1 , w 2 , · · · , w K ] ∈ R d × K .
Therefore, the second order derivative of the empirical risk function is in R dk × dk .
Similar to (18), the high order derivatives of the population risk functions are deﬁned based on vectorized W as well.
In addition, without special descriptions, α = [ α T 1 , α T 2 , · · · , α TK ] T stands for any unit vector that in R dK with α j ∈ R d .
Therefore, we have
[Equation]
Finally, since we focus on order-wise analysis, some constant numbers will be ignored in the majority of the steps.
In particular, we use h 1 ( z ) (cid:38) ( or (cid:46) , (cid:104) )
h 2 ( z ) to denote there exists some positive constant C such that h 1 ( z ) ≥ ( or ≤ , =) C · h 2 ( z ) when z ∈ R is sufﬁciently large. [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The empirical risk function deﬁned in (1);
[Equation]
[Table]
[Equation]
p
[Table]
The population risk function deﬁned in (17);
[Equation]
The value of λδ 2 / ( λδ 2 +
[Table]
λ ˜
[Table]
2 ) ; µ The value of
[Table]
; σ i The i -th largest singular value of W ∗ ; κ The value of σ 1 /σ K ; γ The value of (cid:81)
[Equation]
[Table]
+
[Equation]
D P RELIMINARY L EMMAS
We will ﬁrst start with some preliminary lemmas.
As outlined at the beginning of the supplementary material, Lemma 1 illustrates the local convex region of the population risk function, and Lemma 2 explains the error bound between the population risk and empirical risk functions.
Then, Lemma 3 describes the returned initial model W (0 , 0) via tensor initialization method (Zhong et al., 2017) purely using labeled data.
Next, Lemma 4 is the well known Weyl’s inequality in the matrix setting.
Moreover, Lemma 5 is the concentration theorem for independent random matrices.
The deﬁnitions of the sub-Gaussian and sub-exponential variables are summarized in Deﬁnitions 1 and 2.
Lemmas 6 and 7 serve as the technical tools in bounding matrix norms under the framework of the conﬁdence interval.
Lemma 1.
Given any W ∈ R d × K , let p satisfy
[Equation]
Then, we have
[Equation]
Lemma 2.
Let f and ˆ f be the functions deﬁned in (17) and (1) , respectively.
Suppose the pseudo label is generated through (16) with weights (cid:102) W .
Then, we have
[Equation]
with probability at least 1 − q − d .
Lemma 3 (Initialization, (Zhong et al., 2017)) .
Assuming the number of labeled data satisﬁes
[Equation]
for some large constant q and p ∈ [ 1 K , 1] , the tensor initialization method, which is summarized in
Appendix I, outputs W (0 , 0) such that
[Equation]
with probability at least 1 − q − d .
Lemma 4 (Weyl’s inequality, (Bhatia, 2013)) .
Let B =
A + E be a matrix with dimension m × m .
Let λ i ( B ) and λ i ( A ) be the i -th largest eigenvalues of B and A , respectively.
Then, we have
[Equation]
[Equation]
Lemma 5 ((Tropp, 2012), Theorem 1.6) .
Consider a ﬁnite sequence { Z k } of independent, random matrices with dimensions d 1 × d 2 .
Assume that such random matrix satisﬁes
{+E ( Z k ) = 
+}
and (cid:107) Z k (cid:107) ≤ R almost surely .
Deﬁne
[Equation]
Then for all t ≥ 0 , we have Prob
[Equation]
Deﬁnition 1 (Deﬁnition 5.7, (Vershynin, 2010)) .
A random variable X is called a sub-Gaussian random variable if it satisﬁes
[Equation]
) for all p ≥ 1 and some constant c 1 > 0 .
In addition, we have
[Equation]
27) for all s ∈ R and some constant c 2 > 0 , where (cid:107) X (cid:107) φ 2 is the sub-Gaussian norm of X deﬁned as (cid:107)[-X (cid:107) ψ 2 = sup p ≥ 1 p − 1 / 
-]
[Equation]
Moreover, a random vector X ∈ R d belongs to the sub-Gaussian distribution if one-dimensional marginal α T X is sub-Gaussian for any α ∈ R d , and the sub-Gaussian norm of X is deﬁned as (cid:107) X (cid:107) [Equation] = sup (cid:107) [Equation]
(cid:107)
[Equation] (cid:107) [Equation] Deﬁnition 2 (Deﬁnition 5.13, (Vershynin, 2010)) .
A random variable X is called a sub-exponential random variable if it satisﬁes
[Equation]
28) for all p ≥ 1 and some constant c 3 > 0 .
In addition, we have
[Equation]
29) for s ≤ 1 / (cid:107) X (cid:107) ψ 1 and some constant c 4 > 0 , where (cid:107) X (cid:107) ψ 1 is the sub-exponential norm of X deﬁned as (cid:107) X (cid:107) ψ 1 = sup p ≥ 1 p − 1 ( E | X | p ) 1 /p .
Lemma 6 (Lemma 5.2, (Vershynin, 2010)) .
Let B (0 , 1) ∈
{ α (cid:12)(cid:12) (cid:107) α (cid:107) 2 = 1 , α ∈ R d } denote a unit ball in R d .
Then, a subset S ξ is called a ξ -net of B (0 , 1) if every point z ∈ B (0 , 1) can be approximated to within ξ by some point α ∈ B (0 , 1) , i.e., (cid:107)
z − α (cid:107) 2 ≤ ξ .
Then the minimal cardinality of a ξ -net S ξ satisﬁes
[Equation]
Lemma 7 (Lemma 5.3, (Vershynin, 2010)) .
Let A be an d 1 × d 2 matrix, and let S ξ ( d ) be a ξ -net of B (0 , 1) in R d for some ξ ∈ (0 , 1) .
Then
[Equation]
Lemma 8 (Mean Value Theorem) .
Let U ⊂ R n 1 be open and f : U −→ R n 2 be continuously differentiable, and x ∈ U , h ∈ R n 1 vectors such that the line segment x + t h , 0 ≤ t ≤ 1 remains in U .
Then we have:
[Equation]
where ∇ f denotes the Jacobian matrix of f .
E P ROOF OF T HEOREM
With p = 1 in (17), the population risk function is reduced as
[Equation]
where y = g ( W ∗ ; x ) with x ∼ N (0 , δ 2 I ) and (cid:101) y ∗ =
g ( W ∗ ; (cid:101) x ) with (cid:101) x ∼ N (0 , ˜
δ 2 I ) .
In fact, (32) can be viewed as the expectation of the empirical risk function in (1) given (cid:101) y m
=
g ( W ∗ ; (cid:101) x m ) .
Moreover, the ground-truth model W ∗ is the global optimal to (32) as well.
Lemmas 9 andare the special case of Lemmas 1 and 2 with p = 1 .
The proof of Theorem 2 is followed by the presentation of the two lemmas.
The main idea in proving Theorem 2 is to characterize the gradient descent term by the MVT in Lemma 8 as shown in (36) and (37).
The IVT is not directly applied in the empirical risk function because of its non-smoothness.
However, the population risk functions deﬁned in (17) and (32), which are the expectations over the Gaussian variables, are smooth.
Then, as the distance (cid:107)∇ f ( W ) − ∇ f ( W ∗ ) (cid:107) F is upper bounded by a linear function of (cid:107) W − W ∗ (cid:107) F as shown in (47), we can establish the connection between (cid:107) W ( (cid:96),t +1) − W ∗ (cid:107) F and (cid:107) W ( (cid:96),t ) − W ∗ (cid:107) F as shown in (50).
Finally, by mathematical induction over (cid:96) and t , one can characterize (cid:107) W ( L, 0) − W ∗ (cid:107) F by (cid:107) W (0 , 0) − W ∗ (cid:107) F as shown in (52), which completes the whole proof.
Lemma 9 (Lemma 1 with p = 1 ) .
Let f and ˆ f are the functions deﬁned in (32) and (1) , respectively.
Then, for any W that satisﬁes,
[Equation]
we have
[Equation]
Lemma 10 (Lemma 2 with p = 1 ) .
Let f and ˆ f be the functions deﬁned in (32) and (1) , respectively.
Suppose the pseudo label is generated through (16) with weights (cid:102) W .
Then, we have
[Equation]
with probability at least 1 − q − d .
Proof of Theorem 2.
From Algorithm 1, in the (cid:96) -th outer loop, we have
[Equation]
Since ∇ f is a smooth function and W ∗ is a local (global) optimal to f
[Equation]
then we have
[Equation]
where the last equality comes from MVT in Lemma 8.
For notational convenience [Equation] we use H ( (cid:96),t ) to denote the integration as
[Equation]
Then, we have
[Equation]
Let H ( (cid:96),t ) =
S Λ S T be the eigen-decomposition of H ( (cid:96),t ) .
Then, we deﬁne
A
[Equation]
T
[Equation]
Since (cid:34) S
S (cid:35) (cid:34) S T
S T (cid:35) = (cid:34) I
I (cid:35) , we know A ( β ) and (cid:34) I − η Λ + β I β I
I (cid:35) share the same eigenvalues.
Let γ ( Λ ) i be the i -th eigenvalue of ∇ 2 f ( (cid:98) w ( t ) ) , then the corresponding i -th eigenvalue of (40), denoted by γ ( A ) i , satisﬁes
[Equation]
[Equation]
By simple calculation, we have
[Equation] [Equation]
otherwise . Speciﬁcally, we have γ ( A ) i (0) > γ ( A ) i ( β ) , for ∀ β ∈ (cid:0) 0 , (1 − ηγ ( Λ ) i ) 2 (cid:1) , (43) and γ ( A ) i achieves the minimum γ ( A ) ∗ i =
(cid:12)(cid:12)(cid:12) 1 − (cid:113) ηγ ( Λ ) i (cid:12)(cid:12)(cid:12)
when β = (cid:16) 1 − (cid:113) ηγ ( Λ )
i (cid:17) 2 .
From Lemma 9, for any a ∈ R d with (cid:107) a (cid:107) 2 = 1 , we have a T
[Equation]
, a T ∇ f ( W ( (cid:96)
[Equation]
[Equation]
[Equation]
where γ max = 7( λδ 2 + (cid:101) λ ˜
δ 2 ) K , and γ min = λρ ( δ [Equation] + (cid:101) λρ [Equation]
Therefore, we have
[Equation]
and γ
[Equation]
Λ
[Equation]
Thus, we can select η = (cid:0)
[Equation]
(cid:1) 2 , and (cid:107) A ( β ) (cid:107) 2 can be bounded by
[Equation]
46) where µ ( δ, ˜ δ ) = (cid:16) λρ ( δ
[Equation]
+ (cid:101)
[Equation]
From Lemma 10, we have
[Equation]
Given ε > 0 and ˜ ε > 0 with ε + ˜ ε < 1 , let η
[Equation]
K
[Equation]
δ
[Equation] ˜ [Equation] 168 κ
[Equation]
γK , and η · (cid:101) λ ˜ δ K (cid:114) d log q
[Equation]
t
[Equation] ( δ [Equation] ˜ δ ) (cid:112) 168 κ 2 γK , (48) where we need
[Equation]
and M t ≥ ˜
ε − 2 µ −
[Equation]
(cid:101)
[Equation] ˜
[Equation]
λδ [Equation] (cid:101)
[Equation]
(cid:1)
[Equation]
κ
[Equation]
γK
[Equation]
d
[Equation]
q
[Equation]
Therefore, from (46), (47) and (48), we have
[Equation]
≤
[Equation]
[Equation]
γK
[Equation]
168 κ
[Equation]
γK (cid:17)
[Equation]
when M ≥ 4 d log q .
By mathematical induction on (50) over t
[Equation]
we have
[Equation]
By mathematical induction on (51) over (cid:96) , we have
[Equation]
F P ROOF OF T HEOREM
Instead of proving Theorem 1, we turn to prove a stronger version, as shown in Theorem 3.
One can verify that Theorem 1 is a special case of Theorem 3 by selecting ˆ λ in the order of p and (cid:101) ε is in the order of (2 p − 1) .
The major idea in proving Theorem 3 is similar to that of Theorem 2.
The ﬁrst step is to characterize the gradient descent term on the population risk function by the MVT in Lemma 8 as shown in (58) and (59).
Then, the connection between (cid:107) W ( (cid:96) +1 , 0) − W [ p ] (cid:107) F and (cid:107) W ( (cid:96), 0) − W
[ p ] (cid:107) F are characterized in (64).
Compared with proving Theorem 2, where the induction over (cid:96) holds naturally with large size of labeled data, the induction over (cid:96) requires a proper value of p as shown in (69).
By induction over (cid:96) on (64), the relative error (cid:107) W ( L, 0) − W [ p ] (cid:107) F can be characterized by (cid:107) W (0 , 0) −
W [ p ] (cid:107) F as shown in (71).
Theorem 3.
Suppose the initialization W (0 , 0) satisﬁes with
[Equation]
53) for some constant (cid:101)
ε ∈ (0 , 1 / 2) , where
[Equation]
[Equation]
and
[Equation]
Then [Equation] if the number of samples in (cid:101) D further satisﬁes
[Equation]
56) the iterates { W ( (cid:96),t ) } L,T(cid:96),t =0 converge to W [ p ] with p satisﬁes (53) as
[Equation]
with probability at least 1 − q − d .
Proof of Theorem 3.
From Algorithm 1, in the (cid:96) -th outer loop, we have
[Equation]
[Equation] Since ∇ f is a smooth function and W [ p ] is a local (global) optimal to f [Equation] then we have
[Equation]
where the last equality comes from Lemma 8.
Similar to the proof of Theorem 2, we have
[Equation]
From Lemma 2, we have
[Equation]
When (cid:96) =
[Equation]
following the similar steps from (41) to (46)
[Equation]
we have
[Equation]
) and
[Equation]
Therefore, we have lim T →∞
[Equation]
≤ µ (cid:112) 154 κ
[Equation]
γK 1 − ˜ ε · η · (cid:104)(cid:16) λδ 2 (1 − p )
K (cid:114) d log q N t + (cid:12)(cid:12) λδ 2 · (1 − p ) − (cid:101) λ ˜ δ 2 · p (cid:12)(cid:12) K (cid:17) · (cid:107) W (0 , 0) − W ∗ (cid:107)+ ˜ ε (cid:101) λ ˜ δ 2 · p
K · (cid:114) d log q
M t · (cid:107) W (0 , 0) − W ∗ (cid:107) 2 (cid:105)
[Equation]
(cid:112) 154 κ 2 γK 1 −
[Equation]
ε ·
[Equation]
[Equation]
+ (cid:101)
[Equation]
) · (cid:104)(cid:16) λδ 2 (1 − p )
[Equation]
(cid:114)
[Equation]
q
N t + (cid:12)(cid:12) λδ 2 · (1 − p ) − (cid:101) λ
˜ δ 2 · p (cid:12)(cid:12)
K (cid:17) · (cid:107) W (0 , 0)
[Equation]
W ∗ (cid:107) 2 + ˜ ε (cid:101) λ ˜ δ 2 · p K · (cid:114) d log q M t · (cid:107) W (0 , 0) − W ∗ (cid:107) 2 (cid:105) (cid:39)1 − ˜ ε ·
[Equation]
1 − p
[Equation]
K · (cid:12)(cid:12) (1 − p ) µ ˆ λ − pµ
[Equation]
1 − ˆ λ ) (cid:12)(cid:12)(cid:17) · (cid:107) [Equation] [Equation] 0) −
[Equation]
∗ (cid:107)+ ˜
[Equation]
[Equation]
=1 − ˜ ε · (cid:16) 1 −
[Equation]
µ √ K [Equation] where ˆ λ = λδ
[Equation]
λδ [Equation] To guarantee the convergence in the outer loop, we require
[Equation]
and
[Equation]
Since we have
[Equation]
it is clear that (65) holds if and only if
[Equation]
[Equation]
To guarantee the iterates strictly converges to the desired point
[Equation]
we let [Equation] [Equation] for some larger constant C
[Equation]
which is equivalent to
[Equation]
[Equation]
To make the bound in (69) meaningful, we need
[Equation]
[Equation] When (cid:96) > 1 , following similar steps in (64), we have
[Equation]
Given (69) holds
[Equation]
from (71) [Equation] we have
[Equation]
[Equation]
G D EFINITION AND RELATIVE PROOFS OF ρ
In this section, the formal deﬁnition of ρ is included in Deﬁnition [Equation] and a corresponding claim about ρ is summarized in Lemma 11.
One can quickly check that the ReLU activation function satisﬁes the conditions in Lemma 11.
The major idea in proving Lemma 11 is to show H r ( δ ) and J r ( δ ) in Deﬁnition 3 are in the order of δ r when δ is small.
Deﬁnition 3.
Let H r ( δ ) =
E z ∼N (0 ,δ 2 ) (cid:0) φ (cid:48) ( σ K z ) z r (cid:1) and J r ( δ ) =
E z ∼N (0 ,δ 2 ) (cid:0) φ (cid:48) 2 ( σ K z ) z r (cid:1) .
Then, ρ = ρ ( δ ) is deﬁned as
[Equation]
where σ K is the minimal singular value of W ∗ .
Lemma 11 (Order analysis of ρ ) .
If ρ ( δ ) > 0 for δ ∈ (0 , ξ ) for some positive constant ξ and the sub-gradient of ρ ( δ ) at 0 can be non-zero, then ρ ( δ ) =
Θ( δ 2 ) when δ → 0 + .
Typically, for ReLU activation function, µ in (5) is a ﬁxed constant for all δ, ˜ δ ≤ 1 .
Proof of Lemma 11 .
From Deﬁnition 3, we know that H r ( δ ) =
E z ∼N (0 ,δ 2 ) φ (cid:48) ( σ K z ) z r .
Suppose we have H r ( δ ) =
Θ( δ r ) and J r ( δ ) =
Θ( δ r ) , then from (73) we have
[Equation]
Because ρ is a continuous function with ρ ( z ) > 0 for some z > 0 .
Therefore, ρ (cid:54) = J 0 ( δ ) − H 20 ( δ ) − H 21 ( δ ) when δ → 0 + , otherwise ρ ( z ) < 0 for any z > 0 .
When δ → 0 + , both J 2 ( δ ) − H 21 ( δ ) −
H 22 ( δ ) and H 0 ( δ ) · H 2 ( δ ) − H 21 ( δ ) are in the order of δ 2 , which indicates that µ is a ﬁxed constant when both δ and ˜ δ are close to 0 .
In addition, J 2 ( δ ) − H 21 ( δ ) − H 22 ( δ ) goes to + ∞ while both J 0 ( δ ) − H 20 ( δ ) − H 21 ( δ ) and H 0 ( δ ) · H 2 ( δ ) − H 21 ( δ ) go to −∞ when δ → + ∞ .
Therefore, with a large enough δ , we have [Equation]
which indicates that µ is a strictly decreasing function when δ and ˜ δ are large enough.
Next, we provide the conditions that guarantee H r ( δ ) =
Θ( δ r ) hold, and the relative proof for J r ( δ ) can be derived accordingly following the similar steps as well.
From Deﬁnition 3, we have
[Equation]
where equality ( a ) holds by letting t = zδ .
It is easy to verify that
[Equation]
and both are bounded for a ﬁxed r .
Thus, as long as either φ (cid:48) (0 − ) or φ (cid:48) (0 + ) is non-zero, we have H r ( δ ) =
Θ( δ r ) when δ → 0 + .
If φ has bounded gradient as | φ (cid:48) | ≤ C φ for some positive constant C φ .
Then, we have
[Equation]
Therefore, we have H r ( δ ) =
O ( δ r ) for all δ > 0 when φ has bounded gradient.
Typiclly, for ReLU function, one can directly calculate that H r ( δ ) =
δ r for δ ∈ R , and ρ ( δ ) =
Cδwhen δ ≤ 1 for some constant C = 0 .
Then, it is easy to check that µ is a constant when δ, ˜ δ ≤ 1 .
H P ROOF OF PRELIMINARY LEMMAS H. P ROOF OF L EMMA
The eigenvalues of ∇ 2 f ( · ; p ) at any ﬁxed point W can be bounded in the form of (80) by Weyl’s inequality (Lemma 4).
Therefore, the primary technical challenge lies in bounding (cid:107)∇ 2 f ( W ; p ) − ∇
f ( W [ p ] ; p ) (cid:107) 2 , which is summarized in Lemma 12.
Lemma 13 provides the exact calulation of the lower bound of E x (cid:16) (cid:80) [Equation]
x )
[Equation]
2 when x belongs to Gaussian distribution with zero mean, which is used in proving the lower bound of the Hessian matrix in (81).
Lemma 12.
Let f ( W ; p ) be the population risk function deﬁned in (17) with p and W satisfying (20) .
Then, we have
[Equation]
Lemma 13 (Lemma D.6, (Zhong et al., 2017)) .
For any { w j } K j =1 ∈ R d , let α ∈ R
dK be the unit vector deﬁned in (19) .
When the φ is ReLU function, we have
[Equation]
where ρ ( σ ) is deﬁned in Deﬁnition 3.
Proof of Lemma 1.
Let λ max ( W ) and λ min ( W ) denote the largest and smallest eigenvalues of ∇ 2 f ( W ; p ) at point W , respectively.
Then, from Lemma [Equation] we have
[Equation]
Then
[Equation]
we provide the lower bound of the Hessian matrix of the population function at W [ p ] .
For any α ∈ R dK deﬁned in (19) with (cid:107) α (cid:107) 2 = [Equation] we have
[Equation]
where the last inequality comes from Lemma
[Equation]
Next
[Equation]
the upper bound can be bounded as
[Equation]
For E
[Equation]
, we have
[Equation]
[Equation]
Therefore
[Equation]
we have
[Equation]
Then, given (20), we have
[Equation]
Combining (85) and Lemma 12, we have
[Equation]
[Equation]
Therefore, (86) and (80) completes the whole proof.
H. P ROOF OF L EMMA
The task of bounding of the quantity between (cid:107)∇ ˆ f − ∇ f (cid:107) 2 is dividing into bounding I 1 , I 2 , I [-
-]3 and
I 4 as shown in (89).
I
and I 3 represent the deviation of the mean of several random variables to their expectation, which can be bounded through concentration inequality, i.e, Chernoff bound.
[-I 
 and-]Iand I 4 come from the inconsistency of the output label y and
pseudo label (cid:101) y in the empirical risk function in (1) and population risk function in (17).
The major challenge lies in characterizing the upper bound of I
and I 4 as the linear function of (cid:102) p and p
W ∗
, which is summarized in (96).
Proof of Lemma 2.
From (1), we know that
[Equation]
From (32), we know that
[Equation]
Then, from (17), we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Equation]
I
[Equation]
For any α j ∈ R d with (cid:107) α j (cid:107) 2 ≤ 1 , we deﬁne a random variable Z ( j )
=
(cid:0) φ ( w Tj x ) − φ ( w ∗ Tj x ) (cid:1) α Tj x and Z n ( j ) = (cid:0) φ ( w Tj x n ) − φ ( w ∗ Tj x n ) (cid:1)
α Tj x n as the realization of Z ( j ) for n = 1 , 2 · · · , N .
Then, for any p ∈ N + , we have
[Equation]
where C is a positive constant and the last inequality holds since x ∼ N (0 , δ 2 ) .
From Deﬁnition 2, we know that Z belongs to sub-exponential distribution with (cid:107) Z (cid:107) ψ 1 (cid:46) δ 2 (cid:107) w j − w ∗ j (cid:107) 2 .
Therefore, by Chernoff inequality, we have
[Equation]
for some positive constant C and any s ∈ R .
Let t = δ
[Equation]
(cid:107)
[Equation]
q N and s =
[Equation] [Equation]
w j − w ∗ j
[Equation]
2 · t for some large constant q > 0 , we have
[Equation]
92) with probability at least 1 − q − d .
From Lemma 7, we have
[Equation]
with probability at least 1 − ( q/ 5) − d .
Since q is a large constant [Equation] we release the probability as 1 − q − d for simpliﬁcation.
Similar to Z , we have
[Equation]
w Tj (cid:101) x ) (cid:1)(cid:101) x (cid:13)(cid:13)(cid:13)(cid:46) ˜ δ 2 (cid:107) w j − (cid:101) w j (cid:107) 2 · (cid:114) d log q M (94) with probability at least 1 − q − d . 𝒘𝒘 𝑗𝑗∗ 𝜃𝜃
II I III
IV-A 𝜃𝜃𝒘𝒘 𝑗𝑗
[ 𝑝𝑝 ]
For term E
[Equation]
x ) (cid:1) x (cid:105) , let us deﬁne the angle between w ∗ j and w [ p ] j as θ 1 .
Figure 13 shows the subspace spanned by the vector w ∗ j and (cid:101) w j .
We divide the subspace by 4 pieces, where the gray region denotes area I, and the blue area denotes area II.
Areas III and IV are the symmetries of II and I from the origin, respectively.
Hence, we have
[Equation]
Therefore, we have
[Equation]
From (93), (94) and (96), we have
[Equation]
[Table]
[Equation]
with probability at least 1 − q − d .
In conclusion
[Equation]
let α ∈ R Kd and α j ∈ R d with α
=
[ α T 1 , α T 2 , · · · , α TK ] T , we have
[Equation]
with probability at least 1 − q − d .
H. P ROOF OF L EMMA
The distance of the second order derivatives of the population risk function f ( · ; p ) at point W and W [ p ] can be converted into bounding P 1 , P 2 , P 3 and P 4 , which are deﬁned in (101).
The major idea in proving P 1 is to connect the error bound to the angle between W and W [ p ] .
Similar ideas apply in bounding the other three items as well.
Proof of Lemma 12.
From (17) [Equation] we have
[Equation]
and
[Equation]
where w [ p ] j is the j -th column of W [ p ] .
Then, we have ∂ 2 f
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
For any a ∈ R d with (cid:107) a (cid:107) 2 = 1 , we have
[Equation]
where a ∈ R d .
Let I = φ (cid:48)
( w [ p ] T j 1 x ) (cid:0) φ (cid:48) ( w [ p ] T j 2 x ) − φ (cid:48) ( w Tj 2 x ) (cid:1) · ( a T x ) 2 .
It is easy to verify there exists a group of orthonormal vectors such that B
= { a , b , c , a ⊥ 4 , · · · , a ⊥ d } with { a , b , c } spans a subspace that contains a , w j 2 and w ∗ j 2 .
Then, for any x , we have a unique z = (cid:104) z 1 , z 2 , · · · , z d (cid:105) T such that
[Equation]
Also, since x ∼ N ( 0 , δ 2 I d ) , we have z ∼ N ( 0 , δ 2 I d ) .
Then, we have
[Equation]
where x = z 1 a + z 2 b + z 3 c and f Z ( z 1 , z 2 , z 3 ) is probability density function of ( z 1 , z 2 , z 3 ) .
Next, we consider spherical coordinates with z 1 =
Rcosφ 1 , z 2 = Rsinφ 1 sinφ 2 , z 3 = Rsinφ 1 cosφ 2 .
Hence,
[Equation]
It is easy to verify that φ (cid:48) (cid:0) w Tj 2 x (cid:1) only depends on the direction of x and
[Equation]
only depends on R .
Then, we have
[Equation]
where the inequality (a) is derived from the fact that | cosφ 1 | ≤ 1 and (cid:90)
[Equation]
Deﬁne a set
A 1 = { x | ( w [ p ] T j 2 x )( w Tj 2 x ) < 0 } .
If x ∈ A 1 , then w [ p ] T j 2 x and w Tj 2 x have different signs, which means the value of φ (cid:48) ( w Tj 2 x ) and φ (cid:48) ( w [ p ] T j 2 x ) are different.
This is equivalent to say that
[Equation]
Moreover, if x ∈ A 1 , then we have
[Equation]
Let us deﬁne a set A 2 such that
[Equation]
Hence, we have that
[Equation]
Since x ∼ N ( 0 , δ 2 (cid:107) a (cid:107)
I
) , θ x , w [ p ] j 2 belongs to the uniform distribution on [ − π, π ] , we have
[Equation]
Hence, (104) and (110) suggest that
[Equation]
The same bound that shown in (111) holds for P 2 as well.
P 3 and P 4 satisfy (111) except for changing δ 2 to ˜
δ 2 .
Therefore
[Equation]
we have
[Equation]
where α ∈ R Kd and α j ∈ R d with α =
[ α T 1 , α T 2 , · · · , α TK ] T .
I I NITIALIZATION VIA TENSOR METHOD
In this section, we brieﬂy summarize the tensor initialization in (Zhong et al., 2017) by studying the target function class as
[Equation]
where v ∗ j ∈ R .
Note that for ReLU function, we have v ∗ j φ ( w ∗ Tj x )
= sign ( v ∗ j ) φ ( | v ∗ j | w ∗ Tj x ) .
Without loss of generalization, we can assume v ∗ j ∈ { +1 , − 1 } .
Additionally, it is clear that the function studied in (2) is the special case of (113) when v ∗ j = 1 for all j .
In addition, Theorem 5. in (Zhong et al., 2017) show that the sign of v ∗ j can be directly recovered using tensor initialization, which indicates the the equivalence of (2) and (113) when using tensor initialization.
We ﬁrst deﬁne some high order momenta in the following way:
[Equation]
[Equation]
where E x is the expectation over x and z ⊗ 3 :
= z ⊗ z ⊗ z .
The operator (cid:101) ⊗ is deﬁned as
[Equation]
for any vector v ∈ R d 1 and Z ∈ R d 1 × d 2 .
Following the same calculation formulas in the Claim 5.2 (Zhong et al., 2017), there exist some known constants ψ i , i = 1 , 2 , 3 , such that
[Equation]
where w ∗ j = w ∗ j / (cid:107) w ∗ j (cid:107) 2 in (114)-(116) is the normalization of w ∗ j .
Therefore, we can see that the information of { w ∗ j } Kj =1 are separated as the direction of w j and the magnitude of w j in M 1 , Mand M 3 .
M 1 , M 2 and M 3 can be estimated through the samples (cid:8) ( x n , y n ) (cid:9) N [Table] = [Table]
and let (cid:99) M [-
-][Table] , (cid:99) M [Table]
, (cid:99) Mdenote the corresponding estimates.
First, we will decompose the rank- K tensor M 3 and obtain the { w ∗ j } Kj =1 .
By applying the tensor decomposition method (Kuleshov et al., 2015) to (cid:99) M 3 , the outputs, denoted by (cid:98) w ∗ j , are the estimations of { s j w ∗ j } Kj =1 , where s j is an unknown sign.
Second, we will estimate s j , v ∗ j and (cid:107) w ∗ j (cid:107) 2 through M 1 and M 2 .
Note that M 2 does not contain the information of s j because s 2 j is always 1 .
Then, through solving the following two optimization problem:
[Equation]
The estimation of s j can be given as
[Equation]
Also, we know that | (cid:98) α 1 ,j | is the estimation of (cid:107) w ∗ j (cid:107) and
ˆ v j = sign ( (cid:98) α 1 ,j /s j ) = sign ( (cid:98) α 2 ,j ) .
Thus, W (0) is given as
[Equation]
Subroutine 1 Tensor Initialization Method 1: Input: labeled data D = { ( x n , y n ) } Nn =1 ; 2: Partition D into three disjoint subsets D 1 , D 2 , D 3 ; 3: Calculate (cid:99) M 1 , (cid:99) M 2 following (114), (115) using D 1 , D 2 , respectively; 4: Obtain the estimate subspace (cid:98) V of (cid:99) M 2 ; 5: Calculate (cid:99) M 3 ( (cid:98) V , (cid:98) V , (cid:98) V ) through D 3 ; α = sign
[Equation]
,j ) ,
[Equation]
[Equation]
[Equation] [Equation] K . To reduce the computational complexity of tensor decomposition, one can project (cid:99) M 3 to a lowerdimensional tensor (Zhong et al., 2017).
The idea is to ﬁrst estimate the subspace spanned by { w ∗ j } Kj =1 , and let (cid:98) V denote the estimated subspace.
Moreover, we have
[Equation]
[Equation] Then, one can decompose the estimate (cid:99) M 3 ( (cid:98) V , (cid:98) V , (cid:98) V ) to obtain unit vectors { ˆ s j } Kj =1 ∈ R K .
Since w ∗ lies in the subspace V , we have V V T w ∗ j = w ∗ j .
Then, (cid:98) V ˆ s j is an estimate of w ∗ j .
The initialization process is summarized in Subroutine 1.
J C LASSIFICATION P ROBLEMS
The framework in this paper is extendable to binary classiﬁcation problem. For binary classiﬁcation problem, the output y given input x is deﬁned as [Equation]
[Equation]
with some ground truth parameter W ∗ .
To guarantee the output is within [0 , 1] , the activation function is often used as sigmoid.
For classiﬁcation, the loss function is cross-entropy, and the objective function over labeled data D is deﬁned as
[Equation]
The expectation of objective function can be written as
[Equation]
Please note that (125) is exactly the same as (32) with λ = 1 when the loss function is squared loss.
For cross entropy loss function, the second order derivative of (125) is calculated as
[Equation]
when j (cid:54) =
k .
Refer to (88) in (Fu et al., 2020) or (132) in (Zhang et al., 2020a), we have
[Equation]
Following similar steps in (90), from Deﬁntion 2, we know that α Tj ∂f D ( W ) ∂ w j ∂ w k α k belongs to the subexponential distribution.
Therefore, similiar results for objective function with cross-entropy loss can be established as well.
One can check (Fu et al., 2020) or (Zhang et al., 2020a) for details.
K O NE - HIDDEN LAYER NEURAL NETWORK WITH TOP LAYER WEIGHTS
For a general one-hidden layer neural network, the output of the neural network is deﬁned as
[Equation]
(128) where v = [ v 1 , v 2 , · · · , v K ] ∈ R K .
Then, the target function can be deﬁned as
[Equation]
(129) for some unknown weights W ∗ and v ∗ .
In the following paragraphs, we will provide a short description for the equivalence of (129) and (2) in theoretical analysis.
Note that for ReLU functions, we have v j φ ( w T j x ) =
sign ( v j ) φ ( | v j | w T j x ) .
Without loss of generalization, we can assume v j , v ∗ j ∈ { +1 , − 1 } for all j ∈ [ K ] 5 .
From Appendix I, we know that the sign of v ∗ j can exactly estimated through tensor initialization.
There, we can focus on analysis the neural network in the form as
[Equation]
Considering the objective function in (1) and population risk function in (17)
[Equation]
we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] 131) which is exact the same as (89).
Similar results can be derived for Lemma 12.
Therefore, the conclusions and proofs of Lemma 1 and Lemma 2 does not change at all.
Additionally, ﬁxing the second-layer weights and only training the hidden layer is the state-of-theart practice in analyzing two-layer neural networks (Arora et al., 2019b;a; Allen-Zhu et al., 2019; Safran & Shamir, 2018; Li & Liang, 2018; Brutzkus & Globerson, 2017; Oymak & Soltanolkotabi, 2018; Zhang et al., 2019).
Additionally, as indicated in (Safran & Shamir, 2018), training a onehidden-layer neural network with all v j ﬁxed as 1 has intractable many spurious local minima, which indicates that training problem is not trivial.