T HEORETICAL A NALYSIS OF S ELF -T RAINING WITH D EEP N ETWORKS ON U NLABELED D ATA
A BSTRACT
Self-training algorithms, which train a model to ﬁt pseudolabels predicted by another previously-learned model, have been very successful for learning with unlabeled data using neural networks.
However, the current theoretical understanding of self-training only applies to linear models.
This work provides a uniﬁed theoretical analysis of self-training with deep networks for semi-supervised learning, unsupervised domain adaptation, and unsupervised learning.
At the core of our analysis is a simple but realistic “expansion” assumption, which states that a lowprobability subset of the data must expand to a neighborhood with large probability relative to the subset.
We also assume that neighborhoods of examples in different classes have minimal overlap.
We prove that under these assumptions, the minimizers of population objectives based on self-training and input-consistency regularization will achieve high accuracy with respect to ground-truth labels.
By using off-the-shelf generalization bounds, we immediately convert this result to sample complexity guarantees for neural nets that are polynomial in the margin and Lipschitzness.
Our results help explain the empirical successes of recently proposed self-training algorithms which use input consistency regularization
I NTRODUCTION
Though supervised learning with neural networks has become standard and reliable, it still often requires massive labeled datasets.
As labels can be expensive or difﬁcult to obtain, leveraging unlabeled data in deep learning has become an active research area.
Recent works in semi-supervised learning (Chapelle et al., 2010; Kingma et al., 2014; Kipf & Welling, 2016; Laine & Aila, 2016;
Sohn et al., 2020; Xie et al., 2020) and unsupervised domain adaptation (Ben-David et al., 2010;
Ganin & Lempitsky, 2015; Ganin et al., 2016; Tzeng et al., 2017; Hoffman et al., 2018; Shu et al., 2018; Zhang et al., 2019) leverage lots of unlabeled data as well as labeled data from the same distribution or a related distribution.
Recent progress in unsupervised learning or representation learning (Hinton et al., 1999; Doersch et al., 2015; Gidaris et al., 2018; Misra & Maaten, 2020;
Chen et al., 2020a;b; Grill et al., 2020) learns high-quality representations without using any labels.
Self-training is a common algorithmic paradigm for leveraging unlabeled data with deep networks.
Self-training methods train a model to ﬁt pseudolabels, that is, predictions on unlabeled data made by a previously-learned model (Yarowsky, 1995; Grandvalet & Bengio, 2005; Lee, 2013).
Recent work also extends these methods to enforce stability of predictions under input transformations such as adversarial perturbations (Miyato et al., 2018) and data augmentation (Xie et al., 2019).
These approaches, known as input consistency regularization, have been successful in semi-supervised learning (Sohn et al., 2020; Xie et al., 2020), unsupervised domain adaptation (French et al., 2017;
Shu et al., 2018), and unsupervised learning (Hu et al., 2017; Grill et al., 2020).
Despite the empirical successes, theoretical progress in understanding how to use unlabeled data has lagged.
Whereas supervised learning is relatively well-understood, statistical tools for reasoning about unlabeled data are not as readily available.
Around 25 years ago, Vapnik (1995) proposed the transductive SVM for unlabeled data, which can be viewed as an early version of self-training, yet there is little work showing that this method improves sample complexity (Derbeko et al., 2004).
Working with unlabeled data requires proper assumptions on the input distribution (Ben-David et al., 2008).
Recent papers (Carmon et al., 2019; Raghunathan et al., 2020; Chen et al., 2020c; Kumar et al., 2020; Oymak & Gulcu, 2020) analyze self-training in various settings, but mainly for linear models and often require that the data is Gaussian or near-Gaussian. Kumar et al.
(2020) also analyze self-training in a setting where gradual domain shift occurs over multiple timesteps but assume a small Wasserstein distance bound on the shift between consecutive timesteps.
Another line of work leverages unlabeled data using non-parametric methods, requiring unlabeled sample complexity that is exponential in dimension (Rigollet, 2007; Singh et al., 2009; Urner & Ben-David, 2013).
This paper provides a uniﬁed theoretical analysis of self-training with deep networks for semisupervised learning, unsupervised domain adaptation, and unsupervised learning.
Under a simple and realistic expansion assumption on the data distribution, we show that self-training with input consistency regularization using a deep network can achieve high accuracy on true labels, using unlabeled sample size that is polynomial in the margin and Lipschitzness of the model.
Our analysis provides theoretical intuition for recent empirically successful self-training algorithms which rely on input consistency regularization (Berthelot et al., 2019; Sohn et al., 2020; Xie et al., 2020).
Our expansion assumption intuitively states that the data distribution has good continuity within each class.
Concretely, letting P i be the distribution of data conditioned on class i , expansion states that for small subset S of examples with class i ,
[Equation]
where and c > 1 is the expansion factor.
The neighborhood will be deﬁned to incorporate data augmentation, but for now can be thought of as a collection of points with a small (cid:96) 2 distance to S .
This notion is an extension of the Cheeger constant (or isoperimetric or expansion constant) (Cheeger, 1969) which has been studied extensively in graph theory (Chung & Graham, 1997), combinatorial optimization (Mohar & Poljak, 1993; Raghavendra & Steurer, 2010), sampling (Kannan et al., 1995; Lovász & Vempala, 2007; Zhang et al., 2017), and even in early versions of self-training (Balcan et al., 2005) for the co-training setting (Blum & Mitchell, 1998).
Expansion says that the manifold of each class has sufﬁcient connectivity, as every subset S has a neighborhood larger than S .
We give examples of distributions satisfying expansion in Section 3.1.
We also require a separation condition stating that there are few neighboring pairs from different classes.
Our algorithms leverage expansion by using input consistency regularization (Miyato et al., 2018;
Xie et al., 2019) to encourage predictions of a classiﬁer G to be consistent on neighboring examples:
[Equation]
For unsupervised domain adaptation and semi-supervised learning, we analyze an algorithm which ﬁts G to pseudolabels on unlabeled data while regularizing input consistency.
Assuming expansion and separation, we prove that the ﬁtted model will denoise the pseudolabels and achieve high accuracy on the true labels (Theorem 4.3).
This explains the empirical phenomenon that self-training on pseudolabels often improves over the pseudolabeler, despite no access to true labels.
For unsupervised learning, we consider ﬁnding a classiﬁer G that minimizes the input consistency regularizer with the constraint that enough examples are assigned each label.
In Theorem 3.6, we show that assuming expansion and separation, the learned classiﬁer will have high accuracy in predicting true classes, up to a permutation of the labels (which can’t be recovered without true labels).
The main intuition of the theorems is as follows: input consistency regularization ensures that the model is locally consistent, and the expansion property magniﬁes the local consistency to global consistency within the same class.
In the unsupervised domain adaptation setting, as shown in
Figure 1 (right), the incorrectly pseudolabeled examples (the red area) are gradually denoised by their correctly pseudolabeled neighbors (the green area), whose probability mass is non-trivial (at least c − 1 times the mass of the mistaken set by expansion).
We note that expansion is only required on the population distribution, but self-training is performed on the empirical samples.
Due to the extrapolation power of parametric methods, the local-to-global consistency effect of expansion occurs implicitly on the population.
In contrast, nearest-neighbor methods would require expansion to occur explicitly on empirical samples, suffering the curse of dimensionality as a result.
We provide more details below, and visualize this effect in Figure 1 (left).
To our best knowledge, this paper gives the ﬁrst analysis with polynomial sample complexity guarantees for deep neural net models for unsupervised learning, semi-supervised learning, and unsuper- vised domain adaptation.
Prior works (Rigollet, 2007; Singh et al., 2009; Urner & Ben-David, 2013) analyzed nonparametric methods that essentially recover the data distribution exactly with unlabeled data but require sample complexity exponential in dimension.
Our approach optimizes parametric loss functions and regularizers, so guarantees involving the population loss can be converted to ﬁnite sample results using off-the-shelf generalization bounds (Theorem 3.7).
When a neural net can separate ground-truth classes with large margin, the sample complexities from these bounds can be small, that is, polynomial in dimension.
Finally, we note that our regularizer R ( · ) corresponds to enforcing consistency w.r.t. adversarial examples, which was shown to be empirically helpful for semi-supervised learning (Miyato et al., 2018; Qiao et al., 2018) and unsupervised domain adaptation (Shu et al., 2018).
Moreover, we can extend the notion of neighborhood in (1.1) to include data augmentations of examples, which will increase the neighborhood size and therefore improve the expansion.
Thus, our theory can help explain empirical observations that consistency regularization based on aggressive data augmentation or adversarial training can improve performance with unlabeled data (Shu et al., 2018; Xie et al., 2019; Berthelot et al., 2019; Sohn et al., 2020; Xie et al., 2020;
Chen et al., 2020a).
In summary, our contributions include: 1) we propose a simple and realistic expansion assumption which states that the data distribution has connectivity within the manifold of a ground-truth class 2) using this expansion assumption, we provide ground-truth accuracy guarantees for self-training algorithms which regularize input consistency on unlabeled data, and 3) our analysis is easily applicable to deep networks with polynomial unlabeled samples via off-the-shelf generalization bounds.
1. A DDITIONAL RELATED WORK
Self-training via pseudolabeling (Lee, 2013) or min-entropy objectives (Grandvalet & Bengio, 2005) has been widely used in both semi-supervised learning (Laine & Aila, 2016; Tarvainen & Valpola, 2017; Iscen et al., 2019; Yalniz et al., 2019; Berthelot et al., 2019; Xie et al., 2020; Sohn et al., 2020) and unsupervised domain adaptation (Long et al., 2013; French et al., 2017; Saito et al., 2017; Shu et al., 2018; Zou et al., 2019).
Our paper studies input consistency regularization, which enforces stability of the prediction w.r.t transformations of the unlabeled data.
In practice, these transformations include adversarial perturbations, which was proposed as the VAT objective (Miyato et al., 2018), as well as data augmentations (Xie et al., 2019).
For unsupervised learning, our self-training objective is closely related to BYOL (Grill et al., 2020), a recent state-of-the-art method which trains a student model to match the representations predicted by a teacher model on strongly augmented versions of the input.
Contrastive learning is another popular method for unsupervised representation learning which encourages representations of “positive pairs”, ideally consisting of examples from the same class, to be close, while pushing negative pairs far apart (Mikolov et al., 2013; Oord et al., 2018; Arora et al., 2019).
Recent works in contrastive learning achieve state-of-the-art representation quality by using strong data augmentation to form positive pairs (Chen et al., 2020a;b).
The role of data augmentation here is in spirit similar to our use of input consistency regularization.
Less related to our setting are algorithms which learn representations by solving self-supervised pretext tasks, such as inpainting and predicting rotations (Pathak et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018).
Lee et al.
(2020) theoretically analyze self-supervised learning, but their analysis applies to a different class of algorithms than ours.
Prior theoretical works analyze contrastive learning by assuming access to document data distributed according to a particular topic modeling setup (Tosh et al., 2020) or pairs of independent samples within the same class (Arora et al., 2019).
However, the assumptions required for these analyses do not necessarily apply to vision, where positive pairs apply different data augmentations to the same image, and are therefore strongly correlated.
Other papers analyze information-theoretic properties of representation learning (Tian et al., 2020; Tsai et al., 2020).
Prior works analyze continuity or “cluster” assumptions for semi-supervised learning which are related to our notion of expansion (Seeger, 2000; Rigollet, 2007; Singh et al., 2009; Urner & BenDavid, 2013).
However, these papers leverage unlabeled data using non-parametric methods, requiring unlabeled sample complexity that is exponential in the dimension.
On the other hand, our analysis is for parametric methods, and therefore the unlabeled sample complexity can be low when
Co-training is a classical version of self-training which requires two distinct “views” (i.e., feature subsets) of the data, each of which can be used to predict the true label on its own (Blum & Mitchell, 1998; Dasgupta et al., 2002; Balcan et al., 2005).
For example, to predict the topic of a webpage, one view could be the incoming links and another view could be the words in the page.
The original co-training algorithms (Blum & Mitchell, 1998; Dasgupta et al., 2002) assume that the two views are independent conditioned on the true label and leverage this independence to obtain accurate pseudolabels for the unlabeled data.
By contrast, if we cast our setting into the co-training framework by treating an example and a randomly sampled neighbor as the two views of the data, the two views are highly correlated.
Balcan et al.
(2005) relax the requirement on independent views of co-training, also by using an “expansion” assumption.
Our assumption is closely related to theirs and conceptually equivalent if we cast our setting into the co-training framework by treating neighboring examples are two views.
However, their analysis requires conﬁdent pseudolabels to all be accurate and does not rigorously account for potential propagation of errors from their algorithm.
In contrast, our contribution is to propose and analyze an objective function involving input consistency regularization whose minimizer denoises errors from potentially incorrect pseudolabels.
We also provide ﬁnite sample complexity bounds for the neural network hypothesis class and analyze unsupervised learning algorithms.
Alternative theoretical analyses of unsupervised domain adaptation assume bounded measures of discrepancy between source and target domains (Ben-David et al., 2010; Zhang et al., 2019).
Balcan & Blum (2010) propose a PAC-style framework for analyzing semi-supervised learning, but their bounds require the user to specify a notion of compatability which incorporates prior knowledge about the data, and do not apply to domain adaptation.
Globerson et al.
(2017) demonstrate semisupervised learning can outperform supervised learning in labeled sample complexity but assume full knowledge of the unlabeled distribution.
(Mobahi et al., 2020) show that for kernel methods, self-distillation, a variant of self-training, can effectively amplify regularization.
Their analysis is for kernel methods, whereas our analysis applies to deep networks under data assumptions.
P RELIMINARIES AND NOTATIONS
We let P denote a distribution of unlabeled examples over input space X .
For unsupervised learning,
P is the only relevant distribution.
For unsupervised domain adaptation, we also deﬁne a source distribution P src and let G pl denote a source classiﬁer trained on a labeled dataset sampled from P src .
To translate these deﬁnitions to semi-supervised learning, we set P src and P to be the same, except
P src gives access to labels.
We analyze algorithms which only depend on P src through G pl .
We consider classiﬁcation and assume the data is partitioned into K classes, where the class of x ∈ X is given by the ground-truth G (cid:63) ( x ) for G (cid:63) :
X → [ K ] .
We let P i denote the class-conditional distribution of x conditioned on G (cid:63) ( x ) =
i .
We assume that each example x has a unique label, so P i , P j have disjoint support for i (cid:54) =
j .
Let (cid:98) P (cid:44) { x 1 , . . .
, x n } ⊂ X denote n i.i.d.
unlabeled training examples from P .
We also use (cid:98) P to refer to the uniform distribution over these examples.
We let F : X → R K denote a learned scoring function (e.g. the continuous logits output by a neural network), and G : X → [ K ] the discrete labels induced by F : G ( x ) (cid:44) arg max i F ( x ) i (where ties are broken lexicographically).
Pseudolabels.
Pseudolabeling methods are a form of self-training for semi-supervised learning and domain adaptation where the source classiﬁer G pl : X → [ K ] is used to predict pseudolabels on the unlabeled target data (Lee, 2013).
These methods then train a fresh classiﬁer to ﬁt these pseudolabels, for example, using the standard cross entropy loss: L pl ( F ) (cid:44) E (cid:98) P [ (cid:96) cross-ent
( F ( x ) , G pl ( x ))] .
Our theoretical analysis applies to a pseudolabel-based objective.
Other forms of self-training include entropy minimization, which is closely related, and in certain settings, equivalent to pseudolabeling where the pseudolabels are updated every iteration (Lee, 2013; Chen et al., 2020c).
E XPANSION PROPERTY AND GUARANTEES FOR UNSUPERVISED LEARNING
In this section we will ﬁrst introduce our key assumption on expansion.
We then study the implications of expansion for unsupervised learning.
We show that if a classiﬁer is consistent w.r.t.
input transformations and predicts each class with decent probability, the learned labels will align with ground-truth classes up to permutation of the class indices (Theorem 3.6).
3. E XPANSION PROPERTY
We introduce the notion of expansion.
As our theory studies objectives which enforce stability to input transformations, we will ﬁrst model allowable transformations of the input x by the set B ( x ) , deﬁned below.
We let T denote some set of transformations obtained via data augmentation, and deﬁne B ( x ) (cid:44) { x (cid:48) :
∃ T ∈ T such that (cid:107) x (cid:48) − T ( x ) (cid:107) ≤ r } to be the set of points with distance r from some data augmentation of x .
We can think of r as a value much smaller than the typical norm of x , so the probability P ( B ( x )) is exponentially small in dimension.
Our theory easily applies to other choices of B , though we set this deﬁnition as default for simplicity.
Now we deﬁne the neighborhood of x , denoted by N ( x ) , as the set of points whose transformation sets overlap with that of x :
[Equation]
For S ⊆ X , we deﬁne the neighborhood of S as the union of neighborhoods of its elements: N ( S ) (cid:44) ∪ x ∈ S N ( x ) .
We now deﬁne the expansion property of the distribution P , which lower bounds the neighborhood size of low probability sets and captures connectivity of the distribution in input space.
Deﬁnition 3.1 ( ( a, c ) -expansion) .
We say that the class-conditional distribution P i satisﬁes ( a, c ) - expansion if for all V ⊆ X with P i ( V ) ≤ a , the following holds:
[Equation]
If P i satisﬁes ( a, c ) -expansion for all ∀ i ∈ [ K ] , then we say P satisﬁes ( a, c ) -expansion.
We note that this deﬁnition considers the population distribution, and expansion is not expected to hold on the training set, because all empirical examples are far away from each other, and thus the neighborhoods of training examples do not overlap.
The notion is closely related to the Cheeger constant , which is used to bound mixing times and hitting times for sampling from continuous distributions (Lovász & Vempala, 2007;
Zhang et al., 2017), and small-set expansion , which quantiﬁes connectivity of graphs (Hoory et al., 2006; Raghavendra & Steurer, 2010).
In particular, when the neighborhood is deﬁned to be the collection of points with (cid:96) 2 distance at most r from the set, then the expansion factor c is bounded below by exp( ηr ) , where η is the Cheeger constant (Zhang et al., 2017).
In Section E.1, we use GANs to demonstrate that expansion is a realistic property in vision.
For unsupervised learning, we require expansion with a = 1 / 2 and c > 1 :
Assumption 3.2 (Expansion requirement for unsupervised learning) .
We assume that P satisﬁes (1 / 2 , c ) -expansion on X for c > 1 .
We also assume that ground-truth classes are separated in input space.
We deﬁne the population consistency loss R B ( G ) as the fraction of examples where G is not robust to input transformations: [Equation] such that G
[Equation]
We state our assumption that ground-truth classes are far in input space below:
Assumption 3.3 (Separation) .
We assume P is B -separated with probability 1 − µ by ground-truth classiﬁer G (cid:63) , as follows: R B ( G (cid:63) ) ≤ µ .
Our accuracy guarantees in Theorems 4.3 and 3.6 will depend on µ .
We expect µ to be small or negligible (e.g. inverse polynomial in dimension).
The separation requirement requires the distance between two classes to be larger than 2 r , the (cid:96) 2 radius in the deﬁnition of B ( · ) .
However, r can be much smaller than the norm of a typical example, so our expansion requirement can be weaker than a typical notion of “clustering” which requires intra-class distances to be smaller than inter-class distances.
We demonstrate this quantitatively in the examples below.
As a warm-up, we start with mixture of Gaussians.
Example 3.4 (Mixture of isotropic Gaussians) .
Suppose P is a mixture of K Gaussians P i (cid:44)
N ( τ i , 1 d I d × d ) with isotropic covariance and K < d , corresponding to K separate classes.
1 Suppose the transformation set B ( x ) is an (cid:96) 2 -ball with radius 12 √ d around x , so there is no data augmentation and r = 12 √ d .
Then P satisﬁes (0 . 5 , 1 . 5) -expansion.
Furthermore, if the minimum distance between means satisﬁes min i,j (cid:107) τ i − τ j (cid:107) {+
+} (cid:38) √ log d √ d , then P is B -separated with probability 1 − 1 / poly ( d ) .
In the example above, the population distribution satisﬁes expansion, but the empirical distribution does not .
The minimum distance between any two empirical examples is Ω(1) with high probability, so they cannot be neighbors of each other when r = 12 √ d .
Furthermore, the intra-class distance, which is Ω(1) , is much larger than the distance between the means, which is assumed to be (cid:38) 1 / √ d .
Therefore, trivial distanced-based clustering algorithms on empirical samples do not apply.
Our unsupervised learning algorithm in Section 3.2 can approximately recover the mixture components with polynomial samples, up to O (1 / poly ( d )) error.
Furthermore, this is almost information-theoretically optimal: by total variation distance, Ω( 1 √ d ) distance between the means is required to recover the mixture components.
The example extends to log-concave distributions via more general isoperimetric inequalities (Bobkov et al., 1999).
Thus, our analysis applies to the setting of prior work (Chen et al., 2020c), which studied self-training with linear models on mixtures of Gaussian or log-concave distributions.
The main beneﬁt of our analysis, however, is that it holds for much richer family of distributions than Gaussians, compared to prior work on self-training which only considered Gaussian or nearGaussian distributions (Raghunathan et al., 2020; Chen et al., 2020c; Kumar et al., 2020).
We demonstrate this in the following mixture of manifolds example:
Example 3.5 (Mixture of manifolds) .
Suppose each class-conditional distribution P i over an ambient space R d (cid:48) , where d (cid:48) >
d , is generated by some κ -bi-Lipschitz 2 generator Q i : R d → R d (cid:48) on latent variable z ∈ R d :
[Equation]
We set the transformation set B ( x ) to be an (cid:96) 2 -ball with radius κ 2 √ d around x , so there is no data augmentation and r = κ 2 √ d .
Then, P satisﬁes (0 . 5 , 1 . 5) -expansion.
Figure 1 (right) provides a illustration of expansion on manifolds.
Note that as long as κ (cid:28) d 1 / 4 , the radius κ/ (2 √ d ) is much smaller than the norm of the data points (which is at least on the order of 1 /κ ).
This suggests that the generator can non-trivially scramble the space and still maintain meaningful expansion with small radius.
In Section C.2, we prove the claims made in our examples.
3. P OPULATION GUARANTEES FOR UNSUPERVISED LEARNING
We design an unsupervised learning objective which leverages the expansion and separation properties.
Our objective is on the population distribution, but it is parametric, so we can extend it to the ﬁnite sample case in Section 3.3.
We wish to learn a classiﬁer G : X → [ K ] using only unlabeled data, such that predicted classes align with ground-truth classes.
Note that without observing any labels, we can only learn ground-truth classes up to permutation, leading to the following permutation-invariant error deﬁned for a classiﬁer G :
[Equation]
We study the following unsupervised population objective over classiﬁers G : X → [ K ] , which encourages input consistency while ensuring that predicted classes have sufﬁcient probability.
[Equation]
Here c is the expansion coefﬁcient in Assumption 3.2.
The constraint ensures that the probability of any predicted class is larger than the input consistency loss.
Let ρ (cid:44) min y ∈
[ K ] P ( { x : G (cid:63) ( x ) = y } ) denote the probability of the smallest ground-truth class.
The following theorem shows that when P satisﬁes expansion and separation, the global minimizer of the objective (3.4) will have low error.
Theorem 3.6.
Suppose that Assumptions 3.2 and 3.3 hold for some c, µ such that ρ > max { 2 c − 1 , 2 } µ .
Then any minimizer (cid:98) G of (3.4) satisﬁes
[Equation]
In Section C, we provide the proof of Theorem 3.6 as well as a variant of the theorem which holds for a weaker additive notion of expansion.
The stronger variant is used in this section for ease of interpretation.
By applying the generalization bounds of Section 3.3, we can convert Theorem 3.
into a ﬁnite-sample guarantees that are polynomial in margin and Lipschitzness of the model (see
Theorem D.1).
Our objective is reminiscent of recent methods which achieve state-of-the-art results in unsupervised representation learning: SimCLR (Chen et al., 2020a), MoCov2 (He et al., 2020; Chen et al., 2020b), and BYOL (Grill et al., 2020).
Unlike our algorithm, these methods do not predict discrete labels, but rather, directly predict a representation which is consistent under input transformations, However, our analysis still suggests an explanation for why input consistency regularization is so vital for these methods: assuming the data satisﬁes expansion, it encourages representations to be similar over the entire class, so the representations will capture ground-truth class structure.
Chen et al.
(2020a) also observe that using more aggressive data augmentation for regularizing input stability results in signiﬁcant improvements in representation quality.
We remark that our theory offers a potential explanation: in our framework, strengthening augmentation increases the size of the neighborhood, resulting in a larger expansion factor c and improving the accuracy bound (3.5).
3. F INITE SAMPLE GUARANTEES FOR DEEP LEARNING MODELS
In this section, we show that if the ground-truth classes are separable by a neural net with large robust margin, then generalization can be good.
The main advantage of Theorem 3.6 and Theorem 4.
over prior work is that they analyze parametric objectives, so ﬁnite sample guarantees immediately hold via off-the-shelf generalization bounds.
Prior work on continuity or “cluster” assumptions related to expansion require nonparametric techniques with a sample complexity that is exponential in dimension (Seeger, 2000; Rigollet, 2007; Singh et al., 2009; Urner & Ben-David, 2013).
We apply the generalization bound of (Wei & Ma, 2019b) based on a notion of all-layer margin, though any other bound would work.
The all-layer margin measures the stability of the neural net to simultaneous perturbations to each hidden layer.
Formally, suppose that G ( x ) (cid:44) arg max i F ( x ) i is the prediction of some feedforward neural network F : X → R K which computes the following function: F ( x )
=
W p φ ( · · · φ ( W 1 x ) · · · ) with weight matrices { W i } p
i
=1 .
Let q denote the maximum dimension of any hidden layer.
Let m ( F, x, y ) ≥ 0 denote the all-layer margin at example x for label y , deﬁned formally in Section D.2.
For now, we simply note that m has the property that if
G ( x ) (cid:54) =
y , then m ( F, x, y )
= 0 , so we can upper bound the 0-1 loss by thresholding the all-layer margin: 1 ( G ( x ) (cid:54) =
y ) ≤ 1 ( m ( F, x, y ) ≥ t ) for any t > 0 .
We can also deﬁne a variant that measures robustness to input transformations: m B ( F, x ) (cid:44) min x (cid:48) ∈B ( x ) m ( F, x (cid:48) , arg max i F ( x ) i ) .
The following result states that large all-layer margin implies good generalization for the input consistency loss, which appears in the objective (3.4).
Theorem 3.7 (Extension of Theorem 3.1 of (Wei & Ma, 2019b)) .
With probability 1 − δ over the draw of the training set (cid:98) P , all neural networks G = arg max
i F i of the form F ( x ) (cid:44)
W p φ ( · · · φ ( W 1 x )) will satisfy
[Equation]
for all choices of t > 0 , where ζ (cid:44) O
(cid:16)(cid:112) (log(1 /δ )
+ p log n ) /n
(cid:17) is a low-order term, and (cid:101) O ( · ) hides poly-logarithmic factors in n and d .
A similar bound can be expressed for other quantities in (3.4), and is provided in Section D.2.
In
Section D.1, we plug our bounds into Theorem 3.6 and Theorem 4.3 to provide accuracy guarantees which depend on the unlabeled training set.
We provide a proof overview in Section D.2, and in
Section D.3, we provide a data-dependent lower bound on the all-layer margin that scales inversely with the Lipschitzness of the model, measured via the Jacobian and hidden layer norms on the training data.
These quantities have been shown to be typically well-behaved (Arora et al., 2018;
Nagarajan & Kolter, 2019; Wei & Ma, 2019a).
In Section E.2, we empirically show that explicitly regularizing the all-layer margin improves the performance of self-training.
D ENOISING PSEUDOLABELS FOR SEMI - SUPERVISED LEARNING AND DOMAIN ADAPTATION
We study semi-supervised learning and unsupervised domain adaptation settings where we have access to unlabeled data and a pseudolabeler G pl .
This setting requires a more complicated analysis than the unsupervised learning setting because pseudolabels may be inaccurate, and a student classiﬁer can potentially amplify mistakes made by the pseudolabels.
The evaluation metric is also more challenging because we wish to recover labels exactly, rather than up to permutation.
We design a population objective which measures input transformation consistency and pseudolabel accuracy.
Assuming expansion and separation, we show that the minimizer of this objective will have high accuracy on ground-truth labels.
We assume access to pseudolabeler G pl ( · ) , obtained via training a classiﬁer on the labeled source data in the domain adaptation setting or on the labeled data in the semi-supervised setting.
With access to pseudolabels, we can aim to recover the true labels exactly, rather than up to permutation as in Section 3.2.
For G, G (cid:48) :
X → [ K ] , deﬁne L 0-1 ( G, G (cid:48) )
(cid:44) E P [ 1 ( G ( x ) (cid:54) =
G (cid:48)
( x ))] to be the disagreement between G and G (cid:48) .
The error metric is the standard 0-1 loss on ground-truth labels:
Err ( G ) (cid:44) L 0-1 ( G, G (cid:63) ) .
Let M ( G pl ) (cid:44) { x : G pl ( x ) (cid:54) =
G (cid:63) ( x ) } denote the set of mistakenly pseudolabeled examples.
We require the following assumption on expansion, which intuitively states that each subset of M ( G pl ) has a large enough neighborhood.
Assumption 4.1 ( P expands on sets smaller than M ( G pl ) ) .
Deﬁne ¯ a (cid:44) max i { P i ( M ( G pl )) } to be the maximum fraction of examples in any class which are mistakenly pseudolabeled.
We assume that ¯ a < 1 / 3 and P satisﬁes (¯ a, ¯ c ) -expansion for ¯ c > 3 .
In the sequel, we express our bounds in terms of c (cid:44) min { 1 / ¯ a, ¯ c } .
Note that the above requirement c > 3 is more demanding than the condition c > 1 required in the unsupervised learning setting (Assumption 3.2).
The larger c > 3 accounts for the possibility that mistakes in the pseudolabels can adversely affect the learned classiﬁer in a worst-case manner, whereas for unsupervised learning such a situation cannot occur because pseudolabels are not used.
For the toy distributions in Examples 3.4 and 3.5, we can increase the radius in the deﬁnition of neighborhood by a constant factor of 3 to obtain (0.16, 6)-expansion, which is enough to satisfy the requirement in Assumption 4.1.
On the other hand, Assumption 4.1 is less strict than Assumption 3.2 in the sense that expansion is only required for small sets with mass less than ¯ a , the pseudolabeler’s worst-case error on a class, which can be much smaller than a = 1 / 2 required in Assumption 3.2.
Furthermore, the unsupervised objective (3.4) has the constraint that the input consistency regularizer is not too large, whereas no such constraint is necessary for this setting.
We remark that Assumption 4.1 can also be relaxed to directly consider expansion of subsets of incorrectly pseudolabeled examples, also with a looser requirement on the expansion factor c (see Section B.1).
We design the following objective over classiﬁers G , which ﬁts the classiﬁer to the pseudolabels while regularizing input consistency:
[Equation]
The objective optimizes weighted combinations of R B ( G ) , the input consistency regularizer, and L 0-1 ( G, G pl ) , the loss for ﬁtting pseudolabels, and is related to recent successful algorithms for semi-supervised learning (Sohn et al., 2020; Xie et al., 2020).
We can show that L ( G ) ≥ 0 always holds.
The following lemma bounds the error of G in terms of the objective value.
Lemma 4.2.
Suppose Assumption 4.1 holds.
Then the error of classiﬁer G : X → [ K ] is bounded in terms of consistency w.r.t.
input transformations and accuracy on pseudolabels:
Err ( G ) ≤
L ( G ) .
When expansion and separation both hold, we show that minimizing (4.1) leads to a classiﬁer that can denoise the pseudolabels and improve on their ground-truth accuracy.
Theorem 4.3.
Suppose Assumptions 4.1 and 3.3 hold.
Then for any minimizer (cid:98) G of (4.1) , we have
[Equation]
We provide a proof sketch in Section A, and the full proof in Section B.1.
Our result explains the perhaps surprising fact that self-training with pseudolabeling often improves over the pseudolabeler even though no additional information about true labels is provided.
In Theorem D.2, we translate
Theorem 4.3 into a ﬁnite-sample guarantee by using the generalization bounds in Section 3.3.
At a ﬁrst glance, the error bound in Theorem 4.3 appears weaker than Theorem 3.6 because of the additional dependence on Err ( G pl ) .
This discrepancy is due to weaker requirements on the expansion and the value of the input consistency regularizer.
First, Section 3.2 requires expansion on all sets with probability less than 1 / 2 , whereas Assumption 4.1 only requires expansion on sets with probability less than ¯ a , which can be much smaller than 1 / 2 .
Second, the error bounds in
Section 3.2 only apply to classiﬁers with small values of R B ( G ) , as seen in (3.4).
On the other hand,
Lemma 4.2 gives an error bound for all classiﬁers, regardless of R B ( G ) .
Indeed, strengthening the expansion requirement to that of Section 3.2 would allow us to obtain accuracy guarantees similar to Theorem 3.6 for pseudolabel-trained classiﬁers with low input consistency regularizer value.
E XPERIMENTS
In Section E.1, we provide details for the GAN experiment in Figure 1.
We also provide empirical evidence for our theoretical intuition that self-training with input consistency regularization succeeds because the algorithm denoises incorrectly pseudolabeled examples with correctly pseudolabeled neighbors (Figure 3).
In Section E.2, we perform ablation studies for pseudolabeling which show that components of our theoretical objective (4.1) do improve performance, as intended.
C ONCLUSION
In this work, we propose an expansion assumption on the data which allows for a uniﬁed theoretical analysis of self-training for semi-supervised and unsupervised learning.
Our assumption is realistic for real-world datasets, particularly in vision.
Our analysis is applicable to deep neural networks and can explain why algorithms based on self-training and input consistency regularization can perform so well on unlabeled data.
We hope that this assumption can facilitate future theoretical analyses and inspire theoretically-principled algorithms for semi-supervised and unsupervised learning.
For example, an interesting question for future work is to extend our assumptions to analyze domain adaptation algorithms based on aligning the source and target (Hoffman et al., 2018).
A CKNOWLEDGEMENTS
We would like to thank Ananya Kumar for helpful comments and discussions.
CW acknowledges support from a NSF Graduate Research Fellowship.
TM is also partially supported by the Google Faculty Award, Stanford Data Science Initiative, and the Stanford Artiﬁcial Intelligence Laboratory.
The authors would also like to thank the Stanford Graduate Fellowship program for funding
A P ROOF SKETCH FOR T HEOREM 4.
We provide a proof sketch for Lemma 4.2 for the extreme case where the input consistency regularizer is 0 for all examples, i.e. G ( x ) =
G ( x (cid:48) )
∀ x ∈ X , x (cid:48)
∈ B ( x ) , so R B ( G ) = 0 .
For this proof sketch, we also make an additional restriction to the case when L 0-1 ( G, G pl ) =
Err ( G pl ) .
We ﬁrst introduce some general notation.
For sets U, V ⊆ X , we use U \ V to denote { x : x ∈
U, x / ∈ V } , and ∩ , ∪ denote set intersection and union, respectively.
Let U (cid:44) X \ U denote the complement of U .
Let C i (cid:44) { x : G (cid:63) ( x )
= i } denote the set of examples with ground-truth label i .
For S ⊆ X , we deﬁne N (cid:63) ( S ) to be the neighborhood of S with neighbors restricted to the same class: N (cid:63) ( S ) (cid:44) ∪ i ∈
[ K ] ( N ( S ∩ C i ) ∩ C i ) .
The following key claims will consider two sets: the set of correctly pseudolabeled examples on which the classiﬁer makes mistakes, { x : G ( x ) (cid:54) =
G pl ( x ) and x / ∈
M ( G pl ) } , and the set of examples where both classiﬁer and pseudolabeler disagree with the ground truth, M ( F ) ∩ M ( G pl ) .
The claims below use the expansion property to show that
P ( { x : G ( x ) (cid:54) =
G pl ( x ) and x / ∈ M ( G pl ) } )
> P ( M ( F ) ∩ M ( G pl ))
Claim A.1.
In the setting of Theorem 4.3, deﬁne the set V (cid:44) M ( G ) ∩M
( G pl ) .
Deﬁne q (cid:44)[-
-] Err ( G pl ) c − 1 .
By expansion (Assumption 4.1), if P ( V ) > q , then P ( N (cid:63) ( V ) \ M ( G pl )) > P ( V ) .
A more general version of Claim A.1 is given by Lemma B.7 in Section B.2.
For a visualization of
V and N (cid:63) ( V ) \ M ( G pl ) , refer to Figure 2.
Claim A.2.
Suppose the input consistency regularizer is 0 for all examples, i.e., ∀ x ∈ X , x (cid:48)
∈ B ( x ) , it holds that G ( x ) =
G ( x (cid:48) ) .
Then it follows that
[Equation]
Figure 2 outlines the proof of this claim.
Claim B.5 in Section B provides a more general version of Claim A.2 in the case where R B ( G ) > 0 .
Given the above, the proof of Lemma 4.2 follows by a counting argument.
Proof sketch of Lemma 4.2 for simpliﬁed setting.
Assume for the sake of contradiction that P ( V ) > q .
We can decompose the errors of G on the pseudolabels as follows:
L 0-1 ( G, G pl ) ≥ E [ 1 ( G ( x ) (cid:54) =
G pl ( x ) and x / ∈ M ( G pl ))]
+ E [ 1 ( G ( x ) (cid:54) =
G pl ( x ) and x ∈ M ( G pl ))]
We lower bound the ﬁrst term by P ( V ) by Claims A.1 and A.2.
For the latter term, we note that if x ∈ M ( G pl ) \ V , then G ( x ) =
G (cid:63) ( x ) (cid:54) =
G pl ( x ) .
Thus, the latter term has lower bound
P ( M ( G pl ))
− P ( V ) .
As a result, we obtain
[Equation]
=
Err ( G pl ) which contradicts our simplifying assumption that L 0-1 ( G, G pl ) =
Err ( G pl ) .
Thus, G disagrees with
G (cid:63) at most q fraction of examples in M ( G pl ) .
To complete the proof, we note that G also disagrees with G (cid:63) on at most q fraction of examples outside of M ( G pl ) , or else L 0-1 ( G, G pl ) would again be too high.
B P ROOFS FOR DENOISING PSEUDOLABELS
In this section, we will provide the proof of Theorem 4.3.
Our analysis will actually rely on a weaker additive notion of expansion, deﬁned below.
We show that the multiplicative deﬁnition in
Deﬁnition 3.1 will imply that the additive variant holds.
B. R ELAXATION OF EXPANSION ASSUMPTION FOR PSEUDOLABELING
In this section, we provide a proof of a relaxed version of Theorem 4.3.
We will then reduce Theorem 4.3 to this relaxed version in Section B.2.
It will be helpful to restrict the notion of neighborhood to only examples in the same ground-truth class: deﬁne N (cid:63) ( x ) (cid:44) { x (cid:48) :
x (cid:48) ∈ N ( x ) and G (cid:63) ( x (cid:48) )
=
G (cid:63) ( x ) } and N (cid:63) ( S ) (cid:44) ∪ x ∈ S N (cid:63) ( x ) .
Note that the following relation between N ( S ) and N (cid:63) ( S ) holds in general:
[Equation]
We will deﬁne the additive notion of expansion on subsets of X below.
Deﬁnition B.1 ( ( q, α ) -additive-expansion on a set S ) .
We say that P satisﬁes ( q, α ) -additiveexpansion on S ⊆
X if for all V ⊆ S with P ( V ) >
q , the following holds:
[Equation]
In other words, any sufﬁciently large subset of S must have a sufﬁciently large neighborhood of examples sharing the same ground-truth label.
For the remainder of this section, we will analyze this additive notion of expansion.
In Section B.2, we will reduce multiplicative expansion (Deﬁnition 3.1) to our additive deﬁnition above.
Now for a given classiﬁer, deﬁne the robust set of G , S B ( G ) , to be the set of inputs for which G is robust under B -transformations:
[Equation]
The following theorem shows that if the classiﬁer G is B -robust and ﬁts the pseudolabels sufﬁciently well, classiﬁcation accuracy on true labels will be good.
Theorem B.2.
For a given pseudolabeler G pl : X → { 1 , . . .
, K } , suppose that P has ( q, α ) - additive-expansion on M ( G pl ) for some q, α .
Suppose that G ﬁts the pseudolabels with sufﬁcient accuracy and robustness:
[Equation]
Then G satisﬁes the following error bound:
[Equation]
To interpret this statement, suppose G ﬁts the pseudolabels with error rate at most Err ( G pl ) and (B.1) holds.
Then Err ( G ) ≤ 2( q + R B ( G )) , so if G is robust to B -perturbations on the population distribution, the accuracy of G is high.
Towards proving Theorem B.2, we consider three disjoint subsets of M ( G ) ∩ S B ( G ) :
[Equation]
We ﬁrst bound the probability of M 1 ∪ M
Lemma B.3.
In the setting of Theorem B.2, we have P ( S B ( G ) ∩ M
( G pl ) ∩ M ( G )) ≤ q .
As a result, since it holds that M 1 ∪ M 2 ⊆ S B ( G ) ∩ M ( G pl ) ∩ M ( G ) , it immediately follows that
P ( M 1 ∪ M 2 ) ≤ q .
The proof relies on the following claims.
Claim B.4.
In the setting of Theorem 4.3, deﬁne U (cid:44) N (cid:63) ( S B ( G )
∩ M ( G pl ) ∩ M ( G ))
\ M ( G pl ) .
For any x ∈ U ∩ S B ( G ) , it holds that G pl ( x ) (cid:54) =
G ( x ) and G ( x ) (cid:54) =
G (cid:63) ( x ) .
Proof.
For any x ∈ U ⊆ N (cid:63) ( S B ( G ) ∩ M
( G pl ) ∩ M ( G )) , there exists x (cid:48) ∈ S B ( G ) ∩ M ( G pl ) ∩
M ( G ) such that B ( x ) ∩ B ( x (cid:48) )
(cid:54) = ∅ and G (cid:63) ( x ) =
G (cid:63) ( x (cid:48) ) by deﬁnition of N (cid:63) ( · ) .
Choose z ∈
B ( x ) ∩ B ( x (cid:48) ) .
As x, x (cid:48) ∈ S B ( G ) , by deﬁnition of S B ( G ) we also must have G ( x ) =
G ( z ) =
G ( x (cid:48) ) .
Furthermore, as x (cid:48) ∈ M ( G ) , G ( x (cid:48) ) (cid:54) =
G (cid:63) ( x (cid:48) ) .
Since G (cid:63) ( x ) = G (cid:63) ( x (cid:48) ) , it follows that
G ( x ) (cid:54) =
G (cid:63) ( x ) .
As U ∩ M ( G pl ) = ∅ by deﬁnition of U , G pl much match the ground-truth classiﬁer on U , so
G pl ( x ) =
G (cid:63) ( x ) .
It follows that G ( x ) (cid:54) =
G pl ( x ) , as desired.
Claim B.5.
In the setting of Theorem B.2, deﬁne U (cid:44) N (cid:63) ( S B ( G )
∩ M ( G pl ) ∩ M ( G ))
\ M ( G pl ) .
If P ( S B ( G ) ∩ M ( G pl ) ∩ M ( G )) >
q , then
[Equation]
Proof.
Deﬁne V (cid:44) S B ( G ) ∩ M
( G pl ) ∩ M ( G ) .
By the assumption that M ( G pl ) satiﬁes ( q, α ) - additive-expansion, if P ( V ) >
q holds, it follows that P ( U ) >
P ( V ) + α .
Furthermore, we have U \ S B ( G ) ⊆
S B ( G ) ∪ M ( G pl ) by deﬁnition of U and V as U ∩ M ( G pl ) = ∅ , and so
P ( U \ S B ( G )) ≤ 1 − P ( S B ( G )
∪ M ( G pl )) .
Thus, we obtain
[Equation]
Now we use the principle of inclusion-exclusion to compute
[Equation]
Plugging into the previous, we obtain
[Equation]
P
[Equation]
where we obtained the last line because V = S B ( G ) ∩ M ( G pl ) ∩ M ( G )
⊆
S B ( G ) ∩ M ( G pl ) .
Proof of Lemma B.3.
To complete the proof of Lemma B.3, we ﬁrst compose S B ( G ) into three disjoint sets:
[Equation]
First, by Claim B.4 and deﬁnition of U , we have ∀ x ∈ U ∩S B ( G ) , G ( x ) (cid:54) =
G pl ( x ) and x / ∈ M ( G pl ) .
Thus, it follows that U ∩ S B ( G ) ⊆
S 3 .
Next, we claim that V (cid:48) (cid:44) M
( G pl ) ∩ M ( G ) ∩ S B ( G ) ⊆
S 2 .
To see this, note that for x ∈ V (cid:48) ,
G ( x ) =
G (cid:63) ( x ) and G pl ( x ) (cid:54) =
G (cid:63) ( x ) .
Thus, G ( x ) (cid:54) =
G pl ( x ) , and x ∈ S B ( G ) ∩ M ( G pl ) , which implies x ∈ S 2 .
Assume for the sake of contradiction that P ( S B ( G ) ∩ M ( G pl ) ∩ M ( G )) >
q .
Now we have
P
[Equation]
S
[Equation]
G
[Equation]
≥ P
[Equation]
S
[Equation] [Equation]
P
[Equation]
S
[Equation]
+ P ( S 3 ) ≥ P ( S 1 )
+ P ( S B ( G )
∩ M ( G pl ) ∩ M ( G ))
+ P ( U ∩ S B ( G )) >
P ( S 1 ) + P ( M ( G pl ))
+ P ( S B ( G ))
+ α −(by Claim B.5)
However, we also have
[Equation]
(by the condition in (B.1))
Plugging this in gives us P ( S 1 )
+ P ( S 2 )
+ P ( S 3 ) >
P ( S B ( G )) , a contradiction.
Thus, P ( S B ( G ) ∩
M ( G pl ) ∩ M ( G )) ≤ q , as desired.
The next lemma bounds P ( M 3 ) .
Lemma B.6.
In the setting of Theorem B.2, the following bound holds:
[Equation]
Proof.
The proof will follow from basic manipulation.
First, we note that
[Equation]
pl
[Equation]
x
[Equation]
x ∈
[Equation]
[Equation]
G
[Equation]
G
[Equation]
=
M 1 ∪ { x : G pl ( x ) =
G (cid:63) ( x ) and x ∈ S B ( G ) }
[Equation]
As (B.2) and (B.3) pertain to unions of disjoint sets, it follows that
[Equation]
G pl ( x ) and x ∈ S B ( G ) } ) =
P ( M 1 ) + P ( { x
[Equation]
G
[Equation]
[Equation]
) =
[Equation]
( x ) and x ∈ S B ( G ) } )
Thus, rearranging we obtain
P
[Equation]
≤ [Equation] − Err ( G pl ) Substituting P ( M 1 ) ≤ q from Lemma B.3 gives the desired result.
Proof of Theorem B.2.
To complete the proof, we compute
[Equation]
by Lemmas B.3 and B.6)
B. P ROOF OF T HEOREM 4.
In this section, we complete the proof of Theorem 4.3 by reducing Lemma 4.2 to Theorem B.2.
This requires converting multiplicative expansion to ( q, α ) -additive-expansion, which is done in the following lemma.
Let M i ( G pl ) (cid:44) M ( G pl ) ∩C
i denote the incorrectly pseudolabeled examples with ground-truth class i .
Lemma B.7.
In the setting of Theorem 4.3, suppose that Assumption 4.1 holds.
Then for any β ∈ (0 , c − 1] , P i has ( q, α ) -additive-expansion on M i ( G pl ) for the following choice of q, α :
[Equation]
Proof.
Consider any set S ⊆
M i ( G pl ) with P i ( S ) > βP i
( M i ( G pl ))
c
−. Then by Assumption 4.1,
P i ( N ( S )) ≥ min { cP i ( S ) , 1 } ≥ cP i ( S ) , where we used the fact that P i ( S ) ≤ P i ( M i ( G pl )) and c ≤ 1 P
i ( M i ( G pl )) , so cP i ( S ) ≤ 1 .
Thus, we can obtain
[Equation]
Here the last line used the fact that P i ( S ) > βP i ( M i ( G pl ))
c −.
This implies that P i has ( q, α ) -additiveexpansion on M i ( G pl ) for the ( q, α ) given in (B.4).
We will now complete the proof of Lemma 4.2.
Note that given Lemma 4.2, Theorem 4.3 follows immediately by noting that G (cid:63) satisﬁes L 0-1 ( G (cid:63) , G pl ) =
Err ( G pl ) and R B ( G (cid:63) ) ≤ µ by Assumption 3.3.
We ﬁrst deﬁne the class-conditional pseudolabeling and robustness losses: L ( i ) 0-1 ( G, G (cid:48) ) (cid:44) P i ( { x : G ( x ) (cid:54) = G (cid:48)
( x ) } ) , and R ( i ) B ( G ) (cid:44) E P i [ 1 ( ∃ x (cid:48) ∈ B ( x ) such that G ( x (cid:48) )
(cid:54) =
G ( x ))] .
We also deﬁne the class-conditional error as follows:
Err i ( G )
[Equation]
L ( i ) 0-1 ( G, G (cid:63) ) .
We prove the class-conditional variant of Lemma 4.2 below. Lemma B.8. For any i ∈ [ K ] , deﬁne [Equation]
Then in the setting of Theorem 4.3 where Assumption 4.1 holds, we have the following error bound for class i :
[Equation]
Proof.
First, we consider the case where L ( i ) 0-1 ( G, G pl ) + R ( i ) B ( G ) ≤ ( c − 1)
Err i ( G pl ) .
In this case, we can apply Lemma B.7 with β ∈ (0 , c − 1] chosen such that
[Equation]
We note that P i has ( q, α ) -additive-expansion on M i ( G pl ) for
[Equation]
Now by (B.7), we can apply Theorem B.2 with this choice of ( q, α ) to obtain [Equation] [Equation]
Next
[Equation]
we consider the case where L ( i ) 0-1 ( G, G pl )
+ R ( i ) B ( G ) >
( c − 1)
Err i ( G pl ) .
Note that by triangle inequality
[Equation]
we have
[Equation]
Lemma 4.2 now follows simply by taking the expectation of the bound in (B.6) over all the classes.
C P ROOFS FOR UNSUPERVISED LEARNING
We will ﬁrst prove an analogue of Lemma C.7 for a relaxed notion of expansion.
We will then prove
Theorem 3.6 by showing that multiplicative expansion implies this relaxed notion, deﬁned below:
Deﬁnition C.
( q, ξ ) -constant-expansion) .
We say that distribution P satisﬁes ( q, ξ ) -constantexpansion if for all S ⊆
X satisfying P ( S ) ≥ q and P ( S ∩ C i ) ≤ P ( C i ) / 2 for all i , the following holds:
[Equation]
As before, N (cid:63) ( S ) is deﬁned by ∪ i ∈
[ K ] ( N ( S ∩ C i ) ∩ C i ) .
We will work with the above notion of expansion for this subsection.
We ﬁrst show that a B -robust labeling function which assigns sufﬁcient probability to each class will align with the true classes.
Theorem C.2.
Suppose P satisﬁes ( q, ξ ) -constant-expansion for some q .
If it holds that R B ( G ) < ξ and
[Equation]
there exists a permutation π : [ K ] → [ K ] satisfying the following:
[Equation]
Deﬁne (cid:98) C 1 , . . .
, (cid:98) C K to be the partition induced by G : (cid:98) C i (cid:44) { x : G ( x ) = i } .
Lemma C.3.
In the setting of Theorem C.2, consider any set of the form U (cid:44) S B ( G ) ∩ i ∈I C i ∩ j ∈J (cid:98) C j where I , J are arbitrary subsets of [ K ] .
Then N (cid:63) ( U ) \ U ⊆
S B ( G ) .
Proof.
Consider any x ∈ N (cid:63) ( U ) \ U .
There are two cases.
First, if G ( x ) ∈ J , then by deﬁnition of N (cid:63) ( · ) , x ∈ ∩ i ∈I C
i ∩ j ∈J (cid:98) C j .
However, x / ∈ U , which must imply that x / ∈ S B ( G ) .
Second, if
G ( x ) / ∈ J , by deﬁnition of N (cid:63) ( · ) there exists x (cid:48) ∈
U such that B ( x ) ∩ B ( x (cid:48) )
(cid:54) = ∅ .
It follows that for z ∈ B ( x ) ∩ B ( x (cid:48) ) , G ( z ) =
G ( x (cid:48) )
∈ J .
Thus, since G ( x ) / ∈ J , G ( x ) (cid:54) =
G ( z ) so x / ∈ S B ( G ) .
Thus, it follows that N (cid:63) ( U ) \ U ⊆
S B ( G ) .
Next, we show that every cluster found by G will take up the majority of labels of some ground-truth class.
Lemma C.4.
In the setting of Theorem C.2, ∀ j , ∃ i such that P ( S B ( G ) ∩ C i ∩ (cid:98) C j ) > P ( S B ( G ) ∩C i ).
Proof.
Assume for the sake of contradiction that there exists j such that for all i , P ( S B ( G ) ∩ C i ∩ (cid:98) C j ) ≤ P ( S B ( G ) ∩C i ).
Deﬁne the set
U
i (cid:44) S B ( G )
∩ C i ∩ (cid:98) C j , and U (cid:44) ∪ i
U i
=
S B ( G ) ∩ (cid:98) C j .
Note that { U i } Ki =1 form a partition of U because {C i } Ki =1 are themselves disjoint from one another.
Furthermore, we can apply Lemma C.3 with I =
[ K ] to obtain N (cid:63) ( U ) \ U ⊆
S B ( G ) .
Now we observe that P ( U ) ≥ P ( (cid:98) C j ) − P ( S B ( G )) .
Using the theorem condition that P ( (cid:98) C j ) > 2 P ( S B ( G )) , it follows that
[Equation]
Furthermore for all i we note that
[Equation]
Thus, P ( C i ) ≥ 2 P ( U i ) .
Thus, by ( q, ξ ) -constant-expansion we have
[Equation]
As N (cid:63) ( U ) \ U ⊆ S B ( G ) , this implies R B ( G )
=
P ( S B ( G )) ≥ min { ξ, P ( (cid:98) C j ) / 2 } , a contradiction.
Lemma C.5.
In the setting of Theorem C.2 and Lemma C.4, ∀ j , there exists a unique π ( j ) such that P ( S B ( G ) ∩C
π ( j ) ∩ (cid:98) C j ) >
P ( S B ( G ) ∩C
π ( j ) ), and P ( S B ( G ) ∩C
i ∩ (cid:98) C j ) ≤ P ( S B ( G ) ∩C
ifor i (cid:54)
=
π ( j ) .
Furthermore, π is a permutation from [ K ] to [ K ] .
Proof.
By the conclusion of Lemma C.4, the only way the existence of such a π might not hold is if there is some j where P ( S B ( G ) ∩
C i ∩ (cid:98) C j ) > P ( S B ( G ) ∩C ifor i ∈
{ i 1 , i 2 } , where i 1 (cid:54) =
i 2 .
In this case, by the Pigeonhole Principle, as the conclusion of Lemma C.4 applies for all j ∈ [ K ] and there are K possible choices for i , there must exist i where P ( S B ( G )
∩ C i ∩ (cid:98) C j ) >
P ( S B ( G ) ∩C i )for j ∈ { j 1 , j 2 } , where j 1 (cid:54) =
j 2 .
Then P ( S B ( G ) ∩ C i ∩ (cid:98) C j 1 ) + P ( S B ( G )
∩ C i ∩ (cid:98) C j 2 ) >
P ( S B ( G ) ∩ C i ) , which is a contradiction.
Finally, to see that π is a permutation, note that if π ( j 1 ) =
π ( j 2 ) for j 1 (cid:54) =
j 2 , this would result in the same contradiction as above.
Proof of Theorem C.2.
We will prove (C.1) using π deﬁned in Lemma C.5.
Deﬁne the set U j (cid:44)
S B ( G ) ∩ C π ( j ) ∩ k (cid:54) =
j (cid:98) C k .
Note that U
j = { x : G ( x ) (cid:54) =
j, G (cid:63) ( x ) =
π ( j ) } ∩ S B ( G ) .
Deﬁne
U = ∪ j U j , and note that { U j } Kj =1 forms a partition of U .
Furthermore, we also have U =
x : π ( G ( x )) (cid:54) =
G (cid:63) ( x ) } ∩ S B ( G ) .
We ﬁrst show that P ( U ) ≤ max { q, R B ( G ) } .
Assume for the sake of contradiction that this does not hold.
First, we claim that {N (cid:63) ( U j ) \ U j } kj =1 ⊇ N (cid:63) ( U ) \ U .
To see this, consider any x ∈ C π ( j ) ∩N (cid:63) ( U ) \
U .
By deﬁnition, ∃ x (cid:48) ∈
U such that B ( x (cid:48) )
∩ B ( x ) (cid:54) = ∅ and G (cid:63) ( x ) =
G (cid:63) ( x (cid:48) ) , or x (cid:48) ∈
C π ( j ) .
Thus, it follows that x ∈ N (cid:63) ( C π ( j ) ∩ U ) \ U = N (cid:63) ( U j ) \ U = N (cid:63) ( U j ) \ U j , where the last equality followed from the fact that N (cid:63) ( U j ) and U k are disjoint for j (cid:54) =
k .
Now we apply Lemma C.3 to each N (cid:63) ( U j ) \ U j to conclude that N (cid:63) ( U ) \ U ⊆
S B ( G ) .
Finally, we observe that
[Equation]
(C.3) by the deﬁnition of π in Lemma C.5.
Now we again apply the ( q, ξ ) -constant-expansion property, as we assumed P ( U ) >
q , obtaining
[Equation] P [Equation] U )
[Equation] However, as we showed N (cid:63) ( U ) \ U ⊆ S B ( G ) , we also have R B ( G ) =
P ( S B ( G )) ≥ P ( N (cid:63) ( U ) \
U ) ≥ min { ξ, P ( U ) } .
This contradicts P ( U ) > max { q, R B ( G ) } and R B ( G ) < ξ , and therefore
P ( U ) ≤ max { q, R B ( G ) } .
Finally, we note that { x : π ( G ( x )) (cid:54) =
G (cid:63) ( x ) } ⊆
U ∪ S B ( G ) .
Thus, we ﬁnally obtain
P ( { x : π ( G ( x )) (cid:54) =
G (cid:63) ( x ) } ) ≤ P ( U ) + P ( S B ( G )) ≤ max { q, R B ( G ) }
+ R B ( G )
C. P ROOF OF T HEOREM 3.
In this section, we prove Theorem 3.6 by converting multiplicative expansion to ( q, ξ ) -constantexpansion and invoking Theorem C.2.
The following lemma performs this conversion.
Lemma C.6.
Suppose P satisﬁes (1 / 2 , c ) -multiplicative-expansion (Deﬁnition 3.1) on X .
Then for any choice of ξ > 0 , P satisﬁes (cid:16) ξ c − 1 , ξ (cid:17) -constant expansion.
Proof.
Consider any S such that P ( S ∩ C i ) ≤ P ( C i ) / 2 for all i ∈
[ K ] and P ( S ) > q .
Deﬁne
S i (cid:44) S ∩ C i .
First, in the case where c ≥ 2 , we have by multiplicative expansion
[Equation]
(because c ≥ 2 and P ( S i ) ≤ P ( C i ) / 2 )
Thus, we immediately obtain constant expansion.
Now we consider the case where 1 ≤ c < 2 .
By multiplicative expansion, we must have
[Equation]
(because c < 2 and P ( S i ) ≤ P ( C i ) /
[Equation]
The following lemma states an accuracy guarantee for the setting with multiplicative expansion.
Lemma C.7.
Suppose Assumption 3.2 holds for some c > 1 .
If classiﬁer G satisﬁes
[Equation]
then the unsupervised error is small:
[Equation]
We now prove Lemma C.7, which in turn immediately gives a proof of Theorem 3.6.
Proof of Lemma C.7.
By Lemma C.6, P must satisfy (cid:16) R B ( G ) c − 1 , R B ( G ) (cid:17) -constant-expansion.
As we also have min i P ( { x : G ( x ) = i } ) > max (cid:110) 2 c − 1 , 2 (cid:111) R B ( G ) , we can now apply Theorem C.2 to conclude that there exists permutation π :
[ K ] → [ K ] such that
[Equation]
as desired.
C. J USTIFICATION FOR E XAMPLES 3.4 AND 3.
To avoid the disjointness issue of Example 3.4, we can redeﬁne the ground-truth class G (cid:63) ( x ) to be the most likely label at x .
This also induces truncated class-conditional distributions P 1 , P 2 where the overlap is removed.
We can apply our theoretical analysis to P 1 , P 2 and then translate the result back to P 1 , P 2 , only changing the bounds by a small amount when the overlap is minimal.
To justify Example 3.4, we use the Gaussian isoperimetric inequality (Bobkov et al., 1997), which states that for any ﬁxed p such that P i ( S )
= p where i ∈ { 1 , 2 } , the choice of S minimizing
P i ( N ( S )) is given by a halfspace:
S = H ( p ) (cid:44) { x : w (cid:62) ( x − τ i ) ≤ Φ − 1 ( p ) } for vector w with (cid:107) w (cid:107) = √ d .
It then follows that setting r = 1 √ d , N ( H ( p ))
⊇ { x + t w (cid:107) w (cid:107) 2 : x ∈ H ( p ) , 0 ≤ t ≤ r } ⊇ { x : w (cid:62) ( x − τ i ) ≤ Φ − 1 ( p ) + r √ d } , and thus P ( N ( H ( p ))) ≥ Φ(Φ − 1 ( p ) + r √ d ) .
As
P ( N ( H ( p ))) /P
( H ( p )) is decreasing in p for p < 0 .
5 , our claim about expansion follows.
To see our claim about separation, consider the sets
X i (cid:44) { x : ( x − τ i ) (cid:62) v ij ≤ (cid:107) τ i − τ j (cid:107)− r/ 2 ∀ j } , where v ij (cid:44) τ
j − τ i (cid:107) τ j − τ i (cid:107) 2 .
We note that these sets are β -separated from each other, and furthermore, for the lower bound on (cid:107)
τ
i − τ j (cid:107) in the example, note that X i has probability 1 − µ under P i .
For Example 3.5, we note that for B ( x ) (cid:44) { x (cid:48) : (cid:107) x (cid:48) − x (cid:107) 2 ≤ r } , N ( S ) ⊇ M ( { x (cid:48) :
∃ x ∈
M − 1 ( S ) such that (cid:107) x (cid:48) − x (cid:107) ≤ r/κ } ) .
Thus, our claim about expansion reduces to the Gaussian case.
D A LL -L AYER MARGIN GENERALIZATION BOUNDS D. E ND - TO - END GUARANTEES
In this section, we provide end-to-end guarantees for unsupervised learning, semi-supervised learning, and unsupervised domain adaptation for ﬁnite training sets.
For the following two theorems, we take the notation ˜ O ( · ) as a placeholder for some multiplicative quantity that is poly-logarithmic in n, d .
We ﬁrst provide the ﬁnite-sample guarantee for unsupervised learning.
Theorem D.1.
In the setting of Theorem 3.6 and Section 3.3, suppose that Assumption 3. holds.
Suppose that G = arg max
i F i is parametrized as a neural network of the form F ( x ) (cid:44)
W p φ ( · · · φ ( W 1 x ) · · · ) .
With probability 1 − δ over the draw of the training sample (cid:98) P , if for any choice of t > 0 and { u y } Ky =1 with u y > 0 ∀ y , it holds that
[Equation]
y ∈ [ K ] then it follows that the population unsupervised error is small:
[Equation]
+ ζ where ζ (cid:44) O (cid:18) 1 c − 1 (cid:113) log( K/δ )+ p log n n (cid:19) is a low-order term.
The following theorem provides the ﬁnite-sample guarantee for unsupervised domain adaptation and semi-supervised learning.
Theorem D.2.
In the setting of Theorem 4.3 and Section 3.3, suppose that Assumption 4. holds.
Suppose that G = arg max
i F i is parametrized as a neural network of the form F ( x ) (cid:44)
W p φ ( · · · φ ( W 1 x ) · · · ) .
For any t 1 , t 2 > 0 , deﬁne the following quantities:
[Equation]
where ζ (cid:44) O (cid:18) 1 c − 1 (cid:113) log( K/δ )+ p log n n (cid:19) is a low-order term.
With probability 1 − δ over the draw of the training sample (cid:98) P , for all choices of t 1 , t 2 > 0 , it holds that
[Equation]
D. P ROOFS FOR S ECTION 3.
In this section, we provide a proof sketch of Theorem 3.7.
The proof follows the analysis of (Wei & Ma, 2019b) very closely, but because there are some minor differences we include it here for completeness.
We ﬁrst state additional bounds for the other quantities in our objectives, which are proved in the same manner as Theorem 3.7.
Theorem D.3.
With probability 1 − δ over the draw of the training sample (cid:98) P , all neural networks
G = arg max
i F i of the form F ( x ) (cid:44) W p φ ( · · · φ ( W 1 x )) will satisfy
[Equation]
for all choices of t > 0 , where ζ (cid:44) O (cid:18)(cid:113) log [Equation]
log [Equation] (cid:19) is a low-order term, and (cid:101) O ( · ) hides poly-logarithmic factors in n and d .
Theorem D.4.
With probability 1 − δ over the draw of the training sample (cid:98) P , all neural networks
G = arg max
i F i of the form F ( x ) (cid:44) W p φ [Equation] · · · φ ( W 1 x )) will satisfy
[Equation]
for all choices of y ∈ [ K ] , t > 0 , where ζ (cid:44) O (cid:18)(cid:113) log( [Equation] (cid:19) is a low-order term, and (cid:101) O ( · ) hides poly-logarithmic factors in n and d .
We now overview the proof of Theorem 3.7, as the proofs of Theorem D.3 and D.4 follow identically.
We ﬁrst formally deﬁne the all-layer margin m ( F, x, y ) for neural net F evaluated on example x with label y .
We recall that F computes the function F ( x ) (cid:44) W p φ ( · · · φ ( W 1 x ) · · · ) .
We index the layers of F as follows: deﬁne f 1 ( x ) (cid:44) W 1 x , and f i ( h ) (cid:44) W i φ ( h ) for 2 ≤ i ≤ p , so that
F ( x ) =
f p ◦ · · · ◦ f 1 ( x ) .
Letting δ =
( δ 1 , . .
. , δ p ) denote perturbations for each layer of F , we deﬁne the perturbed output F ( x, δ ) as follows:
[Equation]
Now the all-layer margin m ( F, x, y ) is deﬁned by
[Equation]
[Equation]
subject to arg max i
F [Equation]
= y
As is typical in generalization bound proofs, we deﬁne a ﬁxed class of neural net functions to analyze, expressed as
[Equation]
1 x ) · · · ) : W i ∈
W i ∀ i } where W i is some class of possible instantiations of the i -th weight matrix.
We also overload notation and let W i (cid:44) { h (
cid:55)→ W i h : W i ∈
W i } denote the class of functions corresponding to matrix multiplication by a weight in W i .
Let (cid:107) · (cid:107) op denote the matrix operator norm.
For a function class G , we let N (cid:107)·(cid:107) ( (cid:15), G ) denote the (cid:15) -covering number of G in norm (cid:107) · (cid:107) .
The following condition will be useful for the analysis:
Condition D.5 (Condition A.1 from (Wei & Ma, 2019b)) .
We say that a function class G satisﬁes the (cid:15) − 2 covering condition with respect to norm (cid:107) · (cid:107) with complexity C
(cid:107)·(cid:107) ( G ) if for all (cid:15) > 0 ,
[Equation]
To sketch the proof technique, we only provide the proof of (3.6) in Theorem 3.7, as the other bounds follow with the same argument.
The following lemma bounds R B ( G ) in terms of the robust all-layer margin m B .
Lemma D.6 (Adaptation of Theorem A.1 of (Wei & Ma, 2019b)) .
Suppose that weight matrix mappings W i satisfy Condition D.5 with operator norm (cid:107) · (cid:107) op and complexity function C (cid:107)·(cid:107)
op
( W i ) .
With probability 1 − δ over the draw of the training data, for all t > 0 , all classiﬁers F ∈ F will satisfy
[Equation]
where ζ (cid:44) O (cid:18)(cid:113) log(1 /δ )
+log n n (cid:19) is a low-order term.
The proof of Lemma D.6 mirrors the proof of Theorem A.1 of (Wei & Ma, 2019b).
The primary difference is that because we seek a bound in terms a threshold on the margin whereas (Wei & Ma, 2019b) prove a bound that depends on average margin, we must analyze the generalization of a slightly modiﬁed loss.
Towards proving Lemma D.6, we ﬁrst deﬁne ||| δ ||| (cid:44) (cid:107) ( (cid:107) δ 1 (cid:107) 2 , . . .
, (cid:107) δ p (cid:107) 2 ) (cid:107)for perturbation δ , and ||| F ||| (cid:44) (cid:107) ( (cid:107) W 1 (cid:107) op , . . .
, (cid:107) W p (cid:107) op ) (cid:107) 2 .
We show that m B ( F, x ) is Lipschitz in
F for ﬁxed x with respect to ||| · ||| .
Claim D.7.
Choose F, (cid:98) F ∈ F .
Then for any x ∈ X , | m B ( F, x ) − m B ( (cid:98) F, x ) | ≤ ||| F − (cid:98) F |||
The same conclusion holds if we replace m B with m .
Proof.
We consider two cases:
Case 1: arg max i F ( x ) i = arg max i (cid:98) F ( x ) i .
Let y denote the common value.
In this case, the desired result immediately follows from Claim E.1 of (Wei & Ma, 2019b).
Case 2: arg max i F ( x ) i (cid:54) =
arg max i (cid:98) F ( x ) i .
In this case, the construction of Claim A.1 in (Wei & Ma, 2019b) implies that 0 ≤ m B ( F, x ) ≤ ||| F − (cid:98) F ||| .
(Essentially we choose δ with ||| δ ||| ≤ ||| F − (cid:98) F ||| such that F ( x, δ ) = (cid:98) F ( x ) .)
Likewise, 0 ≤ m B ( (cid:98) F, x ) ≤ ||| F − (cid:98) F ||| .
As a result, it must follow that | m B ( F, x ) − m B ( (cid:98) F, x ) | ≤ ||| F − (cid:98) F ||| .
For t > 0 , deﬁne the ramp loss h t as follows: h t ( a ) = 1 − 1 ( a ≥ 0) min { a/t, 1 }
We now deﬁne the hypothesis class L t (cid:44) { h t ◦ m B ( F, · ) : F ∈ F} .
We now bound the Rademacher complexity of this hypothesis class:
Claim D.8.
In the setting of Lemma D.6, suppose that W i satisﬁes Condition D.5 with operator norm (cid:107) · (cid:107) op and complexity C (cid:107)·(cid:107) op ( W i ) .
Then
[Equation]
As the proof of Claim D.8 is standard, we provide a sketch of its proof.
Proof sketch of Claim D.8.
First, by Lemma A.3 of (Wei & Ma, 2019b), we obtain that F satisﬁes
Condition D.5 with norm ||| · ||| and complexity C |||·||| ( F ) (cid:44) (cid:80) i C (cid:107)·(cid:107)
op ( F i ) .
Now let (cid:98) F be a t(cid:15) -cover of F in ||| · ||| .
We deﬁne the L 2 ( P n ) -norm of a function f : X → R as follows:
[Equation]
Then it is standard to show that
[Equation]
is a (cid:15) -cover of L t in L 2 ( P n ) -norm, because h t is 1 /t -Lipschitz and m B ( F, x ) is 1 -Lipschitz in F for norm ||| · ||| for any ﬁxed x .
It follows that log N L 2 ( P n ) ( (cid:15), L t ) ≤ (cid:22)
[Equation]
|·||| ( F [Equation] t 2 (cid:15) 2 (cid:23) .
Now we apply
Dudley’s Theorem:
[Equation]
A standard computation can be used to bound the quantity on the right, giving the desired result.
Proof of Lemma D.6.
First, by the standard relationship between Rademacher complexity and generalization, Claim D.8 lets us conclude that with probability 1 − δ , for any ﬁxed t > 0 , all F ∈ F satisfy:
[Equation]
We additionally note that h t ( m B ( F, x ))
= 1 when x / ∈ S B ( G ) , because in such cases m B ( F, x )
= 0 .
It follows that 1( x / ∈ S B ( G )) ≤ h t ( m B ( F, x )) .
Thus, we obtain
[Equation]
It remains to show that (D.1) holds for all t .
It is now standard to perform a union bound over choices of t in the form t j (cid:44) t min 2 j , where t min (cid:44) (cid:80) i C (cid:107)·(cid:107) op ( W i ) √ n log n and 0 ≤ j ≤ O (log n ) , so we only sketch the argument here.
We union bound over (D.2) for t =
t j with failure probability δ j = δ/ 2 j +1 , so (D.2) will hold for all t 1 , . . .
, t j max with probability 1 − δ .
For any choice of t , there will either be j such that t/ 2 ≤ t j ≤ t , or (D.1) must trivially hold.
(See Theorem C.1 of (Wei & Ma, 2019b) for a more detailed justiﬁcation.)
As a result, there will be some j such that the right hand side of (D.2) is bounded above by the right hand side of (D.1), as desired.
Proof sketch of Theorem 3.7.
By Lemma B.2 of (Wei & Ma, 2019b), we have C (cid:107)·(cid:107) op ( { W : (cid:107) W (cid:107) F ≤ a } ) =
O ( √ q log qa ) .
Thus, to obtain (3.6), it sufﬁces to apply Lemma D.6 for all choices of a using a standard union bound technique; see for example the proof of Theorem 3.1 in (Wei & Ma, 2019b).
To obtain the other generalization bounds, we can follow a similar argument for Lemma D.6 to prove its analogue for other variants of all-layer margin, and then repeat the same union bound over the weight matrix norms as before.
D. D ATA - DEPENDENT LOWER BOUNDS ON ALL - LAYER MARGIN
We will now provide lower bounds on the all-layer margins used in Theorem 3.7 in the case when the activation φ has ν -Lipschitz derivative.
In this section, it will be convenient to modify the indexing to count the activation as its own layer, so there are 2 p − 1 layers in total.
Let s ( i ) ( x ) denote the (cid:107) · (cid:107)norm of the layer preceding the i -th matrix multiplication, where the parenthesis in the subscript distinguishes between weight indices and layer indices (which also include the activation layers).
Deﬁne ν
j ← i ( x ) to be the Jacobian of the j -th layer with respect to the i − 1 -th layer evaluated at x .
Deﬁne γ ( F ( x ) , y ) (cid:44) F ( x ) y − max i (cid:54) =
y F ( x ) i .
We use the following quantity to measure stability in the layer following W ( i ) :
[Equation]
for a secondary term ψ ( i ) ( x, y ) given by
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
We now have the following lower bounds on m ( F, x, y ) and m B ( F, x ) :
Proposition D.9 (Lemma C.1 from (Wei & Ma, 2019b)) .
In the setting above, if γ ( F ( x ) , y ) > 0 , we have
[Equation]
Furthermore, if γ ( F ( x (cid:48) ) , arg max i F ( x ) i ) > 0 for all x (cid:48) ∈ B ( x ) , then
[Equation] [Equation]
[Table]
E E XPERIMENTS E. E MPIRICAL SUPPORT FOR EXPANSION PROPERTY USING GAN S
In this section we provide additional details regarding the GAN veriﬁcation depicted in Figure(left).
We use 128 by 128 images sampled from a pre-trained BigGAN (Brock et al., 2018).
We categorize images into 10 superclasses chosen in the robustness library of Engstrom et al.
(2019): dog, bird, insect, monkey, car, cat, truck, fruit, fungus, boat.
These superclasses consist of all
ImageNet classes which fall under the category of the superclass.
To sample an image from a superclass, we uniformly sample an ImageNet class from the superclass and then sample from the
GAN conditioned on this class.
We sample 1000 images per superclass and train a ResNet-56 (He et al., 2016) to predict the superclass, achieving 93.74% validation accuracy.
Next, we approximately project GAN images onto the mislabeled set of the trained classiﬁer.
We approximate the projection as follows: we optimize an objective consisting of the (cid:96) 2 distance from the original image and the negative cross entropy loss of the pretrained classiﬁer w.r.t the superclass label.
Letting M denote the GAN mapping, x the original image, y the label, and F the pre-trained classiﬁer, the objective is as follows:
[Equation]
We optimize z for 2000 gradient descent steps using λ ce = 10 and a learning rate of 0 . 0003 , intialized with the same latent variable as was used to generate x .
The resulting M ( z ) is a neighbor of x in the set M ( F ) , the mistakenly labeled set of F .
After performing this procedure on 200 GAN images sampled from each class, we ﬁnd that 20% of these images x have a neighbor x (cid:48)
∈ M ( F ) with (cid:107) x − x (cid:48) (cid:107) 2 ≤ 19 . 765 .
Note that this corresponds to modifying each pixel by 0.024 on average for pixel values in [0, 1].
We use (cid:99) M to denote the set of mislabeled neighbors found this way.
From visual inspection, we ﬁnd that the neighbors appear very visually similar to the original image, suggesting that it is appropriate to regard these images as “neighbors”.
In Figure 1, we visualize typical examples of the neighbors found by this procedure.
Thus, setting B ( x ) =
{ x (cid:48) : (cid:107) x (cid:48) − x (cid:107) 2 ≤ 19 . 7652 } , the set M ( F ) , which has probability 0.0626, has a relatively large neighborhood induced by B of probability 0.2.
This supports our expansion assumption, especially the additive notion in Section B.
Next, we use this same classiﬁer as a pseudolabeler to perform self-training on a dataset ofadditional unlabeled images per superclass, where these images were sampled independently from the 200 GAN images in the previous step.
We add input consistency regularization to the selftraining procedure using VAT (Miyato et al., 2018).
After self-training, the validation accuracy of new classiﬁer (cid:101) G improves to 95.69%.
Furthermore, we evaluate performance of the self-trained classiﬁer (cid:101) G on a subset of (cid:99) M with distance greater than 1 from its neighbor.
We let (cid:99) M (cid:48) denote this subset.
We choose to ﬁlter (cid:99) M this way to rule out cases where the original neighbor was already misclassiﬁed.
We ﬁnd that (cid:101) G achieves 67.27% accuracy
In addition, Figure 3 demonstrates that (cid:101) G is more accurate on examples from (cid:99) M (cid:48) which are closer to the original neighbor used to initialize the projection.
This provides evidence that inputconsistency-regularized self-training is indeed correcting the mistakes of the pseudolabeler by relying on correctly-pseudolabeled neighbors for denoising, because Figure 3 shows that examples which are closer to their neighbors are more likely to be denoised.
Finally, we also remark that Figure 3 provides evidence that the denoising mechanism does indeed generalize from the self-training dataset to the population, because neither examples in (cid:99) M (cid:48) nor their original neighbors appeared in the self-training dataset.
E. P SEUDOLABELING EXPERIMENTS
In this section, we verify that the theoretical objective in (4.1) works as intended.
We consider an unsupervised domain adaptation setting where we perform self-training using pseudolabels from the source classiﬁer.
We evaluate the following incremental steps towards optimizing the ideal objective (4.1), with the aim of demonstrating the improvement from adding each component of our theory:
Source: We train a model on the labeled source dataset and directly evaluate it on the target validation set.
PL: Using the classiﬁer obtained above, we produce pseudolabels on the target training set and train a new classiﬁer to ﬁt these pseudolabels. PL+
VAT : We consider the case when the perturbation set B ( x ) in our theory is given by an (cid:96) 2 ball around x .
We train a classiﬁer to ﬁt pseudolabels while regularizing adversarial robustness on the target domain using the VAT loss of (Miyato et al., 2018), obtaining the following loss over classiﬁer
[Equation]
Note that this loss only enforces true stability on examples where F ( x ) correctly predicts G pl ( x ) .
For pseudolabels not ﬁt by F , the cross-entropy loss discourages the model from being conﬁdent, and therefore the discrete labels may still easily ﬂip under input transformations for such examples.
PL+VAT+AMO: Because the theoretical guarantees in Theorem 4.3 are for the population loss, we apply the AMO algorithm of (Wei & Ma, 2019b) in the VAT loss term to regularize the robust all-layer margin (see Section 3.3).
This encourages robustness on the training set to generalize better.
PL+VAT+AMO+MinEnt: Note that PL+VAT only encourages robustness for examples which ﬁt the pseudolabel, but an ideal classiﬁer should not ﬁt pseudolabels which disagree with the groundtruth.
As the bound in Theorem 4.3 improves with the robustness of F , we aim to also encourage robustness for examples where F does not match G pl .
To this end, we modify the loss to allow the classiﬁer to ignore c fraction of the pseudolabels and optimize min-entropy loss on these examples instead.
We provide additional details on how to select the pseudolabels to ignore below.
MinEnt+VAT+AMO: We investigate the impact of the pseudolabels by removing them from the objective.
We instead rely on the following loss which simply performs entropy minimization on the target while ﬁtting the source dataset:
[Equation]
We include the source loss for training stability.
As before, we apply the AMO algorithm in the VAT loss term to encourage robustness of the classiﬁer to generalize.
Table 1 shows the performance of these methods on six unsupervised domain adaptation benchmarks.
We see that performance improves as we add additional components to the objective to match the theory.
We note that the goal of these experiments is to validate our theory, not to push state-ofthe-art for these datasets, which often relies on domain confusion (Tzeng et al., 2014;
Ganin et al., 2016; Tzeng et al., 2017), which is outside the scope of our theory.
For example, Shu et al.
(2018) achieve strong results on these benchmarks by using a domain confusion technique while optimizing
VAT loss and entropy minimization on the target while training on labeled source data.
Our results for MinEnt+VAT+AMO show that when the domain confusion is removed, performance suffers and is actually worse than training on the source only for all datasets except STL-10 to CIFAR-10.
We provide additional experimental details below.
We use the same dataset setup and model architecture for each dataset as (Shu et al., 2018).
All classiﬁers are optimized using SGD with cosine learning rate and weight decay of 5e-4 and target batch size of 128.
The value of the learning rate is tuned on the validation set for each dataset and method in the range of values { 0 . 03 , 0 . 01 , 0 . 003 , 0 . 001 } .
We choose λ v , the coefﬁcient of the VAT loss, by tuning in the same manner in the range { 3 , 10 , 30 } .
For MinEnt+VAT+AMO, we ﬁx the best hyperparameters for PL+VAT+AMO+MinEnt and tune λ s ∈ { 0 . 25 , 0 .
5 , 1 } and ﬁx λ t = 1 .
We also tune the batch size for the source loss in { 64 , 128 } .
Table 1 depicts accuracies on the target validation set.
We use early stopping and display the best accuracy achieved during training.
All displayed accuracies are on one run of the algorithm, except for the (+MinEnt) method, where we average over 3 independent runs with the same hyperparameters.
To compute the VAT loss (Miyato et al., 2018), we take one step of gradient descent in image space to maximize the KL divergence between the perturbed image and the original.
We then normalize this gradient to (cid:96) 2 norm 1 and add it to the image to obtain the perturbed version.
To incorporate the AMO algorithm of (Wei & Ma, 2019a), we also optimize adversarial perturbations to the three hidden layers preceding pooling layers in the DIRT-T architecture.
The initial values of the perturbations are set to 0, and we jointly optimize them with the perturbation to the input using one step of gradient ascent with a learning rate of 1.
Finally, we provide details on how we choose pseudolabels to ignore for the
PL+VAT+AMO+MinEnt objective.
Some care is required in this step to prevent the optimization objective from falling into bad local minima.
We will maintain a model whose weights are the exponential moving average of the past model weights, F ema .
Every gradient update, the weights of F ema are updated by W ema ← 0 .
999 W ema + 0 .
001 W curr , where W curr is the current model weight after the gradient update.
Our aim is to throw out τ i -fraction of pseudolabels which maximize (cid:96) cross-ent ( F ema ( x ) , G pl ( x )) , where G pl ( x ) is the pseudolabel for example x , and i indexes the current iteration.
We will increase τ i linearly from 0 to its ﬁnal value τ over the course of training.
Towards this goal, we maintain an exponential moving average of the (1 − τ i ) - quantile of the loss, which is updated every iteration using the (1 − τ i ) -quantile of the loss (cid:96) cross-ent ( F ema ( x ) , G pl ( x )) computed on the current batch. We ignore pseudolabels where this loss value is above the maintained exponential moving average for the (1 − τ i ) -th loss quantile.
[Table]