T HEORETICAL A NALYSIS OF C ONSISTENCY R EGULARIZATION WITH L IMITED A UGMENTED D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data.
In this paper, we take a small step in this direction; we present a simple new statistical framework to analyze data augmentation - speciﬁcally, one that captures what it means for one input sample to be an augmentation of another, and also the richness of the augmented set.
We use this to interpret consistency regularization as a way to reduce function class complexity and characterize its generalization performance.
Specializing this analysis for linear regression shows that consistency regularization has strictly better sample efﬁciency as compared to empirical risk minimization (ERM) on the augmented set.
In addition, we also provide generalization bounds under consistency regularization for logistic regression and two-layer neural networks.
We perform experiments that make a clean and apples-to-apples comparison (i e. with no extra modeling or data tweaks) between ERM and consistency regularization using
CIFAR-100 and WideResNet
; these demonstrate the superior efﬁcacy of consistency regularization
I NTRODUCTION
Modern machine learning models, especially deep learning models, require abundant training samples.
Since data collection and human annotation are expensive, data augmentation has been a ubiquitous practice in creating artiﬁcial labeled samples and improving generalization performance.
This practice is corroborated by the fact that the semantics of images remain the same through simple translations like obscuring, ﬂipping, rotation, color jitter, rescaling (Shorten & Khoshgoftaar, 2019).
Conventional algorithms use data augmentation to expand the training data set (Krizhevsky et al., 2012; Simard et al., 1998; Cubuk et al., 2018; Simonyan & Zisserman, 2014; He et al., 2016).
As an alternative, consistency regularization enforces the model to output similar predictions on the original and augmented samples and contributes to many recent state-of-the-art supervised or semisupervised algorithms.
This idea was ﬁrst proposed in (Bachman et al., 2014) and popularized by Laine & Aila (2016); Sajjadi et al.
and gained more attention recently with the success of
FixMatch (Sohn et al., 2020) for semi-supervised few-shot learning, and AdaMatch (Berthelot et al., 2021) for domain adaptation.
Several recent papers (see e.g. (Chen et al., 2020a; Mei et al., 2021; Lyle et al., 2019)) attempt to provide a theoretical understanding of data augmentation (DA); they focus on establishing that augmenting data saves on the number of labeled samples needed for the same level of accuracy.
However, none of these explicitly compare in an apples to apples way the efﬁcacy (in terms of the number of augmented samples) of one algorithmic choice of how to use the augmented samples vs another algorithmic choice.
Another dimension un-explored in previous work is any characterization of the quality of augmentation.
In this paper, we hope to answer the following research question:
Is it possible to develop a theoretical framework to compare the sample efﬁciency of different algorithms that use augmented data?
We present a new theoretical framework that casts consistency regularization as a way to reduce function class complexity, which immediately connects to the well-established theory on generalization and gives rise to a generalization bound for consistency regularization under a general bounded loss function.
When specialized to linear regression, this new theoretical framework shows that the consistency regularization is strictly more sample efﬁcient than empirical risk minimization (ERM) on the augmented dataset.
In addition, using this framework, we also provide generalization bounds under consistency regularization for logistic regression, two-layer neural networks, and expansion-based data augmentations.
As summary, our main contributions are:
• A statistical framework of consistency regularization.
We ﬁrst present a simple new statistical framework to analyze data augmentation - with a formal theoretical deﬁnition of data augmentation and its strength.
We then use this framework to give a generalization bound of consistency regularization and provide instantiations for linear regression, logistic regression, two-layer neural networks, and expansion-based data augmentations.
• Theoretically proving the efﬁcacy of consistency regularization.
When specializing our framework with consistency regularization to linear/logistic regression, it yields a strictly smaller generalization error than ERM with the same augmented data.
• Empirical comparisons between consistency regularization and ERM.
We perform experiments that make a clean and apples-apples comparison (i
e., with no extra modeling or data tweaks) between consistency regularization and ERM using CIFAR-100 and WideResNet.
Our empirical results demonstrate the superior efﬁcacy of consistency regularization.
R ELATED W ORK
Empirical ﬁndings.
Data augmentation (DA) is an essential recipe for almost every state-of-theart supervised learning algorithm since the seminal work of (Krizhevsky et al., 2012)
(see reference therein (Simard et al., 1998; Cubuk et al., 2018; Simonyan & Zisserman, 2014;
He et al., 2016;
Kuchnik & Smith, 2018)).
It started from adding augmented data to the training samples via (random) perturbations, distortions, scales, crops, rotations, and horizontal ﬂips.
More sophisticated variants were subsequently designed; a non-exhaustive list includes Mixup (Zhang et al., 2017),
Cutout (DeVries & Taylor, 2017), and Cutmix (Yun et al., 2019).
The choice of data augmentation and their combinations require domain knowledge and experts’ heuristics, which triggered some automated search algorithm to ﬁnd the best augmentation strategies (Lim et al., 2019; Cubuk et al., 2019).
The effects of different DAs have been systematically explored in (Tensmeyer & Martinez, 2016).
Recent practices not only add augmented data to the training set but also enforce the predictor output to be similar by adding consistency regularization (Bachman et al., 2014; Laine & Aila, 2016;
Sohn et al., 2020).
One beneﬁt of consistency regularization is the feasibility of exploiting unlabeled data.
Therefore input consistency on augmented data also formed a major component to state-of-theart algorithms for semi-supervised learning (Laine & Aila, 2016; Sajjadi et al., 2016; Sohn et al., 2020; Xie et al., 2020), self-supervised learning (Chen et al., 2020b), and unsupervised domain adaptation (French et al., 2017; Berthelot et al., 2021).
Theoretical studies.
Many interpret the effect of DA as some form of regularization (He et al., 2019).
Some work focuses on linear transformations and linear models (Wu et al., 2020) or kernel classiﬁers (Dao et al., 2019).
Convolutional neural networks by design enforce translation equivariance symmetry (Benton et al., 2020; Li et al., 2019); further studies have hard-coded CNN’s invariance or equivariance to rotation (Cohen & Welling, 2016; Marcos et al., 2017; Worrall et al., 2017;
Zhou et al., 2017), scaling (Sosnovik et al., 2019; Worrall & Welling, 2019) and other types of transformations.
A line of work view data augmentation as invariant learning by averaging over group actions (Chen et al., 2020a; Mei et al., 2021; Lyle et al., 2019).
They consider an ideal setting that is equivalent to ERM with all possible augmented data, bringing a clean mathematical interpretation.
We are interested in a more realistic setting with limited augmented data.
In this setting, it is crucial to utilize the limited data with proper training methods, the difference of which cannot be revealed under the previous studied settings.
Some more recent work investigates the feature representation learning procedure with DA for selfsupervised learning tasks (Wen & Li, 2021; HaoChen et al., 2021; von K¨ugelgen et al., 2021; Garg & Liang, 2020).
Cai et al.
; Wei et al.
(2021) studied the effect of data augmentation with label propagation.
Data augmentation is also deployed to improve robustness (Rajput et al., 2019), to facilitate domain adaptation and domain generalization (Cai et al., 2021; Sagawa et al., 2019) .
D ATA A UGMENTATION C ONSISTENCY AND H OW I T L EARNS E FFICIENTLY
In this section, we ﬁrst formally deﬁne data augmentation and introduce the problem setup.
We then deﬁne data augmentation consistency (DAC) regularization and show how it effectively reduces the function class complexity, which connects to a generalization bound for bounded loss functions via Rademacher complexity.
Subsequently, we specialize our general result to linear regression, which ﬁrmly shows that the DAC regularization provably learns more efﬁciently than minimizing the empirical risk on the augmented dataset.
The following section will present more applications (including logistic regression, neural network, etc.).
3. P ROBLEM SETUP AND DATA AUGMENTATION
Consider the standard supervised learning problem setup: x ∈ X are input features, and y ∈ Y is its label (or response).
Let P ∗ be the true distribution of ( x , y ) (i.e., the label distribution follows y ∼ P ∗ ( y | x ) ).
We can then formally deﬁne data augmentation as:
Deﬁnition 1 (Data augmentation) .
For any sample x ∈ X , we say x (cid:48) ∈
X is its augmentation, if and only if P ∗ ( y | x ) =
P ∗ ( y | x (cid:48) ) .
The deﬁnition above speciﬁes what it means for one input sample to be an augmentation of another.
While the deﬁnition covers any x (cid:48) with the same label distribution as x , our results only use the augmented samples that can be achieved via certain transformations (e.g., random cropping, rotation).
However, our deﬁnition does not cover augmentations that alter the labels (e.g., MixUp (Zhang et al., 2017)).
Now we introduce the learning problem on an augmented dataset: Let ( X , y
) ∈
X N × Y N be a training set consisting of N i.i.d.
samples.
Besides the original ( X , y ) , each training sample in it is provided with α augmented samples.
The input features of the augmented dataset can be written as:
[Equation]
where x i is in the original training set and x i,j , ∀ j ∈
[ α ] are the augmentations of x i .
The labels of the augmented samples are kept the same, which can be denoted as (cid:102)
My ∈ Y (1+ α ) N , where (cid:102) M ∈ R (1+ α ) N
× N is a vertical stack of (1 + α ) identity mappings.
Speciﬁcally, when the input x i s are d -dimensional real vectors, we have the following notion of augmentation strength d aug :
Deﬁnition 2 (Strength of augmentations) .
For any δ ∈ (0 , 1) , let
[Equation]
Intuitively, strength of augmentations d aug ( δ ) means that with probability at least 1 − δ , the augmentations perturb at least d aug ( δ ) dimensions; whereas d aug can be intuitively understood as the minimum number of dimensions that the augmentations in (cid:101)
A ( X ) perturbed with high probability.
A lager d aug corresponds to a stronger data augmentations.
For instance, when (cid:101)
A ( X ) = (cid:102) MX almost surely ( e.g. , when the augmentations are identical copies of the original samples, corresponding to the weakest augmentation – no augmentations at all), we have d aug ( δ ) =
d aug = 0 for all δ ∈ (0 , 1) .
On the other hand, if the augmentations are randomly generated, then it is more likely to see larger d aug (i.e., more dimensions being perturbed) with larger α (i.e., more augmentations).
In the next subsection, we formally introduce “data augmentation consistency regularization” and present a generalization bound under bounded loss functions.
We subsequently specialize the bound to linear regression and show that consistency regularization is strictly more sample efﬁcient than empirical risk minimization (ERM) on the augmented dataset.
3. D ATA AUGMENTATION CONSISTENCY REGULARIZATION
Let H =
{ h : X → Y} be a well-speciﬁed function class (e.g. for regression problems, ∃ h ∗ ∈ H , s.t.
h ∗ ( x ) =
E [ y | x ] ) that we hope to learn from.
Without loss of generality, we assume that each function h ∈ H can be expressed as h =
f
h ◦ φ h , where φ h ∈
Φ = { φ : X → W} is a proper representation mapping and f h ∈ F = { f : W → Y} is a predictor on top of the learned representation.
We tend to decompose h such that φ h is a powerful feature extraction function whereas f h can be as simple as a linear combiner.
For instance, in a deep neural network, all the layers before the ﬁnal prediction layer can be viewed as feature extraction φ h , and the predictor f h corresponds to the ﬁnal linear combination of the features.
For a loss function l : Y × Y → R and a metric (cid:37) properly deﬁned on the representation space W , learning with data augmentation consistency (DAC) regularization can be expressed as:
[Equation]
Note that the DAC regularization in Equation (1) can be easily implemented empirically as a regularizer.
Intuitively, DAC regularization penalizes the representation difference between the original sample φ h ( x i ) and the augmented sample φ h ( x i,j ) , with the belief that similar samples (i.e., original and augmented samples) should have similar representations.
When the data augmentations do not alter the labels, it is reasonable to enforce a strong regularization (i.e., λ → ∞ ) - given that the conditional distribution of y does not change.
The function (cid:98) h dac learned with data augmentation consistency regularization can then be written as the solution of a constrained optimization problem:
[Equation]
The constraint in Equation (2) effectively reduces the size of the function class H .
To rigorously capture such reduction, we deﬁne the following data augmentation consistency (DAC) operator over
H .
Given the original training samples X and the augmented dataset (cid:101)
A ( X ) , we have:
Deﬁnition 3 (Data Augmentation Consistency Operator) .
[Equation]
Original Function Class:
Reduced Function Class: can be different.
is the feature mapping of s.t.
Particularly, we assume that a proper representation class Φ is chosen with respect to the augmentations such that h ∗ ∈ T dac (cid:101)
A , X ( H ) .
The DAC operator maps the original function class H to a (potentially much smaller) subset T dac (cid:101) A , X ( H ) , where every function h ∈ T dac (cid:101)
A , X ( H ) gives consistent representation for all samples and their augmentations (i.e., φ h ( x i )
=
φ h ( x i,j ) ).
It is now clear that with DAC regularization, Equation (2) is effectively learning in the function class T dac (cid:101) A , X ( H ) , which is a subset of H .
As we will show in the next subsection, the function class size reduction is the key for efﬁcient learning with data augmentations.
One of the contributions of our paper is to view consistency regularization as function class complexity reduction.
Our next proposition connects the generalization bound to the Rademacher complexity
T dac (cid:101) A , X ( H ) via standard analysis.
We further provide various instantiations in the rest of our paper.
Let L ( h ) denote the population loss induced by h ∈ H , i.e., L ( h ) (cid:44) E ( x ,y ) ∼ P ∗ [ l ( h ( x ) , y )] . Proposition 1 (formally in Proposition 8) .
With Equation (2) , if l is B -bounded and C l -Lipschitz, then for a ﬁxed T dac (cid:101) A , X ( H ) and any δ ∈ (0 , 1) , with probability at least 1 − δ , we have
[Equation]
N .
Remark 1.
Our results can be immediately extended to use unlabeled data.
Notice that the DAC regularization only enforces the same predictions for the original and augmented samples, where no ground truth label is needed.
For clarity of exposition, we focus on the labeled dataset in the main text and defer discussions on unlabeled data to the appendix.
3. E FFICACY OF DAC REGULARIZATION : A GENTLE START WITH LINEAR REGRESSION
To see the efﬁcacy of DAC regularization (i.e., Equation (2)), we revisit a more commonly adopted training method here - empirical risk minimization (ERM) on augmented data:
[Equation]
Now we show that the DAC regularization learns more efﬁciently than ERM.
Consider the following setting: given N observations X ∈ R N × d , the responses y ∈ R N are generated from a linear model y
=
X θ ∗ + (cid:15) , where (cid:15) ∈
R N is zero-mean noise with E (cid:2) (cid:15)(cid:15) (cid:62) (cid:3) =
σ 2 I N .
Recall that (cid:101)
A ( X ) is the entire augmented dataset, and (cid:102)
My corresponds to the labels.
Our next result characterizes the ﬁxed design excess risk of θ on (cid:101)
A ( X ) , which is deﬁned as L ( θ ) (cid:44)
[Equation]
A [Equation] θ − (cid:101) A
[Equation] Under regularity conditions (e.g., x is sub-Gaussian and N is not too small), it is not hard to extend to random design, i.e., the more commonly acknowledged generalization bound with the same order.
Given (cid:101) A ( X ) , notice that by Deﬁnition 2, d aug = rank (cid:16) (cid:101) A ( X ) − (cid:102) MX (cid:17) since there is no randomness in (cid:101)
A , X in ﬁxed design setting.
For a linear regression model to be identiﬁable (i.e., having an unique solution), we assume that (cid:101)
A ( X ) has full column rank.
We then have the following theorem on the excess risks of learning by DAC regularization and by ERM on the augmented dataset.
Theorem 2 (Informal result on linear regression (formally in Theorem 6)) .
Learning with DAC regularization, we have E (cid:104) L ( (cid:98) θ dac )
− L ( θ ∗ ) (cid:105) =
( d − d aug ) σ 2 N , while learning with ERM directly on the augmented dataset, we have E (cid:104) L ( (cid:98) θ da − erm ) − L ( θ ∗ ) (cid:105) =
( d − d aug + d (cid:48) )
σ 2 N , where d (cid:48) ∈ [0 , d aug ] .
Formally, d (cid:48) is deﬁned as d (cid:48)
(cid:44) tr
(( H (cid:101)
A ( X ) − P S )
(cid:102) M (cid:102)
M (cid:62) )
+ α , where H (cid:101)
A ( X ) (cid:44) (cid:101)
A ( X ) (cid:101)
A ( X ) † , and P S is the orthogonal projector onto S (cid:44) (cid:110)(cid:102) MX θ | ∀ θ ∈ R d , s.t. (cid:16) (cid:101)
A ( X ) − (cid:102) MX (cid:17) θ = 0 (cid:111) .
[Figure]
The results precisely match Theorem 2. DA-ERM depends on the d (cid:48) induced by different augmentations, while the DAC regularization works equally well and better than the DA-ERM. Further, both DAC and DA-ERM are affected by the “strength of augmentation”
d aug .
Here we present an explanation for d (cid:48) .
Note that σ 2 · (cid:102) M (cid:102) M (cid:62) is the noise covariance matrix of the augmented dataset.
We have tr (cid:16) P S (cid:102) M (cid:102) M (cid:62) (cid:17) being the variance of (cid:98) θ dac , and tr (cid:16) H (cid:101)
A ( X ) (cid:102) M
(cid:102) M (cid:62) (cid:17) being the variance of (cid:98) θ
da − erm .
Therefore, d (cid:48) ∝ tr (cid:16)(cid:16) H (cid:101)
A ( X ) − P S (cid:17) (cid:102)
M (cid:102) M (cid:62) (cid:17) is used to measure the difference.
When H (cid:101) A ( X ) (cid:54) =
P S (a common scenario as instantiated in Example 1), we have DAC being strictly better than ERM on augmented data.
Example 1.
Consider a 30-dimensional linear regression.
The original training set contains 50 samples.
The inputs x i s are generated independently from N (0 , I 30 ) and we set θ ∗ =
[ θ ∗ c ; 0 ] with θ ∗ c ∼ N (0 , I 5 ) and 0 ∈ R 25 .
The noise variance σ is set to 1 .
We break x into 3 parts [ x c 1 , x e 1 , x e 2 ] and take the following augmentations:
A ([ x c 1 ; x e 1 ; x e 2 ]) =
[ x c 1 ; 2 x e 1 ; − x e 2 ] , x c 1 ∈ R
d
c 1 , x e 1 ∈ R d e 1 , x e 2 ∈ R d e 2 , where d c 1 + d e 1 + d e 2 = 30 .
Notice that the augmentation perturbs x e 1 and x e 2 and leaving x c 1 unchanged, we therefore have d aug = 30 − d c 1 .
By changing d c 1 and d e 1 , we can have different augmentations with different d aug , d (cid:48) .
The results for d aug ∈ { 20 , 25 } and various d (cid:48) s induced by different d e 1 s are presented in
Figure 2.
The excess risks precisely match Theorem 2.
It conﬁrms that the DAC regularization is strictly better than ERM on an augmented dataset for a wide variety of augmentations.
V ARIOUS A PPLICATIONS
Now we ground our general result on the DAC regularization with a set of common applications, including the logistic regression, two-layer neural networks, and expansion-based data augmentations.
For each of the applications, we ﬁrst specify the ground truth distribution P ∗ , the function class H , the loss function l , as well as the augmented dataset (cid:101)
A ( X ) .
Then, we discuss the corresponding excess risk L ( (cid:98) h dac )
− L ( h ∗ ) for the DAC regularization (Equation (2)).
We abridge the analysis in a set of concrete examples with concise arguments while deferring the complete assumptions and theorems to Appendix B. As a supplementary remark, in addition to the popular in-distribution setting where we consider a unique distribution P ∗ for both training and testing, DAC regularization is also known to improve out-of-distribution generalization for settings like domain adaptation. We defer detailed discussion on such advantage of DAC regularization for linear regression in the domain adaptation setting to Appendix B.4.
4. L OGISTIC REGRESSION
Here we consider logistic regressions with X = (cid:8) x ∈ R d (cid:12)(cid:12) (cid:107) x (cid:107) 2 ≤ D (cid:9) , Y = { 0 , 1 } .
For some unkown θ ∗ ∈ R d with (cid:107) θ ∗ (cid:107) ≤ C 0 , we have P ∗ ( y = 1 | x ) = σ (cid:0) θ ∗(cid:62) x (cid:1) , where σ ( · ) is the sigmoid function σ ( z ) (cid:44)
(1 + exp( − z )) .
The function class H is H = (cid:8) h
( x ) = θ (cid:62) x (cid:12)(cid:12) θ ∈ R d , (cid:107) θ (cid:107) 2 ≤ C 0 (cid:9) , such that predictions are given by (cid:98) y
= σ (cid:0) θ (cid:62) x (cid:1) .
For binary classiﬁcation, we use the logistic loss l ( θ (cid:62) x , y )
= − y log( σ ( θ (cid:62) x )) − (1 − y )
log(1 − σ ( θ (cid:62) x )) .
Recall the strength of augmentations d aug from Deﬁnition 2.
Under proper regularity conditions (i.e., x is sub-Gaussian and N is not too small, see Appendix B), we have the following generalization bound for learning logistic regression under DAC regularization.
Theorem 3 (Informal result on logistic regression with DAC (formally in Theorem 9)) .
Learning logistic regression with the DAC regularization h ( x i )
=
h ( x i,j ) , with high probability:
[Equation]
where the strength of augmentation d aug ∈ [0 , d − 1] .
Intuitively, with high probability, minimizing empirical loss on augmented samples gives a generalization bound of L ( (cid:98) h da − erm ) − L ( h ∗ ) (cid:46) max (cid:18) (cid:113) d ( α +1) N , (cid:113)
d − d aug N (cid:19) at best, where the ﬁrst term corresponds to the generalization bound for a d -dimensional logistic regression with ( α + 1) N samples, and the second term follows as the augmentations fail to perturb a ( d − d aug ) -dimensional sub-space (and in which ERM can only rely on the N original samples for learning).
The ﬁrst term will dominate the max when there is limited augmented data (i.e., α is small).
Comparing the two, we see that DAC is more efﬁcient than ERM.
In particular, consider the scenario that the limited data augmentations well perturb the data (e.g., α = 1 and d aug = d − 1 ).
The ERM gives a generalization error that scales as (cid:112) d/N , while DAC yields a dimension-free (cid:112) 1 /N error.
Please see Appendix E for a supporting numerical example, which demonstrates the beneﬁts of DAC over DA-ERM for logistic regression, and empirically veriﬁes the impact of d aug and α that matches with our theoretical analysis.
4. T WO - LAYER NEURAL NETWORK
In this section, we discuss a special case where the result for bounded losses in Proposition 8 can be extended to the unbounded square loss.
With X = R d and Y =
R , we consider a ground truth distribution P ∗ ( y | x ) induced by a two-layer ReLU network:
y [Equation] +
[Equation] [ b ∗ 1 . . .
b ∗ k . . .
b ∗ d ] , for some unknown h ∗ ( x ) (cid:44) (cid:0) x (cid:62) B ∗ (cid:1) + w ∗ where ( · ) + (cid:44) max(0 , · ) elementwisely denotes the ReLU function, b ∗ k ∈ S
d − 1 for all k ∈ [ q ] , and z ∼ N (cid:0) 0 , σ 2 (cid:1) is the Gaussian noise.
In terms of the function class H , for some constant C w ≥ (cid:107) w ∗ [Equation] 1 , we have:
[Equation]
such that h ∗ ∈ H .
We use the standard square loss l ( h ( x ) , y )
=
12 ( h ( x ) − y ) 2 .
Let P N be the projector onto the null space of (cid:101) A ( X ) − (cid:102) MX .
When the augmented sample set (cid:101)
A ( X ) is reasonably diverse (see Appendix B.2), regression over two-layer ReLU networks with the
DAC regularization generalizes as following:
Theorem 4 (Informal result on two-layer neural network with DAC (formally in Theorem 10)) .
Assuming E (cid:104) [Equation] n
[Equation] ni =
[Equation]
P N x i
[Equation]
22 (cid:12)(cid:12)(cid:12) P N (cid:105) ≤ C 2 N for some C N > 0 , learning the two-layer ReLU network with the DAC regularization on (cid:0) x (cid:62) i B (cid:1) + = (cid:0) x (cid:62) i,j B (cid:1) + gives, with high probability:
[Equation]
Recall the strength of augmentations d aug from Deﬁnition 2.
Under proper regularity conditions ( i.e. , x is sub-Gaussian and N is reasonably large, see Appendix B.2), we have C N (cid:46) (cid:112)
d − d aug with high probability.
Analogous to the logistic regression example, applying the
ERM directly on the augmented samples achieves no better than L
( (cid:98) h da − erm ) − L ( h ∗ ) (cid:46) σC
[Equation] max [Equation] (cid:113)
[Equation] N (cid:19) with high probability.
The comparison again illustrates the advantage of DAC regularization over ERM.
The advantage is large when the limited data augmentations are strong (i.e., large d aug and small α ).
4. E XPANSION - BASED DATA AUGMENTATIONS
In this section, we demonstrate that for the classiﬁcation problems, enforcing consistency on a different notion of data augmentations based on expansion also brings a considerable reduction in complexity of the feasible function class.
Concretely, we consider a multi-class classiﬁcation problem: for an arbitrary set X , let Y =
[ K ] , and h ∗ :
X → [ K ] be the ground truth classiﬁer that partitions X : for each k ∈ [ K ] , let X k (cid:44) { x ∈ X | h ∗ ( x ) =
k } , with X i ∩ X j = ∅ , ∀ i (cid:54) =
j .
Here we focus on the expansion-based data augmentations deﬁned as following:
Deﬁnition 4 (Expansion-based data augmentations) .
Let A : X → 2 X be a function generating a set of augmentations A ( x ) from a given x ∈ X that satisﬁes the following:
(a) Nontrivial augmentation and class invariant: { x } (cid:40)
A ( x ) ⊆ { x (cid:48) ∈ X | h ∗ ( x )
= h ∗ ( x (cid:48) ) } for all x ∈ X ; and (b) Non-trivial expansion: for all k ∈
[ K ] , given any ∅ (cid:40) S (cid:40)
X k , there exists some x (cid:48) / ∈ S such that A ( x ) ∩ A ( x (cid:48) )
(cid:54) = ∅ .
We consider a general class of classiﬁers H ⊆
{ h : X → [ K ] } where the ground truth classiﬁer is realizable, h ∗ ∈ H .
With the zero-one loss l 01 ( h ( x ) , y ) =
1 { h ( x ) (cid:54) =
y } , and the corresponding population loss L 01 ( h ) (cid:44) E
x [ 1 { h ( x ) (cid:54) =
h ∗ ( x ) } ] , we learn a classiﬁer from H with the DAC regularization, h ( x )
= h ( x (cid:48) ) , where x (cid:48) s are expansion-based data augmentations generated by an A (Deﬁnition 4).
As a warm-up, we begin with a simpliﬁed setting where we enforce consistency over the population in lieu of a ﬁnite set of training samples as in practice.
Then, learning with the DAC regularization yields a classiﬁer (cid:98)
h dac ∈ T dac A , X ( H ) (cid:44) { h ∈ H | h ( x )
= h ( x (cid:48) )
∀ x ∈ X , x (cid:48)
∈
A ( x ) } .
Theorem 5 (DAC with expansion-based data augmentations over population) .
Learning the classiﬁer with DAC regularization over population, with high probability,
[Equation]
Particularly, with the DAC regularization, the generalization bound of the K -class classiﬁcation problem is dimension-independent but only scales with the number of classes, (cid:101) O ( K ) .
Furthermore, in Appendix B.3, we extend the result to a more practical setting where the consistency is enforced over a ﬁnite training set.
Notably, Theorem 5, along with the ﬁnite train set case in
Appendix B.3, is a reminiscence of Theorem 3.6 and 3.7 by Wei et al.
(2021), as well as Theorem 2.1, 2.2, and 2.3 by Cai et al.
We adapt these existing theories and provide a uniﬁed analysis under our setting, which demonstrates the generality of our framework.
E XPERIMENTS
In this section, we empirically verify that training with DAC learns more efﬁciently than empirical risk minimization on an augmented dataset.
The dataset is derived from CIFAR-100, where we randomly select 10,000 labeled data as the training set (i.e., 100 labeled samples per class).
During the training time, given a training batch, we generate augmentations by RandAugment (Cubuk et al., 2020).
We set the number of augmentations per sample to 7 unless otherwise mentioned.
The experiments focus on comparisons of 1) training with consistency regularization, and 2) empirical risk minimization on the augmented dataset (DA-ERM).
In particular, we use the same network architecture (a WideResNet-28-2 (Zagoruyko & Komodakis, 2016)) and the same training settings (e.g., optimizer, learning rate schedule, etc) for both methods.
We defer the detailed experiment settings to Appendix D. Our test set is the standard CIFAR-100 test set, and we report the average and standard deviation of the testing accuracy of 5 independent runs.
The consistency regularizer is implemented as the l 2 distance of the model’s predictions on the original and augmented samples.
Efﬁcacy of DAC regularization.
We ﬁrst show that the DAC regularization learns more efﬁciently than ERM on the augmented dataset.
The results are listed in Table 1.
Notice that with proper choice of λ (i.e., the multiplicative coefﬁcient before the DAC regularization, see Equation (1)), training with DAC regularization signiﬁcantly improves over DA-ERM.
[Table]
DAC regularization helps more when data is scarce.
Our theoretical results suggest that the
DAC regularization brings more beneﬁts when the training data is scarce.
The data scarcity can be interpreted in two ways: (1) Labeled samples are scarce.
We conduct experiments with different numbers of labeled samples, ranging from 1,000 (i e., 10 images per class) to 20,000 samples (i e., 200 images per class).
We generate 3 augmentations for each of the samples during the training time, and the results are presented in Table 2.
Notice that the DAC regularization gives a bigger improvement over DA-ERM when the labeled samples are scarce.
This matches the intuition that when there are sufﬁcient training samples, data augmentation is less necessary.
Therefore, the difference between different ways of utilizing the augmented samples becomes diminishing.
[Table]
(2) Augmented samples are scarce.
While keeping the number of labeled samples to be we evaluate the performance of the DAC regularization and DA-ERM with different numbers of augmentations.
The number of augmentations for each training sample ranges from 1 to 15, and the results are listed in Table 3.
The DAC regularization offers a more signiﬁcant improvement when the number of augmentations is small.
This clearly demonstrates that the DAC regularization learns more efﬁciently from the limited number of augmentations.
[Table] [Table]
Proper augmentation brings good performance.
To achieve good performance, it is important to have proper data augmentation - it needs to be strong such that it well perturbs the input features, but it should also leave the label distribution unchanged.
Here we experiment with different augmentation strengths, which is the number of different random transformations (e.g., random cropping, ﬂipping, etc.)
applied to the training samples sequentially.
More transformations imply stronger augmentations.
The number of transformations ranges from 1 to 10, and the results are listed in Table 4.
We see that both DA-ERM and the DAC regularization beneﬁts from a proper augmentation.
When the augmentation is too strong (e.g., Augmentation Strength 10, as shown in
Figure 3), the DAC regularization gives a worse performance.
It might be explained by falsely enforcing DAC regularization where the labels of the augmented samples have changed.
[Table] [Table]
Combining with a semi-supervised learning algorithm.
Here we show that the DAC regularization can be easily extended to the semisupervised learning setting.
We take the previously established semisupervised learning method FixMatch (Sohn et al., 2020) as the baseline and adapt the FixMatch by combining it with the DAC regularization.
Namely, besides using FixMatch to learn from the unlabeled data, we additionally generate augmentations for the labeled samples and apply the DAC regularization.
In particular, we focus on the data-scarce regime by only keeping 10,000 labeled samples and at most 20, unlabeled samples.
Results are listed in Table 5.
We see that the DAC regularization also improves the performance of FixMatch when the unlabeled samples are scarce.
This again demonstrates the efﬁciency of learning with DAC regularization.
C ONCLUSION
We present a simple new theoretical framework for understanding the statistical efﬁciency of consistency regularization with limited data augmentations.
In particular, our proposed framework gives a generalization bound for consistency regularization in general cases.
We also provide instantiations for linear regression, logistic regression, two-layer neural networks, and expansion-based data augmentations.
When specialized to linear regression/logistic regression, it shows that consistency regularization yields a strictly smaller generalization error than ERM with augmented data.
We also provide apples-to-apples empirical comparisons between augmented ERM and consistency regularization.
These together demonstrate the superior efﬁcacy of consistency regularization.
Reproducibility Statement Our main theoretical results, including problem setup and (simpliﬁed) theorem statements are in Sections 3 and 4.
The formal theorem statements along with the proofs can be found in Appendices A and B. We present empirical results in Section 5, and the detailed experiment setup is in Appendix D.
We also submit our experiment code in the supplementary material.
A P ROOFS FOR B OUNDED L OSS AND L INEAR R EGRESSION A. P ROOF FOR DAC ON L INEAR R EGRESSION
For ﬁxed (cid:101)
A ( X )
, we recall that d aug = rank (cid:16) (cid:101)
A ( X ) − (cid:102) MX (cid:17) since there is no randomness in (cid:101)
A , X in ﬁx design setting.
Assuming that (cid:101)
A ( X ) admits full column rank, we have the following theorem on the excess risk of DAC and ERM:
Theorem 6 (Formal restatement of Theorem 2 on linear regression.) .
Learning with DAC regularization, we have E (cid:104) L ( (cid:98) θ dac )
− L ( θ ∗ ) (cid:105) =
( d − d aug ) σ 2 N , while learning with ERM directly on the augmented dataset, we have E (cid:104) L ( (cid:98) θ da − erm ) − L ( θ ∗ ) (cid:105) =
( d − d aug + d (cid:48) )
σ 2 N .
d (cid:48) is deﬁned as d (cid:48)
(cid:44) tr
(cid:16)(cid:102) M (cid:62) (cid:16) H (cid:101)
A ( X ) − P S (cid:17) (cid:102) M (cid:17) 1 + α , where d (cid:48) ∈ [0 , d aug ] with H (cid:101)
A ( X ) =
(cid:101)
A ( X ) (cid:16) (cid:101)
A ( X ) (cid:62) (cid:101)
A ( X ) (cid:17) − 1 (cid:101)
A ( X ) (cid:62) and P S ∈
R ( α +1) N × ( α +1) N is the projection matrix to S , and S is deﬁned to be S (cid:44) (cid:110)(cid:102) MX θ | ∀ θ
[Equation]
d , s
[Equation] [Equation]
Proof.
With L ( θ ) (cid:44)
[Equation]
X
[Equation]
2 , the excess risk of ERM on the augmented training set satisﬁes that:
[Equation]
Let C (cid:101)
A ( X ) and C (cid:102)
M denote the column space of (cid:101)
A ( X ) and (cid:102) M , respectively.
Notice that S is a subspace of both C (cid:101)
A ( X ) and C (cid:102) M .
Observing that d aug = rank (cid:16) (cid:101)
A ( X ) − (cid:102) MX (cid:17) =
rank ( P S ) , we have
[Equation]
By the data augmentation consistency constraint, we are essentially solving the linear regression on the ( d − d aug ) -dimensional space (cid:110) θ | ( (cid:101)
A ( X ) − (cid:102) MX ) θ = 0 (cid:111) .
The rest of proof is identical to standard regression analysis, with features ﬁrst projected to S :
[Equation]
B A PPLICATIONS OF L EARNING WITH DAC
To instantiate our general result on the DAC regularization, we start with a concrete application of
Proposition 8 on the logistic regression problem with bounded loss (Section 4.1, Appendix B.1), followed by an extension to the unbounded square loss (Section 4.2, Appendix B.2).
Subsequently, we discuss an alternative notion of data augmentations based on expansion for the classiﬁcation problems (Section 4.3, Appendix B.3).
Finally, we present a supplementary example for the domain adaptation in linear regression (Appendix B.4).
Before diving into the concrete applications, we recall the general setting, and introduce some additional notations used throughout the formal analysis.
DAC regularization with unlabeled data.
Since the DAC regularization leverages only on the unlabeled observable features but not their labels, the unlabeled training set in the DAC regularizer and the labeled training set in the ﬁrst term of Equation (1) for label learning can be considered separately.
For clariﬁcation, in the formal analysis, we distinguish the labeled training set ( X , y ) ∈
X n × Y n from the unlabeled samples X u ∈ X N (possibly with different sizes N ≥ n ), while both sample sets are drawn i.i.d.
from P ∗ .
Particularly, in the supervised learning setting , the DAC regularization is based on the observables from the labeled training set such that X u
= X ( N = n ).
While in the semi-supervised learning setting , X u can be N unlabeled observations drawn i.i.d.
from the marginal distribution P ∗ ( x ) .
For the DAC regularization, we denote the augmentation of the unlabeled samples (excluding the original samples X u , in contrast to the augmented sample (cid:101)
A ( X ) in the main text) as
[Equation]
where for each i ∈ [ N ] , (cid:8) x ui,j (cid:9) j ∈
[ α ] is a set of α augmentations generated from x ui , and let M ∈
R αN × N be the vertical stack of α N × N identity matrices.
Then analogously, with the unlabeled training set X u , we can quantify the strength of the data augmentations (cid:98) A with
[Equation]
, such that 0 ≤ (cid:98) d aug ≤ min ( d, αN ) can be intuitively interpreted as the number of dimensions in the span of the unlabeled samples, Row( X u ) , perturbed by (cid:98) A .
Moreover, we denote N as the ( d − (cid:98) d aug ) -dimensional
null space of (cid:98) A ( X u ) − MX u , and P N as the projection onto N such that
[Equation]
Correspondingly, let N ⊥ be the orthogonal complement of N , and P ⊥N be the projection onto N ⊥ such that P ⊥N = I d − P N .
Then by recalling Deﬁnition 2, we have the following:
Proposition 7.
For any δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
Furthermore, for d aug (cid:44) d aug (1 /n ) , with high probability, (cid:98) d aug ≥ d aug and d − (cid:98) d aug ≤ d − d aug .
In terms of ( X , y ) and X u , the regularization formulation in Equation (1) can be restated as,
[Equation]
and with the notation of (cid:98) A , the corresponding DAC operator is given by
[Equation]
where T dac (cid:98) A , X u and T dac (cid:101) A , X u are equivalent by construction.
Marginal distribution and training set.
Here we formally state the common regularity conditions for the logistic regression (Section 4.1) and the two-layer neural network (Section 4.2) examples in
Assumption 1 and Assumption 2, where we assume X ⊆ R d .
Assumption 1 (Regularity of marginal distribution) .
Let x ∼ P ∗ ( x ) be zero-mean E [ x ] = 0 , with the covairance E [ xx (cid:62) ]
= Σ x (cid:31) 0 .
We assume that ( Σ − 1 / 2 x x ) is ρ 2 -subgaussian 1 , and there exist constants C ≥ c = Θ(1)
such that c
I d (cid:52) Σ x (cid:52) C I d .
Assumption 2 (Sufﬁcient labeled data) .
For ( X , y ) ∈
X n × Y n , with respect to X u and (cid:98) A ( X u ) that characterize (cid:98) d aug , we assume n (cid:29) ρ 4 (cid:16)
d
− (cid:98) d aug (cid:17) .
For a B -bounded loss function l : Y × Y → R ( i.e. , 0 ≤ l ≤ B ) such that for all y ∈ Y , l ( · , y ) :
Y → R is C l -Lipschitz, we have the following:
Proposition 8 (Formal restatement of Proposition 1 on DAC with bounded loss) .
Learning with
DAC regularization (Equation (2) ), for a ﬁxed T dac (cid:98) A , X u ( H ) and any δ ∈ (0 , 1) , with probability at least 1 − δ , we have
[Equation]
where R n ( T dac (cid:98) A , X u ( H )) represents the Rademacher complexity of T dac (cid:98) A , X u ( H ) .
Proof of Proposition 8.
We ﬁrst decompose the expected excess risk as
[Equation]
h dac )
(cid:17)
+ (cid:16)(cid:98) L ( (cid:98) h dac ) − (cid:98) L ( h ∗ ) (cid:17) + (cid:16)(cid:98) L ( h ∗ ) − L ( h ∗ ) (cid:17) , where (cid:98) L ( (cid:98) h dac ) − (cid:98) L ( h ∗ ) ≤ 0 by the basic inequality, and as a consequence, for a ﬁxed H o =
T dac (cid:98) A , X u ( H ) ,
L ( (cid:98) h dac ) − L ( h ∗ ) ≤sup h
[Equation]
dac
[Equation]
A
[Equation]
H ) (cid:12)(cid:12)(cid:12) L
[Equation]
h
[Equation]
−
[Equation]
L
[Equation]
h
[Equation]
(cid:12)(cid:12)(cid:12) =
[Equation]
sup h
[Equation]
o (cid:12)(cid:12)(cid:12) L ( h )
− (cid:98) L ( h ) (cid:12)(cid:12)(cid:12) .
We denote g + ( X , y )
=
sup h ∈H o : L ( h ) − (cid:98) L ( h ) and g − ( X , y )
=
sup h ∈H o : − L ( h )
+ (cid:98) L ( h ) .
Then,
P (cid:104) L ( (cid:98) h dac )
− L ( h ∗ ) ≥ (cid:15) (cid:105) ≤ P (cid:104) g
+ ( X , y ) ≥ (cid:15) 2 (cid:105)
+ P (cid:104) g − ( X , y ) ≥ (cid:15) 2 (cid:105) .
We will derive a tail bound for g + ( X , y ) with the standard inequalities and symmetrization argument Wainwright (2019); Bartlett & Mendelson (2003), while the analogous statement holds for g − ( X , y ) .
Let ( X (1) , y (1) ) be a sample set generated by replacing an arbitrary observation in ( X , y ) with any ( x , y ) ∈
X × Y .
Since l is B -bounded, we have (cid:12)(cid:12) g
+ ( X , y ) − g
+
( X (1) , y (1) ) (cid:12)(cid:12) ≤ B/n .
Leveraging the McDiarmid’s inequality Bartlett & Mendelson (2003),
[Equation]
For an arbitrary sample set ( X , y ) , let (cid:98) L ( X , y )
( h ) =
1 n (cid:80) ni
=1 l ( h ( x i ) , y i ) be the empirical risk of h with respect to ( X , y ) .
Then, by a classical symmetrization argument (e.g., proof of Wainwright (2019) Theorem 4.10), we can bound the expectation: for an independent sample set ( X (cid:48) , y (cid:48) ) ∈
X n × Y n drawn i.i.d.
from P ∗ ,
[Equation]
Since ( X , y ) and ( X (cid:48) , y (cid:48) ) are independent and identically distributed, we can introduce i.i
[Equation]
d.
Rademacher random variables r = { r i ∈ {− 1 , +1 } | i ∈ [ n ] } (independent of both ( X , y ) and ( X (cid:48) , y (cid:48) ) )
such that
[Equation]
r i · ( l ( h ( x (cid:48) i ) , y (cid:48) i ) − l ( h (
[Equation]
i
[Equation]
y i ))
(cid:35)
[Equation]
E ( X , y , r
[Equation]
(cid:34) sup h ∈H o n n (cid:88)
i =
r i · l ( h ( x i ) , y i ) (cid:35) ≤ 2 R n ( l ◦ H o ) where l ◦ H o
= { l ( h ( · ) , · ) :
X × Y → R : h ∈ H o } is the loss function class.
Analogously, E [ g − ( X , y )] ≤ 2 R n ( l ◦ H o ) .
Therefore, with probability at least 1 − δ ,
[Equation]
Finally, since l ( · , y ) is C l -Lipschitz for all y ∈ Y , by Ledoux & Talagrand (2013) Theorem 4.12, we have R n ( l ◦ H o ) ≤ C l
· R n ( H o ) .
(cid:4)
B. F ORMAL R ESULT ON L OGISTIC R EGRESSION
For the logistic regression problem in Section 4.1, with X = (cid:8) x ∈ R d (cid:12)(cid:12) (cid:107) x (cid:107) 2 ≤ D (cid:9) , Y = { 0 , 1 } , training with the DAC regularization can be formulated explicitly as
[Equation]
and yields (cid:98)
[Equation]
[Table]
Theorem 9 (Formal restatement of Theorem 3 on logistic regression with DAC) .
Let
E
[Equation] [Equation]
P N
[Equation]
≤ C 2 N for some C N > 0 .
Then, learning logistic regression with the DAC regularization h ( x ui )
=
h ( x ui,j ) yields that, for any δ ∈ (0 , 1) , with probability at least 1 − δ :
[Equation]
where under Assumption 1 and Assumption 2, C N (cid:46) (cid:112) d − d aug ( δ/ 2) .
Proof of Theorem 9.
Notice that the logistic loss l ( · , y ) is 2-Lipschitz for all y ∈ Y , as ∂l ( z,y ) ∂z
= σ ( z ) − y , whose absolute value is bounded by 2.
Meanwhile, with (cid:107) θ (cid:107) 2 ≤ C 0 and (cid:107) x (cid:107) 2 ≤ D for all θ and x , the logistic loss is also bounded: l (cid:0) θ (cid:62) x , y (cid:1) ≤ log 2+ C 0 D .
Then, applying Proposition 8, we have that for any δ ∈ (0 , 1) , with probability at least 1 − δ/ 2 ,
[Equation]
It left to bound the Rademacher complexity of T dac (cid:98) A , X u ( H ) .
First notice that the sigmoid function σ ( · ) is a bijective function, the data augmentation consistency constraints are equivalent to
[Equation]
Thus
[Equation]
we have
[Equation]
with which we are ready to bound the Rademacher complexity of T dac (cid:98) A , X u ( H ) .
For the emprical Rademacher complexity
[Equation]
we have
[Equation]
n
[Equation] [Equation]
n (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
Converting this to the population Rademacher complexity
[Equation]
we take expectation over X , while conditioned on P N , and recall Jensen’s inequality,
[Equation]
Leveraging Lemma 5, we have that under Assumption 1 and Assumption 2, conditioned on P N , with high probability over X ∼ P ∗ ,
[Equation]
Then we can ﬁnd some C N > 0 with 1 n (cid:80) ni
=1 (cid:107) P N x i (cid:107) 22 ≤ C 2 N such that,
[Equation]
By Proposition 7, we have C N ≤ (cid:112) d − d aug ( δ/ 2) with probability at least 1 − δ/ 2 .
To obtain Theorem 3, we leverage the union bound and suppress the constants such that, with probability at least 1 − δ ,
[Equation]
Finally, taking δ = 2 /n and applying the Jensen’s inequality to the two terms in Equation (4) complete the proof.
[Equation]
B. F ORMAL R ESULT ON T WO - LAYER N EURAL N ETWORK
In the two-layer neural network regression setting described in Section 4.2, training with the DAC regularization can be formulated explicitly as
[Equation]
+ , and yields (cid:98) h dac ( x ) (cid:44) ( x (cid:62) (cid:98) B dac )
+ (cid:98) w dac .
Theorem 10 (Formal restatement of Theorem 4 on two-layer neural network with DAC) .
Sampling from P ∗ that follows Assumption 1, let the augmented training set (cid:98) A ( X u ) satisfy (a) αN ≥ 4 (cid:98)
d aug , (b) (cid:98) A ( X u ) − MX u admits an absolutely continuous distribution over N ⊥ , and P ∗ fulﬁlls (c)
E
[Equation]
n
[Equation]
ni =1 (cid:107) P N x i (cid:107) 22 (cid:12)(cid:12)(cid:12) P N (cid:105) ≤ C 2 N for some C N > 0 .
Then, learning the two-layer ReLU network with the DAC regularization on (cid:16) ( x ui ) (cid:62) B (cid:17) + = (cid:16)(cid:0) x ui,j (cid:1) (cid:62) B (cid:17) + gives that, with high probability,
[Equation]
, where under Assumption 1 and Assumption 2, C N (cid:46) (cid:112) d − d aug ( δ ) with probability at least 1 − δ .
Lemma 1.
Under the assumptions in Theorem 10, every size- (cid:98) d aug subset of rows in (cid:98) A ( X u ) − MX u is linearly independent almost surely.
Proof of Lemma 1.
Since αN > (cid:98) d aug , it is sufﬁcient to show that a random matrix with an absolutely continuous distribution is totally invertible 2 almost surely.
It is known that for any dimension m ∈ N , an m × m square matrix S is singular if det( S ) = 0 where entries of S lie within the roots of the polynomial equation speciﬁed by the determinant.
Therefore, the set of all singular matrices in R m × m has Lebesgue measure zero, λ ( { S ∈ R m × m | det( S ) = 0 } )
= 0 .
Then, for an absolutely continuous probability measure µ with respect to λ , we also have
[Equation]
Since a general matrix R contains only ﬁnite number of submatrices, when R is drawn from an absolutely continuous distribution, by the union bound, P [ R cotains a singular submatrix ]
= 0 .
That is, R is totally invertible almost surely.
(cid:4)
Lemma 2.
Under the assumptions in Theorem 10, the hidden layer in the two-layer ReLU network learns N , the invariant subspace under data augmentations : with high probability,
[Equation]
Proof of Lemma 2.
We will show that for all b k = P N b k + P ⊥N b k , k ∈ [ q ] , P ⊥N b k = 0 with high probability, which then implies that given any x ∈ X , ( x (cid:62) b k ) +
=
( x (cid:62) P N b k ) + for all k ∈ [ q ] .
For any k ∈ [ q ] associated with an arbitrary ﬁxed b k ∈ S d − 1 , let X uk (cid:44) X uk P N + X uk P ⊥N ∈
X
N k be the inclusion-wisely maximum row subset of X u such that X uk b k > 0 element-wisely.
Meanwhile, we denote (cid:98) A ( X uk )
=
M k X uk P N + (cid:98) A ( X uk ) P ⊥N ∈ X
αN
k as the augmentation of X uk where M k ∈ R αN k
× N k is the vertical stack of α identity matrices with size N k × N k .
Then the DAC constraint implies that ( (cid:98) A ( X uk ) − M k X uk ) P ⊥N b k = 0 .
With Assumption 1, for a ﬁxed b k ∈ S d − 1 , P [ x (cid:62) b k > 0] = 12 .
Then, with the Chernoff bound,
[Equation]
which implies that, N k ≥ N 4 with high probability.
Leveraging the assumptions in Theorem 10, αN ≥ 4 (cid:98) d aug implies that αN k ≥ (cid:98) d aug .
Therefore by
Lemma 1, Row (cid:16) (cid:98) A ( X uk ) − M k X uk (cid:17) =
N ⊥ with probability 1 .
Thus, ( (cid:98) A ( X uk ) − M k X uk ) P ⊥N b k = 0 enforces that P ⊥N b k = 0 .
[Equation]
Proof of Theorem 10.
Lemma 2 implies that ( X (cid:98) B dac )
+ =
( XP N (cid:98) B dac )
+ , and therefore,
T dac (cid:98) A , X
[Equation]
In other words, the DAC regularization reduces the feasible set for B such that
[Equation]
Meanwhile, for the square loss, the corresponding excess risk is given by
[Equation]
Leveraging Equation (21) and (22) in Du et al.
(2020), since ( B ∗ , w ∗ ) is feasible under the constraint, by the basic inequality,
[Equation]
where (cid:98) B dac , B ∗ ∈ B and (cid:13) (cid:13)(cid:98) w dac (cid:13) (cid:13) 1 ≤ C w .
Analogously to the Rademacher complexities for the bounded losses, for the square loss, we recall the deﬁnitions of the Gaussian complexities of a vector-valued function class Φ = { φ : X → R q } ,
[Equation]
G
[Equation]
In terms of the Gaussian complexity, for a ﬁxed T dac
[Equation]
A , X u ( H ) ,
[Equation]
[Table]
where the empirical Gaussian complexity can be upper bounded by
[Equation]
Taking expectation over X ∼ P ∗ , we have that for a ﬁxed T dac (cid:98) A , X u ( H ) ( i.e. , conditioned on P N ),
[Equation]
Leveraging Lemma 5, we have that under Assumption 1 and Assumption 2, conditioned on P N , with high probability over X ∼ P ∗ ,
[Equation]
Then we can ﬁnd some C N > 0 with 1 n (cid:80) ni
=1 (cid:107) P N x i (cid:107) 22 ≤ C 2 N such that,
[Equation]
By Proposition 7, we have C N ≤ (cid:112) d − d aug ( δ ) with probability at least 1 − δ .
(cid:4)
B. F ORML RESULTS ON EXPANSION - BASED DATA AUGMENTATION
For the formal analysis of the expansion-based data augmentations, we ﬁrst introduce some helpful notations.
With respect to an augmentation function A : X → 2 X fulﬁlling the conditions in Deﬁnition 4, we deﬁne the neighborhood for an arbitrary x ∈ X , as well as for any S ⊆ X , such that,
[Equation]
Then Deﬁnition 4(b) implies that the corresponding neighborhood function NB : X → 2 X of A has non-trivial expansion :
[Equation]
Intuitively, the non-trivial expansion guarantees that the neighborhood of any proper subset of a class always enlarges the subset within the class.
Remark 2.
The ground truth classiﬁer is invariant throughout the neighborhood : for any augmentation function A : X → 2 X satisfying the conditions in Deﬁnition 4, the corresponding neighborhood function satisﬁes that, given any x ∈ X , h ∗ ( x )
= h ∗ ( x (cid:48) ) for all x (cid:48) ∈ NB ( x ) .
Notice that the previous data augmentation strength (cid:98) d aug (Deﬁnition 2) is not well deﬁned for Deﬁnition 4 (since X needs not be a subset of R d ).
Therefore we need a new notion of data augmentation strength.
Remark 3.
The strength of expansion-based data augmentations (Deﬁnition 4) is characterized by the size of A ( x ) .
Intuitively, for any x ∈ X , the augmentations can be stronger ( i.e. , there exists x (cid:48) ∈
A ( x ) that is more different from x ) if A ( x ) is inclusion-wisely larger.
In addition, for an arbitrary classiﬁer h ∈ H , we denote the majority label with respect to h for each class,
[Equation]
along with the respective class-wise local and global minority sets,
[Equation]
Inﬁnite unlabeled data.
As a warm-up, we simplify the setting by enforcing consistency over the population such that learning with the DAC constraints can be expressed as,
[Equation]
We can also formulate algorithm in terms of the DAC operator,
[Equation]
Lemma 3.
Given any h ∈ T dac A , X ( H ) , for each k ∈ [ K ] , there exists some (cid:98) y k ∈
[ K ] such that h ( x ) = (cid:98) y k for all x ∈ X k .
Proof of Lemma 3.
It is sufﬁcient to show that M k = ∅ for all k ∈ [ K ] .
By contradiction, suppose
M k (cid:54) = ∅ .
Then by the non-trivial expansion assumption in Deﬁnition 4, ( NB ( M k ) \ M k ) ∩
X k (cid:54) = ∅ .
That is, there exists some x ∈ ( NB ( M k ) \ M k ) ∩ X k .
But on one hand, x ∈ NB ( M k ) implies that we can ﬁnd some x (cid:48)
∈ M k and x (cid:48)(cid:48) ∈
A ( x ) ∩ A ( x (cid:48) ) .
Therefore by the consistency constraint h ∈ T dac A , X ( H ) , h ( x (cid:48)(cid:48) )
=
h ( x ) =
h ( x (cid:48) ) .
On the other hand, x ∈ X k \ M k implies that h ( x ) = (cid:98) y k (cid:54)
=
h ( x (cid:48) ) , and leads to a contradiction.
[Equation]
Restating Theorem 5 with the formal notations in the appendix: with high probability,
[Equation]
Proof of Theorem 5.
By Lemma 3, since
[Equation]
where {
[Equation] [Equation]
[Equation] [Equation]
] , the consistency constraint yields a ﬁnite feasible classiﬁer class with (cid:12)(cid:12) T dac A , X ( H ) (cid:12)(cid:12) =
K ! ≤ K K .
Meanwhile, since A ( x ) ⊆ { x (cid:48) ∈ X | h ∗ ( x ) = h ∗ ( x (cid:48) ) } , by construction, we know that T dac A ,
X ( H ) is realizable ( i.e. , h ∗ ∈ T dac A , X ( H ) such that (cid:98) L dac 01 ( h ∗ ) = 0 ).
Therefore,
[Equation]
where for every classiﬁer h ∈ T dac A , X ( H ) ,
[Equation]
Then by the union bound, we have
[Equation]
The proof is completed by upper bounding the above, P (cid:104)
L 01 (cid:16)(cid:98)
h dac X (cid:17) > (cid:15) (cid:105) ≤ 1 /n , when taking
[Equation]
and observing that L 01 ( h ∗ )
= 0 .
[Equation]
Finite unlabeled data.
For the ﬁnite unlabeled data case, we start by introducing some additional concepts and notations.
We concretize the classiﬁer class H with a proper function class
F ⊆ (cid:8) f : X → R K (cid:9) such that H =
(cid:110)
h ( x ) (cid:44) argmax k ∈ [ K ] f ( x ) k (cid:12)(cid:12)(cid:12) f ∈ F (cid:111) .
Speciﬁcally, we adapt the existing setting in Wei et al.
; Cai et al.
and consider F as a class of fully connected neural networks.
To constrain the feasible hypothesis class through the DAC regularization with ﬁnite unlabeled observations X u =
[ x u 1 , . . .
, x uN ] (cid:62) , we leverage, from Wei & Ma (2021), the notion of all-layer-margin, m : F × X × Y
→
R ≥ 0 , that measures the maximum possible perturbation in all layers of f while maintaining the prediction y .
Precisely, given any f ∈ F such that f ( x ) =
W p ϕ ( . . .
ϕ ( W 1 x ) . . . )
for some activation function ϕ : R → R and parameters (cid:8)
W ι ∈ R d ι × d ι − 1 (cid:9) p ι =1 , we can write f = f 2 p − 1 ◦ · · · ◦ f 1 where f 2 ι − 1 ( x ) =
W ι x for all ι ∈ [ p ] and f 2 ι ( z ) =
ϕ ( z ) for ι ∈ [ p − 1] .
For an arbitrary set of perturbation vectors δ = ( δ 1 , . . .
δ 2 p − 1 ) such that δ 2 ι − 1 , δ 2 ι ∈ R d ι for all ι , let f ( x , δ ) be the perturbed neural network deﬁned recursively such that
[Equation]
The all-layer-margin m ( f, x , y ) measures the minimum norm of the perturbation δ such that f ( x , δ ) fails to provide the classiﬁcation y ,
[Equation]
With the notion of all-layer-margin established, for any A : X → 2 X that satisﬁes conditions in
Deﬁnition 4, the robust margin is deﬁned as
[Equation]
With merely ﬁnite unlabeled data, comparing to enforcing consistency over population, we need stronger assumptions on data augmentations in addition to the ones in Deﬁnition 4, as speciﬁed in
Deﬁnition 5 below.
Deﬁnition 5 ( ( F , τ ) -expansion-based data augmentation) .
With respect to the given function class
F and some proper constant τ > 0 , Let A : X → 2 X be a function that satisﬁes conditions in
Deﬁnition 4, and additionally,
[Equation]
Then for any sample x ∈ X , x (cid:48) ∈ X is a ( F , τ ) -expansion-based data augmentation of x induced by A if x (cid:48) ∈
A ( x ) .
Remark 4.
The existence of the ( F , τ ) -expansion-based data augmentations relies on the suitable choice of τ with respect to the given F .
For a ﬁxed F , as τ increases, Equation (8) imposes stronger assumption on A ( i.e. , A ( x ) is forced to be inclusion-wisely smaller, where A becomes weaker), while better theoretical guarantee (Theorem 12) can be achieved.
We consider a class of p -layer fully connected neural networks with maximum width q ,
[Equation]
where with W ι ∈ R d ι × d ι − 1 for all ι ∈ [ p ] , q (cid:44) max ι =0 ,...,p d ι .
The goal is to learn a classiﬁer (cid:98)
h dac
X
u = argmax k ∈
[ K ] (cid:98) f dacn,N ( x ) k where (cid:98) f dacn,N ∈ F is a fully connected neural network.
With respect to F , we choose a suitably large τ > 0 such that there exists a A : X → 2 X fulﬁlling the conditions in Deﬁnition 5.
To enforce sufﬁciently strong consistency of f with ﬁnite unlabeled samples, we incorporate the DAC regularization with a positive robust margin such that,
[Equation]
In terms of the DAC operator, the algorithm can be restated as (cid:98) h dac X u (cid:44) argmin h ∈ T
[Equation]
A
[Equation]
u (
[Equation]
with
[Equation]
Deﬁnition 6 (Expansion assumptions, Wei et al.
; Cai et al.
The marginal distribution
P ∗ ( x ) satisﬁes
(a) ( q, ξ ) -constant expansion if given any S ⊆ X with P ∗ ( S ) ≥ q and P ∗ ( S ∩ X k ) ≤ 12 for all k ∈
[ K ] , P ∗ ( NB ( S )) ≥ min { P ∗ ( S ) , ξ } + P ∗ ( S ) ; (b) ( a, c ) -multiplicative expansion if for all k ∈ [ K ] , given any S ⊆ X with P ∗ ( S ∩ X k ) ≤ a , P ∗ ( NB ( S )
∩ X k ) ≥ min { c · P ∗ ( S ∩ X k ) , 1 } .
Proposition 11 (Wei et al.
Theorem 3.7, Cai et al.
Proposition 2.2) .
For any δ ∈ (0 , 1) , with probability at least 1 − δ/ 2 , there exists some µ such that
[Equation]
for all h ∈ T dac A , X u ( H ) , where (cid:101) O ( · ) hides polylogarithmic factors in N and d .
Theorem 12 (DAC with expansion-based data augmentations over ﬁnite samples) .
With proper choices of τ > 0 , for any A : X → 2 X inducing ( F , τ ) -expansion-based data augmentations, learning the classiﬁer with DAC regularization, Equation (9) , provides that, for any δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
In speciﬁc, for some q < 12 , c > 1 , with a sufﬁciently large unlabeled sample size N such that µ ≤ 14 min { c − 1 , 1 } ,
(a) when P ∗ ( x ) satisﬁes ( q, 2 µ ) -constant expansion,
[Equation]
(b) while when P ∗ ( x ) satisﬁes ( 12 , c ) -multiplicative expansion,
[Equation]
where we recall from Proposition 11 that,
[Equation]
Lemma 4 (Cai et al.
(2021), Lemma A.1) .
For any h ∈ T dac A , X u ( H ) , when P ∗ satisﬁes
(a) ( q, 2 µ ) -constant expansion with q < 12 , P ∗ ( M ) ≤ max { q, 2 µ } ; (b) (cid:0) 12 , c (cid:1) -multiplicative expansion with c > 1 + 4 µ , P ∗ ( M ) ≤ max (cid:110)
µ c − 1 , 2 µ (cid:111) .
Proof of Lemma 4.
We start with the proof for Lemma 4 (a).
By deﬁnition of M k and (cid:98) y k , we know that M k = M ∩ X k ≤ 12 .
Therefore, for any 0 < q < 12 , one of the following two cases holds:
(i) P ∗ ( M ) <
q ; (ii) P ∗ ( M ) ≥ q .
Since P ∗ ( M ∩ X k ) < 12 for all k ∈
[ K ] holds by construction, with the ( q, 2 µ ) -constant expansion, P ∗ ( NB ( M ))
≥ min { P ∗ ( M ) , 2 µ } + P ∗ ( M ) .
Meanwhile, since the ground truth classiﬁer h ∗ is invariant throughout the neighborhoods,
NB ( M
k ) ∩ NB
M k (cid:48)
) =
∅ for
k (cid:54) =
k (cid:48) , and therefore NB ( M ) \ M = (cid:83) K k =1 NB ( M k ) \ M k with each NB ( M k ) \ M k disjoint.
Then, we observe that for each x ∈ NB ( M ) \ M , here exists some k = h ∗ ( x ) such that x ∈ NB ( M k ) \ M k .
x ∈ X k \ M k implies that h ( x ) = (cid:98) y k , while x ∈ NB ( M k ) suggests that there exists some x (cid:48)
∈
A ( x ) ∩ A ( x (cid:48)(cid:48) )
where x (cid:48)(cid:48) ∈
M k such that either h ( x (cid:48) )
= (cid:98) y k
and h ( x (cid:48) ) (cid:54)
=
h ( x (cid:48)(cid:48) ) for x (cid:48) ∈
A ( x (cid:48)(cid:48) ) , or h ( x (cid:48) )
(cid:54) =
(cid:98) y k
and h ( x (cid:48) ) (cid:54)
=
h ( x ) for x (cid:48) ∈
A ( x ) .
Therefore, we have
[Equation]
Moreover, since P ∗ ( NB ( M )) − P ∗ ( M ) ≤ P ∗ ( NB ( M ) \ M ) ≤ 2 µ , we know that min { P ∗ ( M ) , 2 µ } + P ∗ ( M ) ≤ P ∗ ( NB ( M )) ≤ P ∗ ( M ) + 2 µ.
That is, P ∗ ( M ) ≤ 2 µ .
Overall, we have P ∗ ( M ) ≤ max { q, 2 µ } .
To show Lemma 4 (b), we recall from Wei et al.
Lemma B.6 that for any c > 1 + 4 µ , (cid:0) 12 , c (cid:1) -multiplicative expansion implies (cid:16)
µ c − 1 , 2 µ (cid:17) -constant expansion.
Then leveraging the proof for Lemma 4 (a), with q = 2 µ c − 1 , we have P ∗ ( M ) ≤ max (cid:110)
µ c − 1 , 2 µ
[Equation]
Proof of Theorem 12.
To show Equation (10), we leverage the proof of Proposition 8, and observe that B = 1 with the zero-one loss.
Therefore, when conditioned on T dac A , X u ( H ) , for any δ ∈ (0 , 1) , with probability at least 1 − δ/ 2 ,
[Equation]
For the upper bounds of the Rademacher complexity, let (cid:101) µ (cid:44) sup h ∈ T dac A , X u
( H ) P ∗ ( M ) where M denotes the global minority set with respect to h ∈ T dac A , X u ( H ) .
Lemma 4 suggests that
(a) when P ∗ satisﬁes ( q, 2 µ ) -constant expansion for some q < 12 , (cid:101) µ ≤ max { q, 2 µ } ; while (b) when P ∗ satisﬁes ( 12 , c ) -multiplicative expansion for some c > 1 + 4 µ , (cid:101) µ ≤
[Equation]
µ min { c − 1 , 1 } .
Then, it is sufﬁcient to show that, conditioned on T dac A , X u ( H ) ,
[Equation]
To show this, we ﬁrst consider a ﬁxed set of n observations in X , X
=
[ x 1 , . . .
, x n ] (cid:62) ∈
X n .
Let the number of distinct behaviors over X in T dac A , X u ( H ) be
[Equation]
[Table]
Then, by the Massart’s ﬁnite lemma, the empirical rademacher complexity with respect to X is upper bounded by
[Equation]
n .
By the concavity of (cid:112) log ( · ) , we know that,
[Equation]
n
[Equation]
 ≤
[Equation]
n
[Equation]
Since P ∗ ( M ) ≤ (cid:101) µ ≤ 12 for all h ∈ T dac A , X u ( H ) , we have that, conditioned on T dac A , X u ( H ) ,
[Equation]
dac
[Equation]
(cid:18)
[Equation]
(cid:19)
[Equation]
Plugging this into Equation (12) yields Equation (11).
Finally, the randomness in T dac A , X u ( H ) is quantiﬁed by (cid:101) µ, µ , and upper bounded by Proposition 11.
(cid:4)
B. S UPPLEMENTARY E XAMPLE : DOMAIN ADAPTATION
As a supplementary example, we demonstrate the possible failure of the ERM on augmented training set, and how the DAC regularization can serve as a remedy, with an illustrative linear regression problem in the domain adaptation setting: where the training samples are drawn from some source distribution P s , while the excess risk is tested over a related but different distribution P t , known as the target distribution.
Speciﬁcally, assuming that E P s [ y | x ] and E P t [ y | x ] are distinct, but there exists some invariant feature subspace X r ⊂ X such that P s [ y | x ∈ X r ] =
P t [ y | x ∈ X r ] , we aim to demonstrate the advantage of the DAC regularization over the ERM on augmented training set, with a provable separation in the respective excess risks.
[Equation]
Source and target distributions.
Formally, the source and target distributions are concretized with the causal graph in Figure 4.
For both P s and P t , the observable feature x is described via a linear generative model in terms of two latent features, the ‘invariant’ feature ζ iv ∈ R d iv and the ‘environmental’ feature ζ e ∈ R d e :
[Equation]
where S =
[ S iv , S e ] ∈ R d × ( d iv + d e ) ( d iv + d e ≤ d ) consists of orthonormal columns.
Let the label y depends only on the invariant feature ζ iv for both domains,
[Equation]
for some θ ∗ ∈ Range ( S iv ) such that P s [ y | ζ iv ] =
P t [ y | ζ iv ] , while the environmental feature ζ e is conditioned on y , ζ iv , (along with the Gaussian noise z ), and varies across different domains e with E P s [ y | x ] (cid:54) =
E P t [ y | x ] .
In other words, with the square loss l ( h ( x ) , y )
=
12 ( h ( x ) − y ) 2 , the optimal hypotheses that minimize the expected excess risk over the source and target distributions are distinct.
Therefore, learning via the ERM with training samples from P s can overﬁt the source distribution, in which scenario identifying the invariant feature subspace Range ( S iv ) becomes indispensable for achieving good generalization in the target domain.
Moreover, we assume the following regularity conditions on the source and target distributions:
Assumption 3 (Regularity conditions for P s and P t ) .
Let P s satisfy Assumption 1, and P t satisfy the following: let E P t [ xx (cid:62) ] (cid:31) 0 , and
(a) for the invariant feature, c t,iv I d iv (cid:52) E P t [ ζ iv ζ (cid:62) iv ] (cid:52) C t,iv I d iv for some C t,iv ≥ c t,iv = Θ(1) ; (b) for the environmental feature, E P t [ ζ e ζ (cid:62) e ] (cid:60) c
t,e I d e for some c t,e > 0 , and E P t [ z · ζ e ] = 0 .
Training samples and data augmentations.
For a fair comparison between learning with the
DAC regularization and the ERM on augmented training set, we restrict to the supervised learning setting: X u = X ∈ X n .
Recall that we denote the augmented training sets, including and excluding the original samples, respectively with
[Equation]
In particular, we consider a set of augmentations that only perturb the environmental feature ζ e , while keep the invariant feature ζ iv intact,
[Equation]
We recall the notion (cid:98) d aug = rank (cid:16) (cid:98) A ( X ) − MX (cid:17) = rank (cid:16) (cid:101)
A ( X ) − (cid:102) MX (cid:17) (notice that 0 ≤ (cid:98) d aug ≤ d e ), and assume that X and (cid:98) A ( X ) are representative enough:
Assumption 4 (Diversity of X and (cid:98) A ( X ) ) .
( X , y ) ∈
X n ×Y n is sufﬁciently large with n (cid:29) ρ
4 d iv , θ ∗ ∈ span { x i | i ∈ [ n ] } , and (cid:98) d
aug = d e .
Excess risks in target domain.
Learning from the linear hypothesis class H = (cid:8) h ( x ) = θ (cid:62) x (cid:12)(cid:12) θ ∈ R d (cid:9) , with the DAC regularization on h ( x i )
=
h ( x i,j ) , we have (cid:98)
[Equation]
argmin
[Equation]
while with the ERM on augmented training set,
[Equation]
where M and (cid:102)
M denote the vertical stacks of α and 1+ α identity matrices of size n × n , respectively as denoted earlier.
We are interested in the excess risk on P t :
L t ( θ ) − L t ( θ ∗ ) where L t ( θ ) (cid:44)
E P t ( x ,y ) (cid:2) 12 ( y − x (cid:62) θ ) 2 (cid:3) .
Theorem 13 (Domain adaptation with DAC) .
Under Assumption 3(a) and Assumption 4, (cid:98) θ dac satisﬁes that, with constant probability,
[Equation]
Theorem 14 (Domain adaptation with ERM on augmented samples) .
Under Assumption 3 and
Assumption 4, (cid:98) θ dac and (cid:98) θ da − erm satisﬁes that,
[Equation]
for some EER e > 0 .
In contrast to (cid:98) θ dac where the DAC constraints enforce S (cid:62) e (cid:98) θ dac =
with a sufﬁciently diverse (cid:98) A ( X ) (Assumption 4), the ERM on augmented training set fails to ﬁlter out the environmental feature in (cid:98) θ da − erm : S (cid:62) e (cid:98) θ da − erm (cid:54)
= 0 .
As a consequence, the expected excess risk of (cid:98) θ da − erm in the target domain can be catastrophic when c t,e → ∞ , as instantiated by Example 2.
Proofs and instantiation.
We ﬁrst recall from the beginning of Appendix B that
[Equation]
is the orthogonal projector onto the dimension- ( d − (cid:98) d aug ) null space of (cid:98) A ( X ) − MX .
Furthermore, let P iv (cid:44) S iv S (cid:62) iv and P e (cid:44) S e S (cid:62) e be the orthogonal projectors onto the invariant and environmental feature subspaces, respectively,
such that x = S iv ζ iv + S e ζ e =
( P iv + P e ) x for all x .
Proof of Theorem 13.
By construction Equation (13), (cid:16) (cid:98)
A
[Equation]
X ) − MX
[Equation]
P iv = 0 , and it follows that
P iv (cid:52) P N .
Meanwhile from Assumption 4, (cid:98) d aug = d e implies that dim ( P N ) =
d iv .
Therefore,
P iv = P N , and the data augmentation consistency constraints can be restated as
[Equation]
Then with θ ∗ ∈ span { x i | i ∈ [ n ] } from Assumption 4,
[Equation]
where (cid:98) Σ
X iv (cid:44) 1 n P iv X (cid:62) XP iv .
Since (cid:98) θ dac − θ ∗ ∈
Col ( S iv ) , we have
E P t (cid:104) z · x (cid:62) P e ( (cid:98) θ dac − θ ∗ ) (cid:105) = 0 .
Therefore, let Σ x ,t (cid:44) E P t [ xx (cid:62) ] , with high probability,
E P s (cid:104)
L t ( (cid:98) θ dac )
− L t ( θ ∗ ) (cid:105) =
[Equation]
P s (cid:20) 1 2 (cid:13)(cid:13)(cid:13)(cid:98) θ dac − θ ∗ (cid:13)(cid:13)(cid:13)
[Equation]
Σ
[Equation]
Proof of Theorem 14.
Let (cid:98) Σ (cid:101)
A ( X ) (cid:44)
[Equation]
1+ α ) n (cid:101)
A ( X ) (cid:62) (cid:101)
A ( X ) .
Then with θ ∗ ∈ span { x i | i ∈ [ n ] } from Assumption
[Equation]
we have θ ∗ = (cid:98) Σ † (cid:101)
A ( X ) (cid:98) Σ (cid:101)
A ( X ) θ ∗ .
Since θ ∗ ∈
Col ( S iv )
[Equation]
(cid:102) MX θ ∗ =
(cid:102)
MXP iv θ ∗ = (cid:101)
A ( X ) θ ∗ .
Then, the ERM on the augmented training set yields
[Equation]
Meanwhile with E P t [ z · ζ e ] = 0 from Assumption 3, we have E P t [ z · P e x ] = 0 .
Therefore, by recalling that Σ x ,t (cid:44) E P t [ xx
[Equation] [Equation]
such that the expected excess risk can be expressed as
[Equation]
where let (cid:98) Σ (cid:101)
A ( X e ) (cid:44) P e (cid:98) Σ (cid:101)
A ( X ) P e ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We denote
[Equation]
, and observe that
[Equation]
>
[Equation]
Finally, we complete the proof by partitioning the lower bound for the target expected excess risk of (cid:98) θ
da − erm into the invariantand environmental parts such that
[Equation]
[Table]
[Equation]
Now we construct a speciﬁc domain adaptation example with a large separation ( i.e. , proportional to d e ) in the target excess risk between learning with the DAC regularization ( i.e. , (cid:98) θ dac ) and with the
ERM on augmented training set ( i.e. , (cid:98) θ da − erm ).
Example 2.
We consider P s and P t that follow the same set of relations in Figure 4, except for the distributions over e where P s ( e ) (cid:54) =
P t ( e ) .
Precisely, let the environmental feature ζ e depend on ( ζ iv , y, e ) :
[Equation]
sign
[Equation]
, where e ∼ N ( 0 , I d e ) for P s ( e ) and e ∼ N ( 0 , σ t I d e ) for P t ( e ) , σ t ≥ c t,e (recall c t,e from
Assumption 3).
Assume that the training set X is sufﬁciently large, n (cid:29) d e + log (1 /δ ) for some given δ ∈ (0 , 1) .
Augmenting X with a simple by common type of data augmentations – the linear transforms, we let
[Equation]
and deﬁne ν
[Equation] [Equation]
∪
[Equation]
σ max
[Equation]
∈ [ α ]
[Equation]
and ν 2 (cid:44) σ min
[Equation]

[Equation]
where σ min ( · ) and σ max ( · ) refer to the minimum and maximum singular values, respectively.
Then under Assumption 3 and Assumption 4, with constant probability,
[Equation]
Proof of Example 2.
With the speciﬁed distribution, for E =
[ e 1 ; . . .
; e n ] ∈ R n × d e ,
[Equation]
=
[Equation]
By Lemma 5, under Assumption 3 and Assumption 4, we have that with high probability, 0 .
9 I d e (cid:52)n E (cid:62) E (cid:52) 1 .
I d e .
Therefore with E and z being independent,
[Equation]
and the rest follows from Theorem 14.
[Equation]
C T ECHNICAL L EMMAS
Lemma 5.
We consider a random vector x ∈ R d with E [ x ] = 0 , E [ xx (cid:62) ]
= Σ , and x = Σ − 1 / 2 xbeing ρ 2 -subgaussian.
Given an i.i.d.
sample of x , X =
[ x 1 , . . .
, x n ] (cid:62) , for any δ ∈ (0 , 1) , if n (cid:29) ρ 4 d , then 0 .
9 Σ (cid:52) 1 n X (cid:62) X (cid:52) 1 . 1 Σ with probability high probability.
Proof.
We ﬁrst denote P X (cid:44) ΣΣ † as the orthogonal projector onto the subspace X ⊆ R d supported by the distribution of x .
With the assumptions E [ x ] = 0 and E [ xx (cid:62) ]
= Σ , we observe that E [ x ] =and E (cid:2) xx (cid:62) (cid:3) =
E (cid:2) xΣ − 1 x (cid:62) (cid:3) =
P X .
Given the sample set X of size n (cid:29) ρ 4 ( d + log(1 /δ )) for some δ ∈ (0 , 1) , we let U = 1 n (cid:80) n i =1 x i Σ − 1 x (cid:62) i − P X .
Then the problem can be reduced to showing that, with probability at least 1 − δ , (cid:107) U (cid:107) 2 ≤ 0 . 1 .
For this, we leverage the (cid:15) -net argument as following.
For an arbitrary v ∈ X ∩ S d − 1 , we have
[Equation]
where, given x i being ρ 2 -subgaussian, v (cid:62) x i is ρ 2 -subgaussian.
Since
[Equation]
we know that (cid:0) v (cid:62) x
i (cid:1) 2 − 1 is 16 ρ 2 -subexponential.
Then, we recall the Bernstein’s inequality,
[Equation]
Let N ⊂ X ∩ S d − 1 be an (cid:15) 1 -net such that | N | =
e O ( d ) .
Then for some 0 < (cid:15) 2 ≤ 16 ρ 2 , by the union bound,
[Equation]
whenever n > 2 ( 16 ρ 2 ) 2 (cid:15)(cid:0) Θ ( d ) + log 1 δ (cid:1) .
By taking δ = exp (cid:18) − 14 (cid:16) (cid:15) 2 16 ρ 2 (cid:17) 2 n (cid:19) , we have that max v ∈ N (cid:12)(cid:12) v (cid:62) Uv (cid:12)(cid:12) ≤ (cid:15) 2 with high probability when n > 4 (cid:16) 16 ρ 2 (cid:15) 2 (cid:17) 2 Θ ( d ) , and taking n (cid:29) ρ 4 d is sufﬁcient.
Now for any v ∈ X ∩ S d − 1 , there exists some v (cid:48) ∈
N such that (cid:107) v − v (cid:48) (cid:107) 2 ≤ (cid:15) 1 .
Therefore, (cid:12)(cid:12) v (cid:62) Uv (cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) v (cid:48)(cid:62) Uv (cid:48)
+
[Equation]
Taking the supremum over v ∈ S d − 1 , with probability at least 1 − δ ,
[Equation]
With (cid:15) 1 = 13 and (cid:15) 2 = 145 , we have (cid:15) 2 2 − (1+ (cid:15) 1 ) 2 = 110 .
Overall, if n (cid:29) ρ 4 d , then with high probability, we have (cid:107) U (cid:107) 2 ≤ 0 . 1 .
[Equation]
Lemma 6.
Let U ⊆
R d be an arbitrary non-trivial subspace in R d , and g ∼ N ( 0 , I d ) be a
Gaussian random vector.
Then for any continuous and C l -Lipschitz function ϕ : R → R ( i.e. , | ϕ ( u ) − ϕ
( u (cid:48) )
| ≤ C l · | u − u (cid:48) | for all u, u (cid:48) ∈ R ),
[Equation]
where ϕ acts on u entry-wisely, ( ϕ ( u ))
j
= ϕ ( u j ) .
In other words, the Gaussian width of the image set ϕ
( U ) (cid:44) (cid:8) ϕ ( u ) ∈ R d | u ∈ U (cid:9) is upper bounded by that of U scaled by the Lipschitz constant.
Proof.
[Equation]
D E XPERIMENT D ETAILS
In this section, we provide the details of our experiments.
Our code is adapted from the publicly released repo: https://github.com/kekmodel/FixMatch-pytorch .
Dataset: Our training dataset is derived from CIFAR-100, where the original dataset contains 50, training samples of 100 different classes.
Out of the original 50,000 samples, we randomly select 10,000 labeled data as training set (i.e., 100 labeled samples per class).
To see the impact of different training samples, we also trained our model with dataset that contains 1,000 and 20,000 samples.
Evaluations are done on standard test set of CIFAR-100, which contains 10,000 testing samples.
Data Augmentation:
During the training time, given a training batch, we generate corresponding augmented samples by RandAugment (Cubuk et al., 2020).
We set the number of augmentations per sample to 7, unless otherwise mentioned.
To generate an augmented image, the RandAugment draws n transformations uniformaly at random from 14 different augmentations, namely { identity, autoContrast, equalize, rotate, solarize, color, posterize, contrast, brightness, sharpness, shear-x, shear-y, translate-x, translate-y } .
The RandAugment provides each transformation with a single scalar (1 to 10) to control the strength of each of them, which we always set to 10 for all transformations.
By default, we set n = 2 (i.e., using 2 random transformations to generate an augmented sample).
To see the impact of different augmentation strength, we choose n ∈ { 1 , 2 , 5 , 10 } .
Examples of augmented samples are shown in Figure 3.
Parameter Setting: The batch size is set to 64 and the entire training process takes 2 15 steps.
During the training, we adopt the SGD optimizer with momentum set to 0.9, with learning rate for step i being 0 .
03 × cos (cid:16) i × 7 π 2 15 × 16 (cid:17) .
Additional Settings for the semi-supervised learning results: For the results on semi-supervised learning, besides the 10,000 labeled samples, we also draw additionally samples (ranging from 5,000 to 20,000) from the training set of the original CIFAR-100.
We remove the labels of those additionally sampled images, as they serve as “unlabeled”
E I LLUSTRATIVE E XAMPLE
[Figure]
The results precisely match Theorem 3 and the discussion afterward.
As suggested by the theoretical analysis, the performance of DAC only depends on the d aug , but not α .
Further, given the same augmented dataset, DA-ERM is always worse than DAC.
The gap is particularly signiﬁcant when the number of augmentations is small (i.e., a small α ).
Here we present an illustrative example for logistic regression, which numerically shows the beneﬁts of DAC over ERM, and also clearly demonstrates the impact of different d aug and α .
Example 3.
Consider a 30-dimensional logistic regression.
The original training set contains 50 samples.
The inputs x i s are generated independently from N (0 , I 30 ) and we set θ ∗ =
[ θ ∗ c ; 0 ] with θ ∗ c ∼ N (0 , I 3 ) and 0 ∈ R 27 .
To generate the augmentations, we ﬁrst specify a parameter d aug and leave the ﬁrst 30 − d aug elements of each x i unchanged, and replace the later d aug elements of each x i with a new vector randomly generated from N (0 , I d aug ) .
Further, we generate α augmentations for each of the x i .
For any α ≥ 1 , the augmentation will perturb d aug coordinates with probability 1.
The results for d aug ∈ { 20 , 25 } and various α s are presented in Figure 5.
The test error clearly matches Theorem 3 and the discussion afterward:
The generalization performance of DAC only relies on d aug but
not α , and larger d aug gives smaller testing error.
The performance of DA-ERM crucially depends on α , when α is small (i.e., limited augmentations), the performance between DAC and DA-ERM is very signiﬁcant.
And when we further increases α , DA-ERM can only approach to DAC but not out-perform DAC.