C O - COMPLEXITY : A N E XTENDED P ERSPECTIVE ON G ENERALIZATION E RROR
Anonymous authors
Paper under double-blind review
A BSTRACT
It is well known that the complexity of a classiﬁer’s function space controls its generalization gap, with two important examples being VC-dimension and Rademacher complexity (R-Complexity).
We note that these traditional generalization error bounds consider the ground truth label generating function (LGF) to be ﬁxed.
However, if we consider a scenario where the LGF has no constraints at all, then the true generalization error can be large, irrespective of training performance, as the values of the LGF on unseen data points can be largely independent of the values on the training data.
To account for this, in this work, we consider an extended characterization of the problem, where the ground truth labels are generated by a function within another function space, which we call the generator space.
We ﬁnd that the generalization gap in this scenario depends on the R-Complexity of both the classiﬁer and the generator function spaces.
Thus, we ﬁnd that, even if the R-Complexity of the classiﬁer is low and it has a good training ﬁt, a highly complex generator space could worsen generalization performance, in accordance with the no free lunch theorem.
Furthermore, the characterization of a generator space allows us to model constraints, such as invariances (translation and scale in vision) or local smoothness.
Subsequently, we propose a joint entropy-like measure of complexity between function spaces (classiﬁer and generator), called co-complexity, which leads to tighter bounds on the generalization error in this setting.
Co-complexity captures the similarities between the classiﬁer and generator spaces.
It can be decomposed into an invariance co-complexity term, which measures the extent to which the classiﬁer respects the invariant transformations in the generator, and a dissociation co-complexity term, which measures the ability of the classiﬁer to differentiate separate categories in the generator.
Our major ﬁnding is that reducing the invariance co-complexity of a classiﬁer, while maintaining
its dissociation co-complexity, improves the training error and reduces the generalization gap.
Furthermore, our results, when specialized to the previous setting where the LGF is ﬁxed, lead to potentially tighter generalization error bounds.
Theoretical results are supported by empirical validation on the CNN architecture and its transformation-equivariant extensions.
Co-complexity showcases a new side to the generalization abilities of classiﬁers and can potentially be used to improve their design.
I NTRODUCTION
In the context of supervised classiﬁcation, a major factor for consideration is the generalization error of the classiﬁer, i.e., how good a classiﬁer generalizes to test (unseen) data points.
The notion of overﬁtting describes the case when the test error signiﬁcantly exceeds the training error.
Naturally, the objective for building a robust classiﬁer entails the minimization of this generalization gap , to avoid overﬁtting.
To that end, statistical studies on generalization error (Blumer et al.
; Bartlett & Mendelson (2003))
ﬁnd that complexity measures on the classiﬁer function space, F , often directly control the generalization gap of a classiﬁer.
Two prominent examples of such measures include the
Rademacher Complexity R m ( F ) (Bartlett & Mendelson (2003)) and the VC dimension (Blumer et al.
V C ( F ) .
Both measures directly estimate the ﬂexibility of a function space, i.e., how likely is it for F to contain functions that can ﬁt any random labelling over a set of data points.
In this paper, we work with Rademacher complexity and propose extensions that provide a new perspective on generalization error.
From a statistical perspective, the generalization gap can be understood through convergence bounds on the error function, i.e., the expected deviation of the error function on the test data compared to the training data.
Traditional generalization error bounds (Bartlett & Mendelson, 2003) state that function complexity (i.e., R m ( F ) ) directly corresponds to the generalization gap.
Thus, higher R m ( F ) usually leads to a greater generalization gap and slower convergence.
Although the original R m ( F ) was proposed for binary classiﬁcation, similar results have been shown for multi-class settings and a larger variety of loss functions (Xu et al., 2016;
Liao et al., 2018).
Note that R m ( F ) is over the entire function space and thus global in nature.
Local forms of Rademacher complexity, which involve restricting the function space and lead to minimum error on the training data samples, have been proposed (Bartlett et al., 2005; 2002).
Apart from function complexity based measures, there is also considerable work which uses an information theoretic perspective (Xu & Raginsky, 2017; Russo &
Zou, 2020; Bu et al., 2020; Haghifam et al., 2020) in treating the subject.
1. W HY THE G ROUND T RUTH L ABEL G ENERATING F UNCTION M ATTERS
We deﬁne the label generating function for a classiﬁcation problem as the function which generates the true labels for all possible datapoints.
Note that most generalization error bounds, including the traditional ones, primarily are introspective in nature, i.e., they consider the size and ﬂexibility of the classiﬁer’s function space F .
The main direction proposed in this work is the investigation of the unknowability of the ground truth label generating functions (LGF), using another function space which we call the generator space.
The generalization error bounds in Bartlett & Mendelson (2003) state that the difference in test and training performance is roughly bounded above by the Rademacher Complexity of the classiﬁer’s
function space (i.e., R m ( F ) ).
In other words, whatever the training error, the test error will always be likely to be greater by an amount R m ( F ) on average.
We note that, in deriving the original bound, a major assumption is that the LGF g is ﬁxed and knowable from the data.
We now outline our main argument for taking the generator space into account.
The LGF is indeed ﬁxed, i.e., there cannot be two different ground truth label generating functions applicable to the same problem.
However, our primary emphasis is on the fact that the true LGF will always be unknown , i.e., for any ﬁnite training data containing data-label pairs ( z i , g ( z i )) , we would only truly know the output of the label generating function on the given training data samples.
Only when we have inﬁnite training data samples, the values of LGF are known at each z ∈ R d .
In this work, we denote the function space of all possible LGFs, within which the true LGF is contained, as the generator space.
Note that the generator space arises due to the unknowability of the LGF.
We show in this work that due to the generator space, the true generalization gap is greater than the Rademacher complexity of the classiﬁer, and also depends on the Rademacher complexity of the generator.
Note that the size of the generator space, which dictates its complexity, will be dependent on the amount of constraints that the LGF has: if the LGF has no constraints at all, then generator spaces are larger,
whereas if the
LGF is constrained to be smooth/invariant to many transformations, generator spaces are smaller (as the set of functions which are smooth and invariant are also much smaller, for instance in vision).
Let us consider the case where the LGF g has no constraints at all, i.e., it is sampled from a generator space G which contains all possible functions g : R d −→ {− 1 , 1 } .
In this case, the function g is expected to have no structure and behaves like a random function, and thus the expected test accuracy of any classiﬁer will be 50% (i.e., random chance).
Therefore, even if the classiﬁer function f ∈ F produces a very good ﬁt on the training data and F happens to have a low complexity measure
R m ( F ) , the generalization performance will still be poor, as no knowledge of the LGF values on the unseen datapoints is available from the training samples.
This is in contrast to the generalization error bounds based on Rademacher complexity, which would estimate that the classiﬁer should have good generalization performance (i.e., low test error), as both R m ( F ) and training error are low.
Note that although a typical training dataset extracted from this LGF may be hard to ﬁt using a low-complexity classiﬁer F , there will be examples of training instances with non-zero probability on which a low-complexity classiﬁer can have a good ﬁt.
The takeaway from this example is that the structure of the data (here represented using the complexity of the generator space) can additionally dictate whether a classiﬁer can generalize.
Note that, in this scenario, the expected generalization performance would be better if the LGF had more structure.
Figure 1 illustrates the role of both generator and classiﬁer spaces in generalization via the two example scenarios discussed earlier.
In both examples, the same low-complexity classiﬁer has a good ﬁt on the training data, thus R m ( F ) is low.
In example (a), the LGF has no constraints; while, in example (b), the LGF has constraints such as local smoothness.
In example (a), the classiﬁer clearly exhibits poor generalization performance on the test data.
This agrees with our previous argument that if the LGF has no constraints, generalization is essentially impossible.
In example (b), the classiﬁer shows much more robust generalization performance on the test data, due to the LGF having signiﬁcantly more structure.
Furthermore, it is intuitively clear that a classiﬁer function space F which has a high overlap with the generator space G (and therefore its constraints) should yield good generalization performance.
This shows, that in addition to the function spaces F and G individually affecting the generalization gap, the similarities between F and G are also important.
Both of these perspectives play leading roles in our construction of generator-and-classiﬁer aware complexity measures and the associated novel generalization error bounds.
1. R ELATED W ORK
To the best of our knowledge, the approach we are proposing to study generalization performance is novel and there is not much directly related work.
Here, we describe some examples of works which discuss relevant concepts.
The no free lunch theorem proposed in Wolpert & Macready (1997) indirectly sheds light on the behaviour of the LGFs.
However, it does not incorporate ways to reduce variability in the LGFs by considering constraints related to the classiﬁcation problem.
Invariance constraints in learning algorithms were studied in Sokolic et al.
(2017), where the input space was factored into invariant transformations, similar to what we also do in this work.
In doing so, the complexity of the data was indirectly explored, based on the number of invariant transformations present in the input space.
However, the generalization bounds were derived with an assumption of perfectly invariant classiﬁer function spaces, which is not applicable for CNNs and their variants (as shown in Kauderer-Abrams (2017)).
In another relevant study (Jin et al.
, a cover-complexity measure of a single dataset was proposed, and the generalization error of fully connected networks was analyzed with respect to the same.
However, invariances in the dataset and the learning algorithm were neglected, i e., the similarities between the generator and the classiﬁer spaces were not studied.
1. K EY C ONTRIBUTIONS
The contributions of this work are as follows
We propose a novel complexity measure between the classiﬁer function space F and the generator function space G called co-complexity , which we use to derive new, more accurate global estimates of generator-aware generalization error bounds (Theorem 3 and 4).
Co-complexity considers not only the complexity of the generator and classiﬁer function spaces, but also the similarities between them.
Doing so allows for the a more exhaustive look into generalization error.
We decompose co-complexity into two different measures of complexity, invariance co-complexity and dissociation co-complexity, which are used to derive new generalization error bounds, including bounds on the expected training error (Theorem 6).
We ﬁnd that reducing invariance co-complexity while keeping the dissociation co-complexity unchanged, helps reduce the generalization gap (low variance) while maintaining low training error (low bias).
This emphasizes
the importance of having classiﬁers that share invariance properties with generator spaces, e.g., rotation-invariant CNNs (Cohen & Welling (2016b)) on MNIST-Rot (Larochelle et al.
We present empirical validation of co-complexity measures of CNN and its scale-equivariant and rotation-equivariant counterparts (SE-CNN in Sosnovik et al.
(2019), RE-CNN in Cohen &
Welling (2016b)), which explains their superior generalization ability compared to MLPs.
Our proposed error bounds are easily specialized to the case where ground truth label function is ﬁxed, leading to potentially tighter generalization error bounds (see Appendix A).
Although our proposed measures are global in nature, local variants can be derived via similar extensions used in
Bartlett et al.
D EFINITIONS
Assume that we have m number of d -dimensional i.i.d sampled instances S =
[ z 1 , z 2 , .., z m ] drawn from some distribution P , and another set of m i.i.d sampled instances S (cid:48) =
[ z (cid:48) 1 , z (cid:48) 2 , .., z (cid:48) m ] also drawn from P .
Deﬁne σ =
[ σ 1 , σ 2 , .., σ m ] where σ i are i.i.d random variables following the Rademacher distribution ( Pr ( σ i = +1)
=
Pr ( σ i = − 1) = 0 . 5 ).
F and G are two function spaces from R d −→ {− 1 , 1 } and we assume that they are deﬁned at all points in R d .
In the context of our problem,
F will be the classiﬁer’s function space, whereas G will be the generator space.
Rademacher Complexity (Bartlett & Mendelson (2003)):
First, we provide the deﬁnition of
Rademacher Complexity :
[Equation]
It can be seen that R m ( F ) indicates the noisy-label ﬁtting ability of the classiﬁer’s function space F .
Correlated Rademacher Complexity: We propose a modiﬁed form of the original Rademacher complexity, called Correlated Rademacher Complexity , R Cm ( F ) , which is deﬁned as follows:
[Equation]
We have 0 ≤ R Cm ( F ) ≤ 1 / 2 .
We also show that R Cm ( F ) ≤ R m ( F ) (see Appendix C.1).
It has been argued that R m ( F ) (and therefore R Cm ( F ) ) can be considered as ”entropy” measures of the entire function space F (Anguita et al.
Note that, like R m ( F ) , R Cm ( F ) is also eventually depends on the noisy label ﬁtting ability of F , but computes it via ability of the function space to assign the same or a different label to two random points
z i and z (cid:48) i (via f ( z i ) f
( z (cid:48) i ) ).
Thus we denote it as the correlated Rademacher complexity of F .
Co-Complexity: Now we propose various complexity measures between two separate function spaces F and G .
Similar to R m ( F ) and R Cm ( F ) , these measures assess the noisy label ﬁtting abilities of the union of the function spaces F and G .
In doing so, these measures compute a joint-entropy like metric over the two function spaces.
First, we deﬁne the Co-complexity between F and G as follows.
[Equation]
Some of the properties of co-complexity are as follows:
P1 R m ( F , G ) =
R m ( G , F ) , i.e., co-complexity is symmetric.
P2 R m ( F , G ) ≥ R Cm ( F ) and R m ( F , G ) ≥ R Cm ( G ) , i.e., the co-complexity between F and G is always greater than the individual correlated Rademacher complexities of F and G .
P3 R m ( F , G ) ≤ R Cm ( F ) + R Cm ( G ) ≤ R m ( F ) + R m ( G ) , i.e., the co-complexity between F and G is upper bounded by the sum of the Rademacher complexities of F and G .
P4 R m ( F , G ) behaves like a joint-entropy measure of F and G .
To see this, let us deﬁne I m ( F , G )
=
R Cm ( F ) + R Cm ( G ) − R m ( F , G ) , called the mutual co-complexity between F and G .
We later ﬁnd that I m ( F , G ) behaves like a mutual-information measure.
This, coupled with the fact that R Cm ( F ) and R Cm ( G ) can be construed as entropy measures of F and G , implies the result.
Invariance Co-Complexity: Next, we quantify some of the properties of the ground truth generator space G in terms of its invariance transformations.
We deﬁne the Invariance Classes of G as follows:
[Equation]
where { τ i ( ., θ i ) | i = 1 , 2 , · · · , n } are functions from R d −→ R d , and θ i represents the extent of the transformation
τ i .
Each θ i ∈ R is a scalar and takes on a set of admissible values depending on the transformation (possibly inﬁnite).
The functions are constrained such that, ∀ τ ∈ I C ( G ) , ∀ g ∈ G , g ( τ ( z ) , t )
=
g ( z ) for all data points z ∈ R d , and for all admissible values of the transformation parameter t .
Also, note that setting θ i =
leads to the identity function, i.e., τ i ( z, 0)
= 1 for all z ∈ R d and all i .
Based on the above we deﬁne a transformation-indicator function I ( z, z i ) , such that I ( z, z i ) =
if z = τ 1 ( ., t 1 ) ◦ τ 2 ( ., t 2 ) ◦ .... ◦
τ n ( ., t n )( z i ) , for a certain t 1 , t 2 , ...
t n and otherwise I ( z, z i ) = 0 .
Additionally, with respect to a generator space G , we also deﬁne the invariance extended set of a datapoint z i ∈ R d as follows:
[Equation]
Note that that invariance extended set of z will always have the same ground truth label as z .
Next, we deﬁne the Invariance Co-complexity between two function spaces F and G as follows:
[Equation]
, where z (cid:48)
i ∼ P ( z ) I
( z, z i ) , ∀ i .
Note that R
Im ( F , G ) ≤ 1 .
Each data point z (cid:48)
i in S (cid:48) is contained within the invariance extended set
I G ( z i ) of z i , and is sampled according to the un-normalized distribution P ( z ) I ( z, z i ) over z ∈ R d .
The invariance co-complexity between F and G indicates the degree to which F obeys the invariance transformations within G .
For instance, a low R Im ( F , G ) would indicate that f ( z i ) f
( z (cid:48) i )
= 1 (thus f ( z i ) =
f ( z (cid:48) i ) for most z i and z (cid:48)
i which are related via some invariance transformation in G .
Dissociation Co-Complexity: We now deﬁne the Dissociation Co-complexity between F and G when the corresponding datapoints in S and S (cid:48) are not related by any invariance transformation.
That is, for all z i ∈ S and z (cid:48) i ∈ S (cid:48) , z (cid:48)
i (cid:54)∈ I G ( z i ) , and is sampled from the distribution P ( z )(1 − I
( z, z i )) over z ∈ R d .
The dissociation co-complexity can then be deﬁned as:
[Equation]
, where z (cid:48)
[Equation]
Note that R D m ( F , G ) measures the average ﬂexibility of F in its label assignment to any two points
z i and z (cid:48) i which are not related via any invariance transformation in G .
Thus, a larger R Dm ( F , G ) would indicate that the classiﬁer function space F is able to easily assign separate labels to datapoints which are not related via any invariance transformation in G .
We also deﬁne a variant of R Dm ( F , G ) , in which S contains only one instance, instead of m instances, denoted as R D, 1 m ( F , G ) .
R D, 1 m ( F , G ) is deﬁned as follows.
[Equation]
, where z (cid:48)
i ∼ P ( z )(1 − I ( z, z
[Equation]
) , ∀ i.
[Equation]
Note that R Dm ( F , G ) ≤ 0 .
5 and R D, 1 m ( F , G ) ≤ 0 .
Rademacher Smoothness: For our ﬁnal result in Theorem 6, we assume that the function space F is Rademacher smooth w.r.t.
G .
We deﬁne Rademacher Smoothness as follows.
If F is Rademacher smooth w.r.t.
G , then any quantity of the form E S,S (cid:48) (cid:2) sup f ∈F (cid:0) 1 m (cid:80) mi =1 f ( z i ) f
( z (cid:48) i ) g
( z i ) g
( z (cid:48) i )
σ i (cid:1)(cid:3) or
E S,S (cid:48) (cid:2) sup f ∈F ,g ∈G (cid:0) 1 m (cid:80) mi =1 f
( z i ) f
( z (cid:48) i ) g
( z i ) g
( z (cid:48) i )
σ i (cid:1)(cid:3) , where the expectation is over datapoint permutations S, S (cid:48) , should lie between the two following cases : (i) S (cid:48)
∈
I G ( S ) (i.e., z (cid:48)
i ∼ P ( z )( I ( z, z i )) for all i ) and (ii) S (cid:48) /
∈
I G ( S ) (i.e., z (cid:48)
i ∼ P ( z )(1 − I ( z, z i )) for all i ).
Then, for some non-zero value of α , 0 ≤ α ≤ 1 , the following holds:
[Equation]
Similarly, the same applies to E S,S
[Equation]
sup f ∈F
[Equation]
g ∈G
[Equation]
m
[Equation]
g
[Equation]
z
[Equation]
g
[Equation]
z
[Equation]
σ i
[Equation]
T HEORETICAL R ESULTS
We now present a series of results which extend the generalization framework using generator spaces.
The proofs of all our results are available in Appendix C. For the following results, we assume that the classiﬁer’s function space F contains the constant function f ( z )
=
c ∀ z , c ∈ {− 1 , 1 } .
We use the deﬁnitions provided in the previous section.
Speciﬁcally, F is be the classiﬁer’s function space and G is the generator space.
Given that the data labels are generated by g ∈ G , we extend the deﬁnition of the sampled instances S by adding the output labels, i.e.,
S =
[( z 1 , g ( z 1 )) , ( z 2 , g ( z 2 )) , .., ( z m , g ( z m ))] .
Then for f ∈ F , we denote the 0-1 loss on S by
[Equation]
We also denote the generalization error over the data samples generated by distribution P as
[Equation]
First, we describe the generalization error bound originally proposed in Bartlett & Mendelson (2003).
Theorem 1.
( Bartlett & Mendelson (2003) )
For 0 < δ < 1 , with probability p ≥ 1 − δ , we have
[Equation]
The above theorem assumes that the ground truth function is completely knowable from the given training examples, and therefore the generator space contains only a single element (Bartlett &
Mendelson (2003)).
For all of our results that follow, we assume the extended case where the generator space can contain more than one element due to the unknowability of the LGF, and construct generalization error bounds that consider the complexity of G as well.
Also note that the following results hold for every choice of f ∈ F and g ∈ G .
First we present the extended Theorem 1 when the LGF is unknowable ( R m ( G ) > 0 ).
Theorem 2.
For 0 < δ < 1 , with probability p ≥ 1 − δ , we have,
[Equation]
Remark 1.
This result states that the complexity of the generator space also directly contributes towards increasing the generalization gap.
In problems where the LGF is known to be heavily constrained and structured, R m ( G ) will be low, reducing the expected generalization gap of classiﬁers, and vice-versa.
Note that in the knowable ground truth scenario where G has a single element, we have R m ( G ) = 0 , which returns the error bound in Theorem 1.
This result shows that larger complexity of G results in a greater generalization gap.
We demonstrate this with neural network generator spaces of varying complexity (see Appendix B.3).
Intuitively, the additional R m ( G ) term comes from the fact that given the training data labels, there still exists a subspace of functions in G which are potential candidates for the ground truth function.
The following result presents tightens Theorem 2 using co-complexity.
Theorem 3.
For 0 < δ < 1 , with probability p ≥ 1 − δ , we have
[Equation]
Remark 2.
Co-complexity measures the degree of similarity between F and G , instead of simply adding their respective complexities, and therefore is tighter than Theorem 2 (co-complexity P3 ).
We also ﬁnd that the generalization gap, for the case when the roles are reversed, is unchanged (see corollary 3.1 in Appendix C).
This demonstrates an inherent symmetry in the problem.
Also note that
Theorem 3 leads to potentially tighter bounds when specialized to the conventional setting where
R m ( G ) =
(Appendix A).
The following result outlines a lower bound on the generalization error, using co-complexity.
Theorem 4.
For 0 < δ < 1 , with probability p ≥ 1 − δ ,
[Equation]
Remark 3.
This result, coupled with Theorem 3, demonstrates that if one interprets the (cid:99) err S ( f ) term as the bias of the classiﬁer, R m ( F , G ) can be interpreted as the variance.
The following result addresses the joint-entropy like behaviour of R m ( F , G ) .
Theorem 5.
We are given the mutual complexity measure I m ( F , G ) as deﬁned in co-complexity
P4 .
Consider another ground truth generator space G (cid:48) , such that I m ( G (cid:48) , G ) = 0 , i.e., G and G (cid:48) are independent.
Then we have,
[Equation]
Remark 4.
Let H ( X ) represents the Shannon entropy of the random variable X
and I ( X ; Y ) represents the mutual information between random variables X and Y .
Then, it is known that
H ( X ) ≥ I ( X ; Y )
+ I ( X ; Y (cid:48) ) , when Y and Y (cid:48) are independent random variables ( I ( Y ; Y (cid:48) )
= 0) .
This observation, coupled with the fact that R Cm ( F ) can be considered as an entropy measure of the function space (see section 2) indicates that the quantity I m ( F , G ) behaves like a mutual information estimate between the function spaces F and G .
Furthermore, as R m ( F , G ) =
R Cm ( F ) + R Cm ( G ) −
I m ( F , G ) , we can see that R m ( F , G ) behaves as a joint entropy measure of two function spaces.
The following result demonstrates how the co-complexity measure R m ( F , G ) can be decomposed into separate co-complexity measures.
Lemma 1.
Consider function spaces F and G , such that F is Rademacher smooth w.r.t.
G .
Let us deﬁne R C,Dm ( G ) as 12 × E σ,S,S (cid:48) ,S (cid:48) /
∈I G ( S ) (cid:2) sup g ∈G (cid:0) 1 m (cid:80) mi =1 g ( z i ) g
( z (cid:48) i )
σ i (cid:1)(cid:3) .
For some non-negative real constant 0 ≤ α ≤ 1 , we then have
[Equation]
17) where R Im ( F , G ) and R Dm ( F , G ) are the invariance and dissociation co-complexity, respectively.
Remark 5.
This decomposition allows us to differentiate the impact of R
Im ( F , G ) and R Dm ( F , G ) on the generalization error bound.
Note that value of α here will be proportional to the cardinality of the invariance transformation classes I G of G .
Using this, we proceed to our ﬁnal result, where we express the generalization error bound in Theorem 3, in terms of the invariance and dissociation co-complexities.
For purposes of simpliﬁcation, we denote R
Im ( F , G ) as R Im , R Dm ( F , G ) as R Dm and similarly for R D, 1 m .
Theorem 6.
Consider function spaces F and G , such that F is Rademacher smooth w.r.t.
G .
Let err P denote the generalization error for the functions f ∈ F which showcase the best ﬁt on the training data samples (averaged across all g in G ).
R C,Dm ( G ) is deﬁned in Lemma 1.
For some non-negative real constants 0 ≤ α, β ≤ 1 and 0 < δ < 1 , with probability p ≥ 1 − δ , we have
[Equation]
Remark 6.
The ﬁrst term in the generalization error bound in (18) represents an upper bound on the average training error of the function which best ﬁts the training data.
We ﬁnd that smaller R
I leads to a smaller generalization gap (variance), while keeping the training error unchanged.
However, the same cannot be said for the dissociation co-complexity R D .
Thus, for a ﬁxed R D and R D, 1 , classiﬁer function spaces with smaller R I will lead to smaller generalization error.
Also note that when the generator space only contains a single element, R C,Dm ( G ) = 0 .
The next proposition gives interpretable bounds on invariance and dissociation co-complexity.
Let z
=
z 1 , z 2 , ..., z m , and z’ =
z (cid:48) 1 , ..., z (cid:48) m , where z k , z (cid:48)
k ∈ R d ∀ k .
Deﬁne the well known growth function of binary valued functions F as A = Π F ( m ) =
max z |{ f ( z k ) : k
= 1 , 2 , ..., m, f ∈ F}| .
Deﬁne the invariance-constrained and dissociation-constrained growth functions of F w.r.t.
G as:
[Equation]
Proposition 1.
Deﬁne R
[Equation]
∈F (cid:0) 1 m (cid:80) m i =1 f ( z i ) f
( z (cid:48) i )
σ i (cid:1)(cid:3) , where z (cid:48)
i ∈ I
G
( z i ) , ∀ i .
Note that R I (cid:48) m
( F , G ) ≤ R
Im ( F , G ) .
Given the deﬁnitions above, we have R I (cid:48) m
( F , G ) ≤ (cid:113) log B 2 m ≤ (cid:113) log Am and R Dm ( F , G ) ≤ (cid:113) log C 2 m ≤ (cid:113) log Am .
Remark 7.
Reducing Π I F |G ( m ) while keeping Π D F |G ( m ) unchanged would result in low invariance co-complexity without affecting dissociation co-complexity, which can’t be achieved by simply reducing Π F ( m ) (or the V C ( F ) ).
This shows that although R Im and R Dm are not completely independent, careful construction of F makes it possible to reduce R Im while maintaining R Dm .
E XPERIMENTS AND D ISCUSSIONS
Our experiments (on MNIST and STL-10) explore the implications of Theorem 6, which states that low R Im while maintaining R Dm improves generalization while keeping training error unchanged.
As all of our proposed complexity metrics assume binary classiﬁers, we create subsets of these datasets containing only two randomly chosen categories, which converts the problem into a binary classiﬁcation scenario.
4. M EASURING I NVARIANCE C O -C OMPLEXITY
As invariance co-complexity is deﬁned w.r.t.
a transformation class ( I G ), we compute the invariance co-complexity (in (6)) of various networks shown in Table 1 for four different transformations.
For each transformation τ , we choose the datapoint pairs S and S (cid:48) in (6) such that z (cid:48) i =
τ ( z i , t ) , where t is the transformation parameter.
The parameter t is randomly chosen as follows: 0 ◦ -180 ◦ for rotation, 0.7-1.2 for scale, 0 ◦ -90 ◦ for shear angle and 0.5-4 pixels for translation.
The computed R I m values indicate the extent to which these networks naturally allow for invariance to various transformation types.
To approximately compute R Im , we take 1000 randomly weighted networks to construct F to compute (6).
We average the results over 100 batches of data ( S and S (cid:48) ), each containingexamples ( m = 1000 ) from their respective datasets.
We compute R
Im for multi-layered perceptrons (MLPs), CNNs and their transformation-equivariant extensions: scale-equivariant CNN (SE-CNN in Sosnovik et al.
and rotation-equivariant CNN (RE-CNN in Cohen & Welling (2016b)).
Note that the scale-equivariant CNN and the rotation-equivariant CNN are known to outperform the vanilla CNN on MNIST and its rotation and scale extensions (Sosnovik et al.
; Cohen &
Welling (2016b)).
Hence, these transformation-equivariant extensions showcase better generalization performance.
One of the objectives of this experiment is to verify whether the invariance and dissociation co-complexities of these networks indeed point to better generalization performance, as is observed in literature.
In every case in Table 1, we ﬁnd that CNNs have lower R Im than MLPs.
As expected, we see that
SE-CNN and RE-CNN have even lower R Im .
CNNs are primarily expected to be robust to translation, due to max-pooling layers.
However, even for other transformations such as rotation and shear, the
R Im of the CNN is less than that of an MLP.
Please see Appendix B.2 for a more detailed analysis on how invariance co-complexity varies with choice of the transformation parameter.
[Table]
[Table]
4. M EASURING D ISSOCIATION C O -C OMPLEXITY
Smaller invariance co-complexity can be inconsequential, if the network’s ability to discriminate between two different categories in G also suffers.
Hence, we measure the dissociation co-complexity (in (8)) to check whether lower values of R Im also affect their discriminatory potential.
We compute
R Dm similarly to R
Im , by using 1000 randomly initialized networks (for the supremum in (8)), and then averaging the measure for 100 batches of data ( S and S (cid:48) ).
Table 2 shows the results, where we ﬁnd that the values of R Dm for all studied networks are very close to each other.
Therefore, only the invariance co-complexity of these architectures will majorly decide the generalization error bounds in Theorem 6.
This shows that, in spite of better invariance capabilities of CNNs (lower R Im ) and its transformation-equivariant extensions, those networks preserve discriminatory potential necessary for differentiating images that belong to different categories.
R EFLECTIONS
The proposed co-complexity measures lead to an extended perspective on generalization error by accounting for ground truth generator spaces.
The objective of introducing the generator space is to consider the structure in the ground truth labelling functions.
New error bounds are proposed which consider various aspects of interaction between generator and classiﬁer function spaces.
Co-complexity can be decomposed into two separate components, invariance co-complexity
( R Im ) and dissociated co-complexity ( R Dm ), which are found to measure the degree to which the classiﬁer’s function space obeys the invariance transformations in the data ( R Im ), and also the degree to which the classiﬁer is able to differentiate between separate categories in the data ( R Dm ).
If we interpret
R m ( F , G ) as the variance of the classiﬁer (see Theorems 3 and 4) and the training error as the bias , we see that R
Im and R Dm affect the training error (bias) and generalization gap (variance) differently (see Theorem 6).
Theorem 6 outlines a clear objective for reducing variance while maintaining low bias: reduce R Im while keeping R Dm and R D, 1 m unchanged.
We note that monitoring R Im
and R Dm can be useful for ﬁnding better learning architectures for a speciﬁc problem.
Furthermore, it is also clear that classiﬁcation problems which contain more invariance constraints, R Im plays a greater role in controlling the generalization gap (as α will be higher).
Experiments on MNIST and STL-10 reveal that the invariance co-complexity R Im for CNNs and their transformation-equivariant extensions is always lower than MLPs, while their dissociation co-complexities, R Dm and R D, 1 m , are comparable to MLPs (see Appendix B.1 for experiments which measure R D, 1 m ).
Furthermore, transformation-equivariant versions of CNN show lower R Im than the vanilla CNN itself, while maintaining R Dm , i.e., they have low variance and low bias.
Appendix B. presents results when neural networks of varying complexity are used as generator spaces and veriﬁes that higher complexity generator spaces lead to a larger generalization gap.
A A PPENDIX - T IGHTENING E XISTING R ESULTS
For the following results we primarily work in the setting where the ground truth generator space only contains a single element, which pertains to the conventional treatment of generalization error bounds with Rademacher complexity.
Note that in this setting we can still deﬁne the invariance classes of
G which we will make use of.
Furthermore we primarily work with the correlated Rademacher complexity R Cm ( F ) which was deﬁned in section 2.
We note that R Cm ( F ) involves two instantiations of the dataset in S and S (cid:48) .
Furthermore, we also observe that the ﬁnal R Cm ( F ) depends on how each datapoint
z i in S is paired with the corresponding datapoint z (cid:48)
i in S (cid:48) ,
[Equation]
For the following results, we deﬁne the invariance-matched correlated Rademacher complexity
R C inv m ( F ) , which chooses a pairing ( i, M ( i )) between the points in S and S (cid:48) , such that
[Equation]
That is, the matching function M is chosen such that there exists the largest number of pairings
z i , z (cid:48) M ( i )
where z (cid:48)
M ( i ) is present the invariance extended set of z i .
Thus, the invariance-matched correlated complexity can then be deﬁned as
[Equation]
where M is chosen using equation 20.
Given this, we have the following theorem.
Theorem 7.
Consider the classiﬁer function space F , and the previous deﬁned correlated
Rademacher complexity of F , R Cm ( F ) .
Furthermore, let us assume that the generator space only contains a single element, i.e., the conventional setting ( R m ( G ) = 0 ).
Then we have for any f ∈ F , with a probability p ≥ 1 − δ , we have
[Equation]
Proof.
First, we will show that for any f ∈ F , with a probability p ≥ 1 − δ ,
[Equation]
This result trivially follows from Theorem 3, when G contains a single element denoted by g 0 , as then
[Equation]
Note that as R Cm ( F ) ≤ R m ( F ) , this is a tighter global bound that the original Rademacher bound.
Then, the result immediately follows when we recognize that changing the ordering of z i and z (cid:48)
i does not affect the value of the generalization gap in (57).
Furthermore, the matching M for each sampled
S, S (cid:48) is independent of the choice of the Rademacher variables σ i , and thus the subsequent steps follow in the same way as in (67).
Subsequently, we still have with a probability p ≥ 1 − δ ,
[Equation]
Note that when the invariance co-complexity of F w.r.t.
G is smaller than its dissociation cocomplexity, then the above is expected to be a tighter bound than (23).
This is because the invarianceaware matching function M leads to a greater number of pairings z i , z (cid:48) i where z (cid:48)
i ∈ I
G ( z i ) (thus lies in its invariance extended set).
B A PPENDIX - A DDITIONAL E XPERIMENTS B. E XPERIMENT : M EASURING D ISSOCIATION C O - COMPLEXITY ( R D, 1 m ( F , G ) )
In the main paper, we measure the invariance co-complexities R
Im ( F , G ) and the dissociation cocomplexities R Dm ( F , G ) of four different network types: Multi-layer Perceptron (MLP), Vanilla
CNN (CNN), Scale-Equivariant CNN (SE-CNN in Sosnovik et al.
and Rotation Equivariant
CNN (RE-CNN in Cohen & Welling (2016b)).
In this experiment, we measure the single-sample dissociation co-complexity R D, 1 m ( F , G ) of these networks, as R D, 1 m ( F , G ) also affects the training error in the generalization error bound in Theorem 6.
Please note that the datasets used in our experiments (MNIST and STL-10), have 10 categories.
However, since the invariance and dissociation co-complexities can only be computed binary classiﬁers, we use random subsets of the data which contain only two categories, for computing the value of co-complexity.
Table 3 shows the results.
We ﬁnd that in both datasets, the R D, 1 m ( F , G ) values of the tested networks are very close to each other.
This is consistent with the dissociation co-complexity R Dm ( F , G ) , shown in Table 2 of the main paper.
Together with those results, it is clear that only the invariance co-complexities R Im ( F , G ) will mainly contribute to the generalization error of these networks, as the dissociation co-complexities R D, 1 m ( F , G ) and R Dm ( F , G ) do not show any signiﬁcant variation.
For example, from Table 1 and 2 of the main paper, we see that the invariance co-complexity (rotation) of the RE-CNN (0.357) is smaller than that of the MLP (0.424), whereas the R D, 1 m ( F , G ) (0.4492 vs.
0.4472) and R Dm ( F , G ) (0.438 vs. 0.4415) of these two networks do not show much variation.
[Table]
B. E XPERIMENT : M EASURING I NVARIANCE C O - COMPLEXITY FOR DIFFERENT TRANSFORMATION PARAMETER CHOICES
In addition to the experiments which estimate the overall invariance co-complexity of the various network baselines, we also showcase how R Im depends on the exact transformation parameter t for each case, in Fig. 3.
Note that equivariant networks usually have low R
Im for most values of t , except for the RE-CNN which is tuned to be invariant to rotations of 90 ◦ , and therefore only showcases signiﬁcant drops in R
Im when t is near 90 ◦ or 180 ◦ .
B. E XPERIMENT : T ESTING G ENERALIZATION E RROR B OUNDS WITH N EURAL N ETWORK G ENERATOR S PACES
We consider a setting where F is the space of linear classiﬁers and the generator space G is the space of relu-activated neural networks with a single hidden layer containing H hidden neurons.
We have
R m ( F ) ∝ (cid:112) 1 /m and R m ( G ) ∝ (cid:112) H/m .
Closed form expressions for co-complexity R m ( F , G ) are non-trivial to compute, but we use the property R m ( F , G ) ≤ R m ( F ) + R m ( G ) (Theorem 2).
Theorem 2 implies that only increasing the complexity of G (by increasing H ), should lead to a steady increase in the difference between training and testing error (i.e. the generalization gap).
We execute this scenario with two variations: m = 10 and m = 100 .
H was chosen from { 2 , 4 , 8 , 16 , 32 } .
Note that the data distribution is kept ﬁxed for all H .
Therefore, in this experiment, only the label generating function is changed.
We observe that the expected generalization gap steadily increases (ranged between 0.6%-8.9%) as we increase the complexity of the generator class (for both m = 10 and m = 100 ).
Furthermore, we ﬁnd a very high degree of correlation (0.982 for m = 10 and 0.976 for m = 100 ) between the generalization gap trend predicted by Theorem 2 ( ≈ (cid:112) H/m ) and the empirically observed average generalization gap.
This shows that the complexity of the generator class also directly controls the generalization gap, when everything else is unchanged.
This result clearly highlights that when additional information about the generator space is known, more accurate trends of generalization error can be estimated by using both R m ( G ) and R m ( F ) .
C A PPENDIX - P ROOF OF T HEORETICAL R ESULTS
In what follows, we provide the proofs of Theorems 2 to 6.
The proof of Theorem 1 of the main paper is available in Bartlett & Mendelson (2003).
We also prove, in Lemma 2, the properties of co-complexity stated in Section 5 of the main paper.
We use the deﬁnitions provided in Section 5 of the main paper.
when 2-layer neural networks of variable hidden neuron number H , are chosen as the label generating function, and a single layer neural network is chosen as the classiﬁer.
The generalization gap trend is plotted for two scenarios, (a) m = 10 and (b) m = 100 , where m is the number of training examples.
As here R m ( F ) is ﬁxed, but R m ( G ) ∝ √ H changes with H , we plot the generalization gap averages against √ H .
Note that the trend is linear in nature in both cases (correlation of ≈ 0 .
98 for both), showing that the generalization gap depends directly on the complexity of the generator space.
Please note that for the following proofs, a variable z which follows the Rademacher distribution ( Pr ( z = +1) =
Pr ( z = − 1) = 0 . 5 ) are referred to as Rademacher variable.
C. P ROOFS OF P ROPERTIES P1-P3 OF C O - COMPLEXITY
We begin with the proof of the properties of co-complexity ( P1-P3 ) and a property of the correlated
Rademacher complexity, which are described in Section 5 of the main paper.
Lemma 2.
For any two function spaces F and G , the following statements hold.
[Equation]
[Equation] [Equation]
[Equation] [Equation]
Proof.
(i)
To prove the ﬁrst statement, it is trivial to see that interchanging f and g in the expression of R m ( F , G ) keeps it unchanged.
Thus, R m ( F , G ) =
R m ( G , F ) .
(ii) Let us deﬁne σ (cid:48)
i = g
( z i ) g
( z (cid:48) i )
σ i .
To prove the second statement, we choose any ﬁxed function g 0 within G and we have,
[Equation]
Here, we use the fact that σ (cid:48)
i is also a Rademacher variable.
(iii)
We can prove the third statement in the same manner, ﬁxing f 0 ∈ F instead.
(iv)
The fourth statement can be proven as follows.
First, we note that f ( z i ) f
( z (cid:48) i ) and g ( z i )
g
( z (cid:48) i ) are variables which take the value of 1 or − 1 .
Thus, f ( z i ) f ( z (cid:48) i ) g
( z i ) g
( z (cid:48) i ) can be expressed as
[Equation]
i ) − g
( z i ) g
( z (cid:48) i ) v i ) , where v i ∈ {− 1 , − 1 } depends on f ( z i ) f
( z (cid:48) i ) and g ( z i )
g
( z (cid:48) i ) together, but is independent of each of them individually.
We can therefore write,
[Equation]
Now
[Equation]
consider any ﬁxed σ = { σ 0 , ..., σ m } = { σ 00 , ..., σ 0 m }
[Equation]
and ﬁx S and S (cid:48) .
First
[Equation]
let us denote
[Equation] [Equation]
Let us denote the ﬁnal v i for f opt and g opt by v opti .
Note that v opti here depends on f opt ( z i ) f opt ( z (cid:48) i ) and g opt ( z i ) g opt
( z (cid:48) i ) .
Note that v opt 1 , v opt 2 , ..v optm will be independent of each other, as σ 1 , σ 2 , ..
σ m are independent.
Also observe that v opt 1 , v opt 2 , ..v optm will be independent of σ 01 , σ 0 2 , ..
σ 0 m respectively, as they only depend on f opt and g opt .
We deﬁne σ (cid:48)
i = v opti σ 0 i .
We note that σ (cid:48) 1 , σ (cid:48) 2 , ..σ (cid:48) m will be independent across all samples, and Pr ( σ (cid:48)
i = +1) =
Pr ( σ (cid:48)
i = − 1) = 0 .
5 , same as σ i .
Thus σ (cid:48) 1 , σ (cid:48) 2 , ..σ (cid:48) m are i.i.d
Rademacher variables like σ 1 , σ 2 , ..σ m .
We thus note,
[Equation]
where in the ﬁnal supremum σ (cid:48) i are kept ﬁxed at their original values ( σ (cid:48)
i = v opti σ 0 i ).
We then have,
[Equation]
The ﬁnal result R Cm ( F ) ≤ R m ( F ) can be shown as follows.
To prove this, we use the fact that f
[Equation]
z
[Equation]
z
[Equation]
i ) =
[Equation]
− | f ( z i ) − f ( z (cid:48) i ) | .
We have,
[Equation]
Here, v i takes values in {− 1 , 1 } , depending on whether f ( z i ) or f ( z (cid:48) i ) is greater.
Note that as σ i is a Rademacher variable, v i σ i is also a Rademacher variable.
We also note that v i is independent of f ( z i ) and
f ( z (cid:48) i ) individually.
Thus we have,
[Equation]
where σ (cid:48) i = v i σ i .
As − σ (cid:48)
i is also a Rademacher variable, we ﬁnally have
[Equation]
This completes the proofs.
C. P ROOF OF T HEOREM
The following result proves a generalization error bound which incorporates R m ( F ) and R m ( G ) .
Theorem 2.
For 0 < δ < 1 , with probability p ≥ 1 − δ , we have
[Equation]
Proof.
First, we reiterate that (cid:99) err S ( f ) =
(cid:80) mi
=1 (1 − f
( z i ) g truth ( z i )) / 2 m , for some g truth ∈ G , and similarly for err P ( f ) , where m −→ ∞ .
Then, we retrace the steps of the original proof in Bartlett & Mendelson (2003), noting that
[Equation]
Here we initially only consider functions g in G such that g ( z 0 ) =
g truth ( z 0 ) , g ( z 1 ) =
g truth ( z 1 ) , ..., g ( z m ) =
g truth ( z m ) .
Notice, that differently from (Bartlett & Mendelson (2003)), the supremum eventually considers the generator space G in addition to the classiﬁer’s function space F .
We deﬁne φ ( S ) =
sup f
[Equation]
,g ∈G ( err P ( f ) − (cid:99) err S ( f )) , and notice that
[Equation] [Equation]
This leads to the application of McDiarmid’s Inequality (McDiarmid (1989)) on the concentration bounds for φ ( S ) , which results in the the following bound, with a probability of at least 1 − δ ,
[Equation]
Finally, we express E S
[ φ ( S )] as follows.
[Equation]
The last inequality (Jensen’s inequality (Cover & Thomas (2012)) is applicable because sup is a convex function.
We proceed by multiplying the terms with Rademacher variables σ i , which randomly take on {− 1 , 1 } with equal probability.
This keeps the expected value unchanged.
Note that E
[ σ i ] = 0 .
In what follows, we express f ( z i ) g
( z i ) = 1 − | f
( z i ) − g
( z i ) | .
This is enabled because both have their range in {− 1 , 1 } .
[Equation]
Note that multiplication with the Rademacher variables does not change the value of the expectation, as the Rademacher variable σ i simply controls whether z i and z (cid:48)
i belong to S and S (cid:48)
( σ i = 1) , or to
S (cid:48) and S (cid:48)
( σ i = − 1 ).
Coupled with the fact that E [ σ i ] = 0 , we note that this keeps the expectation in (59) unchanged.
Now, note that ( | f ( z i ) − g ( z i ) | ) σ i = ( f ( z i ) − g
( z i ))
v i σ i .
Here v i is also a variable that takes its values in {− 1 , 1 } .
Since σ i is a Rademacher variable, v i σ i = σ (cid:48)
i is also a
Rademacher variable.
In what follows, we use the fact that v i is independent of f i and g i individually.
[Equation]
Similarly, we can derive the same for E σ,S,S
[Equation]
∈F
[Equation]
Thus, ﬁnally, we can represent the upper bound on E [ φ ( S )] as follows .
[Equation]
This leads to the ﬁnal form of the generalization error bound.
With probability at least 1 − δ , we have
[Equation]
C. P ROOF OF T HEOREM
The following statements propose the generalization error bound in terms of co-complexity.
Theorem 3.
For 0 < δ < 1 , with probability p ≥ 1 − δ , we have
[Equation]
Proof.
We proceed in the same way as in appendix C.2, until we arrive at (57).
In what follows, we note that we can write f ( z i ) g
( z i )
− f
( z (cid:48) i ) g
( z (cid:48) i ) =
[Equation]
1 − f
[Equation]
z i ) g
( z i ) f
( z (cid:48) i ) g
( z (cid:48) i ))
v i , for some binary valued variable v i with range in {− 1 , 1 } .
We also multiply the original Rademacher variable σ i to (57)
[Equation]
and derive the following set of bounds.
[Equation]
The last step follows from the fact that E [ σ i ] = 0 , and the assumption that σ (cid:48)
i = v i σ i , are also i.i.d
Rademacher variables.
This assumption will hold for most well behaved classiﬁers, for which less than half of the terms in (68) are greater than zero (which is applicable for classiﬁers with not too large R m ( F ) ).
As 12 × E σ
[Equation]
,S,
[Equation]
= R m ( F , G ) , we have the ﬁnal generalization bound as follows.
With probability ≥ 1 − δ ,
[Equation]
The above theorem leads to the following corollary.
Corollary 3.1.
We consider the hypothetical case where the roles are reversed, i.e., when the generator function space G is used to ﬁt the labels, which are now generated by a function f ∈ F .
For any function g ∈ G , for 0 < δ < 1 , with probability p ≥ 1 − δ , we have
[Equation]
Proof.
This is a direct outcome of the fact that R m ( F , G ) =
R m ( G , F ) .
When the data labels in S is generated by functions in F rather than G , we will simply have with probability p ≥ 1 − δ ,
[Equation]
Note that here the labels in S are generated by a certain f ∈ F .
This completes the proof.
C.
P ROOF OF T HEOREM
The following result outlines the lower bound on the error function.
Theorem 4.
For 0 < δ < 1 , with probability p ≥ 1 − δ ,
[Equation]
Proof.
First, we note that
[Equation]
Next, we denote
[Equation]
We then proceed similarly to Theorem 2 and Theorem 3, and we can show that
[Equation]
Furthermore, note that
[Equation]
This allows us to apply another consequence of Mcdiarmid’s inequality, which leads to the following, with probability p ≥ 1 − δ ,
[Equation]
This completes the proof.
C. P ROOF OF T HEOREM
Now, we outline a set of results pertaining to the joint-entropy like behaviour of R m ( F , G ) .
Theorem 5.
We are given the mutual complexity measure I m ( F , G ) as deﬁned in co-complexity P (Section 5 of main paper).
We consider an alternative ground truth generator space G (cid:48) , such that
I m ( G (cid:48) , G ) = 0 , i
[Equation]
e., G and G (cid:48) are independent.
Then we have,
[Equation]
Proof.
To prove the above, note that it is sufﬁcient to prove the following:
[Equation]
We elaborate on the left hand side of the above inequality as follows
[Equation]
We introduce i
[Equation]
i.d Rademacher variables { (cid:15) 0 , ..., (cid:15) m } , by multiplying them with the terms within
R m ( F , G (cid:48) ) .
As E [ (cid:15) i ] = 0 ∀ i , this maneuver does not change the value of the expectation
[Equation]
≥
[Equation]
m (cid:88) i
= f
[Equation]
z
[Equation]
z
[Equation]
(cid:15)
[Equation]
g (cid:48) (
[Equation]
i
[Equation]
g (cid:48) (
[Equation]
(cid:48) i ))
σ i v i (cid:33)(cid:35) (86)
Now note that one can re-express g
( z i ) g
( z (cid:48) i )
+ (cid:15)
i g (cid:48) ( z i ) g (cid:48)
( z (cid:48)
i ) as (1 + (cid:15) i g
( z i ) g
( z (cid:48) i ) g (cid:48)
( z i ) g (cid:48)
( z (cid:48) i )) v i .
Here v i ∈ {− 1 , 1 } ∀ i .
Note that the value of v i depends on the values of g ( z i ) , g ( z (cid:48) i ) , g (cid:48)
( z i ) , g (cid:48)
( z (cid:48) i ) and (cid:15) i .
However, note that with so many dependencies v i ultimately is independent of all of these terms, individually.
We have,
[Equation]
Note that the expectation is over a ﬁxed values of σ, (cid:15), S and S (cid:48) .
For the next step, we recognize that v i σ i = σ (cid:48)
i is a Rademacher variable itself in the ﬁrst term to the R.H.S of the inequality above, as v i is independent of f ( z i ) and f ( z (cid:48) i ) .
For a ﬁxed value of those parameters, we then consider
[Equation]
Then, we can compute a lower bound for the expression in (87) as follows,
[Equation]
It is important to note here that the variable (cid:15) i σ i v i f ∗ ( z i ) f ∗ ( z (cid:48)
i ) is a Rademacher variable, but however, is not yet independent of the other terms being multiplied to it.
The terms (cid:15) i , σ i , f ∗ ( z i ) and f ∗ ( z (cid:48) i ) are all independent of g ( z i ) , g ( z (cid:48) i ) , g (cid:48)
( z i )& g (cid:48)
( z (cid:48) i ) , but however, the same cannot be said of v i .
Therefore, to resolve this issue we note that the product g ( z i ) g ( z (cid:48) i ) g (cid:48) ( z i ) g (cid:48)
( z (cid:48)
i ) itself can be expressed as another function g (cid:48)(cid:48)
( z i , z (cid:48) i ) =
g ( z i ) g
( z (cid:48) i ) g (cid:48)
( z i ) g (cid:48)
( z (cid:48) i ) , as an instance of another function space G (cid:48)(cid:48) and the second term in (89) can then be re-expressed as,
[Equation]
Note that although v i was not independent of the g ( z i ) , g ( z (cid:48) i ) , g (cid:48)
( z i )& g (cid:48)
( z (cid:48)
i ) , it is independent of the product g
( z i ) g
( z (cid:48) i ) g (cid:48)
( z i ) g (cid:48)
( z (cid:48) i ) , and thus independent of g (cid:48)(cid:48)
( z i , z (cid:48) i ) .
Thus, now we can ﬁnally express the variables { σ (cid:48) i
=
( (cid:15) i σ i v i f ∗ ( z i )
f ∗ ( z (cid:48) i ))
| i = 1 , 2 , · · · , m } as i.i.d Rademacher variables themselves.
This leads to the lower bound as follows:
[Equation] [Equation]
The last step follows from the fact that I m ( G (cid:48) , G ) = 0 , that is, the function spaces G and G (cid:48) are independent (
i.e. R m ( G , G (cid:48) )
=
R Cm ( G ) + R Cm ( G (cid:48) ) ).
This concludes our proof.
We now prove a corollary to Theorem 5, where we discuss implications of Theorem 5 on the generalization error bound in Theorem 3.
Corollary 5.1.
Given a function space G (cid:48) , such that I m ( G (cid:48) , G ) = 0 .
Suppose that R m ( F ) =
I m ( F , G ) + I m ( F , G (cid:48) )
+ (cid:15) , for some (cid:15) ≥ 0 .
We have with probability p ≥ 1 − δ ,
[Equation]
Proof.
This is a direct consequence of the fact that R m ( F ) ≥ I m ( F , G )
+ I m ( F , G (cid:48) ) , combined with the generalization bound in Theorem 3.
Now we proceed to the ﬁnal set of results, where we decompose the co-complexity into two other complexity terms, as deﬁned in Section 5 of the main paper.
First, we show how the co-complexity measure R m ( F , G ) can be decomposed into R
Im ( F , G ) and R Dm ( F , G ) .
C. P ROOF OF T HEOREM
The following results discuss the impact of dissociation co-complexity and invariance co-complexity on generalization error bounds.
Lemma 2.
Consider function spaces F and G , such that F is Rademacher smooth w.r.t.
G .
Let us deﬁne R C,Dm ( G ) as 12 ×
[Equation] [Equation]
) σ
[Equation]
For some non-negative real constant 0 ≤ α ≤ 1 , we then have
[Equation]
where R Im ( F , G ) and R Dm ( F , G ) are the invariance co-complexity and the dissociation co-complexity, respectively.
Proof.
As the functions are Rademacher smooth, we can decompose the expectation over S, S (cid:48) to only the ones where S (cid:48) ∈
I G ( S ) , and the ones where S (cid:48) / ∈
I G ( S ) .
We begin from (67) as follows.
[Equation]
Then, using the Rademacher smoothness constraint
[Equation]
we have for a certain 0 ≤ α ≤ 1 ,
R m
[Equation]
≤ α
E
[Table]
,S,S (cid:48) ,S (cid:48) ∈I G ( S ) (cid:34) sup f ∈F (cid:32)m m (cid:88) i = (1 − f
( z i ) f
( z (cid:48) i ))
σ i (cid:33)(cid:35)
+ (1 − α )
( R Dm ( F , G ) + R C,Dm ( G )) (99) ≤ α
E S,S (cid:48) ,S (cid:48)
∈I G ( S ) (cid:34) sup f ∈F (cid:32)m m (cid:88) i = (1 − f
( z i ) f
( z (cid:48) i ))
(cid:33)(cid:35) + (1 − α )
( R Dm ( F , G ) + R C,Dm ( G ))
(100) =
α R Im ( F , G )
+ (1 − α )
( R Dm ( F , G ) + R C,Dm ( G )) .
We use the fact that E σ,S,S (cid:48) ,S (cid:48) / ∈I G ( S ) (cid:104) sup f ∈F ,g ∈G (cid:16) 1 m (cid:80) mi =1 f ( z i ) f
( z (cid:48) i ) g
( z i ) g
( z (cid:48) i ) σ i(cid:17)(cid:105) ≤ R Dm ( F , G )
+
R C,Dm ( G ) , which follows trivially from a derivation similar to that of co-complexity P3 in Lemma( R m ( F , G ) ≤ R Cm ( F ) + R Cm ( G ) ).
This completes the proof.
Using this, we proceed to the proof of Theorem 6, where we re-express the generalization error bound in terms of the invariance and dissociation co-complexities, including bounds on the training error.
For purposes of simpliﬁcation, we denote
Theorem 6.
Consider function spaces F and G , such that F is Rademacher smooth w.r.t.
G and
F contains the constant function f ( z )
= c, ∀ z , c ∈ {− 1 , 1 } .
Let err P denote the generalization error for the functions f ∈ F which showcase the best ﬁt on the training data samples.
For some non-negative real constants 0 ≤ α, β ≤ 1 and 0 < δ < 1 , with probability p ≥ 1 − δ , we have
[Equation]
Note: Recall that R C,Dm ( G ) is deﬁned in Lemma 2.
Proof.
We note that err P is the generalization error for the case when only the best ﬁtting function f opt = arg minf ∈F ( (cid:99) err S ( f )) is chosen for each training data set S , and is averaged over all possible ground truth functions g ∈ G .
Note that the labels in S is subject to change, depending on the choice of g .
Thus for ﬁxed datapoints z 1 , z 2 , .., z m ∈ S , (53) changes to
[Equation]
We now denote φ (cid:48)
( S ) =
E g ∈G [inf f ∈F ( (cid:99) err S ( f opt ))]
+ sup f ∈F ,g ∈G ( err P ( f ) − (cid:99) err S ( f )) .
Let us also denote ψ ( S ) =
E g ∈G [inf f ∈F ( (cid:99) err S ( f opt ))] and γ ( S ) =
sup f ∈F ,g ∈G ( err P ( f ) − (cid:99) err S ( f )) .
We then have the following for a ﬁxed f opt .
[Equation]
It is easy to show that sup
z 1 ,z 2 ,...,z n ,z (cid:48)
i ∈Z | ψ ( z 1 , z 2 , ..., z i , ...., z m ) − ψ ( z 1 , z 2 , ..., z (cid:48) i , ...., z m ) | ≤ 1 m and sup z 1 ,z 2 ,...,z n ,z (cid:48) i ∈Z | γ ( z 1 , z 2 , ..., z i , ...., z m ) − γ ( z 1 , z 2 , ..., z (cid:48)
i , ...., z m ) | ≤ 1 m .
Applying Mcdiarmid’s inequality, we have with probability p ≥ 1 − δ ,
[Equation] [Equation]
We already have that E S (cid:2) sup f ∈F ,g ∈G ( err P ( f ) − (cid:99) err S ( f )) (cid:3) ≤ R m ( F , G ) ≤ α R Im +(1 − α )
( R Dm +
R C,Dm ( G ) , for a certain 0 ≤ α ≤ 1 (Lemma 2).
The ﬁrst term (expected training error) can be expressed as,
[Equation]
As iterated in the main statement of this theorem, we assume that the classiﬁer function space is complex enough to allow for an error rate ≤ 50% (chance level for binary classiﬁcation) on the training data itself.
This indicates that we can safely assume that sup f ∈F (cid:0) 1 m (cid:80) i f
( z i ) g
( z i ) (cid:1) ≥ 0 .
We then have,
[Equation]
where we make use of the fact that f ( z (cid:48) ) g ( z (cid:48) )
f ( z (cid:48) ) g ( z
[Equation]
= 1 .
Here
[Equation]
the datapoint z (cid:48) is sampled from the same distribution P .
Thus we have,
[Equation]
As before
[Equation]
we assume that the function space f is Rademacher smooth,
i
[Equation]
e. for a certain 0 ≤ β ≤ 1 , we can express
[Equation] [Equation]
The last step above involves the reasonable assumption that the function space is going to be on average worse at ﬁtting random noise labels, rather than a labelling provided by an element of the generator space G .
Finally, we note that
[Equation]
116) simply by choosing the constant function f ( z ) =
c , which is assumed to be present within the classiﬁer’s function space F .
Also, note that E S,z (cid:48) ,S/ ∈I G ( z (cid:48) ) ,σ (cid:2) sup f ∈F (cid:0) 1 m (cid:80) i f
( z i ) f
( z (cid:48) )
σ i (cid:1)(cid:3)
= 2 R D, 1 m ( F , G ) .
We then have
[Equation]
This leads to the ﬁnal upper bound for err P by replacing the terms in (108) by the above.
With probability p ≥ 1 − δ , we then have,
[Equation]
Finally, we provide the proof of Proposition 1, which gives interpretable bounds on invariance co-complexity and dissociation co-complexity.
Proposition 1.
Deﬁne the well known growth function of binary valued functions F as A = Π F ( m ) =
max z |{ f ( z k ) : k
= 1 , 2 , ..., m, f ∈ F}| .
Deﬁne the invariance-constrained and dissociation-constrained growth functions of F w.r.t.
G as:
[Equation]
Deﬁne R
[Equation]
, where z (cid:48)
[Equation]
∈
I G ( z i ) , ∀ i .
Note that R I (cid:48) m
( F , G ) ≤ R
Im ( F , G ) .
Given the deﬁnitions above, we have R Im ( F , G ) ≤ (cid:113) log B 2 m ≤ (cid:113) log
Am and R Dm ( F , G ) ≤ (cid:113) log C 2 m ≤ (cid:113) log Am .
Proof.
First we show that R I (cid:48) m
( F , G ) ≤ R
Im ( F , G ) .
This follows from the fact that
[Equation]
The proof of the main proposition immediately follows from the application of Massart’s ﬁnite class lemma, which states that for a ﬁnite subset K of R m , and Rademacher variables σ 1 , σ 2 , ..., σ m , we have
[Equation]
This immediately leads to the well known result for Rademacher complexity R m ( F ) , which states
[Equation]
The result above directly follows from the fact that the cardinality of the ﬁnite subset K is upper bounded by the growth function Π F ( m ) , when K is the set of all possible function outputs from functions f ∈ F , on any m points.
In our case, as we have that
[Equation]
we can directly apply Massart’s lemma, by the using the invariance-constrained growth function
Π I F |G ( m ) to obtain the ﬁrst result as follows.
First, we choose the ﬁnite subset K such that it contains all possible outputs of the function product f
( z i ) f
( z (cid:48) i ) when S, S (cid:48) is chosen such that each element in S (cid:48) lies in the invariance extended subset of each corresponding element in S .
For this
K , the cardinality |K| is bounded above by the previously deﬁned growth function Π I F |G ( m ) .
This immediately leads to the ﬁrst result,
[Equation]
We can similarly show the equivalent result for dissociation co-complexity R Dm ( F , G ) , to obtain
[Equation]
Finally, we recognize that the growth functions Π I F |G ( m ) and Π D F |G ( m ) are bounded above by the cardinality of all possible function values taken in S, S (cid:48) , which in turn is bounded above by Π F ( m ) × Π F ( m ) .
This implies Π I F |G ( m ) ≤ Π F ( m ) 2 and Π D F |G ( m )
≤ Π F ( m ) 2 , from which we then have
[Equation]
and
[Equation]
This concludes the proof.
D N EURAL N ETWORK A RCHITECTURE D ETAILS
Here we present the architectures that were used in our experiments.
As mentioned in section 4, four different architectures were tested.
In Table 4, we show the layer-wise details of each architecture.
Note that all architectures share a similar parameter count, and also evoke a similar dissociation co-complexity as a result (see Table 2 and 3).
Note that the input to all these architectures is an image of size 28 × 28 (same size as in the MNIST dataset).
For STL-10, we resized the input ( 96 × 96 ) to 28 × 28 and used the same architectures for consistency.
[Table]