On Robust Multiclass Learnability
Anonymous Author(s)
AffiliationAddress email
Abstract
This work analyzes the robust learning problem in the multiclass setting.
Under theframework of Probably Approximately Correct (PAC) learning, we first show thatthe graph dimension and the Natarajan dimension, which characterize the standardmulticlass learnability, are no longer applicable in robust learning problem.
We thengeneralize these notions to the robust learning setting, denoted as the adversarialgraph dimension (AG-dimension) and the adversarial Natarajan dimension (ANdimension).
Upper and lower bounds of the sample complexity of robust multiclasslearning are rigorously derived based on the AG-dimension and AN-dimension,respectively.
Moreover, we calculate the AG-dimension and AN-dimension of theclass of linear multiclass predictors, and show that the graph (Natarajan) dimensionis of the same order as the AG(AN)-dimension.
Finally, we prove that the AGdimension and AN-dimension are not equivalent.
Introduction
Learning models that are robust to adversarial perturbations has attracted significant research attentionin recent years.
Notably, prior work on this subject has largely studied the theory of robust learnabilityin the context of binary supervised learning.
However, many important problems require classificationinto a great number of target classes.
For example, in image object recognition and language modelsbuilding[1, 2], the number of classes scales as the number of possible objects or the dictionary sizerespectively.
It is thus of both practical and theoretical interest to generalize the robust binary learningtheory to the robust multiclass setting.
In the standard multiclass learning setting, the finiteness of the Natarajan dimension or graphdimension [3] (both of which are generalized from the VC-dimension [4]) is necessary and sufficientto ensure learnability [5].
Recent work has shown that the finiteness of the VC-dimension is neithersufficient nor necessary for (proper) robust binary classification [6].
Inspired by [6], this paperdemonstrates that the Natarajan/graph dimension can no longer characterize robust learnability in themulticlass problem.
Meanwhile, some questions naturally arise: In multiclass setting, how can robustlearning be ensured?
And what is necessary for robust learning?
To answer these questions, we generalize the corrupted hypothesis classes [7], which arise fromstandard hypothesis classes in the binary setting in the presence of an adversary, and use them toreduce the robust learning problem to a non-robust one.
By considering the graph dimension and
Natarajan dimension of the corrupted hypothesis classes—which are defined as the adversarial graphdimension (AG-dimension) and adversarial Natarajan dimension (AN-dimension) respectively—wederive the upper and lower bounds of sample complexity of the robust multiclass learning problem.
We then analyze the AG-dimension and the AN-dimension of linear multiclass predictor class,showing that the AG(AN)-dimension and the graph (Natarajan) dimension are of the same order.
Since it is proven that the graph dimension and the Natarajan dimension are equivalent when thetarget class is finite [5], is it possible to use the AG-dimension and AN-dimension to bound eachother?
Our work suggests that the equivalent relationship does not hold, which is illustrated byconstructing a hypothesis class with finite AN-dimension but not robustly learnable.
Related work.
[8, 9, 10, 11] focus on the robust binary classification problem.
[3] characterize themulticlass learnability in non-robust setting.
[12, 5, 13] define a large family of notions of dimensions,all of which generalize the VC-dimension and may be used to estimate the sample complexity ofmulticlass classification.
[14] calculate the graph dimension and the Natarajan dimension of linearmulticlass predictor class.
[15] upper bounds the adversarial risk of linear multiclass classifiersbased on Rademacher complexity, which cannot directly derive bounds on sample complexity ofgeneral robust multiclass learning problems.
The theory on robust multiclass learnability for generalhypothesis class is less explored.
This work takes some steps towards solving this problem.
Preliminaries
This section introduces some basic notations and concepts used throughout the present study.
In the remainder of this article, R , N , R + and R d represent the sets of real numbers, natural numbers,non-negative real numbers and d -dimensional vectors over R , respectively.
We denote the set{ 1 , . . . , n } (for n ∈ N ) by [ n ] .
If A and B are sets, we use B A to denote the collection of allmappings from A to B and 2 A to denote the power set of A, that is the collection of all subsets of A.
Let C
= { c 1 , . .
, c m } ⊂ X and C ′
⊂ C be a subset of C,
then I C ′ ∈ [ m ] denotes the index set of
C ′ .
Let H be a class of functions defined in X , then H| C represents the restriction of H to C, that is
H C
= { ( h ( c 1 ) , . . .
, h ( c m )) :
h ∈ H} .
We denote the indicator function by 1 ( event ) , that is 1 if anevent happens and 0 otherwise.
Finally, ∥ · ∥ p represents the ℓ p norm.
The robust learning problem is formalized as follows.
Let X =
R d be the instance space and
Y = { 1 , . . .
, k } be the label space.
For an unknown distribution D over X × Y 1 and a hypothesisclass H ⊂ Y X , the goal of robust learning is to find a function f ∈ H , based on n labeled trainingsamples S
=
( x i , y i ) } ni =1 drawn independent and identically distributed (i.i.d.) from D , such thatunder a small perturbation U : X → 2 X , the adversarial risk
[Equation]
is minimal for the 0-1 loss l (ˆ y, y ) ≜
1 (ˆ y ̸ = y ) .
The perturbation U ( x ) is required to be nonempty, so some choice of x ′ is always available.
Onechoice for the perturbation is the p -norm ball ( p ≥ 1 ) with a small radius r , i.e. U ( x ) =
{ z ∈ X :∥
z − x ∥ p ≤ r } .
Selecting r = 0 gives the identity perturbation : I ( x ) ≜ { x } .
Note that in this case,the problem is reduced to a standard learning problem, and R I ( f ; D ) is called the standard risk of fover D .
The mechanism of finding a function is called a learning algorithm .
In this paper, we focus on properlearning algorithms, that is, learning algorithms will always pick a function from H .
Formally, alearning algorithm is a function A : ∪ ∞ n =
( X → Y ) n → H .
A common choice of robust learningalgorithm is to learn through adversarial empirical risk minimization or AERM for short:
[Equation]
where ˆ D n is the empirical distribution generated by S, i.e. for ( x, y ) ∼ ˆ D n , ( x, y ) is equal to ( x i , y i )with probability 1 /n for each i ∈ { 1 , . .
. , n } .
Learning algorithms like AERM do not always produce a function that achieves the optimal adversarial risk, because a training set consisting of finite samples is not always sufficiently informative to represent D .
Thus, we introduce the definition of robust probably approximately correct (PAC)learnability, in the realizable and agnostic setting [7, 6]:
Definition 1 (Agnostic Robust PAC Learnability) .
Under perturbation U , a hypothesis class H isrobustly PAC learnable in the agnostic setting if there is a function m ag H , U : (0 , 1) 2 → N and alearning algorithm A : ∪ ∞ n =
( X × Y ) n → H with the following property.
For every ϵ, δ ∈ (0 , 1)and every distribution D over X × Y , let S = { ( x i , y i ) } ni
=1 be n samples i.i.d generated by D s.t.n > m ag H , U ( ϵ, δ ) .
Then with probability at least 1 − δ ,
[Equation]
The function m ag H , U is defined as the sample complexity of learning H under perturbation U in theagnostic setting.
Definition 2 (Realizable Robust PAC Learnability) .
Under perturbation U , a hypothesis class His robustly PAC learnable in the realizable setting if there is a function m re H , U : (0 , 1) 2 → N and alearning algorithm A : ∪ ∞ n =
( X × Y ) n → H with the following property.
For every ϵ, δ ∈ (0 , 1)and every distribution D over X × Y , let S = { ( x i , y i ) } ni
=1 be n samples i.i.d generated by D s.t.n > m re H , U ( ϵ, δ ) .
Then with probability at least 1 − δ ,
R U ( A ( S ); D ) ≤ ϵ.
The function m re H , U is defined as the sample complexity of learning H under perturbation U in therealizable setting.
These definitions agree with the standard PAC learnability
when U =
I , which has been widelystudied in both binary ( k = 2 ) and multiclass ( k > 2 ) setting.
We recall the known result regardingthe sample complexity of binary learning.
Recall the definition of the Vapnik-Chervonenkis dimension(VC-dimension) [4]:
Definition 3 (VC-dimension) .
Let H ⊂ { 0 , 1 } X be a hypothesis class and let S ⊂ X .
We say that Sis shattered by H if H| S
= { 0 , 1 } S .
The VC-dimension of H , denoted by VC ( H ) , is the maximalcardinality of a set that is shattered by H .
The following theorem, based on the VC-dimension, characterizes the sample complexity of learningbinary hypothesis classes.
Theorem 1 ([4] and [16]) .
There are absolute constants C 1 , C 2 > 0 such that for every H ⊂ { 0 , 1 } X ,
[Equation]
and
[Equation]
It is natural to seek a generalization of the VC-dimension to hypothesis classes of non-binary functions.
We recall two generalizations, both introduced by [3]:
Definition 4 (Graph dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H
G-shatters S if there exists an f : S → Y such that for every T ⊂ S, there is a g ∈ H such that ∀ x ∈ T, g ( x ) =
f
( x ) , and ∀ x ∈ S \ T, g ( x ) ̸ = f ( x ) .
The graph dimension of H , denoted by d G ( H ) , is the maximal cardinality of a set that is G-shatteredby H .
Definition 5 (Natarajan dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H N-shatters S if there exists f 1 , f 2 : S
→ Y such that ∀ x ∈ S, f 1 ( x ) ̸
= f 2 ( x ) , and for every
T ⊂ S, there is a g ∈ H such that
∀ x ∈ T, g ( x ) = f 1 ( x ) , and ∀ x ∈ S \ T, g ( x ) =
f 2 ( x ) .
Both the graph dimension and the Natarajan dimension coincide with the VC-dimension for k = 2 .
Based on these notions, the following theorem provides the upper and lower bounds of samplecomplexity of standard multiclass learning:
Theorem 2 ([3] and [5]) .
There are absolute constants C 1 , C 2 > 0 such that for every H ⊂ Y X ,
[Equation]
and
[Equation]
AG-dimension and upper bounds for robust multiclass learning
We start by showing that finite graph dimension is not sufficient for robust multiclass learnability.
We then introduce the notion of corrupted hypotheses, presented by [7].
By computing the graphdimension of these hypotheses, which is defined as adversarial graph dimension, we can upper boundthe sample complexity of robust multiclass learning problem in both realizable and agnostic settings.
3.1 Finite graph dimension is not sufficient for robust multiclass learning
Theorem 2 shows that the finiteness of the graph dimension is a sufficient condition for multiclasslearnability in non-robust setting.
While in adversarial setting, for hypothesis classes with finite graphdimension, one cannot ensure the robust PAC learnability of H , even if d G ( H ) = 1 .
Theorem 3.
There exists a hypothesis class H with d G ( H )
= 1 and an adversary U such that H isnot robustly PAC learnable with respect to U .
To prove this theorem, We first present a lemma, which is generalized from Lemma 3 in [6].
Lemma 1.
Let m ∈ N .
Then, there exists a hypothesis class H⊂Y X such that for any learning rule
A X × Y ) ∗ → H , there exists a distribution D over X × Y such that:
There exists a function f ∗ ∈ H with R U ( f ∗ ; D ) = 0 .2.
With probability of at least 1 / 7 over the choice of S ∼ D m we have that R U
( A ( S ); D ) ≥ 1 / 8 .
The proof of this lemma is presented in Appendix.
We now proceed with the proof of Theorem 3.
Proof of Theorem 3: The argument follows closely a proof of an analogous result by [6] for binaryrobust learning, but generalizes the constructions and analyzes to match the definition of graphdimension in the multiclass setting.
For each m ∈ N , let X m =
{ x ( m ) 1 , . .
, x ( m ) 3 m } be a set thatcontains 3 m distinct points in X subject to ∀ x i , x j ∈ ∪ ∞ m =1 X m , if x i ̸
= x j , then U ( x i ) ∩U ( x j ) = ∅ .
And we construct H m on X m as follows.
For each b ∈ { 0 , 1 } 3 m , we construct a set Z b : Initialize Z b = ∅ , for each i ∈ [3 m ] , if b i = 1 then pick a point z ∈ U ( x ( m ) i ) such that z / ∈ Z b ′ for eachb ′ ̸ = b, and add it to Z b .
Let h b : X → Y be a hypothesis such that h b ( x ) =
if and only if x / ∈ Z band x / ∈ X m ′ for m ′ ̸ = m.
Furthermore, for fixed m ∈ N and x ∈ X m ′ s.t. m
′ ̸
= m , we need all h bmap x into a same label y x ( ̸ = 1) , where y x is consistent for all m. That is,h b ( x
[Equation]
=   ¬ 1 , x ∈ Z b , y x ( ̸ = 1) , ∃ m ′ ̸
= m, x ∈ X m ′ , 1 , otherwise, where ¬ 1 is some label that is not equal to 1 .
Then define
[Equation] h b : b ∈ { 0 , 1 } 3 m , 3 m (cid:88) i =
b i = m } .
Let H = ∪ ∞ m =1 H m .
We claim that d G ( H ) ≤ 1 .
That is, if pick two points x 1 , x 2 ∈ X such thatx 1 ̸ = x 2 , H cannot G-shatters { x 1 , x 2 } .
since for all h ∈ H and x ∈ X , there are at most two cases of h ( x ) , one of which is 1.
Hence the only f : { x 1 , x 2 } → S in Definition 4 that needs to beconsidered is f ( x 1 ) =
f ( x 2 )
= 1 .
There are six cases to consider:
(1) There exists m ∈ N such that z 1 , z 2 ∈ X m .
Then ∀ h b ∈
H m , x 1 , x 2 / ∈ Z b by the construction ofZ b .
Hence h b ( x 1 ) =
h b ( x 2 ) = 1 .
And ∀
h ∈ H m ′ for m ′ ̸
= m, we have h ( x 1 ) =
y x 1 ̸ = 1 , h ( x 2 ) =
y x 2 ̸ = 1 .
So we only obtain labelings (1 , 1) and ( ¬ 1 , ¬ 1) .(2)
There exists m ∈ N
such that x 1 , x 2 ∈ U
( X m ) \ X m .
There are two sub-cases to consider.
(i) ∃ bs.t.
h b ∈ H m and x 1 , x 2 ∈ Z b , then we have h b ( x 1 ) ̸
= 1 and h b ( x 2 ) ̸
= 1 .
For the remaining h ∈ Hwe have h ( x 1 )
=
h ( x 2 ) = 1 .
So we only obtain labelings (1 , 1) and ( ¬ 1 , ¬ 1) in this sub-case.
(ii)For the remaining sub-cases, labeling ( ¬ 1 , ¬ 1) cannot be obtained since x 1 , x 2 / ∈
X m for all m andthere is no b such that x 1 , x 2 are in Z b in the same time.(3) x 1 ∈ X m and x 2 ∈ X m ′ for m ̸
= m ′ .
In this case, we have h ( x 1 )
= 1 , h ( x 2 ) ̸
= 1 for h ∈ H m ,and h ( x 1 ) ̸
= 1 , h ( x 2 )
= 1 for h ∈ H m ′ .
For those predictors that are in neither H m nor H m ′ , wehave h ( x 1 ) ̸
= 1 , h ( x 2 ) ̸
= 1 .
Hence we cannot label both points x 1 and x 2 with (1 ,
x 1 ∈ X m and x 2 ∈ U ( X m ′ )
\ X m ′ for m ̸
= m ′ .
Observe that h ( x 1 ) =
if and only if h ∈ H m .
But for all h ∈ H m , h always labels x 2 with 1 .
Hence labeling (1 , ¬ 1) cannot be obtained.(5)
x 1 ∈ U ( X
m ) \ X m and x 2 ∈ U ( X m ′ )
\ X m ′ for m ̸
= m ′ .
Similar to the discussion above, for allh ∈ H , either h ( x 1 ) = 1 or h ( x 2 )
= 1 .
So we cannot obtain labeling ( ¬ 1 , ¬ 1) .
The remaining cases,
i
e. x , x / ∈ U (
X
m )
for all
m
∈
N .
In this case we can only obtain labeling(
1) since for all h ∈ H , we have
h
x = h ( x 2 ) =
The arguments above show that d G ( H ) ≤ 1 .
It remains to show H is not robustly PAC learnable.
By Lemma 1, it follows that for any learning rule A : ( X × Y ) ∗ → H and for any m ∈ N , thereexists a distribution D over X m × Y where there exists a predictor h ∗ ∈ H with R U
( h ∗ ; D )
= 0 ,
butwith probability at least 1 / 7 over S ∼ D m , R U
( A ( S ); D ) > 1 /
if A ( S ) ∈
H m .
Classifiers in H m ′where m ′ ̸
= m are also non-robust since they make mistakes on points in X m .
This concludes that
H is not robustly PAC learnable, which completes the proof.
3.2 Adversarial graph dimension
Consider a given hypothesis f : X → Y .
A labeled adversarial sample (˜ x, y ) is classified correctly if˜ x ∈ f − 1 ( y ) .
A labeled example ( x, y ) is classified correctly if U ( x ) ⊂ f − 1 ( y ) .
Following [7], let˜ Y =
Y ∪ {⊥} , where ⊥ is the special “always wrong” output, i.e. l ( y, ⊥ )
=
l ( ⊥ , y ) =
1 for everyy ∈ ˜ Y .
We define the mapping κ U : Y X
→ ˜ Y X :
[Equation]
The corrupted set of hypotheses induced by perturbation U is then defined by ˜ H
= { κ U ( f ) : f ∈ H} .
Lemma 2 (Lemma 2 in [7]) .
For any perturbation U and distribution D , R U ( f ; D )
=
R I ( κ U ( f ); D ) .
This lemma bridges the equivalence between robustly learning H and learning ˜ H without adversary,which enables us to use standard techniques to bound the sample complexity.
Intuitively, we canderive a sufficient condition by considering the graph dimension of ˜ H , which can be equivalentlyformalized as follows:
Definition 6 (Adversarial Graph Dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H adversarially G-shatters S if there exists an f : X → Y such that for every T ⊂ S, there is a g ∈ H such that
[Equation]
and
[Equation]
The adversarial graph dimension (AG-dimension) of H , denoted by d U G ( H ) , is the maximal cardinalityof a set that is adversarially G-shattered by H .
Proposition 1.
Let H ⊂
Y X be a hypothesis class and let ˜ H be the corrupted set of hypothesesinduced by perturbation U .
Then we have
[Equation]
The proof is presented in Appendix.
Intuitively, by combining the above proposition with Lemma 2 and Theorem 2, we can upper boundthe sample complexity of robust multiclass learning.
Formally:
Theorem 4.
There are absolute constants C > 0 such that for every H ⊂ Y X ,
[Equation]
[Equation]
ϵ (cid:19) , and m ag H , U ( ϵ, δ ) ≤ C (cid:18)
d U G ( H ) + ln( 1 δ
[Equation]
ϵ(cid:19) .
Proof.
Let H ⊂
Y X be a hypothesis class with d U G ( H ) =
d. For every f ∈ H , define ¯ f : X × Y →{ 0 , 1 } by setting ¯ f ( x, y )
=
if and only if f ( x ′ )
=
y for all x ′ ∈ U ( x ) .
Note that according to ourconstruction, we have
[Equation] [Equation]
In the realizable case, let ¯ H
= { ¯ f : f ∈ H} .
From the definition, we can see that VC ( ¯ H ) =
d U G ( H ) .
By Theorem 1, there exists C > 0 such that when we draw m > C ( dϵ ln( 1 ϵ )
+ 1 ϵ ln( 1 δ ))
samples
i.i.d.from some distribution D on X × Y , then with probability at least 1 − δ we have P ( x,y )
∼D [ ¯ f ( x, y ) ̸ =1] ≤ ϵ. Plugging 2 into this inequality, we complete the proof for the realizable case and it is similarfor the agnostic case.
AN-dimension and lower bounds for robust multiclass learning
In this section, we discuss the necessary conditions for robust multiclass learnability.
4.1 Finite Natarajan
dimension is not necessary for robust multiclass learning
We first show that the finiteness of ordinary Natarajan dimension is also not necessary for robustmulticlass learnability.
Theorem 5.
There exist H , U such that d N ( H ) =
∞ but H is robustly PAC learnable under U , i.e.m ag H , U ( ϵ, δ ) < ∞ .
The proof follows the construction from [6], which shows that finite VC-dimension is not necessaryfor robust binary learnability.
We leave the proof details to Appendix.
4.2 Adversarial Natarajan dimension
Following the idea of generalizing the graph dimension to the AG-dimension, for a given hypothesisclass H , one can consider the Natarajan dimension of its corrupted hypothesis class to lower boundthe sample complexity of robust learnability.
However, there are several natural but different waysin which the notion of N-shattering can be generalized: that is, we can either let f 2 that witnesses˜
H N-shattering S take the value “ ⊥ ” or not.
Both notions derive a necessary condition for robustlearnability.
In this work, we choose the stronger notion, i.e. we need the entire perturbation sets tobe N-shattered by H .
Definition 7 (Adversarial Natarajan Dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H adversarially N-shatters S if there exist f 1 , f 2 : X → Y such that ∀ y ∈ S, f 1 ( y ) ̸
= f 2 ( y ) , and for every T ⊂ S, there is a g ∈ H such that
[Equation]
and
[Equation]
The adversarial Natarajan dimension (AN-dimension) of H , denoted by d U N ( H ) , is the maximalcardinality of a set that is adversarially N-shattered by H .
Based on this definition, we present that:
Theorem 6.
There are absolute constants C > 0 such that for every H ⊂ Y X , m re H , U ( ϵ, δ ) ≥ C (cid:18)
d U N ( H ) + ln( 1 δ ) ϵ (cid:19) , and m ag H , U ( ϵ, δ ) ≥ C (cid:18)
d U N ( H ) + ln( 1 δ ) ϵ(cid:19) .
Proof.
Let H ⊂
Y X be a hypothesis class such that d U N ( H ) =
d, and let H d = { 0 , 1 } [ d ] be thefamily of all mappings from [ d ] to { 0 , 1 } .
We want to show that for every learning algorithm A for
H and every ϵ, δ ∈ (0 , 1) , there exists a learning algorithm
¯
A satisfying: if there exists a functionm re A , U : (0 , 1) 2 → N such that when A witnesses m > m re A , U ( ϵ, δ ) samples (denoted as S )
drawni.i.d from some distribution D , P [ R U
( A ( S ); D ) ≤ ϵ ] > 1 − δ.
Then when ¯ A witnesses m samples(denoted as S ′ ) drawn i.i.d from some distribution D ′ on [ d ] ×{ 0 , 1 } , P [ R I ( ¯ A ( S ′ ); D ′ ) ≤ ϵ ] > 1 − δholds.
By this claim, we have m re ¯ A , I ≤ m re A , U , thus m re H d , I ≤ m re H , U . Note that VC ( H d ) = d, so by
Theorem 1 we have m re H , U ( ϵ, δ ) ≥ C ( d +ln(1 /δ )
ϵ ) for some constant C > 0 (independent of H ), andsimilarly for the agnostic case.
We now prove the claim.
Let { s 1 , . . .
, s d } ⊂ X be a set and let f 0 , f 1 be functions that witnessthe adversarial N-shattering of H .
Given a sample set S ′ ≜ { ( x i , y i ) } mi =1 ⊂ [ d ] × { 0 , 1 } , let g =
A ( { s x i , f y i ( s x i ) } mi =1 ) .
Consider the learning algorithm ¯
A : ∪ ∞ n [-=
 ( Y-]=0 ([ d ] × { 0 , 1 } ) n → H d :¯ A ( S ′ )
( j ) = (cid:26) 1 , g ( s ′ j ) = f 1 ( s j ) , ∀ s ′ j ∈ U ( s j ) , 0 , otherwise.
Denote ¯ A ( S ′ ) by f.
By the definition of adversarial N-shattering, it can be derived that f ( j ) = 0 ifand only if g ( s ′ j ) =
f 2 ( s j ) , ∀ s ′ j ∈ U ( s j ) .
By the construction we have
[Equation]
where ˆ D ′ m is defined by P ˆ D ′ m
( X = s
j , Y = f
y ( s j ))
=
P D ′
( J = j, B = y ) .
The AG/AN-dimension of linear multiclass predictors
In this section, we calculate the AG-dimension and AN-dimension for linear multiclass predictorsunder bounded l p perturbation, motivated by the binary results of [7] and the non-adversarial results of[14].
Recall the definition of linear multiclass predictors.
Let Ψ : X × Y → R dk be a class-sensitivefeature mapping [17]:
[Equation]
] , where x ( i ) represents the i th coordinate of x.
The class of linear multiclass predictors is defined as
[Equation]
We cite that the graph dimension and Natarajan dimension of H Ψ is given by ( d − 1)( k − 1) ≤d N ( H Ψ )
≤ d G ( H Ψ ) ≤
O ( dk ln( dk ))
We then present the theorems, showing that the AGdimension and AN-dimension also satisfy this bound.
Theorem 7.
Let U ( x ) =
{ z ∈ X : ∥
z − x ∥ p ≤ r } for some r ∈ R + and p ∈ R + ∪ {∞} , and let H Ψ be as defined in 3.
Then, the AN-dimension of H Ψ satisfies
d U N ( H Ψ ) =
d N ( H Ψ ) .
Proof.
Since adversarial N-shattering implies N-shattering, we have d U N ( H Ψ )
≤ d N ( H Ψ ) .
To provethe inverse inequality, it should be noted that the classifiers in H Ψ are linear, hence if H Ψ N-shatters
S , it also N-shatters the set with some x ∈ S replaced by ∀ x ′ ∈
l x , where l x is the half-line whichterminates at 0 and goes through x. For those x ∈ S that cannot be adversarially N-shattered by H Ψ ,we can always find some x ′ ∈ l x such that U ( x ′ ) does not cross the decision boundary of g T thatwitnesses the N-shattering for all T ∈ S.
Corollary 1.
Under the conditions of Theorem 7, ( d − 1)( k − 1) ≤ d U N ( H Ψ ) ≤ dk.
Proof.
Combine Theorem 7 and ( d − 1)( k − 1) ≤ d N ( H Ψ ) ≤ dk (Corollary 29.8 in [18]).
Theorem 8.
Let U ( x ) =
{ z ∈ X : ∥
z − x ∥ p ≤ r } for some r ∈ R + and p ∈ R + ∪ {∞} , and let
H Ψ be as defined in 3.
Then, the AG-dimension of H Ψ satisfies
[Equation]
To prove this result, we first present a lemma, which can be viewed as the generalization of Sauer’s
Lemma [19] in the adversarial setting.
We first define the restriction of a corrupted class.
Let ˜ H bethe corrupted hypothesis class induced by U .
Let S = { x 1 , . . .
, x m } ⊂ X .
Fix a f ∈ Y S , for eachh ∈ H , define h f : S → { 0 , 1 } such that h f ( x i )
=
if and only if κ U ( h )( x i ) =
f
( x i ) , i ∈ [ m ] .
Define H f
= { h f : h ∈ H} .
The restriction of ˜ H to S is defined by:˜
H| S =
H f ∗ | S , where f ∗ = arg max
[Equation]
Lemma 3.
Let U be an adversary and H be a hypothesis class with d U G ( H ) ≤
d G < ∞ .
Let ˜ H be the corrupted hypothesis class induced by U .
Let S = { x 1 , . . .
, x m } ⊂ X .
Then, for all m > d G
+we have
[Equation]
Proof.
By Sauer’s Lemma [19] and the fact that VC( H f ) ≤ d U G ( H ) for all f ∈ Y S .
We now prove Theorem 8.
The construction idea follows the proof of non-adversarial results in [14].
Proof of Theorem 8: The lower bound follows from Theorem 7 and the fact that d U G ( H ) ≥
d U N ( H )for all H .
To upper bound d G :=
d U G ( H ) , let S =
( x 1 , . .
, x d G ) ∈ R
d be a set which is adversarially
G-shattered by H Ψ , and let f : S → Y be a function that witnesses the shattering.
For every( i, j ) ∈ [ d G ] × [ k ] , define z i,j = Ψ( x i , f ( x i ))
− Ψ( x i , j ) .
Denote Z = { z i,j | ( i, j ) ∈
[ d G ] × [ k ] } .
We now show that there exists a injective mapping from subsets of S to W dkf | Z , where W dk is theclass of halfspace classifiers over R dk .
For each T ⊂ S, by the definition of adversarial G-shattering, ∃ h T ∈ H Ψ such that
[Equation]
Let w T ∈ R
dk be the vector defines h T , 4 implies
[Equation]
and
[Equation]
It can be easily derived from 5 that ∀ i ∈ I S \ T , ∃ j ∈ [ k ] ,
∃ z ′ ∈ V ( z i,j ) s.t.
⟨ w T , z ′ ⟩ < 0 , where V is the p -norm ball in R dk with radius r.
And from 5 we can assume w.l.o.g that ∀ i ∈ I T , ∀ j ∈ [ k ] , ∀ z ′ ∈
V ( z i,j ) , ⟨ w T , z ′ ⟩ ≥ 0 .
This is because if there exists some z ′ ∈ V ( z ) s.t. ⟨ w T , z ′ ⟩ < 0 , notice that z ′ =
V ( c ′ )
+ z i,j for some bounded c ′ , we can replace x i by some x ′ i without destroying the adversarial
G-shattering (see the trick used in proof of Theorem 7), to ensure ⟨ w T , Ψ( x ′ i , f ( x i ))
− Ψ( x ′ i , j ) ⟩ > ⟨ w T , c ⟩ , ∀ c ∈ V (0) .
Consequently, for every T ∈ S we obtain a label pattern P T : [ d G ] ×
[ k ] → { 0 , 1 } :
[Equation]
It satisfies that ∀ i ∈ I T , j ∈ [ k ] , P S ( i, j )
= 1 and ∀ i ∈ I S \ T , ∃ j ∈ [ k ] , P T ( i, j )
= 0 , which implies
P T ̸ = P T ′ for T ̸ = T ′ .
And by definition P T is connected with a unique element in W dkf | Z , whichcompletes the proof of our claim.
Note that the AG-dimension of halfspace classifier class with respect to U coincides with the definitionof adversarial VC-dimension in [7].
Thus it satisfies that d U G ( W dk )
=
dk
+ 1 by Theorem 2 in [7].
Consequently, we have
[Equation]
where we have used Lemma 3 in step (i).
We conclude that d G ≤ O ( dk ln dk ) .
AG-dimension and AN-dimension are not equivalent
In previous sections, we have shown that the finiteness of the AN-dimension is a necessary conditionfor robust learnability, and the finiteness of the AG-dimension is a sufficient condition for robustlearnability.
Furthermore, we analyze these definitions on the linear multiclass predictor class,finding that the AG-dimension (AN-dimension) matches the order of the graph dimension (Natarajandimension) for this class.
The following question therefore naturally arises: are these two quantitiesequivalent?
In [5], it is proven that for every hypotheses class H ⊂
Y X ,
[Equation]
That is, in standard learning setting, if k ≤ ∞ then the definition of the Natarajan dimension and thegraph dimension are indeed equivalent, hence both the Natarajan dimension and graph dimensioncan characterize the standard learnability of a hypothesis class.
However, in the adversarial setting( U ̸ = I ), the generalized definitions derived above cannot bound each other.
To show this, we claimthat finite AN-dimension is not sufficient for robust learnability.
Theorem 9.
There exist H , U such that d U N ( H ) =
but H is not robustly PAC learnable under U ,i.e. m ag H ,
U ( ϵ, δ ) = ∞ .
Proof.
Let U be the 2 -norm ball: U ( x ) =
{ z ∈ X : ∥
z − x ∥ 2 ≤ r } for all x ∈ X and some r > 0 .
We construct H as follow.
Pick a sequence { x i } ∞ i =1 ⊂ X such that for any i ̸
= j, U ( x i ) ∩U ( x j ) = ∅ .
Let f : X → Y be a function satisfying: there exists z
i ∈ U ( x i ) \ x i such that f ( z i ) ̸
= f
( x i ) for eachi ∈ N .
For each sequence b = { b i } ∞ i =1 ⊂ { 0 , 1 } , define h b : X → Y :h [Equation] ( x ) =
  f ( x i ) , x ∈ U ( x i )
\ z i , i ∈ N , f ( x i ) , x = z i and b i = 1 , f ( z i ) , x = z i and b i = 0 , 1 , otherwise.
Let H =
{ h b : b is a sequence in { 0 , 1 }} .
Using the same approach employed in the proof of Lemma1, for any learning rule A : ( X × Y ) ∗ → H , there exists a distribution D over X × Y such that thereexists a function f ∗ ∈ H with R U ( f ∗ ; D ) = 0 , and with probability of at least 1 / 7 over the choiceof S ∼ D
m we have that R U
( A ( S ); D ) ≥ 1 / 8 , by considering the distribution family D containingdistributions that are uniform over 2 m points in { ( x 1 , f ( x 1 )) , . .
, ( x 3 m , f ( x 3 m )) } .
We completethe proof by noting that d U N ( H )
= 0 , since there is no single r -ball that is adversarially N-shatteredby H .
Discussion
In this work, we construct the concepts of AG-dimension and AN-dimension to upper bound andlower bound the sample complexity of robust multiclass learning, respectively. However, it is not yetclear whether finite AG-dimension is also necessary for robust multiclass learning.
Establishing acomplexity measure that characterizes robust learnability still remains an open question.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using
/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...(
a) Did you include the full text of instructions given to participants and screenshots, ifapplicable
[N/A
(b) Did you describe any potential participant risks , with links to Institutional Review
Board ( IRB) approvals , if applicable?
[N/A
(c)
Did
you include the estimated hourly wage paid to participants and
the
total
amountspent on participant compensation?
N/A