On Robust Multiclass Learnability
Anonymous Author(s)
AffiliationAddress email
Abstract
This work analyzes the robust learning problem in the multiclass setting.
Under theframework of Probably Approximately Correct (PAC) learning, we first show thatthe graph dimension and the Natarajan dimension, which characterize the standardmulticlass learnability, are no longer applicable in robust learning problem.
We thengeneralize these notions to the robust learning setting, denoted as the adversarialgraph dimension (AG-dimension) and the adversarial Natarajan dimension (ANdimension).
Upper and lower bounds of the sample complexity of robust multiclasslearning are rigorously derived based on the AG-dimension and AN-dimension,respectively.
Moreover, we calculate the AG-dimension and AN-dimension of theclass of linear multiclass predictors, and show that the graph (Natarajan) dimensionis of the same order as the AG(AN)-dimension.
Finally, we prove that the AGdimension and AN-dimension are not equivalent.
Introduction
Learning models that are robust to adversarial perturbations has attracted significant research attentionin recent years.
Notably, prior work on this subject has largely studied the theory of robust learnabilityin the context of binary supervised learning.
However, many important problems require classificationinto a great number of target classes.
For example, in image object recognition and language modelsbuilding[1, 2], the number of classes scales as the number of possible objects or the dictionary sizerespectively.
It is thus of both practical and theoretical interest to generalize the robust binary learningtheory to the robust multiclass setting.
In the standard multiclass learning setting, the finiteness of the Natarajan dimension or graphdimension [3] (both of which are generalized from the VC-dimension [4]) is necessary and sufficientto ensure learnability [5].
Recent work has shown that the finiteness of the VC-dimension is neithersufficient nor necessary for (proper) robust binary classification [6].
Inspired by [6], this paperdemonstrates that the Natarajan/graph dimension can no longer characterize robust learnability in themulticlass problem.
Meanwhile, some questions naturally arise: In multiclass setting, how can robustlearning be ensured?
And what is necessary for robust learning?
To answer these questions, we generalize the corrupted hypothesis classes [7], which arise fromstandard hypothesis classes in the binary setting in the presence of an adversary, and use them toreduce the robust learning problem to a non-robust one.
By considering the graph dimension and
Natarajan dimension of the corrupted hypothesis classes—which are defined as the adversarial graphdimension (AG-dimension) and adversarial Natarajan dimension (AN-dimension) respectively—wederive the upper and lower bounds of sample complexity of the robust multiclass learning problem.
We then analyze the AG-dimension and the AN-dimension of linear multiclass predictor class,showing that the AG(AN)-dimension and the graph (Natarajan) dimension are of the same order.
Since it is proven that the graph dimension and the Natarajan dimension are equivalent when thetarget class is finite [5], is it possible to use the AG-dimension and AN-dimension to bound eachother?
Our work suggests that the equivalent relationship does not hold, which is illustrated byconstructing a hypothesis class with finite AN-dimension but not robustly learnable.
1.1 Overview of techniques
Our Theorems 3 and 5, stating that the finiteness of graph dimension is not sufficient and thefiniteness of Natarajan dimension is not necessary for robust learning, respectively, are demonstratedby constructing counter examples.
The arguments follow closely the proofs of analogous results butgeneralize the constructions and analyzes to match the definition of graph/Natarajan dimension in themulticlass setting. Our main results, namely Theorems 4 and 6, are high probability bounds of sample complexity that aresufficient/necessary for robust PAC learning, respectively. Compared to standard PAC learning, robust PAC learning is mainly different in the definition of loss function, which considers the maximumof 0-1 loss over a perturbation set.
Thus to prove these bounds, we need to design a way that takesthe disturbance set into account to control the adversarial risk in a high probability.
[7] presents thenotion of corrupted hypothesis class and use it to derive an upper bound for binary robust learning byconsidering the VC-dimension of the corrupted hypothesis class.
This approach shed some light onour work.
However, generalizing to multiclass setting in this way faces some difficulties, since thereare quite a few definitions that can be generalized to multiclass setting, e.g. corrupted hypothesis class,shattering coefficient and VC-dimension in the context of adversarial setting.
To make it more natural,we define the corrupted hypothesis class in multiclass setting, which does not involve fixed labelpatterns, and then study the graph/Natarajan dimension of a corrupted hypothesis class.
For clarity,we analytically abstract these definitions as the AG/AN-dimension of hypothesis class, making itget rid of the notion of corrupted hypothesis class.
There are some beneficial efforts by defining inthis way: one is to intuitively correspond the graph/Natarajan dimension to the AG/AN-dimension,as well as the bounds derived by these notions.
Another benefit is to make it possible to transfercommon multiclass proof techniques to adversarial setting, namely constructing auxiliary binaryclassification problems and using them to control the risk we focus on.
Theorems 7 and 8 give bounds of AG/AN-dimension for linear multiclass predictors H Ψ .
To showthese results, we need to prove two non-trivial bounds, namely the upper bound of AG-dimensionand the lower bound of AN-dimension.
For the upper bound, we gain our construction idea from[8], which calculates the graph dimension of multiclass SVM.
Yet their proof make use of Sauer’s
Lemma [9], which is only applicable for standard setting.
To fill this gap, we present Lemma 2the generalization of Sauer’s Lemma in adversarial setting - showing that the growth function in thecontext of adversarial and multiclass setting, which is defined very finely, also increases polynomiallyas the number of samples m > d U G + 1 , where d U G is the AG-dimension of some hypothesis class.
Asfor the lower bound, we claim that for linear classifiers, its AN-dimension equals to its Natarajandimension.
We demonstrate this by geometric analysis, which makes use of the linearity of predictorsin H Ψ .
We prove Theorem 9 by constructing a novel counter example to show the finiteness of ANdimension is not sufficient for robust multiclass learning, which reveals that the AG-dimensionand AN-dimension are not equivalent.
1.2 Related work
[10, 11, 12, 13] focus on the robust binary classification problem.
[3] characterize the multiclasslearnability in non-robust setting.
[14, 5, 15] define a large family of notions of dimensions, all ofwhich generalize the VC-dimension and may be used to estimate the sample complexity of multiclassclassification.
[8] calculate the graph dimension and the Natarajan dimension of linear multiclasspredictor class.
[16] upper bounds the adversarial risk of linear multiclass classifiers based on Rademacher complexity, which cannot directly derive bounds of sample complexity of general robust multiclass learning problems.
The theory on robust multiclass learnability for general hypothesisclass is less explored.
This work takes some steps towards solving this problem.
Preliminaries
This section introduces some basic notations and concepts used throughout the present study.
In the remainder of this article, R , N , R + and R d represent the sets of real numbers, natural numbers,non-negative real numbers and d -dimensional vectors over R , respectively.
We denote the set{ 1 , . . . , n } (for n ∈ N ) by [ n ] .
If A and B are sets, we use B A to denote the collection of allmappings from A to B and 2 A to denote the power set of A, that is the collection of all subsets of A.
Let C
= { c 1 , . .
, c m } ⊂ X and C ′
⊂ C be a subset of C,
then I C ′ ∈ [ m ] denotes the index set of
C ′ .
Let H be a class of functions defined in X , then H| C represents the restriction of H to C, that is
H C
= { ( h ( c 1 ) , . . .
, h ( c m )) :
h ∈ H} .
We denote the indicator function by 1 ( event ) , that is 1 if anevent happens and 0 otherwise.
Finally, ∥ · ∥ p represents the ℓ p norm.
The robust learning problem is formalized as follows.
Let X =
R d be the instance space and
Y = { 1 , . . .
, k } be the label space.
For an unknown distribution D over X × Y 1 and a hypothesisclass H ⊂ Y X , the goal of robust learning is to find a function f ∈ H , based on n labeled trainingsamples S
=
( x i , y i ) } ni =1 drawn independent and identically distributed (i.i.d.) from D , such thatunder a small perturbation U : X → 2 X , the adversarial risk
[Equation]
is minimal for the 0-1 loss l (ˆ y, y ) ≜
1 (ˆ y ̸ = y ) .
The perturbation U ( x ) is required to be nonempty, so some choice of x ′ is always available.
Onechoice for the perturbation is the p -norm ball ( p ≥ 1 ) with a small radius r , i.e. U ( x ) =
{ z ∈ X :∥
z − x ∥ p ≤ r } .
Selecting r = 0 gives the identity perturbation : I ( x ) ≜ { x } .
Note that in this case,the problem is reduced to a standard learning problem, and R I ( f ; D ) is called the standard risk of fover D .
The mechanism of finding a function is called a learning algorithm .
In this paper, we focus on properlearning algorithms, that is, learning algorithms will always pick a function from H .
Formally, alearning algorithm is a function A : ∪ ∞ n =
( X → Y ) n → H .
A common choice of robust learningalgorithm is to learn through adversarial empirical risk minimization or AERM for short:
[Equation]
where ˆ D n is the empirical distribution generated by S, i.e. for ( x, y ) ∼ ˆ D n , ( x, y ) is equal to ( x i , y i )with probability 1 /n for each i ∈ { 1 , . .
. , n } .
Learning algorithms like AERM do not always produce a function that achieves the optimal adversarial risk, because a training set consisting of finite samples is not always sufficiently informativeto represent D .
Thus, we introduce the definition of robust probably approximately correct (PAC)learnability, in the realizable and agnostic setting [7, 6]:
Definition 1 (Agnostic Robust PAC Learnability) .
Under perturbation U , a hypothesis class H isrobustly PAC learnable in the agnostic setting if there is a function m ag H , U : (0 , 1) 2 → N and alearning algorithm A : ∪ ∞ n =
( X × Y ) n → H with the following property.
For every ϵ, δ ∈ (0 , 1)and every distribution D over X × Y , let S = { ( x i , y i ) } ni
=1 be n samples i.i.d generated by D s.t.n > m ag H , U ( ϵ, δ ) .
Then with probability at least 1 − δ ,
[Equation]
The function m ag H , U is defined as the sample complexity of learning H under perturbation U in theagnostic setting.
Definition 2 (Realizable Robust PAC Learnability) .
Under perturbation U , a hypothesis class His robustly PAC learnable in the realizable setting if there is a function m re H , U : (0 , 1) 2 → N and alearning algorithm A : ∪ ∞ n =
( X × Y ) n → H with the following property.
For every ϵ, δ ∈ (0 , 1)and every distribution D over X × Y , let S = { ( x i , y i ) } ni
=1 be n samples i.i.d generated by D s.t.n > m re H , U ( ϵ, δ ) .
Then with probability at least 1 − δ ,
R U ( A ( S ); D ) ≤ ϵ.
The function m re H , U is defined as the sample complexity of learning H under perturbation U in therealizable setting.
These definitions agree with the standard PAC learnability
when U =
I , which has been widelystudied in both binary ( k = 2 ) and multiclass ( k > 2 ) setting.
We recall the known result regardingthe sample complexity of binary learning.
Recall the definition of the Vapnik-Chervonenkis dimension(VC-dimension) [4]:
Definition 3 (VC-dimension) .
Let H ⊂ { 0 , 1 } X be a hypothesis class and let S ⊂ X .
We say that Sis shattered by H if H| S
= { 0 , 1 } S .
The VC-dimension of H , denoted by VC ( H ) , is the maximalcardinality of a set that is shattered by H .
The following theorem, based on the VC-dimension, characterizes the sample complexity of learningbinary hypothesis classes.
Theorem 1 ([4] and [17]) .
There are absolute constants C 1 , C 2 > 0 such that for every H ⊂ { 0 , 1 } X , [Equation] ϵ
[Equation]
, and
[Equation]
It is natural to seek a generalization of the VC-dimension to hypothesis classes of non-binary functions.
We recall two generalizations, both introduced by [3]:
Definition 4 (Graph dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H
G-shatters S if there exists an f : S → Y such that for every T ⊂ S, there is a g ∈ H such that [Equation] and ∀ x ∈ S \ T, g ( x [Equation]
f
[Equation] x [Equation]
The graph dimension of H , denoted by d G ( H ) , is the maximal cardinality of a set that is G-shatteredby H .
Definition 5 (Natarajan dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H N-shatters S if there exists f 1 , f 2 : S
→ Y such that ∀ x ∈ S, f 1 ( x ) ̸
= f 2 ( x ) , and for every
T ⊂ S, there is a g ∈ H such that
[Equation] The Natarajan dimension of H , denoted by d N ( H ) , is the maximal cardinality of a set that is
N-shattered by H .
Both the graph dimension and the Natarajan dimension coincide with the VC-dimension for k = 2 .
Based on these notions, the following theorem provides the upper and lower bounds of samplecomplexity of standard multiclass learning:
Theorem 2 ([3] and [5]) .
There are absolute constants C 1 , C 2 > 0 such that for every H ⊂ Y X ,
[Equation]
C(cid:18) d G ( H ) ln 1 ϵ + ln 1 δ ϵ (cid:19) , and C(cid:18)
[Equation]
N ( H )
[Equation]
1 δ ϵ(cid:19)
[Equation]
H , I
[Equation]
ϵ, δ )
[Equation]
(cid:18) d G ( H )
[Equation]
ln 1 δ ϵ(cid:19) .
AG-dimension and upper bounds for robust multiclass learning
We start by showing that finite graph dimension is not sufficient for robust multiclass learnability.
We then introduce the notion of corrupted hypotheses, presented by [7].
By computing the graphdimension of these hypotheses, which is defined as adversarial graph dimension, we can upper boundthe sample complexity of robust multiclass learning problem in both realizable and agnostic settings.
3.1 Finite graph dimension is not sufficient for robust multiclass learning
Theorem 2 shows that the finiteness of the graph dimension is a sufficient condition for multiclasslearnability in non-robust setting.
While in adversarial setting, for hypothesis classes with finite graphdimension, one cannot ensure the robust PAC learnability of H , even if d G ( H ) = 1 .
Theorem 3.
There exists a hypothesis class H with d G ( H )
= 1 and an adversary U such that H isnot robustly PAC learnable with respect to U .
[-Let h b : X → Y be a hypothesis such that h b ( x ) = 
-]
The argument follows closely a proof of an analogous result by [6] for binary robust learning.
[-Observe that h ( x 1 ) = 
-]
Theproof is presented in Appendix.
[-( A ( S ); D ) > 1 / 
-]
3.2 Adversarial graph dimension
Consider a given hypothesis f : X → Y .
A labeled adversarial sample (˜ x, y ) is classified correctly if˜ x ∈ f − 1 ( y ) .
A labeled example ( x, y ) is classified correctly if U ( x ) ⊂ f − 1 ( y ) .
Following [7], let˜ Y =
Y ∪ {⊥} , where ⊥ is the special “always wrong” output, i.e. l ( y, ⊥ )
=
l ( ⊥ , y ) =
1 for everyy ∈ ˜ Y .
We define the mapping κ U : Y X
→ ˜ Y X :
[Equation]
The corrupted set of hypotheses induced by perturbation U is then defined by ˜ H
= { κ U ( f ) : f ∈ H} .
Lemma 1 (Lemma 2 in [7]) .
For any perturbation U and distribution D , R U ( f ; D
[Equation]
f [Equation] D ) .
This lemma bridges the equivalence between robustly learning H and learning ˜ H without adversary,which enables us to use standard techniques to bound the sample complexity.
Intuitively, we canderive a sufficient condition by considering the graph dimension of ˜ H , which can be equivalentlyformalized as follows:
Definition 6 (Adversarial Graph Dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H adversarially G-shatters S if there exists an f : X → Y such that for every T ⊂ S, there is a g ∈ H such that
[Equation]
and
[Equation]
The adversarial graph dimension (AG-dimension) of H , denoted by d U G ( H ) , is the maximal cardinalityof a set that is adversarially G-shattered by H .
Proposition 1.
Let H ⊂
Y X be a hypothesis class and let ˜ H be the corrupted set of hypothesesinduced by perturbation U .
Then we have
[Equation]
The proof is presented in Appendix.
Intuitively, by combining the above proposition with Lemma 1 and Theorem 2, we can upper boundthe sample complexity of robust multiclass learning.
Formally:
Theorem 4.
There are absolute constants C > 0 such that for every H ⊂ Y X , m re H , U ( ϵ, δ ) ≤ C (cid:18) [Equation] 1 ϵ
[Equation]
ln( 1 δ
[Equation]
ϵ (cid:19) , and m ag H , U ( ϵ, δ ) ≤ C (cid:18)
[Equation]
G ( H [Equation] ( 1 δ
[Equation]
ϵ [Equation] (cid:19) .
Proof.
Let H ⊂
Y X be a hypothesis class with d U G ( H ) =
d. For every f ∈ H , define ¯ f : X × Y →{ 0 , 1 } by setting ¯ f ( x, y )
=
if and only if f ( x ′ )
=
y for all x ′ ∈ U ( x ) .
Note that according to ourconstruction, we have
[Equation]
In the realizable case, let ¯ H
= { ¯ f : f ∈ H} .
From the definition, we can see that VC ( ¯ H ) =
d U G ( H ) .
By Theorem 1, there exists C > 0 such that when we draw m > C ( dϵ ln( 1 ϵ )
+ 1 ϵ ln( 1 δ ))
samples
i.i.d.from some distribution D on X × Y , then with probability at least 1 − δ we have P ( x,y )
∼D [ ¯ f ( x, y ) ̸ =1] ≤ ϵ. Plugging 2 into this inequality, we complete the proof for the realizable case and it is similarfor the agnostic case.
AN-dimension and lower bounds for robust multiclass learning
In this section, we discuss the necessary conditions for robust multiclass learnability.
4.1 Finite Natarajan dimension is not necessary for robust multiclass learning
We first show that the finiteness of ordinary Natarajan dimension is also not necessary for robustmulticlass learnability.
Theorem 5.
There exist H , U such that d N ( H ) =
∞ but H is robustly PAC learnable under U , i.e.m ag H , U ( ϵ, δ ) < ∞ .
The proof follows the construction from [6], which shows that finite VC-dimension is not necessaryfor robust binary learnability.
We leave the proof details to Appendix.
4.2 Adversarial Natarajan dimension
Following the idea of generalizing the graph dimension to the AG-dimension, for a given hypothesisclass H , one can consider the Natarajan dimension of its corrupted hypothesis class to lower boundthe sample complexity of robust learnability.
However, there are several natural but different waysin which the notion of N-shattering can be generalized: that is, we can either let f 2 that
witnesses˜
H N-shattering S take the value “ ⊥ ” or not.
Both notions derive a necessary condition for robustlearnability.
In this work, we choose the stronger notion, i.e. we need the entire perturbation sets tobe N-shattered by H .
Definition 7 (Adversarial Natarajan Dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H adversarially N-shatters S if there exist f 1 , f 2 : X → Y such that ∀ y ∈ S, f 1 ( y ) ̸
= f 2 ( y ) , and for every T ⊂ S, there is a g ∈ H such that
[Equation]
and
[Equation]
The adversarial Natarajan dimension (AN-dimension) of H , denoted by d U N ( H ) , is the maximalcardinality of a set that is adversarially N-shattered by H .
Based on this definition, we present that:
Theorem 6.
There are absolute constants C > 0 such that for every H ⊂ Y X , m re H , U [Equation] ϵ, δ [Equation]
≥
[Equation] and m [Equation]
Proof.
Let H ⊂
Y X be a hypothesis class such that d U N ( H ) =
d, and let H d = { 0 , 1 } [ d ] be thefamily of all mappings from [ d ] to { 0 , 1 } .
We want to show that for every learning algorithm A for
H and every ϵ, δ ∈ (0 , 1) , there exists a learning algorithm
¯
A satisfying: if there exists a functionm re A , U : (0 , 1) 2 → N such that when A witnesses m > m re A , U ( ϵ, δ ) samples (denoted as S )
drawni.i.d from some distribution D , P [ R U
( A ( S ); D ) ≤ ϵ ] > 1 − δ.
Then when ¯ A witnesses m samples(denoted as S ′ ) drawn i.i.d from some distribution D ′ on [ d ] ×{ 0 , 1 } , P [ R I ( ¯ A ( S ′ ); D ′ ) ≤ ϵ ] > 1 − δholds.
By this claim, we have m re ¯ A , I ≤ m re A , U , thus m re H d , I ≤ m re H , U .
Note that VC ( H d ) =
d, so by
Theorem 1 we have m re H , U ( ϵ, δ ) ≥ C ( d +ln(1 /δ )
ϵ ) for some constant C > 0 (independent of H ), andsimilarly for the agnostic case.
We now prove the claim.
Let { s 1 , . . .
, s d } ⊂ X be a set and let f 0 , f 1 be functions that witnessthe adversarial N-shattering of H .
Given a sample set S ′ ≜ { ( x i , y i ) } mi =1 ⊂ [ d ] × { 0 , 1 } , let g =
A ( { s x i , f y i ( s x i ) } mi =1 ) .
Consider the learning algorithm ¯
A : ∪ ∞ n =0 ([ d ] × { 0 , 1 } ) n → H d :¯ A ( S ′ )
( j ) =
(cid:26) 1 , g ( s ′ j ) =
f 1 ( s j ) , ∀ s ′ j ∈ U ( s j ) , 0 , otherwise.
Denote ¯ A ( S ′ ) by f.
By the definition of adversarial N-shattering, it can be derived that f ( j ) = 0 ifand only if g ( s ′ j ) =
f 2 ( s j ) , ∀ s ′ j ∈ U ( s j ) .
By the construction we have
E ( j,y ) ∼D ′ [ 1 ( f ( j ) ̸ = y )]
= E ( j,y ) ∼D ′ [ sup s ′ j ∈U ( s j ) 1 ( g ( s ′ j ) ̸ = f y ( s j ))]
= E ( x,y ) ∼ ˆ D ′ m R U ( g ; ˆ D ′ m ) , where ˆ D ′ m is defined by P ˆ D ′ m
( X = s
j , Y = f
y ( s j ))
=
P D ′
( J = j, B = y ) .
The AG/AN-dimension of linear multiclass predictors
In this section, we calculate the AG-dimension and AN-dimension for linear multiclass predictorsunder bounded l p perturbation, motivated by the binary results of [7] and the non-adversarial resultsof [8].
Recall the definition of linear multiclass predictors.
Let Ψ : X ×Y → R dk be a class-sensitivefeature mapping [18]:
[Equation]
where x ( i ) represents the i th coordinate of x.
The class of linear multiclass predictors is defined as
[Equation]
We cite that the graph dimension and Natarajan dimension of H Ψ is given by ( d − 1)( k − 1) ≤d N ( H Ψ )
≤ d G ( H Ψ ) ≤
O ( dk ln( dk ))
We then present the theorems, showing that the AGdimension and AN-dimension also satisfy this bound.
We start by calculating the AN-dimension of H Ψ .
Specifically, one can show that the AN-dimensionof H Ψ equals to the standard Natarajan dimension of H Ψ .
Theorem 7.
Let U ( x ) =
{ z ∈ X : ∥
z − x ∥ p ≤ r } for some r ∈ R + and p ∈ R + ∪ {∞} , and let
H Ψ be as defined in 3.
Then, the AN-dimension of H Ψ satisfies
d U N ( H Ψ ) =
d N ( H Ψ ) .
Proof.
Since adversarial N-shattering implies N-shattering, we have d U N ( H Ψ )
≤ d N ( H Ψ ) .
To provethe inverse inequality, it should be noted that the classifiers in H Ψ are linear, hence if H Ψ N-shatters
S , it also N-shatters the set with some x ∈ S replaced by ∀ x ′ ∈
l x , where l x is the half-line whichterminates at 0 and goes through x. For those x ∈ S that cannot be adversarially N-shattered by H Ψ ,we can always find some x ′ ∈ l x such that U ( x ′ ) does not cross the decision boundary of g T thatwitnesses the N-shattering for all T ∈ S. Note that the Natarajan dimension of H Ψ satisfies ( d − 1)( k − 1) ≤ d N ( H Ψ ) ≤ dk (Corollary 29.8in [19]). Combine this result with Theorem 7, we have:
Corollary 1.
Under the conditions of Theorem 7, ( d − 1)( k − 1) ≤ d U N ( H Ψ ) ≤ dk.
Next, we present the AG-dimension of H Ψ :
Theorem 8.
Let U ( x ) =
{ z ∈ X : ∥
z − x ∥ p ≤ r } for some r ∈ R + and p ∈ R + ∪ {∞} , and let
H Ψ be as defined in 3.
Then, the AG-dimension of H Ψ satisfies
[Equation]
To prove this result, we first present a lemma, which can be viewed as the generalization of Sauer’s
Lemma [9] in the adversarial setting.
We first define the restriction of a corrupted class.
Let ˜ H bethe corrupted hypothesis class induced by U .
Let S = { x 1 , . . .
, x m } ⊂ X .
Fix a f ∈ Y S , for eachh ∈ H , define h f : S → { 0 , 1 } such that h f ( x i )
=
if and only if κ U ( h )( x i ) =
f
( x i ) , i ∈ [ m ] .
Define H f
= { h f : h ∈ H} .
The restriction of ˜ H to S is defined by:
[Equation]
where f ∗ = arg max f ∈Y S
[Equation]
Lemma 2.
Let U be an adversary and H be a hypothesis class with d U G ( H ) ≤
d G < ∞ .
Let ˜ H be the corrupted hypothesis class induced by U .
Let S = { x 1 , . . .
, x m } ⊂ X .
Then, for all m > d G
+we have
[Equation]
Proof.
By Sauer’s Lemma [9] and the fact that VC( H f ) ≤ d U G ( H ) for all f ∈ Y S .
We now prove Theorem 8.
The construction idea follows the proof of the non-adversarial results in[8].
Proof of Theorem 8: The lower bound follows from Theorem 7 and the fact that d U G ( H ) ≥
d U N ( H )for all H .
To upper bound d G :=
d U G ( H ) , let S =
( x 1 , . .
, x d G ) ∈ R
d be a set which is adversarially
G-shattered by H Ψ , and let f : S → Y be a function that witnesses the shattering.
For every( i, j ) ∈ [ d G ] × [ k ] , define z i,j = Ψ( x i , f ( x i ))
− Ψ( x i , j ) .
Denote Z = { z i,j | ( i, j ) ∈
[ d G ] × [ k ] } .
We now show that there exists a injective mapping from subsets of S to W dkf | Z , where W dk is theclass of halfspace classifiers over R dk .
For each T ⊂ S, by the definition of adversarial G-shattering, ∃ h T ∈ H Ψ such that
[Equation]
Let w T ∈ R
dk be the vector defines h T , 4 implies
[Equation]
and
[Equation]
It can be easily derived from 5 that ∀ i ∈ I S \ T , ∃ j ∈ [ k ] ,
∃ z ′ ∈ V ( z i,j ) s.t.
⟨ w T , z ′ ⟩ < 0 , where V is the p -norm ball in R dk with radius r.
And from 5 we can assume w.l.o.g that ∀ i ∈ I T , ∀ j ∈ [ k ] , ∀ z ′ ∈
V ( z i,j ) , ⟨ w T , z ′ ⟩ ≥ 0 .
This is because if there exists some z ′ ∈ V ( z ) s.t. ⟨ w T , z ′ ⟩ < 0 , notice that z ′ =
V ( c ′ )
+ z i,j for some bounded c ′ , we can replace x i by some x ′ i without destroying the adversarial
G-shattering (see the trick used in proof of Theorem 7), to ensure ⟨ w T , Ψ( x ′ i , f ( x i ))
− Ψ( x ′ i , j ) ⟩ > ⟨ w T , c ⟩ , ∀ c ∈ V (0) .
Consequently, for every T ∈ S we obtain a label pattern P T : [ d G ] ×
[ k ] → { 0 , 1 } :
[Equation]
It satisfies that ∀ i ∈ I T , j ∈ [ k ] , P S ( i, j )
= 1 and ∀ i ∈ I S \ T , ∃ j ∈ [ k ] , P T ( i, j )
= 0 , which implies
P T ̸ = P T ′ for T ̸ = T ′ .
And by definition P T is connected with a unique element in W dkf | Z , whichcompletes the proof of our claim.
Note that the AG-dimension of halfspace classifier class with respect to U coincides with the definitionof adversarial VC-dimension in [7].
Thus it satisfies that d U G ( W dk )
=
dk
+ 1 by Theorem 2 in [7].
Consequently, we have
[Equation]
where we have used Lemma 2 in step (i).
We conclude that d G ≤ O ( dk ln dk ) .
AG-dimension and AN-dimension are not equivalent
In previous sections, we have shown that the finiteness of the AN-dimension is a necessary conditionfor robust learnability, and the finiteness of the AG-dimension is a sufficient condition for robustlearnability.
Furthermore, we analyze these definitions on the linear multiclass predictor class,finding that the AG-dimension (AN-dimension) matches the order of the graph dimension (Natarajandimension) for this class.
The following question therefore naturally arises: are these two quantitiesequivalent?
In [5], it is proven that for every hypotheses class H ⊂
Y X ,
[Equation]
That is, in standard learning setting, if k ≤ ∞ then the definition of the Natarajan dimension and thegraph dimension are indeed equivalent, hence both the Natarajan dimension and graph dimensioncan characterize the standard learnability of a hypothesis class.
However, in the adversarial setting( U ̸ = I ), the generalized definitions derived above cannot bound each other.
To show this, we claimthat finite AN-dimension is not sufficient for robust learnability.
Theorem 9.
There exist H , U such that d U N ( H ) =
but H is not robustly PAC learnable under U ,i.e. m ag H ,
U ( ϵ, δ ) = ∞ .
Proof.
Let U be the 2 -norm ball: U ( x ) =
{ z ∈ X : ∥
z − x ∥ 2 ≤ r } for all x ∈ X and some r > 0 .
We construct H as follow.
Pick a sequence { x i } ∞ i =1 ⊂ X such that for any i ̸
= j, U ( x i ) ∩U ( x j ) = ∅ .
Let f : X → Y be a function satisfying: there exists z
i ∈ U ( x i ) \ x i such that f ( z i ) ̸
= f
( x i ) for eachi ∈ N .
For each sequence b = { b i } ∞ i =1 ⊂ { 0 , 1 } , define h b : X → Y :h b ( x ) =
  f
( x i ) , x ∈ U ( x i )
\ z i , i ∈ N , f ( x i ) , x = z i and b i = 1 , f ( z i ) , x = z i and b i = 0 , 1 , otherwise.
Let H =
{ h b : b is a sequence in { 0 , 1 }} .
Using the same approach employed in the proof of Lemma3, for any learning rule A : ( X × Y ) ∗ → H , there exists a distribution D over X × Y such that thereexists a function f ∗ ∈ H with R U ( f ∗ ; D ) = 0 , and with probability of at least 1 / 7 over the choiceof S ∼ D
m we have that R U
( A ( S ); D ) ≥ 1 / 8 , by considering the distribution family D containingdistributions that are uniform over 2 m points in { ( x 1 , f ( x 1 )) , . .
, ( x 3 m , f ( x 3 m )) } .
We completethe proof by noting that d U N ( H )
= 0 , since there is no single r -ball that is adversarially N-shatteredby H .
Conclusion
In this work, we show that the graph/Natarajan dimension cannot characterize robust multiclasslearnability. And we construct the concepts of AG-dimension and AN-dimension to upper bound andlower bound the sample complexity of robust multiclass learning. We further analyze the AG/ANdimension of linear multiclass predictors. Finally, we prove that AG-dimension and AN-dimensionare not equivalent.
However, it is not yet clear whether finite AG-dimension is also necessary forrobust multiclass learning.
Establishing a complexity measure that characterizes robust learnabilitystill remains an open question.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/
curating contains personally identifiable information or offensive content?
N/ A]
If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots , if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]