On Robust Multiclass Learnability
Abstract
This work analyzes the robust learning problem in the multiclass setting.
Under the framework of Probably Approximately Correct (PAC) learning, we first show that the graph dimension and the Natarajan dimension, which characterize the standard multiclass learnability, are no longer applicable in robust learning problem.
We then generalize these notions to the robust learning setting, denoted as the adversarial graph dimension (AG-dimension) and the adversarial Natarajan dimension (ANdimension).
Upper and lower bounds of the sample complexity of robust multiclass learning are rigorously derived based on the AG-dimension and AN-dimension, respectively.
Moreover, we calculate the AG-dimension and AN-dimension of the class of linear multiclass predictors, and show that the graph (Natarajan) dimension is of the same order as the AG(AN)-dimension.
Finally, we prove that the AGdimension and AN-dimension are not equivalent.
Introduction
Learning models that are robust to adversarial perturbations has attracted significant research attention in recent years[1, 2, 3].
Notably, prior work on this subject has largely studied the theory of robust learnability in the context of binary supervised learning.
However, many important problems require classification into a great number of target classes.
For example, in image object recognition and language models building[4, 5, 6, 7, 8, 9], the number of classes scales as the number of possible objects or the dictionary size respectively.
It is thus of both practical and theoretical interest to generalize the robust binary learning theory to the robust multiclass setting.
In the standard multiclass learning setting the finiteness of the Natarajan dimension or graph dimension [10] (both of which are generalized from the VC-dimension [11]) is necessary and sufficient to ensure learnability [12].
Recent work has shown that the finiteness of the VC-dimension is neither sufficient nor necessary for (proper) robust binary classification [13].
Inspired by [13], this paper demonstrates that the Natarajan/graph dimension can no longer characterize robust learnability in the multiclass problem.
Meanwhile, some questions naturally arise: In multiclass setting, how can robust learning be ensured?
And what is necessary for robust learning?
To answer these questions, we generalize the corrupted hypothesis classes [14], which arise from standard hypothesis classes in the binary setting in the presence of an adversary, and use them to reduce the robust learning problem to a non-robust one.
By considering the graph dimension and
Natarajan dimension of the corrupted hypothesis classes—which are defined as the adversarial graph dimension (AG-dimension) and adversarial Natarajan dimension (AN-dimension) respectively—we derive the upper and lower bounds of sample complexity of the robust multiclass learning problem.
We then analyze the AG-dimension and the AN-dimension of linear multiclass predictor class, showing that the AG(AN)-dimension and the graph (Natarajan) dimension are of the same order.
Since it is proven that the graph dimension and the Natarajan dimension are equivalent when the target class is finite [12], is it possible to use the AG-dimension and AN-dimension to bound each other?
Our work suggests that the equivalent relationship does not hold, which is illustrated by constructing a hypothesis class with finite AN-dimension but not robustly learnable.
Related work.
[15, 16, 17, 18] focus on the robust binary classification problem.
[10] characterize the multiclass learnability in non-robust setting.
[19, 12, 20] define a large family of notions of dimensions, all of which generalize the VC-dimension and may be used to estimate the sample complexity of multiclass classification.
[21] calculate the graph dimension and the Natarajan dimension of linear multiclass predictor class.
[22] upper bounds the adversarial risk of linear multiclass classifiers based on Rademacher complexity, which cannot directly derive bounds of sample complexity of general robust multiclass learning problems.
The theory on robust multiclass learnability for general hypothesis class is less explored.
This work takes some steps towards solving this problem.
Preliminaries
This section introduces some basic notations and concepts used throughout the present study.
In the remainder of this article, R , N , R + and R d represent the sets of real numbers, natural numbers, non-negative real numbers and d -dimensional vectors over R , respectively.
We denote the set { 1 , . . . , n } (for n ∈ N ) by [ n ] .
If A and B are sets, we use B A to denote the collection of all mappings from A to B and 2 A to denote the power set of A, that is the collection of all subsets of A.
Let C
= { c 1 , . .
, c m } ⊂ X and C ′
⊂ C be a subset of C,
then I C ′ ∈ [ m ] denotes the index set of
C ′ .
Let H be a class of functions defined in X , then H| C represents the restriction of H to C, that is
H C
= { ( h ( c 1 ) , . . .
, h ( c m )) :
h ∈ H} .
We denote the indicator function by 1 ( event ) , that is 1 if an event happens and 0 otherwise.
Finally, ∥ · ∥ p represents the ℓ p norm.
The robust learning problem is formalized as follows.
Let X =
R d be the instance space and
Y = { 1 , . . .
, k } be the label space.
For an unknown distribution D over X × Y 2 and a hypothesis class H ⊂ Y X , the goal of robust learning is to find a function f ∈ H , based on n labeled training samples S
= { ( x i , y i ) } ni =1 drawn independent and identically distributed (i.i.d.) from D , such that under a small perturbation U : X → 2 X , the adversarial risk
[Equation]
is minimal for the 0-1 loss l (ˆ y, y ) ≜
1 (ˆ y ̸ = y ) .
The perturbation U ( x ) is required to be nonempty, so some choice of x ′ is always available.
One choice for the perturbation is the p -norm ball ( p ≥ 1 ) with a small radius r , i.e. U ( x ) =
{ z ∈ X : ∥
z − x ∥ p ≤ r } .
Selecting r = 0 gives the identity perturbation : I ( x ) ≜ { x } .
Note that in this case, the problem is reduced to a standard learning problem, and R I ( f ; D ) is called the standard risk of f over D .
The mechanism of finding a function is called a learning algorithm .
In this paper, we focus on proper learning algorithms, that is, learning algorithms will always pick a function from H .
Formally, a learning algorithm is a function A : ∪ ∞ n =
( X × Y ) n → H .
A common choice of robust learning algorithm is to learn through adversarial empirical risk minimization or AERM for short:
[Equation]
where ˆ D n is the empirical distribution generated by S , i.e. for ( x, y ) ∼ ˆ D n , ( x, y ) is equal to ( x i , y i ) with probability 1 /n for each i ∈ { 1 , . .
. , n } .
Learning algorithms like AERM do not always produce a function that achieves the optimal adversarial risk, because a training set consisting of finite samples is not always sufficiently informative to represent D .
Thus, we introduce the definition of robust probably approximately correct (PAC) learnability, in the realizable and agnostic setting [14, 13]:
Definition 1 (Agnostic Robust PAC Learnability) .
Under perturbation U , a hypothesis class H is robustly PAC learnable in the agnostic setting if there is a function m ag H , U : (0 , 1) 2 → N and a learning algorithm A : ∪ ∞ n =
( X × Y ) n → H with the following property.
For every ϵ, δ ∈ (0 , 1) and every distribution D over X × Y , let S = { ( x i , y i ) } ni
=1 be n samples i.i.d generated by D s.t. n > m ag H , U ( ϵ, δ ) .
Then with probability at least 1 − δ ,
[Equation]
The function m ag H , U is defined as the sample complexity of learning H under perturbation U in the agnostic setting.
Definition 2 (Realizable Robust PAC Learnability) .
Under perturbation U , a hypothesis class H is robustly PAC learnable in the realizable setting if there is a function m re H , U : (0 , 1) 2 → N and a learning algorithm A : ∪ ∞ n =
( X × Y ) n → H with the following property.
For every ϵ, δ ∈ (0 , 1) and every distribution D over X × Y , let S = { ( x i , y i ) } ni
=1 be n samples i.i.d generated by D s.t. n > m re H , U ( ϵ, δ ) .
Then with probability at least 1 − δ ,
[Equation]
The function m re H , U is defined as the sample complexity of learning H under perturbation U in the realizable setting.
These definitions agree with the standard PAC learnability
when U =
I , which has been widely studied in both binary ( k = 2 ) and multiclass ( k > 2 ) setting.
We recall the known result regarding the sample complexity of binary learning.
Recall the definition of the Vapnik-Chervonenkis dimension (VC-dimension)
Definition 3 (VC-dimension) .
Let H ⊂ { 0 , 1 } X be a hypothesis class and let S ⊂ X .
We say that S is shattered by H if H| S
= { 0 , 1 } S .
The VC-dimension of H , denoted by VC ( H ) , is the maximal cardinality of a set that is shattered by H .
The following theorem, based on the VC-dimension, characterizes the sample complexity of learning binary hypothesis classes.
Theorem 1 ([11] and [23]) .
There are absolute constants C 1 , C 2 > 0 such that for every H ⊂ { 0 , 1 } X ,
[Equation]
and
[Equation]
It is natural to seek a generalization of the VC-dimension to hypothesis classes of non-binary functions.
We recall two generalizations, both introduced by [10]:
Definition 4 (Graph dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H
G-shatters S if there exists an f : S → Y such that for every T ⊂ S, there is a g ∈ H such that ∀ x ∈ T, g ( x ) =
f
( x ) , and ∀ x ∈ S \ T, g ( x ) ̸
[Equation]
f ( x ) . The graph dimension of H , denoted by d G ( H ) , is the maximal cardinality of a set that is G-shattered by H .
Definition 5 (Natarajan dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H N-shatters S if there exists f 1 , f 2 : S
→ Y such that ∀ x ∈ S, f 1 ( x ) ̸
= f 2 ( x ) , and for every
T ⊂ S, there is a g ∈ H such that
[Equation]
The Natarajan dimension of H , denoted by d N ( H ) , is the maximal cardinality of a set that is
N-shattered by H .
Both the graph dimension and the Natarajan dimension coincide with the VC-dimension for k = 2 .
Based on these notions, the following theorem provides the upper and lower bounds of sample complexity of standard multiclass learning:
Theorem 2 ([10] and [12]) .
There are absolute constants C 1 , C 2 > 0 such that for every H ⊂ Y X ,
C
[Equation]
ϵ
[Equation]
and
[Equation]
AG-dimension and upper bounds for robust multiclass learning
We start by showing that finite graph dimension is not sufficient for robust multiclass learnability.
We then introduce the notion of corrupted hypotheses, presented by [14].
By computing the graph dimension of these hypotheses, which is defined as adversarial graph dimension, we can upper bound the sample complexity of robust multiclass learning problem in both realizable and agnostic settings.
3.1 Finite graph dimension is not sufficient for robust multiclass learning
Theorem 2 shows that the finiteness of the graph dimension is a sufficient condition for multiclass learnability in non-robust setting.
While in adversarial setting, for hypothesis classes with finite graph dimension, one cannot ensure the robust PAC learnability of H , even if d G ( H ) = 1 .
Theorem 3.
There exists a hypothesis class H with d G ( H )
= 1 and an adversary U such that H is not robustly PAC learnable with respect to U .
To prove this theorem, We first present a lemma, which is generalized from Lemma 3 in [13].
Lemma 1.
Let m ∈ N . Then, there exists a hypothesis class H ⊂ Y X such that for {+any learning rule A : ∪ ∞ n =
+}
( X × Y ) n → H , there exists a distribution D over X × Y such that:
There exists a function f ∗ ∈ H with R U ( f ∗ ; D ) = 0 .
With probability of at least 1 /
7 over the choice of S ∼ D m we have that R U
( A ( S ); D ) ≥ 1 / 8 . The proof of this lemma is presented in Appendix. We now proceed with the proof of Theorem 3. Proof of Theorem 3.
The argument follows closely a proof of an analogous result by [13] for binary robust learning, but generalizes the constructions and analyzes to match the definition of graph dimension in the multiclass setting.
For each m ∈ N , let X m =
{ x ( m ) 1 , . .
, x ( m ) 3 m } be a set that contains 3 m distinct points in X subject to ∀ x i , x j ∈ ∪ ∞ m =1 X m , if x i ̸
= x j , then U ( x i ) ∩U ( x j ) = ∅ .
And we construct H m on X m as follows.
For each b ∈ { 0 , 1 } 3 m , we construct a set Z b : Initialize
Z b = ∅ , for each i ∈ [3 m ] , if b i = 1 then pick a point z ∈ U ( x ( m ) i )
such that z / ∈ Z b ′ for each b ′ ̸ = b, and add it to Z b .
{+Let h b : X → Y be a hypothesis such that h b ( x ) = 
+}
if and only if x / ∈ Z b and x / ∈ X m ′ for m ′ ̸ = m.
Furthermore, for fixed m ∈ N and x ∈ X m ′ s.t. m
′ ̸
= m , we need all h b map x into a same label y x ( ̸ = 1) , where y x is consistent for all m. That is, h b ( x ) =
  ¬ 1 , x ∈ Z b , y x
( ̸ = 1) , ∃ m ′ ̸
= m, x ∈ X m ′ , 1 , otherwise, where ¬ 1 is some label that is not equal to 1 .
Then define
H m = { h b : b ∈ { 0 , 1 } 3 m , 3 m (cid:88) i =
b i = m } .
Let H = ∪ ∞ m =1 H m .
We claim that d G ( H ) ≤ 1 .
That is, if pick two points x 1 , x 2 ∈ X such that x 1 ̸ = x 2 , H cannot G-shatters { x 1 , x 2 } .
since for all h ∈ H and x ∈ X , there are at most two cases of h ( x ) , one of which is 1.
Hence the only f : { x 1 , x 2 } → Y in Definition 4 that needs to be considered is f ( x 1 ) =
f ( x 2 )
= 1 .
There are six cases to consider:
(1) There exists m ∈ N
such that z 1 , z 2 ∈ X m .
Then ∀ h b ∈
H m , x 1 , x 2 / ∈ Z b by the construction of Z b .
Hence h b ( x 1 ) =
h b ( x 2 ) = 1 .
And ∀
h ∈ H m ′ for m ′ ̸
= m, we have h ( x 1 ) =
y x 1 ̸ = 1 , h ( x 2 ) =
y x 2 ̸ = 1 .
So we only obtain labelings (1 , 1) and ( ¬ 1 , ¬ 1) .
(2) There exists m ∈ N
such that x 1 , x 2 ∈ U ( X m )
\ X m .
There are two sub-cases to consider.
(i)
∃ b s.t.
h b ∈ H m and x 1 , x 2 ∈ Z b , then we have h b ( x 1 ) ̸
= 1 and h b ( x 2 ) ̸
= 1 .
For the remaining h ∈ H we have h ( x 1 )
=
h ( x 2 ) = 1 .
So we only obtain labelings (1 , 1) and ( ¬ 1 , ¬ 1) in this sub-case.
(ii)
For the remaining sub-cases, labeling ( ¬ 1 , ¬ 1) cannot be obtained since x 1 , x 2 / ∈
X m for all m
and there is no b such that x 1 , x 2 are in Z b in the same time.
(3) x 1 ∈ X m and x 2 ∈ X m ′ for m ̸
= m ′ .
In this case, we have h ( x 1 )
= 1 , h ( x 2 ) ̸
= 1 for h ∈ H m , and h ( x 1 ) ̸
= 1 , h ( x 2 )
= 1 for h ∈ H m ′ .
For those predictors that are in neither H m nor H m ′ , we have h ( x 1 ) ̸ = 1 , h ( x 2 ) ̸ = 1 .
Hence we cannot label both points x 1 and x 2 with (1 , 1) .
(4) x 1 ∈ X m and x 2 ∈ U ( X m ′ )
\ X m ′ for m ̸
= m ′ .
{+Observe that h ( x 1 ) = 
+}
if and only if h ∈ H m .
But for all h ∈ H m , h always labels x 2 with 1 .
Hence labeling (1 , ¬ 1) cannot be obtained.
(5) x 1 ∈ U ( X m )
\ X m and x 2 ∈ U ( X m ′ )
\ X m ′ for m ̸
= m ′ .
Similar to the discussion above, for all h ∈ H , either h ( x 1 )
= 1 or h ( x 2 )
= 1 .
So we cannot obtain labeling ( ¬ 1 , ¬ 1) .
(6) The remaining cases, i.e. x 1 , x
2 / ∈ U ( X m ) for all m ∈ N .
In this case we can only obtain labeling (1 , 1) since for all h ∈ H , we have h ( x 1 )
=
h ( x 2 ) = 1 .
The arguments above show that d G ( H ) ≤ 1 .
It remains to show H is not robustly PAC learnable.
By
{+Lemma 1, it follows that for any learning rule A : ∪ ∞ n =
+}
( X × Y ) n → H and for any m ∈ N , there exists a distribution D over X m × Y where there exists a predictor h ∗ ∈ H with R U
( h ∗ ; D )
= 0 , but with probability at least 1 / 7 over S ∼ D m ,
R U ( A ( S ); D ) > 1 / 8 if A ( S ) ∈
H m .
Classifiers in H m ′ where m
′ ̸
= m are also non-robust since they make mistakes on points in X m .
This concludes that
H is not robustly PAC learnable, which completes the proof.
3.2 Adversarial graph dimension
Consider a given hypothesis f : X → Y .
A labeled adversarial sample (˜ x, y ) is classified correctly if ˜ x ∈ f − 1 ( y ) .
A labeled example ( x, y ) is classified correctly if U ( x ) ⊂ f − 1 ( y ) .
Following [14], let ˜ Y =
Y ∪ {⊥} , where ⊥ is the special “always wrong” output, i.e. l ( y, ⊥ )
=
l ( ⊥ , y ) =
1 for every y ∈ ˜ Y .
We define the mapping κ U : Y X
→ ˜ Y X :
[Equation]
The corrupted set of hypotheses induced by perturbation U is then defined by ˜ H
= { κ U ( f ) : f ∈ H} .
Lemma 2 (Lemma 2 in [14]) .
For any perturbation U and distribution D , R U ( f ; D ) =
R I ( κ U ( f ); D ) .
This lemma bridges the equivalence between robustly learning H and learning ˜ H without adversary, which enables us to use standard techniques to bound the sample complexity.
Intuitively, we can derive a sufficient condition by considering the graph dimension of ˜ H , which can be equivalently formalized as follows:
Definition 6 (Adversarial Graph Dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H adversarially G-shatters S if there exists an f : X → Y such that for every T ⊂ S, there is a g ∈ H such that
[Equation]
and
[Equation]
The adversarial graph dimension (AG-dimension) of H , denoted by d U G ( H ) , is the maximal cardinality of a set that is adversarially G-shattered by H .
Proposition 1.
Let H ⊂
Y X be a hypothesis class and let ˜ H be the corrupted set of hypotheses induced by perturbation U .
Then we have
[Equation]
The proof is presented in Appendix.
Intuitively, by combining the above proposition with Lemma 2 and Theorem 2, we can upper bound the sample complexity of robust multiclass learning.
Formally:
Theorem 4.
There are absolute constants C > 0 such that for every H ⊂ Y X , m re
[Equation]
[Equation]
, δ
[Equation]
and m ag H , U ( ϵ, δ
[Equation]
(cid:18) d U G ( H ) + ln
[Equation]
[Equation]
ϵ
[Equation]
Proof.
Let H ⊂
Y X be a hypothesis class with d U G ( H ) =
d. For every f ∈ H , define ¯ f : X × Y → { 0 , 1 } by setting ¯ f ( x, y )
=
if and only if f ( x ′ )
=
y for all x ′ ∈ U ( x ) .
Note that according to our construction, we have
[Equation]
In the realizable case, let ¯ H
= { ¯ f : f ∈ H} .
From the definition, we can see that VC ( ¯ H ) =
d U G ( H ) .
By Theorem 1, there exists C > 0 such that when we draw m > C ( dϵ ln( 1 ϵ )
+ 1 ϵ ln( 1 δ ))
samples i.i.d.
from some distribution D on X × Y , then with probability at least 1 − δ we have P ( x,y )
∼D [ ¯ f ( x, y ) ̸ = 1] ≤ ϵ. Plugging 2 into this inequality, we complete the proof for the realizable case and it is similar for the agnostic case.
AN-dimension and lower bounds for robust multiclass learning
In this section, we discuss the necessary conditions for robust multiclass learnability.
4.1 Finite Natarajan dimension is not necessary for robust multiclass learning
We first show that the finiteness of ordinary Natarajan dimension is also not necessary for robust multiclass learnability.
Theorem 5.
There exist H , U such that d N ( H ) =
∞ but H is robustly PAC learnable under U , i.e. m ag H , U ( ϵ, δ ) < ∞ . Proof.
The proof follows the construction from [13], which shows that finite VC-dimension is not necessary for robust binary learnability. Let H =
Y X be the family of all mappings from X to Y and U ( x ) =
X be an all-powerful adversary.
It is easy to see that ∀ n ∈ N , there exist n distinct points in R d that are N-shattered by H , hence d N ( H )
= ∞ . Given a distribution D on X × Y , let S = { ( x i , y i ) } mi =1 be m samples drawn i.i.d from D and A be a learning rule such that A ( S ) returns a constant-label predictor h : [Equation]
According to the weak law of large numbers, ˆ D m → D as m → ∞ , where ˆ D m is the empirical distribution generated by S, we have P (cid:2)
k m = arg max k ∈Y P ( x,y ) ∼D
[ y = k ] (cid:3) → 1 as m → ∞ .
Hence for all δ ∈ (0 , 1) there exists a m 0 ∈ N such that when m > m 0 , R U ( h ; D ) reaches the optimal adversarial risk in H , i.e. R U ( h ; D ) =
arg min f ∈H R U ( f ; D ) with probability at least 1 − δ.
4.2 Adversarial Natarajan dimension
Following the idea of generalizing the graph dimension to the AG-dimension, for a given hypothesis class H , one can consider the Natarajan dimension of its corrupted hypothesis class to lower bound the sample complexity of robust learnability.
However, there are several natural but different ways in which the notion of N-shattering can be generalized: that is, we can either let f 2 that witnesses ˜
H N-shattering S take the value “ ⊥ ” or not.
Both notions derive a necessary condition for robust learnability.
In this work, we choose the stronger notion, i.e. we need the entire perturbation sets to be N-shattered by H .
Definition 7 (Adversarial Natarajan Dimension) .
Let H ⊂
Y X be a hypothesis class and let S ⊂ X .
We say that H adversarially N-shatters S if there exist f 1 , f 2 : X → Y such that ∀ y ∈ S, f 1 ( y ) ̸
= f 2 ( y ) , and for every T ⊂ S, there is a g ∈ H such that
[Equation]
and
[Equation]
The adversarial Natarajan dimension (AN-dimension) of H , denoted by d U N ( H ) , is the maximal cardinality of a set that is adversarially N-shattered by H .
Based on this definition, we present that:
Theorem 6.
There are absolute constants C > 0 such that for every H ⊂ Y X ,
[Equation]
U
[Equation]
Proof.
Let H ⊂
Y X be a hypothesis class such that d U N ( H ) =
d, and let H d = { 0 , 1 } [ d ] be the family of all mappings from [ d ] to { 0 , 1 } .
We want to show that for every learning algorithm A for
H and every ϵ, δ ∈ (0 , 1) , there exists a learning algorithm
¯
A satisfying: if there exists a function m re A , U : (0 , 1) 2 → N such that when A witnesses m > m re A , U ( ϵ, δ ) samples (denoted as S ) drawn i.i.d from some distribution D , P [ R U ( A ( S ); D ) ≤ ϵ ] > 1 − δ.
Then when ¯ A witnesses m samples (denoted as S ′ ) drawn i.i.d from some distribution D ′ on [ d ] ×{ 0 , 1 } , P [ R I ( ¯ A ( S ′ ); D ′ ) ≤ ϵ ] > 1 − δ holds.
By this claim, we have m re ¯ A , I ≤ m re A , U , thus m re H d , I ≤ m re H , U .
Note that VC ( H d ) =
d, so by
Theorem 1 we have m re H , U ( ϵ, δ ) ≥ C ( d +ln(1 /δ )
ϵ ) for some constant C > 0 (independent of H ), and similarly for the agnostic case.
We now prove the claim.
Let { s 1 , . . .
, s d } ⊂ X be a set and let f 0 , f 1 be functions that witness the adversarial N-shattering of H .
Given a sample set S ′ ≜ { ( x i , y i ) } mi =1 ⊂ [ d ] × { 0 , 1 } , let g
= A ( { s x i , f y i ( s x i ) } mi =1 ) .
Consider the learning algorithm ¯
A : ∪ ∞ n =0 ([ d ] × { 0 , 1 } ) n → H d : ¯ A ( S ′ )
( j ) =
(cid:26) 1 , g ( s ′ j ) =
f 1 ( s j ) , ∀ s ′ j ∈ U ( s j ) , 0 , otherwise.
Denote ¯ A ( S ′ ) by f.
By the definition of adversarial N-shattering, it can be derived that f ( j ) = {+
+}
if and only if g ( s ′ j ) = f 2 ( s j ) , ∀ s ′ j ∈ U ( s j ) .
By the construction we have
E ( j,y ) ∼D ′ [ 1 ( f ( j ) ̸ = y )]
=
E ( j,y ) ∼D ′ [ sup s ′ j ∈U ( s j )
[Equation]
m ) , where ˆ D ′ m is defined by P ˆ D ′ m ( X = s j , Y = f
y ( s j ))
=
P D ′
( J = j, B = y ) .
The AG/AN-dimension of linear multiclass predictors
In this section, we calculate the AG-dimension and AN-dimension for linear multiclass predictors under bounded l p perturbation, motivated by the binary results of [14] and the non-adversarial results of [21].
Recall the definition of linear multiclass predictors.
Let Ψ : X × Y → R dk be a class-sensitive feature mapping [24]:
[Equation]
] , where x ( i ) represents the i th coordinate of x.
The class of linear multiclass predictors is defined as
[Equation]
We cite that the graph dimension and Natarajan dimension of H Ψ is given by ( d − 1)( k − 1)
≤ d N ( H Ψ ) ≤ d G ( H Ψ ) ≤ O ( dk ln( dk ))
We will demonstrate that the AG-dimension and ANdimension also satisfy this bound in the remainder of this section.
To this end, we start by calculating the AN-dimension of H Ψ .
Specifically, one can show that the AN-dimension of H Ψ equals to the standard Natarajan dimension of H Ψ .
Theorem 7.
Let U ( x ) =
{ z ∈ X : ∥
z − x ∥ p ≤ r } for some r ∈ R + and p ∈ R + ∪ {∞} , and let
H Ψ be as defined in 3.
Then, the AN-dimension of H Ψ satisfies
d U N ( H Ψ ) =
d N ( H Ψ ) .
Proof.
Since adversarial N-shattering implies N-shattering, we have d U N ( H Ψ )
≤ d N ( H Ψ ) .
To prove the inverse inequality, it should be noted that the classifiers in H Ψ are linear, hence if H Ψ N-shatters
S , it also N-shatters the set with some x ∈ S replaced by ∀ x ′ ∈
l x , where l x is the half-line which terminates at 0 and goes through x. For those x ∈ S that cannot be adversarially N-shattered by H Ψ , we can always find some x ′ ∈ l x such that U ( x ′ ) does not cross the decision boundary of g T that witnesses the N-shattering for all T ∈ S.
Note that the Natarajan dimension of H Ψ satisfies ( d − 1)( k − 1) ≤ d N ( H Ψ ) ≤ dk (Corollary 29. in [25]).
Combine this result with Theorem 7, we have:
Corollary 1.
Under the conditions of Theorem 7, ( d − 1)( k − 1) ≤ d U N ( H Ψ ) ≤ dk.
Next, we present the AG-dimension of H Ψ :
Theorem 8.
Let U ( x ) =
{ z ∈ X : ∥
z − x ∥ p ≤ r } for some r ∈ R + and p ∈ R + ∪ {∞} , and let
H Ψ be as defined in 3.
Then, the AG-dimension of H Ψ satisfies
[Equation]
To prove this result, we first present a lemma, which can be viewed as the generalization of Sauer’s
Lemma [26] in the adversarial setting.
We first define the restriction of a corrupted class.
Let ˜ H be the corrupted hypothesis class induced by U .
Let S = { x 1 , . . .
, x m } ⊂ X .
Fix a f ∈ Y S , for each h ∈ H , define h f : S → { 0 , 1 } such that h f ( x i )
=
if and only if κ U ( h )( x i ) =
f
( x i ) , i ∈ [ m ] .
Define H f
= { h f : h ∈ H} .
The restriction of ˜ H to S is defined by:
[Equation]
where f ∗ = arg max f ∈Y S (cid:12)(cid:12)(cid:12) H f | S (cid:12)(cid:12)(cid:12) .
Lemma 3.
Let U be an adversary and H be a hypothesis class with d U G ( H ) ≤
d G < ∞ .
Let ˜ H be the corrupted hypothesis class induced by U .
Let S = { x 1 , . . .
, x m } ⊂ X .
Then, for all m > d G
+we have
[Equation]
Proof.
By Sauer’s Lemma [26] and the fact that VC( H f ) ≤ d U G ( H ) for all f ∈ Y S .
We now prove Theorem 8.
The construction idea follows the proof of the non-adversarial results in [21].
Proof of Theorem 8: The lower bound follows from Theorem 7 and the fact that d U G ( H ) ≥
d U N ( H ) for all H .
To upper bound d G :=
d U G ( H ) , let S =
( x 1 , . .
, x d G ) ∈ R
d be a set which is adversarially
G-shattered by H Ψ , and let f : S → Y be a function that witnesses the shattering.
For every ( i, j ) ∈ [ d G ] × [ k ] , define z i,j = Ψ( x i , f ( x i ))
− Ψ( x i , j ) .
Denote Z = { z i,j | ( i, j ) ∈
[ d G ] × [ k ] } .
We now show that there exists a injective mapping from subsets of S to W dkf | Z , where W dk is the class of halfspace classifiers over R dk .
For each T ⊂ S, by the definition of adversarial G-shattering, ∃ h T ∈ H Ψ such that
[Equation]
and ∀ x ∈ S \ T, ∃ x ′ ∈ U ( x ) , h T ( x ′ )
[Equation]
= f
[Equation]
x )
[Equation]
Let w T ∈ R
dk be the vector defines h T , 4 implies
[Equation]
and
[Equation]
x [Equation] It can be easily derived from 5 that ∀ i ∈ I S \ T , ∃ j ∈ [ k ] ,
∃ z ′ ∈ V ( z i,j ) s.t.
⟨ w T , z ′ ⟩ < 0 , where V is the p -norm ball in R dk with radius r.
And from 5 we can assume w.l.o.g that ∀ i ∈ I T , ∀ j ∈ [ k ] , ∀ z ′ ∈
V ( z i,j ) , ⟨ w T , z ′ ⟩ ≥ 0 .
This is because if there exists some z ′ ∈ V ( z ) s.t. ⟨ w T , z ′ ⟩ < 0 , notice that z ′ =
V ( c ′ )
+ z i,j for some bounded c ′ , we can replace x i by some x ′ i without destroying the adversarial
G-shattering (see the trick used in proof of Theorem 7), to ensure ⟨ w T , Ψ( x ′ i , f ( x i ))
− Ψ( x ′ i , j ) ⟩ > ⟨ w T , c ⟩ , ∀ c ∈ V (0) .
Consequently, for every T ∈ S we obtain a label pattern P T : [ d G ] ×
[ k ] → { 0 , 1 } :
P T ( z i,j ) = 1 ( min z ′ ∈V ( z i,j ) ⟨ w T , z ′ ⟩ ≥ 0) .
It satisfies that ∀ i ∈ I T , j ∈ [ k ] , P S ( i, j )
= 1 and ∀ i ∈ I S \ T , ∃ j ∈ [ k ] , P T ( i, j )
= 0 , which implies
P T ̸ = P T ′ for T ̸ = T ′ .
And by definition P T is connected with a unique element in W dkf | Z , which completes the proof of our claim.
Note that the AG-dimension of halfspace classifier class with respect to U coincides with the definition of adversarial VC-dimension in [14].
Thus it satisfies that d U G ( W dk )
=
dk
+ 1 by Theorem 2 in [14].
Consequently, we have
[Equation]
where we have used Lemma 3 in step (i).
We conclude that d G ≤ O ( dk ln dk ) .
AG-dimension and AN-dimension are not equivalent
In previous sections, we have shown that the finiteness of the AN-dimension is a necessary condition for robust learnability, and the finiteness of the AG-dimension is a sufficient condition for robust learnability.
Furthermore, we analyze these definitions on the linear multiclass predictor class, finding that the AG-dimension (AN-dimension) matches the order of the graph dimension (Natarajan dimension) for this class.
The following question therefore naturally arises: are these two quantities equivalent?
In [12], it is proven that for every hypotheses class H ⊂
Y X ,
[Equation]
That is, in standard learning setting, if k ≤ ∞ then the definition of the Natarajan dimension and the graph dimension are indeed equivalent, hence both the Natarajan dimension and graph dimension can characterize the standard learnability of a hypothesis class.
However, in the adversarial setting ( U ̸ = I ), the generalized definitions derived above cannot bound each other.
To show this, we claim that finite AN-dimension is not sufficient for robust learnability.
Theorem 9.
There exist H , U such that d U N ( H ) =
but H is not robustly PAC learnable under U , i.e. m ag H ,
U ( ϵ, δ ) = ∞ .
Proof.
Let U be the 2 -norm ball: U ( x ) =
{ z ∈ X : ∥
z − x ∥ 2 ≤ r } for all x ∈ X and some r > 0 .
We construct H as follow.
Pick a sequence { x i } ∞ i =1 ⊂ X such that for any i ̸
= j, U ( x i ) ∩U ( x j ) = ∅ .
Let f : X → Y be a function satisfying: there exists z
i ∈ U ( x i ) \ x i such that f ( z i ) ̸
= f
( x i ) for each i ∈ N .
For each sequence b = { b i } ∞ i =1 ⊂ { 0 , 1 } , define h b : X → Y : [Equation]
1 , otherwise.
Let H =
{ h b : b is a sequence in { 0 , 1 }} .
Using the same approach employed in the proof of Lemma 1, for any learning rule A : {+∪ ∞ n =
+} ( X × Y ) n → H , there exists a distribution D over X × Y such that there exists a function f ∗ ∈ H with R U ( f ∗ ; D ) = 0 , and with probability of at least 1 / 7 over the choice of S ∼ D
m we have that R U
( A ( S ); D ) ≥ 1 / 8 , by considering the distribution family D containing distributions that are uniform over 2 m points in { ( x 1 , f ( x 1 )) , . .
, ( x 3 m , f ( x 3 m )) } .
We complete the proof by noting that d U N ( H ) = 0 , since there is no single r -ball that is adversarially N-shattered by H .
Conclusion
In this work, we construct the concepts of AG-dimension and AN-dimension to upper bound and lower bound the sample complexity of robust multiclass learning, respectively.
We further analyze the AG/AN-dimension of linear multiclass predictors and prove that AG-dimension and AN-dimension are not equivalent in general.
However, establishing a complexity measure that characterizes robust learnability still remains an open question. Acknowledgements This work is supported by the National Natural Science Foundation of China under Grant 61976161.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[N/A] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]