Original: P ROTECT THE WEAK : C LASS - FOCUSED ONLINE LEARNING FOR ADVERSARIAL TRAINING
Revised: P ROTECT THE WEAK : C LASS - FOCUSED ONLINE LEARNING FOR ADVERSARIAL TRAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original:  BSTRACT
Revised: A BSTRACT

Original: Adversarial training promises a defense against adversarial perturbations in terms of average accuracy.
Revised: Adversarial training promises a defense against adversarial perturbations in terms of average accuracy.

Original: In this work, we identify that the focus on the average accuracy metric can create vulnerabilities to the “weakest” class.
Revised: In this work, we identify that the focus on the average accuracy metric can create vulnerabilities to the “weakest” class

Original: For instance, on
Revised: For instance , on

Original: CIFAR10, where the average accuracy is 47%, the worst class accuracy can be as low as 14%.
Revised: CIFAR10, where the average accuracy is %, the worst class accuracy can be as low as 14%

Original: The performance sacriﬁce of the weakest class can be detrimental for real-world systems, if indeed the threat model can adversarially choose the class to attack.
Revised: The performance sacriﬁce of the weakest class can be detrimental for real- world systems if indeed the threat model can adversarially choose the class to attack

Original: To this end, we propose to explicitly minimize the worst class error, which results in a min- max- max optimization formulation.
Revised: To this end , we propose to explicitly minimize the worst class error which results in a min max max optimization formulation

Original: We provide high probability convergence guarantees of the worst class loss for our method, dubbed as class focused online learning (CFOL), which can be plugged into existing training setups with virtually no overhead in computation.
Revised: We provide high probability convergence guarantees of the worst class loss for our method, dubbed as class focused online learning (CFOL , using techniques from the online learn ing community. CFOL can be plugged into existing training setups with virtually no overhead in computation

Original: We observe signiﬁcant improvements on the worst class accuracy of 30% for CIFAR10.
Revised: We observe signiﬁcant improvements on the worst class accuracy of % for CIFAR10.

Original: We also observe consistent behavior across CIFAR100 and STL10.
Revised: We also observe consistent behavior across CIFAR100 and STL10.

Original: Intriugingly, we ﬁnd that minimizing the worst case can even sometimes improve the average.
Revised: Intriugingly, we ﬁnd that minimizing the worst case can even sometimes improve the average.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The susceptibility of neural networks to adversarial noise (Goodfellow et al., 2014; Szegedy et al., 2013) has been a grave concern over the launch of such systems in real-world applications.
Revised: The susceptibility of neural networks to adversarial noise (Goodfellow et al., 2014; Szegedy et al., 2013) has been a grave concern over the launch of such systems in real-world applications.

Original: Several techniques defending such attacks that optimize the average performance have been proposed (Papernot et al., 2016; Raghunathan et al., 2018; Guo et al., 2017; Madry et al., 2017; Zhang et al., 2019).
Revised: Several techniques defending such attacks that optimize the average performance have been proposed (Papernot et al., 2016; Raghunathan et al., 2018; Guo et al., 2017; Madry et al., 2017; Zhang et al., 2019).

Original: In response, even stronger attacks have been proposed (Carlini & Wagner, 2016; Engstrom et al., 2018; Carlini, 2019).
Revised: In response, even stronger attacks have been proposed (Carlini & Wagner, 2016; Engstrom et al., 2018; Carlini, 2019).

Original: Indeed, recent studies demonstrate that regardless of the defense there exists an attack that can lower the average performance of the system (Shafahi et al., 2018).
Revised: Indeed, recent studies demonstrate that regardless of the defense there exists an attack that can lower the average performance of the system (Shafahi et al., 2018).

Original: In this work, we argue that the average performance is not the only criterion that is of interest for realworld applications.
Revised: In this work, we argue that the average performance is not the only criterion that is of interest for real-world applications.

Original: For classiﬁcation, in particular, optimizing the average performance provides no guarantees for the “weakest” class.
Revised: For classiﬁcation, in particular, optimizing the average performance provides no guarantees for the “weakest” class.

Original: This is critical in scenarios where an attacker can pick the class adversarially.
Revised: This is critical in scenarios where an attacker can pick the class adversarially in addition to the adversarial perturbation.

Original: It turns out that the worst performing class can indeed be much worse than the average in adversarial training.
Revised: It turns out that the worst performing class can indeed be much worse than the average in adversarial training.

Original: This difference is already present in clean training but we critically observe, that the gap between the average and the worst, is greatly exacerbated in adversarial training.
Revised: This difference is already present in clean training but we critically observe, that the gap between the average and the worst, is greatly exacerbated in adversarial training.

Original: This gap can already be observed on CIFAR10 where the accuracy across classes is far from uniform with 47% average robust accuracy while the worst class is 14% (see Figure 1).
Revised: This gap can already be observed on CIFAR where the accuracy across classes is far from uniform with 47% average robust accuracy while the worst class is 14% (see Figure 1).

Original: The effect is even more prevalent when more classes are present as in CIFAR100 where we observe that the worst class has zero accuracy while the best has 70% (see Appendix C where we include multiple other datasets).
Revised: The effect is even more prevalent when more classes are present as in CIFAR100 where we observe that the worst class has zero accuracy while the best has 70% (see Appendix C where we include multiple other datasets).

Revised: Despite the focus on adverarial training, we note that the same effect can be observed for robust evaluation after clean training (see Figure 4).

Original: This dramatic drop in accuracy for the weakest classes begs for different approaches than the classical empirical risk minimization (ERM), which focuses squarely on the average loss.
Revised: This dramatic drop in accuracy for the weakest classes begs for different approaches than the classical empirical risk minimization (ERM), which focuses squarely on the average loss.

Original: We suggest a simple alternative, which we dub class focused online learning (CFOL), that can be plugged into existing adversarial training procedures.
Revised: We suggest a simple alternative, which we dub class focused online learning (CFOL), that can be plugged into existing adversarial training procedures.

Original: Instead of minimizing the average performance over the dataset we sample from an adversarial distribution over classes that is learned jointly with the model parameters.
Revised: Instead of minimizing the average performance over the dataset we sample from an adversarial distribution over classes that is learned jointly with the model parameters.

Original: In this way we aim at becoming robust to an attacker that can adversarially choose the class .
Revised: In this way we aim at becoming robust to an attacker that can adversarially choose the class .

Original: Concretely, we make the following contributions: • We propose a simple solution which relies on the classical bandit algorithm from the online learning literature, namely the Exponential-weight algorithm for Exploration and Exploita-
Revised: The focus of this paper is thus on the robust accuracy of the weakest classes instead of the average robust accuracy.

Original: a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h
Revised: a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h

Original: o r s e s h
Revised: o r s e s h

Original: i p t
Revised: i p t

Original: r u c k 0.
Revised: r u c k 0.

Original: C l e a n t e s t a cc u r a c y
Revised: C l e a n t e s t a cc u r a c y

Original: Clean training a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h
Revised: Clean training a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h

Original: o r s e s h
Revised: o r s e s h

Original: i p t
Revised: i p t

Original: r u c k 0.
Revised: r u c k 0.

Original: R o b u s t t e s t a cc 
Revised: R o b u s t t e s t a cc u

Original: [Figure]
Revised: [Figure]

Revised: y

Revised: [Figure]

Original: tion (Exp3) (Auer et al., 2002).
Revised: training However, when considering the some classes perform worse than Concretely, we make the following contributions: • We propose a simple solution which relies on the classical bandit algorithm from the online learning literature, namely the Exponential-weight algorithm for Exploration and Exploitation (Exp3) (Auer et al., 2002).

Original: The method is directly compatible with standard adversarial training procedures (Madry et al., 2017), by replacing the empirical distribution with an adaptively learned adversarial distribution over classes.
Revised: The method is directly compatible with standard adversarial training procedures (Madry et al., 2017), by replacing the empirical distribution with an adaptively learned adversarial distribution over classes.

Original: • We carry out extensive experiments comparing CFOL against three strong baselines across three datasets, where we consistently observe that CFOL improves the weakest classes.
Revised: • We carry out extensive experiments comparing CFOL against three strong baselines across three datasets, where we consistently observe that CFOL improves the weakest classes.

Original: • We support the empirical results with high probability convergence guarantees for the worst class accuracy and establish direct connection with the conditional value at risk (CVaR) (Rockafellar et al., 2000) uncertainty set from the distributional robust optimization.
Revised: • We support the empirical results with high probability convergence guarantees for the worst class accuracy and establish direct connection with the conditional value at risk (CVaR) (Rockafellar et al., 2000) uncertainty set from the distributional robust optimization.

Original: Overview We deﬁne our new objective in Section 2, followed by the description of the proposed method in Section 3.
Revised: Overview We deﬁne our new objective in Section 2, followed by the description of the proposed method in Section 3.

Original: In Section 3.1 we prove convergence guarantees.
Revised: In Section 3.1 we prove convergence guarantees.

Original: We then move to the empirical work in Section 4, where we observe and resolve the issue of robust overﬁtting in the context of the worst class.
Revised: We then move to the empirical work in Section 4, where we observe and resolve the issue of robust overﬁtting in the context of the worst class.

Original: Our main empirical ﬁndings are covered in Section 5 and we conclude in Section 6.
Revised: Our main empirical ﬁndings are covered in Section 5 and we conclude in Section 6.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Adversarial examples Goodfellow et al.
Revised: Adversarial examples Goodfellow et al.

Original: ; Szegedy et al.
Revised: ; Szegedy et al.

Original: (2013) are the ﬁrst to make the important observation that deep neural networks are vulnerable to small adversarially perturbation of the input.
Revised: (2013) are the ﬁrst to make the important observation that deep neural networks are vulnerable to small adversarially perturbation of the input.

Original: Since then, there has been a growing body of literature addressing this safety critical issue, spanning from certiﬁed robust model (Raghunathan et al., 2018), distillation (Papernot et al., 2016), input augmentation (Guo et al., 2017), to adversarial training (Madry et al., 2017; Zhang et al., 2019).
Revised: Since then, there has been a growing body of literature addressing this safety critical issue, spanning from certiﬁed robust model (Raghunathan et al., 2018), distillation (Papernot et al., 2016), input augmentation (Guo et al., 2017), to adversarial training (Madry et al., 2017; Zhang et al., 2019).

Original: We focus on adversarial training in this paper.
Revised: We focus on adversarial training in this paper.

Original: While certiﬁed robustness is desirable, adversarial training remains one of the most successful defenses in practice.
Revised: While certiﬁed robustness is desirable, adversarial training remains one of the most successful defenses in practice.

Original: In a concurrent work Tian et al. (2021) independently observe the non-uniform accuracy over classes in adversarial training, further strengthening the case that lack of class-wise robustness is indeed an issue.
Revised: In a concurrent work Tian et al. (2021) independently observe the non-uniform accuracy over classes in adversarial training, further strengthening the case that lack of class-wise robustness is indeed an issue.

Original: However, they mainly focus on constructing an attack that can enlarge this disparity.
Revised: However, they mainly focus on constructing an attack that can enlarge this disparity.

Original: Minimizing the maximum Connected to our method is that of focused online learning (FOL) (Shalev-Shwartz & Wexler, 2016), which similarly takes a bandit approach, but instead re-weights the distribution over the N training examples, independent of the class label.
Revised: Minimizing the maximum Connected to our method is that of focused online learning (FOL) (Shalev-Shwartz & Wexler, 2016), which similarly takes a bandit approach, but instead re-weights the distribution over the N training examples, independent of the class label.

Original: This naturally leads to a convergence rate in terms of the number of examples N instead of the number of classes k for which usually k (cid:28) N .
Revised: This naturally leads to a convergence rate in terms of the number of examples N instead of the number of classes k for which usually k (cid:28) N .

Original: We compare in more detail theoretically and empirically in Section 3.1 and
Revised: We compare in more detail theoretically and empirically in Section 3.1 and

Original: Section 5 respectively.
Revised: Section 5 respectively.

Original: Interpolations between average and maximum loss have been considered in various other settings: for class imbalanced datasets (Lin et al., 2017), in federated learning (Li et al., 2019), and more generally the tilted empirical risk minimization (Li et al., 2020; Lee et al., 2020).
Revised: Interpolations between average and maximum loss have been considered in various other settings: for class imbalanced datasets (Lin et al., 2017), in federated learning (Li et al., 2019), and more generally the tilted empirical risk minimization (Li et al., 2020; Lee et al., 2020).

Original: Distributional robust optimization The accuracy over the worst class can be seen as a particular re-weighing the data distribution which adversarially assigns all weight to a single class.
Revised: Distributional robust optimization The accuracy over the worst class can be seen as a particular re-weighing the data distribution which adversarially assigns all weight to a single class.

Original: Worst case perturbation of the data distribution have more generally been studied under the framework of distributional robust stochastic optimization (DRO) (Ben-Tal et al., 2013; Shapiro, 2017).
Revised: Worst case perturbation of the data distribution have more generally been studied under the framework of distributional robust stochastic optimization (DRO) (Ben-Tal et al., 2013; Shapiro, 2017).

Original: Instead of attempting to minimizing the empirical risk on a training distribution P 0 , this framework considers some uncertainty set around the training distribution
Revised: Instead of attempting to minimizing the empirical risk on a training distribution P 0 , this framework considers some uncertainty set around the training distribution

Original: U ( P 0 ) and seeks to minimize the worst case risk within this set, sup Q ∈U ( P 0 )
Revised: U ( P 0 ) and seeks to minimize the worst case risk within this set, sup Q ∈U ( P 0 )

Original: E x ∼ Q
Revised: E x ∼ Q

Original: [ (cid:96) ( x )] .
Revised: [ (cid:96) ( x )] .

Original: A choice of uncertainty set, which has been given signiﬁcant attention in the community, is conditional value at risk (CVaR), which aims at minimizing the weighted average of the tail risk (Rockafellar et al., 2000).
Revised: A choice of uncertainty set, which has been given signiﬁcant attention in the community, is conditional value at risk (CVaR), which aims at minimizing the weighted average of the tail risk (Rockafellar et al., 2000; Levy et al., 2020; Kawaguchi & Lu, 2020;

Original: Recently, scalable methods for minimizing CVaR, with convergence rate independent of samples N and dimensionality d , were studied in detail for convex problems both in its primal and dual formulation (Levy et al., 2020; Kawaguchi & Lu, 2020; Fan et al., 2017). Bandit approaches has also been used to minimize CVaR by casting it as a subset selection problem in Curi et al. (2019) where they empirically demonstrate its success.
Revised: Fan et al., 2017; Curi et al., 2019).

Original: CVaR hasalso been specialized to a re-weighting over class labels, namely labeled conditional value at risk (LCVaR) (Xu et al., 2020).
Revised: CVaR has been specialized to a re-weighting over class labels, namely labeled conditional value at risk (LCVaR) (Xu et al., 2020).

Original: This was originally derived in the context of imbalanced dataset to re-balance the classes.
Revised: This was originally derived in the context of imbalanced dataset to re-balance the classes.

Original: It is still applicable in our setting and we thus provide a comparison.
Revised: It is still applicable in our setting and we thus provide a comparison.

Original: The original empirical work of (Xu et al., 2020) only considers the full-batch setting.
Revised: The original empirical work of (Xu et al., 2020) only considers the full-batch setting.

Original: We complement this by demonstrating LCVaR in a stochastic setting.
Revised: We complement this by demonstrating LCVaR in a stochastic setting.

Original: In Duchi et al.
Revised: In Duchi et al.

Original: ; Duchi & Namkoong (2018) they are, similarly to our setting, interested in uniform performance over various groups.
Revised: ; Duchi & Namkoong (2018) they are, similarly to our setting, interested in uniform performance over various groups.

Original: However, these groups are assumed to be latent subpopulations, which introduces signiﬁcant complications.
Revised: However, these groups are assumed to be latent subpopulations, which introduces signiﬁcant complications.

Original: It is thus concerned with a different setting, an example being training on a dataset implicitly consisting of multiple text corpora.
Revised: It is thus concerned with a different setting, an example being training on a dataset implicitly consisting of multiple text corpora.

Original: CFOL can also be formulated in the framework of DRO by choosing an uncertainty set that can re-weight the k class-conditional risks.
Revised: CFOL can also be formulated in the framework of DRO by choosing an uncertainty set that can re-weight the k class-conditional risks.

Original: The precise deﬁnition is given in Appendix B.1.
Revised: The precise deﬁnition is given in Appendix B.1.

Original: We further establish a direct connection between the uncertainty sets of CFOL and CVaR that we make precise in Appendix B.1, which also contains a summary of the most relevant related methods in Table 4.
Revised: We further establish a direct connection between the uncertainty sets of CFOL and CVaR that we make precise in Appendix B.1, which also contains a summary of the most relevant related methods in Table 4.

Original:  P ROBLEM FORMULATION AND PRELIMINARIES
Revised:  P ROBLEM FORMULATION AND PRELIMINARIES

Original: Notation The underlying data distribution is denoted by D with examples x ∈ R d and classes y ∈ [ k ] .
Revised: Notation The underlying data distribution is denoted by D with examples x ∈ R d and classes y ∈ [ k ] .

Original: A given iteration is characterized by t ∈ [ T ] , while p yt indicates the y th index of the t th iterate.
Revised: A given iteration is characterized by t ∈ [ T ] , while p yt indicates the y th index of the t th iterate.

Original: The indicator function is denoted with 1 { boolean } and unif( n ) indicates the uniform distribution over n elements.
Revised: The indicator function is denoted with 1 { boolean } and unif( n ) indicates the uniform distribution over n elements.

Original: An overview of the notation is provided in Appendix A.
Revised: An overview of the notation is provided in Appendix A.

Original: In classiﬁcation, we are normally interested in minimizing the population risk E ( x,y )
Revised: In classiﬁcation, we are normally interested in minimizing the population risk E ( x,y )

Original: ∼D [ (cid:96) ( θ, x, y )] over our model parameters θ ∈ R p , where (cid:96) is some loss function of θ and example x ∈ R d with an associated class y ∈ [ k ] .
Revised: ∼D [ (cid:96) ( θ, x, y )] over our model parameters θ ∈ R p , where (cid:96) is some loss function of θ and example x ∈ R d with an associated class y ∈ [ k ] .

Original: Madry et al.
Revised: Madry et al.

Original: (2017) formalized adversarially training by modifying this objective with an adversarial perturbation to each example.
Revised: (2017) formalized adversarially training by modifying this objective with an adversarial perturbation to each example.

Original: That is, we instead want to ﬁnd a parameterization θ of our predictive model which solves the following optimization problem:
Revised: That is, we instead want to ﬁnd a parameterization θ of our predictive model which solves the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where each x is now perturbed by adversarial noise δ ∈ S ⊆ R d .
Revised: where each x is now perturbed by adversarial noise δ ∈ S ⊆ R d .

Original: Common choices of S include norm-ball constraints as in Madry et al.
Revised: Common choices of S include norm-ball constraints as in Madry et al.

Original: or bounding some notion of perceptual distance (Laidlaw et al., 2020).
Revised: or bounding some notion of perceptual distance (Laidlaw et al., 2020).

Original: When the distribution over classes is uniform this is implicitly minimizing the average loss over all class.
Revised: When the distribution over classes is uniform this is implicitly minimizing the average loss over all class.

Original: This does not guarantee high accuracy for the worst class as illustrated in Figure 1, since we only know with certainty that max ≥ avg .
Revised: This does not guarantee high accuracy for the worst class as illustrated in Figure 1, since we only know with certainty that max ≥ avg .

Original: Instead, we will focus on a different objective, namely minimizing the worst class-conditioned risk :
Revised: Instead, we will focus on a different objective, namely minimizing the worst class-conditioned risk :

Original: [Equation]
Revised: [Equation]

Original: This follows the philosophy that“ a chain is only as strong as its weakest link. ”
Revised: This follows the philosophy that “ a chain is only as strong as its weakest link ”.

Original: In our particular setting it models a scenario where the attacker can adversarially choose what class the model is evaluated on.
Revised: In our particular setting it models a scenario where the attacker can adversarially choose what class the model is evaluated on.

Original: In safety critical application, such as autonomous driving, modeling even just a single class wrong can still have catastrophic consequences.
Revised: In safety critical application, such as autonomous driving, modeling even just a single class wrong can still have catastrophic consequences.

Original: As the maximum in equation 2 is a discrete maximization problem its treatment requires more care.
Revised: As the maximum in Equation 2 is a discrete maximization problem its treatment requires more care.

Original: We will take a common approach and construct a convex relaxation to this problem in Section 3. 
Revised: We will take a common approach and construct a convex relaxation to this problem in Section 3. a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h o r s e s h

Revised: i p t

Original:   M ETHOD
Revised: r u c k 0. p Class sampling distribution  M ETHOD

Original: Since we do not have access to the true distribution D , we will instead minimize over the provided empirical distribution.
Revised: Since we do not have access to the true distribution D , we will instead minimize over the provided empirical distribution.

Original: Let N y be the set of data point indices for class y such that the total number of examples is N = (cid:80) ky
Revised: Let N y be the set of data point indices for class y such that the total number of examples is N = (cid:80) ky

Original: =1 |N y | .
Revised: =1 |N y | .

Original: Then, we are interested in minimizing the maximum empirical class-conditioned risk ,
Revised: Then, we are interested in minimizing the maximum empirical class-conditioned risk ,

Original: [Equation]
Revised: [Equation]

Original: We relax this discrete problem to a continuous problem over the simplex ∆ k ,
Revised: We relax this discrete problem to a continuous problem over the simplex ∆ k ,

Original: [Equation]
Revised: [Equation]

Original: Note that equality is attained when p is a dirac on the argmax over classes.
Revised: Note that equality is attained when p is a dirac on the argmax over classes.

Original: Equation 4 leaves us with a two-player zero-sum game between the model parameters θ and the class distribution p .
Revised: Equation 4 leaves us with a two-player zero-sum game between the model parameters θ and the class distribution p .

Original: From the perspective of p it is just a linear objective, albeit adversarially picked by the model.
Revised: From the perspective of p it is just a linear objective, albeit adversarially picked by the model.

Original: This immediately let us use a no-regret algorithm under simplex constraint, namely
Revised: This immediately let us use a no-regret algorithm under simplex constraint, namely

Original: Hedge (Freund & Schapire, 1997):
Revised: Hedge (Freund & Schapire, 1997):

Original: [Equation]
Revised: [Equation]

Original:  To show convergence for (Hedge) the loss needs to satisfy certain assumptions.
Revised: Hedge [Equation] To show convergence for (Hedge) the loss needs to satisfy certain assumptions.

Original: In our case of classiﬁcation, the loss is the zero-one loss (cid:96) ( θ, x, y )
Revised: In our case of classiﬁcation, the loss is the zero-one loss (cid:96) ( θ, x, y )

Original: = 

Revised: =

Original: [ h θ ( x ) (cid:54) =
Revised: [ h θ ( x ) (cid:54) =

Original: y ] , where h θ ( · ) is the predictive model.
Revised: y ] , where h θ ( · ) is the predictive model.

Original: Hence, the loss is bounded, which is a sufﬁcient requirement.
Revised: Hence, the loss is bounded, which is a sufﬁcient requirement.

Original: Note that (Hedge) relies on zero-order information of the loss, which we indeed have available.
Revised: Note that (Hedge) relies on zero-order information of the loss, which we indeed have available.

Original: However, in the current form, (Hedge) requires so called full information over the k dimensional loss vector.
Revised: However, in the current form, (Hedge) requires so called full information over the k dimensional loss vector.

Original: In other words, we need to compute (cid:98) L y ( θ ) for all y ∈ [ k ] at every iteration.
Revised: In other words, we need to compute (cid:98) L y ( θ ) for all y ∈ [ k ] at every iteration.

Revised: When accessing the data through mini-batches, it is strictly not possible to compute even an estimate for each y if the number of classes is larger than the batch size.

Original: Following the seminal work of Auer et al.
Revised: Following the seminal work of Auer et al.

Original: (2002), we instead construct an unbiased estimator of the k dimensional loss vector (cid:98) L ( θ t ) :
Revised: (2002), we instead construct an unbiased estimator of the k dimensional loss vector (cid:98) L ( θ t ) :

Original: = ( (cid:98) L 1 ( θ t ) , ..., (cid:98)
Revised: = ( (cid:98) L 1 ( θ t ) , ..., (cid:98)

Original: L k ( θ t )) (cid:62) based on a sampled class y t from some distribution y t ∼ p t .
Revised: L k ( θ t )) (cid:62) based on a sampled class y t from some distribution y t ∼ p t .

Original: This stochastic formulation further lets us estimate the class conditioned risk (cid:98)
Revised: This stochastic formulation further lets us estimate the class conditioned risk (cid:98)

Original: L y t ( θ t ) with an unbiased sample i ∼ unif( |N y t | ) .
Revised: L y t ( θ t ) with an unbiased sample i ∼ unif( |N y t | ) .

Original: This leaves us with the following estimator, 
Revised: This leaves us with the following estimator, (cid:101)

Original: [Equation]
Revised: [Equation]

Original: otherwise 
Revised: otherwise (5)

Revised: Algorithm parameters: a step rule ModelUpdate for the model satisfying Assumption 1, adversarial step-size η > 0 , uniform mixing parameter γ = 1 / 2 , and the loss L y,i ( θ ) .

Original: {+Initialization: Set w 0 = 
+}
Revised: {+Initialization: Set w 0 = 
+}

Revised: such that q 0 and p 0 are uniform.

Revised: foreach t in 0 ..

Revised: T do y t ∼ p t ; // sample class i t ∼ unif( |N y t | ) ; /

Original: [Equation]
Revised: [Equation]

Revised: sample uniformly from class

Original: where L y,i ( θ ) :=
Revised: [Equation] // update model parameters [Equation] [Equation]

Revised: construct estimator

Revised: [Equation]

Revised: // update the adv.

Revised: class distribution

Revised: [Equation]

Revised: end foreach where L y,i ( θ ) :=

Original: max δ ∈S (cid:96) ( θ, x i + δ, y ) .
Revised: max δ ∈S (cid:96) ( θ, x i + δ, y ) .

Original: It is easy to verify that this estimator is unbiased,
Revised: It is easy to verify that this estimator is unbiased,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For ease of presentation the estimator only uses a single sample but this can trivially be extended to a mini-batch where classes are drawn i.i.d.
Revised: For ease of presentation the estimator only uses a single sample but this can trivially be extended to a mini-batch where classes are drawn i.i.d.

Original: from p t .
Revised: from p t .

Original: We could pick p t to be the learned adversarial distribution, but it is well known that this can lead to unbounded regret if some p ty is small (see Appendix A for deﬁnition of regret).
Revised: We could pick p t to be the learned adversarial distribution, but it is well known that this can lead to unbounded regret if some p ty is small (see Appendix A for deﬁnition of regret).

Original: Auer et al.
Revised: Auer et al.

Original: (2002) resolves this problem with Exp3, which instead learns a distribution q t ,but then mixes it with a uniform distribution, to eventually sample from p ty = γ 1 k + (1 − γ )
Revised: (2002) resolves this problem with Exp3, which instead learns a distribution q t , then mixes it with a uniform distribution, to eventually sample from p ty = γ 1 k + (1 − γ )

Original: q ty where γ ∈ (0 , 1) .
Revised: q ty where γ ∈ (0 , 1) .

Original: Intuitively, this enforces exploration.
Revised: Intuitively, this enforces exploration.

Original: In the general case this γ needs to be picked carefully and small enough, but similarly to Shalev-Shwartz & Wexler (2016), we show that γ = 1 / 2 sufﬁce in our setting.
Revised: In the general case this γ needs to be picked carefully and small enough, but similarly to Shalev-Shwartz & Wexler (2016), we show that γ = 1 / 2 sufﬁce in our setting.

Original: Our algorithm thus updates q t +1 with (Hedge) using the estimator (cid:101)
Revised: Our algorithm thus updates q t +1 with (Hedge) using the estimator (cid:101)

Original: L t with sample drawn from p t and subsequently plays p t +1 .
Revised: L t with sample drawn from p t and subsequently plays p t +1 .

Original: CFOL in conjunction with the simultaneous update of the minimization player can be found in algorithm 1 with an example of a learned distribution p t in Figure 2.
Revised: CFOL in conjunction with the simultaneous update of the minimization player can be found in algorithm 1 with an example of a learned distribution p t in Figure 2.

Original: Practically, the scheme bears negligible computational overhead over ERM since the softmax required to sample is of the same dimensionality as the softmax used in the forward pass through the model.
Revised: Practically, the scheme bears negligible computational overhead over ERM since the softmax required to sample is of the same dimensionality as the softmax used in the forward pass through the model.

Original: This computation is negligible in comparison with backpropagating through the entire model.
Revised: This computation is negligible in comparison with backpropagating through the entire model.

Original: For further details on the implementation we refer to Appendix C.
Revised: For further details on the implementation we refer to Appendix C.

Original: Algorithm parameters: for the model satisfying Assumption 1, adversarial step-size η > 0 , uniform mixing parameter γ = 1 / 2 , and the loss L y,i ( θ ) .

Original: [-Initialization: Set w 0 = 
-]
Revised: [-Initialization: Set w 0 = 
-]

Original: such that q 0 and p 0 are uniform.

Original: foreach t in 0 ..

Original: [-T do y t ∼ p t ; // sample class i t ∼ unif( |N y t | ) ; // sample uniformly from class θ t +
-]
Revised: [-T do y t ∼ p t ; // sample class i t ∼ unif( |N y t | ) ; // sample uniformly from class θ t +
-]

Original: =

Original: ModelUpdate( θ t , L y t ,i t ( θ t )) ; // update model parameters (cid:101)

Original: [-L ty = 
-]
Revised: [-L ty = 
-]

Original: { y = y t } L y,i t ( θ t ) /p ty

Original: ∀ y ; // construct estimator w t +1 y =

Original: w ty − η (cid:101) L ty ∀ y ; // update the adv.

Original: class distribution q t +1 y =

Original: exp (cid:0) w t +1 y (cid:1) / (cid:80) ky

Original: =1 exp (cid:0) w t +1 y (cid:1) ∀

Original: y ; p t +1 y

Original: =

Original: γ 1 k + (1 − γ ) q t +1 y ∀

Original: y ; end foreach

Original: [Figure]

Original: 3. C ONVERGENCE RATE
Revised: 3. C ONVERGENCE RATE

Original: To understand what kind of result we can expect, it is worth entertaining a hypothetical worst case scenario.
Revised: To understand what kind of result we can expect, it is worth entertaining a hypothetical worst case scenario.

Original: Imagine a classiﬁcation problem where one class is much harder to model than the remaining classes.
Revised: Imagine a classiﬁcation problem where one class is much harder to model than the remaining classes.

Original: We would expect the learning algorithm to require exposure to examples from the hard class in order to model that class appropriately—otherwise the classes would not be distinct.
Revised: We would expect the learning algorithm to require exposure to examples from the hard class in order to model that class appropriately—otherwise the classes would not be distinct.

Original: From this one can see why ERM might be slow.
Revised: From this one can see why ERM might be slow.

Original: The algorithm would naively pass over the entire dataset in order to improve the hard class using only the fraction of examples belonging that class.
Revised: The algorithm would naively pass over the entire dataset in order to improve the hard class using only the fraction of examples belonging that class.

Original: In contrast, if we can adaptively focus on the difﬁcult class, we can avoid spending time on classes that are already improved sufﬁciently.
Revised: In contrast, if we can adaptively focus on the difﬁcult class, we can avoid spending time on classes that are already improved sufﬁciently.

Original: As long as we can adapt fast enough, as expressed through the regret of the adversary, we should be able to improve on the convergence rate for the worst class.
Revised: As long as we can adapt fast enough, as expressed through the regret of the adversary, we should be able to improve on the convergence rate for the worst class.

Original: We will now make this intuition precise by establishing a high probability convergence guarantee for the worst class loss analogue to that of FOL.
Revised: We will now make this intuition precise by establishing a high probability convergence guarantee for the worst class loss analogue to that of FOL.

Original: For this we will assume that the model parameterized by θ enjoys a so called mistake bound of C (Shalev-Shwartz et al., 2011, p. 288).
Revised: For this we will assume that the model parameterized by θ enjoys a so called mistake bound of C ( -Shwartz et al., 2011, p. 288)

Original: Theproof is deferred to appendix A.
Revised: The is deferred to A

Original: Assumption 1.
Revised: Assumption 1.

Original: For any sequence of classes ( y 1 , ..., y T ) ∈
Revised: For any sequence of classes ( y 1 , ..., y T ) ∈

Original: [ k ] T and class conditioned sample indices ( i 1 , ..., i T ) with i t ∈ N y t the model enjoys the following bound for some C  < ∞  ,
Revised: [ k ] T and class conditioned sample indices ( i 1 , ..., i T ) with i t ∈ N y t the model enjoys the following bound for some C (cid:48) < ∞ and C = max { k log k, C (cid:48) } ,

Original: [Equation]
Revised: [Equation]

Original:  With this at hand we are ready to state the main theorem.
Revised: Remark 1 . The requirement C ≥ k log k will be needed to satisfy the mild step-size requirement η ≤ 2 k in Lemma 1. In most settings the smallest C (cid:48) is some fraction of the number of iterations T , which in turn is much larger than the number of classes k , so C = C (cid:48) . With this at hand we are ready to state the convergence of the worst class-conditioned empirical risk.

Original: Theorem 1.
Revised: Theorem 1.

Original: If algorithm 1 is run on bounded rewards L y t ,i t ( θ t ) ∈ [0 , 1] ∀ t with step-size η = (cid:112) log k/ (4 kC ) , mixing parameter γ = 1 / 2 and the model satisﬁes Assumption 1 for which C ≥ k log k , then after T iterations with probability at least 1 − δ ,max [Equation] [ k ]
Revised: If algorithm 1 is run on bounded rewards L y t ,i t ( θ t ) ∈ [0 , 1] ∀ t with step-size η = (cid:112) log k/ (4 kC ) , mixing parameter γ = 1 / 2 and the model satisﬁes Assumption 1, then after T iterations with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: for an ensemble of size n where t j iid ∼ unif( T ) for j ∈ [ n ] .
Revised: for an ensemble of size n where t j iid ∼ unif( T ) for j ∈ [ n ] .

Original: Remark 1 .

Original: The additional requirement C ≥ k log k is needed for satisfying the mild step-size requirement η ≤ 2 k in Lemma 1.

Original: This is the case in most settings as the mistake bound C is some fraction of the number of iterations T , which in turn is much larger than the number of classes k .

Original: To contextualize Theorem 1, let us consider the simple case of linear binary classiﬁcation mentioned in Shalev-Shwartz & Wexler (2016).
Revised: To contextualize Theorem 1, let us consider the simple case of linear binary classiﬁcation mentioned in Shalev-Shwartz & Wexler (2016).

Original: In this setting SGD needs O ( CN ) iterations to obtain a consistent hypothesis.
Revised: In this setting SGD needs O ( CN ) iterations to obtain a consistent hypothesis.

Original: In contrast, the iteration requirement of FOL decomposes into a sum (cid:101) O
Revised: In contrast, the iteration requirement of FOL decomposes into a sum (cid:101) O

Original: ( C + N ) which is much smaller since both C and N are large.
Revised: ( C + N ) which is much smaller since both C and N are large.

Original: When we are only concerned with the convergence of the worst class we show that CFOL can converge as (cid:101)
Revised: When we are only concerned with the convergence of the worst class we show that CFOL can converge as (cid:101)

Original: O ( C + k ) where usually k (cid:28) C .
Revised: O ( C + k ) where usually k (cid:28) C .

Original: Connecting this back to our motivational example in the beginning of this section, this sum decomposition exactly captures our intuition.
Revised: Connecting this back to our motivational example in the beginning of this section, this sum decomposition exactly captures our intuition.

Original: That is, adaptively focusing the class distribution can avoid the learning algorithm from needlessly going over all k classes in order to improve just one of them.
Revised: That is, adaptively focusing the class distribution can avoid the learning algorithm from needlessly going over all k classes in order to improve just one of them.

Original: From Theorem 1 we also see that we need an ensemble of size n = Ω( (cid:112) log( k/δ ) /ε 2 ) , which has only mild dependency on the number of classes k .
Revised: From Theorem 1 we also see that we need an ensemble of size n = Ω( (cid:112) log( k/δ ) /ε 2 ) , which has only mild dependency on the number of classes k .

Original: If we wanted to drive the worst class error ε to zero the dependency on ε would be problematic.
Revised: If we wanted to drive the worst class error ε to zero the dependency on ε would be problematic.

Original: However, for adversarial training, even in the best case of the CIFAR10 dataset, the average error is larger than 1 / 2 .
Revised: However, for adversarial training, even in the best case of the CIFAR10 dataset, the average error is larger than 1 / 2 .

Original: We can expect the worst class error to be even worse, so that only a small n is required.
Revised: We can expect the worst class error to be even worse, so that only a small n is required.

Original: In practice, a single model turns out to sufﬁce.
Revised: In practice, a single model turns out to sufﬁce.

Original: Note that the maximum upper bounds the average, so by minimizing this upper bound as in Theorem 1, we are implicitly still minimizing the usual average loss.
Revised: Note that the maximum upper bounds the average, so by minimizing this upper bound as in Theorem 1, we are implicitly still minimizing the usual average loss.

Original: In addition, Theorem 1 shows that a mixing parameter of γ = 1 / 2 is sufﬁcient for minimizing the worst class loss.
Revised: In addition, Theorem 1 shows that a mixing parameter of γ = 1 / 2 is sufﬁcient for minimizing the worst class loss.

Original: Effectively, the average loss is still directly being minimized, but only through half of the sampled examples.
Revised: Effectively, the average loss is still directly being minimized, but only through half of the sampled examples.

Original:  W ARMUP : R OBUST OVERFITTING
Revised:  E XPERIMENTAL SETUP : OVERCOMING ROBUST OVERFITTING

Original: Robust overﬁtting is a well-documented challenge in adversarial training
Revised: Robust overﬁtting is a well-documented challenge in adversarial training

Original: (Rice et al., 2020; Pang et al., 2020).
Revised: (Rice et al., 2020; Pang et al., 2020).

Original: This phenomenon is characterized by a rapid drop in the validation accuracy shortly after the ﬁrst piecewise step-size decay.
Revised: This phenomenon is characterized by a rapid drop in the validation accuracy shortly after the ﬁrst piecewise step-size decay.

Original: The reported models in the literature are indeed early stopped prior to the second step-size decay (e.g., in Madry et al.
Revised: The reported models in the literature are indeed early stopped prior to the second step-size decay (e.g., in Madry et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: This is particularly problematic when interested in the worst class accuracy, which might not be aligned with the average performance across training epochs.
Revised: This is particularly problematic when interested in the worst class accuracy, which might not be aligned with the average performance across training epochs.

Original: Below, we conduct preliminary experiments on CIFAR10 conﬁrming this problem and propose a solution.
Revised: Below, we conduct preliminary experiments on CIFAR10 conﬁrming this problem and propose a solution.

Original: We adopt the setupused by Zhang et al. (2019) for adversarial training with PGD.
Revised: We adopt the setup for adversarial training with PGD in Madry et al. (2017) with the stepsize scheduling from Zhang et al.

Revised: We consider adversarial training on the empirical distribution (ERM-AT) and adversarial training on the jointly learned adversarially class distribution (CFOL-AT).

Original: With ERM the worst class has only 14% robust test accuracy when early stopping based on the average accuracy.
Revised: With ERM-AT the worst class has only 14% robust test accuracy when early stopping based on the average accuracy.

Original: Early stopping based on the worst class can improve on the worst class accuracy, but only marginally, as we show in the supplementary (see Table 5).
Revised: Early stopping based on the worst class can improve on the worst class accuracy, but only marginally, as we show in the supplementary (see Table 5).

Original: This minor improvement comes at a huge cost for the average accuracy which drops from 47% to 38%.
Revised: This minor improvement comes at a huge cost for the average accuracy which drops from 47% to 38%.

Original: Ideally both metrics should instead increase monotonically.
Revised: Ideally both metrics should instead increase monotonically.

Revised: [Table]

Original: For CFOL the problem of robust overﬁtting presents itself differently.
Revised: For CFOL-AT the problem of robust overﬁtting presents itself differently.

Original: The worst class accuracy improves much more rapidly, reaching 27% even when early stopped based on the average accuracy.
Revised: The worst class accuracy improves much more rapidly, reaching 27% even when early stopped based on the average accuracy.

Original: Note that this is not surprising and is in line with the convergence rate derived in Section 3.1.
Revised: Note that this is not surprising and is in line with the convergence rate derived in Section 3.1.

Original: The implication is that we are required to run CFOL for a shorter time to get consistent performance and carefully select the step-size decay point.
Revised: The implication is that we are required to run CFOL-AT for a shorter time to get consistent performance and carefully select the step-size decay point.

Original: This limited training time, due to the earlier overﬁtting, unfortunately restricts the expressivity of the model.
Revised: This limited training time, due to the earlier overﬁtting, unfortunately restricts the expressivity of the model.

Original: We show comparison with both ERM with similar early step-size decay and also more standard step-size decay of Zhang et al.
Revised: We show comparison with both ERM-AT with similar early step-size decay and also more standard step-size decay of Zhang et al.

Original: (2019) in Table 1.
Revised: (2019) in Table 1.

Original: Although we see a signiﬁcant improvement from 14% to 27% in terms of the worst class we ideally want the method be stable for more iterations.
Revised: Although we see a signiﬁcant improvement from 14% to 27% in terms of the worst class we ideally want the method be stable for more iterations.

Original: To avoid the root problem of overﬁtting, we regularize the training objective.
Revised: To avoid the root problem of overﬁtting, we regularize the training objective.

Original: We adopt temporal ensembling (TE) which is a regularization technique that was shown to mitigate overﬁtting in adversarial training (Laine & Aila, 2016; Dong et al., 2021).
Revised: We adopt temporal ensembling (TE) which is a regularization technique that was shown to mitigate overﬁtting in adversarial training (Laine & Aila, 2016; Dong et al., 2021).

Original: This allows us to reliably run CFOL underproblem [Table] exactly the same training conﬁguration as ERM and thus prevents further hyperparameter tuning.
Revised: This allows us to reliably run CFOL-AT under exactly the same training conﬁguration as ERM-AT and thus prevents further hyperparameter tuning.

Original: Additionally, it leads to a more fair comparison with ERM, which enjoys an improvement in terms of the worst class  accuracy from 14% to 21% .
Revised: Additionally, TE leads to a more fair comparison with ERM-AT, which enjoys an improvement in terms of the worst class robust accuracy from 14% to 21% .

Original: We make use of this additional regularization in all subsequent experiments.
Revised: We make use of this additional regularization in all subsequent experiments.

Revised: In this work we focus on the robust accuracy.

Revised: However, we add a cautionary note, since we observe that TE can lead to a reduction in clean accuracy (see Table 6).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We consider the common adversarial setting where the constraint set of the attacker S is (cid:96) ∞
Revised: We consider the common adversarial setting where the constraint set of the attacker S is (cid:96) ∞

Original: -bounded attacks.
Revised: -bounded attacks.

Original: We test on three datasets with different dimensionality, number of examples per class and number of classes.
Revised: We test on three datasets with different dimensionality, number of examples per class and number of classes.

Original: Speciﬁcally, we consider CIFAR10, CIFAR100 and STL10 (Krizhevsky et al., 2009; Coates et al., 2011) (see Appendix C.2 for further details).
Revised: Speciﬁcally, we consider CIFAR10, CIFAR100 and STL10 (Krizhevsky et al., 2009; Coates et al., 2011) (see Appendix C.2 for further details).

Original: Hyper-parameters: Unless otherwise noted, we use the standard  training setup of a ResNet network (He et al., 2016) with a learning rate τ = 0 .
Revised: Hyper-parameters Unless otherwise noted, we use the standard adversarial training setup of a ResNet50 network (He et al., 2016) with a learning rate τ = 0 .

Original: 1 , momentum of 0 . 9 , weight decay of 5 · 10 − 4 , batch size of 128 and piece-wise constant weight decay of 0 . 1 at epoch 75 and 90 for a total of 100 epochs (Zhang et al., 2019), as well as regularizing with temporal ensembling as explained in Section 4.
Revised: 1 , momentum of 0 . 9 , weight decay of 5 · 10 − 4 , batch size of 128 (Madry et al., 2017) with a piece-wise constant weight decay of 0 . 1 at epoch 75 and 90 for a total of 100 epochs according to Zhang et al. (2019), as well as regularizing with temporal ensembling as explained in Section 4.

Original: For the attack we similarly adopt the common attack radius of 8 / 255 using 7 steps of projected gradient descent (PGD) with a step-size of 2 / 255 (Madry et al., 2017).
Revised: For the attack we similarly adopt the common attack radius of 8 / 255 using 7 steps of projected gradient descent (PGD) with a step-size of 2 /(Madry et al., 2017).

Original: For STL we adopt the learning rate τ = 0 .
Revised: For STL10 we adopt the learning rate τ = 0 .

Original: 01 and use a batch-size of 64 to ﬁt it inside our hardware.
Revised: 01 and use a batch-size of 64 to ﬁt it inside our hardware.

Original: For evaluation we use the stronger attack of 20 step PGD.
Revised: For evaluation we use the stronger attack of 20 step PGD.

Original: Baselines: With this setup we compare our proposed method, CFOL, against empirical risk minimization (ERM), labeled conditional value at risk (LCVaR)
Revised: Baselines With this setup we compare our proposed method, CFOL, against empirical risk minimization (ERM), labeled conditional value at risk (LCVaR)

Original: (Xu et al., 2020) and focused online learning (FOL) (Shalev-Shwartz & Wexler, 2016).
Revised: (Xu et al., 2020) and focused online learning (FOL) (Shalev-Shwartz & Wexler, 2016).

Revised: We add the sufﬁx ”AT” to all methods to indicate that the training examples are adversarially perturbed according to adversarial training of Madry et al.

Original: We consider ERM as the core baseline, while we also implement FOL and LCVarR as alternative methods that can improve the worst performing class.
Revised: We consider ERM-AT as the core baseline, while we also implement FOL-AT and LCVaR-AT as alternative methods that can improve the worst performing class.

Original: For fair comparison, and to match existing literature, we do early stopping based on the average robust accuracy on the hold-out set.
Revised: For fair comparison, and to match existing literature, we do early stopping based on the average robust accuracy on the hold-out set.

Original: More details on hyperparameters and implementation can be found in Appendix C.1 and Appendix C.3 respectively.
Revised: More details on hyperparameters and implementation can be found in Appendix C.1 and Appendix C.4 respectively.

Revised: In Table 10 we additionally provide experiments for a variant of CFOL-AT which instead reweighs the gradients.

Original: Metrics: We report the average accuracy, the worst class accuracy and the accuracy across the 20% worst classes (referred to as the 20% tail) for both clean ( A clean ) and robust accuracy ( A rob ).
Revised: Metrics We report the average accuracy, the worst class accuracy and the accuracy across the 20% worst classes (referred to as the 20% tail) for both clean ( acc clean ) and robust accuracy ( acc rob ).

Revised: The mean and standard deviation in all tables are computed over 5 independent executions.

Revised: We note that the aim is not to be state-of-the-art but rather provide a fair comparison between the methods.

Original: The ﬁrst core experiment is conducted on CIFAR10.
Revised: The ﬁrst core experiment is conducted on CIFAR10.

Original: In Table 2 the quantitative results are reported with the accuracy per class illustrated in Figure 3.
Revised: In Table 2 the quantitative results are reported with the accuracy per class illustrated in Figure 3.

Original: The results reveal that all methods other than ERM improve the worst performing class with CFOL obtaining higher accuracy in the weakest class than all methods in both the clean and the robust case.
Revised: The results reveal that all methods other than ERM-AT improve the worst performing class with CFOL-AT obtaining higher accuracy in the weakest class than all methods in both the clean and the robust case.

Original: Interestingly, FOL performs better than LCVaR, so for the next experiments we simply consider FOL as the non-ERM comparison method.
Revised: Interestingly, FOL-AT performs better than LCVaR-AT, so for the next experiments we simply consider FOL as the non-ERM comparison method.

Original: The results for the remaining datasets can be found in Table 3.
Revised: The results for the remaining datasets can be found in Table 3.

Original: The results exhibit similar patterns to the experiment on CIFAR10, where CFOL improves both the worst performing class and the 20% tail.
Revised: The results exhibit similar patterns to the experiment on CIFAR10, where CFOL-AT improves both the worst performing class and the 20% tail.

Original: In the supplementary, we also provide results for early stopped models using the best worst class accuracy from the validation set (see Table 6).
Revised: In the supplementary, we also provide results for early stopped models using the best worst class accuracy from the validation set (see Table 7), larger attack radius (see Table 8), and a different test time attack (see Table 9).

Original: In all cases, we observe that CFOL has consistently improved accuracy with respect to the weakest class (and the 20% tail).
Revised: In all cases, we observe that CFOL-AT has consistently improved accuracy with respect to the weakest class (and the 20% tail).

Original: We note that this modiﬁcation is not sufﬁcient to make ERM competitive with CFOL.
Revised: We note that early stopping based on the worst class is not sufﬁcient to make ERM-AT competitive with CFOL-AT.

Original: We ﬁnd that CFOL consistently improves the accuracy for both the worst class and the 20% tail across the three datasets.
Revised: We ﬁnd that CFOL-AT consistently improves the accuracy for both the worst class and the 20% tail across the three datasets.

Original: At the same time, the average performance only suffers a minor reduction.
Revised: At the same time, the average performance only suffers a minor reduction.

Original: The improvement seem to be even more noticeable in terms of the clean accuracy.
Revised: The improvement seem to be even more noticeable in terms of the clean accuracy.

Original: Interestingly,
Revised: Interestingly,

Original: FOL improves even the average accuracy on CIFAR10 while having small variance, suggestion that sometimes focusing on the worst cases can improve the average.
Revised: FOL-AT improves even the average accuracy on CIFAR10 while having small variance, suggestion that sometimes focusing on the worst cases can improve the average.

Original: [Table]
Revised: CFOL-AT signiﬁcantly improves the robust accuracy for both the worst class and the 20% tail, while only incurring a small reduction in the average robust accuracy in comparison with ERM-AT. ERM-AT CFOL-AT LCVaR-AT

Revised: FOL-AT

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: test accuracy for CFOL and ERM over classes.
Revised: [Figure] [Table] [Figure] [Table]

Original: The horizontal black and red line depicts the average and worst class accuracy over the classes respectively.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: The improvement in the minimum accuracy is noteable when using CFOL, while there is only marginal difference in the average accuracy.
Revised: [Figure] [Table] [Figure] provement in the minimum accuracy is notable when using CFOL-AT, while there is only marginal difference in the average accuracy.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we have introduced a method for class focused online learning (CFOL), which samples from an adversarial learned distribution over classes.
Revised: In this work, we have introduced a method for class focused online learning (CFOL), which samples from an adversarial learned distribution over classes.

Original: We establish high probability convergence results of the worst class for CFOL through a specialized regret analysis.
Revised: We establish high probability convergence results of the worst class for CFOL through a specialized regret analysis.

Original: In the context of adversarial examples this is motivated by an adversarial threat model in which the attacker chooses what class to evaluate on in addition to the perturbation.
Revised: In the context of adversarial examples this is motivated by an adversarial threat model in which the attacker chooses what class to evaluate on in addition to the perturbation.

Original: We conduct a thorough empirical validation on three datasets.
Revised: We conduct a thorough empirical validation on three datasets.

Original: The empirical results for adversarial training consistency demonstrate the improvement over the weakest classes.
Revised: The empirical results for adversarial training consistently demonstrate the improvement over the weakest classes.

Revised: The work opens up for multiple interesting research avenues.

Original: Our method can also be applied in non-adversarial settings and in the future we intend to look further into the challenging cases of non-uniform distribution over classes.
Revised: Firstly, our method can also be applied in non-adversarial settings, and in the future we intend to look further into the challenging cases of non-uniform distribution over classes.

Revised: Secondly, it remains open to establish generalization bounds for CFOL.

Revised: Finally, it is interesting to understand why adversarial perturbations leads to inhomogeneous accuracies over classes.

Original: A C ONVERGENCE ANALYSIS A. P RELIMINARY AND NOTATION
Revised: A C ONVERGENCE ANALYSIS A. P RELIMINARY AND NOTATION

Original: Consider the abstract online learning problem where for every t ∈ [ T ] the player chooses an action x t and subsequently the environment reveals the loss function l ( · , y t ) .
Revised: Consider the abstract online learning problem where for every t ∈ [ T ] the player chooses an action x t and subsequently the environment reveals the loss function l ( · , y t ) .

Original: As traditional in the online learning literature we will measure performance in terms of regret, which compares our sequence of choices { x t } Tt =1 with a ﬁxed strategy in hindsight u ,
Revised: As traditional in the online learning literature we will measure performance in terms of regret, which compares our sequence of choices { x t } Tt =1 with a ﬁxed strategy in hindsight u ,

Original: [Equation]
Revised: [Equation]

Original: If we, instead of minimizing over losses l , maximize over rewards r we deﬁne regret as,
Revised: If we, instead of minimizing over losses l , maximize over rewards r we deﬁne regret as,

Original: [Equation]
Revised: [Equation]

Original: When we allow randomized strategy, as in the bandit setting, R T becomes a random variable that we wish to upper bound with high probability.
Revised: When we allow randomized strategy, as in the bandit setting, R T becomes a random variable that we wish to upper bound with high probability.

Original: For convenience we include an overview over the notation deﬁned and used in Section 2 and Section 3 below.
Revised: For convenience we include an overview over the notation deﬁned and used in Section 2 and Section 3 below.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Number of iterations
Revised: Number of iterations

Original: [Equation]
Revised: [Equation]

Original: The y th index of the t th iterate
Revised: The y th index of the t th iterate

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The indicator function
Revised: The indicator function

Original: [Equation]
Revised: [Equation]

Original: Discrete uniform distribution over n elements
Revised: Discrete uniform distribution over n elements

Original: [Equation]
Revised: [Equation]

Original: Set of data point indices for class y ∈ [ k ]
Revised: Set of data point indices for class y ∈ [ k ]

Original: [Equation]
Revised: [Equation]

Original: The size of the data set
Revised: The size of the data set

Original: [Equation]
Revised: [Equation]

Original: Loss on a particular example
Revised: Loss on a particular example

Original: [Equation]
Revised: [Equation]

Original: The empirical class-conditioned risk
Revised: The empirical class-conditioned risk

Original: [Equation]
Revised: [Equation]

Original: The vector of all empirical class-conditioned risks
Revised: The vector of all empirical class conditioned risks

Original: L ty :=
Revised: L ty :=

Original: L y,i ( θ t )
Revised: L y,i ( θ t )

Original: Class-conditioned estimator at iteration t with i ∼ unif( |N y | )
Revised: Class-conditioned estimator at iteration t with i ∼ unif( |N y | )

Original: A. C ONVERGENCE RESULTS
Revised: A. C ONVERGENCE RESULTS

Original: We restate algorithm 1 while leaving out the details of the classiﬁer for convenience.
Revised: We restate Algorithm 1 while leaving out the details of the classiﬁer for convenience.

Original: Initialize wsuch that q 0 and p 0 are uniform.
Revised: Initialize wsuch that q 0 and p 0 are uniform.

Original: Then Exp3 proceeds for every t ∈ [ T ] as follows:
Revised: Then Exp3 proceeds for every t ∈ [ T ] as follows:

Original: Draw class y t ∼ p t 2.
Revised: Draw class y t ∼ p t 2.

Original: Observe a scalar reward L ty t 3.
Revised: Observe a scalar reward L ty t 3.

Original: Construct estimator (cid:101)
Revised: Construct estimator (cid:101)

Original: L ty =
Revised: L ty =

Original: L ty t 1 { y
Revised: L ty t 1 { y

Original: [Equation]
Revised: [Equation]

Original: y t } /p ty
Revised: y t } /p ty

Original: [Equation]
Revised: [Equation]

Original: y 4.
Revised: y 4.

Original: Update distribution
Revised: Update distribution

Original: [Equation]
Revised: [Equation]

Original: We can bound the regret of Exp3 (Auer et al., 2002) in our setting, even when the mixing parameter γ is not small as otherwise usually required, by following a similar argument as Shalev-Shwartz &
Revised: We can bound the regret of Exp3 (Auer et al., 2002) in our setting, even when the mixing parameter γ is not small as otherwise usually required, by following a similar argument as Shalev-Shwartz &

Original: Wexler (2016).
Revised: Wexler (2016).

Original: For this we will use the relationship between p and q throughout.
Revised: For this we will use the relationship between p and q throughout.

Original: From p y = γk
Revised: From p y = γk

Original: + (1 − γ )
Revised: + (1 − γ )

Original: q y it can easily be veriﬁed that,
Revised: q y it can easily be veriﬁed that,

Original: [Equation]
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: If Exp3 is run on bounded rewards L ty t ∈ [0 , 1] ∀ t with η ≤ γ/k then
Revised: If Exp3 is run on bounded rewards L ty t ∈ [0 , 1] ∀ t with η ≤ γ/k then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We can write the estimator vector as (cid:101)
Revised: We can write the estimator vector as (cid:101)

Original: L t =
Revised: L t =

Original: L tyt p tyt e y t where e y t is a canonical basis vector.
Revised: L tyt p tyt e y t where e y t is a canonical basis vector.

Original: By assumption we have L ty t ∈ [0 , 1] .
Revised: By assumption we have L ty t ∈ [0 , 1] .

Original: Combining this with the bound on 1 / p y in equation 11 we have (cid:101)
Revised: Combining this with the bound on 1 / p y in Equation 11 we have (cid:101)

Original: L ty ≤ k / γ for all y .
Revised: L ty ≤ k / γ for all y .

Original: This lets us invoke (Shalev-Shwartz et al., 2011, Thm 2.22) as long as the step-size η is small enough.
Revised: This lets us invoke (Shalev-Shwartz et al., 2011, Thm 2.22) as long as the step-size η is small enough.

Original: Speciﬁcally, we have that if η ≤ 1 / (cid:101)
Revised: Speciﬁcally, we have that if η ≤ 1 / (cid:101)

Original: L ty ≤ γ / k
Revised: L ty ≤ γ / k

Original: [Equation]
Revised: [Equation]

Original: By expanding (cid:101)
Revised: By expanding (cid:101)

Original: L ty and applying the inequalities from equation 11 we can get rid of the dependency on q and p ,
Revised: L ty and applying the inequalities from Equation 11 we can get rid of the dependency on q and p ,

Original: [Equation]
Revised: [Equation]

Original: where the last line uses the boundedness assumption L ty ∈ [0 , 1] ∀ y .
Revised: where the last line uses the boundedness assumption L ty ∈ [0 , 1] ∀ y .

Original: [Equation]
Revised: [Equation]

Original: It is apparent that we need to control the sum of losses.
Revised: It is apparent that we need to control the sum of losses.

Original: We will do so by assuming that the model admits a mistake bound as in Shalev-Shwartz & Wexler (2016); Shalev-Shwartz et al.
Revised: We will do so by assuming that the model admits a mistake bound as in Shalev-Shwartz & Wexler (2016); Shalev-Shwartz et al.

Original: Assumption 1.
Revised: Assumption 1.

Original: For any sequence of classes ( y 1 , ..., y T ) ∈
Revised: For any sequence of classes ( y 1 , ..., y T ) ∈

Original: [ k ] T and class conditioned sample indices ( i 1 , ..., i T ) with i t ∈ N y t the model enjoys the following bound for some C  < ∞ ,
Revised: [ k ] T and class conditioned sample indices ( i 1 , ..., i T ) with i t ∈ N y t the model enjoys the following bound for some C (cid:48) < ∞ and

Original: [Equation]
Revised: [Equation]

Original: Recall that we need to bound L t y : =
Revised: Recall that we need to bound L ty :=

Original: L y,i t ( θ t ) , where y are picked adversarially and i t are sampled uniformly, so the above is sufﬁcient.
Revised: L y,i t ( θ t ) , where y are picked adversarially and i t are sampled uniformly, so the above is sufﬁcient.

Original: Lemma 2.
Revised: Lemma 2.

Original: If the model satisﬁes Assumption 1for which C ≥ k log k and algorithm 1 is run on bounded rewards L y t ,i t ( θ t ) ∈ [0 , 1] ∀ t with step-size η = (cid:112) log k/ (4 kC ) and mixing coefﬁcient γ = 1 / 2 then 
Revised: If the model satisﬁes Assumption 1 and algorithm 1 is run on bounded rewards L y t ,i t ( θ t ) ∈ [0 , 1] ∀ t with step-size η = (cid:112) log k/ (4 kC ) and mixing coefﬁcient γ = 1 / 2 then T [Equation]

Revised: u

Original: [Equation]
Revised: [Equation]

Revised: ≤ 6 C.

Original: Proof.
Revised: Proof.

Original: By expanding and rearranging Lemma 1,
Revised: By expanding and rearranging Lemma 1,

Original: [Equation]
Revised: [Equation]

Original: as long as η ≤ γ / k .
Revised: as long as η ≤ γ / k .

Original: Then by using our favorite inequality in equation 11 and under Assumption 1, log k
Revised: Then by using our favorite inequality in Equation 11 and under Assumption 1, log k

Original: [Equation]
Revised: [Equation]

Original: To maximize (1 − γ ) γ ) we now pick γ = 1 / 2 so that,
Revised: To maximize (1 − γ ) γ ) we now pick γ = 1 / 2 so that,

Original: [Equation]
Revised: [Equation]

Original: For η , notice that it appears in two of the terms.
Revised: For η , notice that it appears in two of the terms.

Original: To minimize the bound respect to η we pick η = (cid:112) log k/ 
Revised: To minimize the bound respect to η we pick η = (cid:112) log k/ (4 kC )

Original: [Equation]

Original: 4 kC ) such that log k η
Revised: such that log k η

Original: [Equation]
Revised: [Equation]

Original: , which leaves us with,
Revised: , which leaves us with,

Original: [Equation]
Revised: [Equation]

Original: From the ﬁrst term it is clear that if C ≥ k log k  then the original step-size requirement of η ≤ 1 / 2 k is satisﬁes.
Revised: From the ﬁrst term it is clear that since C ≥ k log k by assumption then the original step-size requirement of η ≤ 1 / 2 k is satisﬁes.

Original: In this case we can additionally simplify further,
Revised: In this case we can additionally simplify further,

Original: [Equation]
Revised: [Equation]

Original: This still only gives us a bound on a stochastic object.
Revised: This still only gives us a bound on a stochastic object.

Original: We will now relate it to the empirical class conditional risk (cid:98) L y ( θ ) by using standard concentration bounds.
Revised: We will now relate it to the empirical class conditional risk (cid:98) L y ( θ ) by using standard concentration bounds.

Original: To be more precise, we want to show that by picking u = e y in (cid:104) u, (cid:101)
Revised: To be more precise, we want to show that by picking u = e y in (cid:104) u, (cid:101)

Original: L t (cid:105) we concentrate to (cid:98) L y ( θ ) .
Revised: L t (cid:105) we concentrate to (cid:98) L y ( θ ) .

Original: Following Shalev-Shwartz &
Revised: Following Shalev-Shwartz &

Original: Wexler (2016)
Revised: Wexler (2016)

Original: we adopt their use of a Bernstein’s type inequality.
Revised: we adopt their use of a Bernstein’s type inequality.

Original: Lemma 3 (e.g.
Revised: Lemma 3 (e.g.

Original: Audibert et al.
Revised: Audibert et al.

Original: (2010, Thm.
Revised: (2010, Thm.

Original: Let A 1 , ..., A T be a martingale difference sequence with respect to a Markovian sequence B 1 , ..., B T and assume | A t | ≤ V and E (cid:2)
Revised: Let A 1 , ..., A T be a martingale difference sequence with respect to a Markovian sequence B 1 , ..., B T and assume | A t | ≤ V and E (cid:2)

Original: A 2 t | B 1 , . . .
Revised: A 2 t | B 1 , . . .

Original: , B t (cid:3) ≤ s .
Revised: , B t (cid:3) ≤ s .

Original: Then for any δ ∈ (0 , 1) ,
Revised: Then for any δ ∈ (0 , 1) ,

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: t=
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: If algorithm 1 is run on bounded rewards L y t ,i t ( θ t ) ∈ [0 , 1] ∀ t with step-size η = (cid:112) log k/ (4 kC ) , mixing coefﬁcient γ = 1 / 2 and the model satisﬁes Assumption 1 for which C ≥ k log k , then for any y ∈
Revised: If algorithm 1 is run on bounded rewards L y t ,i t ( θ t ) ∈ [0 , 1] ∀ t with step-size η = (cid:112) log k/ (4 kC ) , mixing coefﬁcient γ = 1 / 2 and the model satisﬁes Assumption 1, then for any y ∈

Original: [ k ] we obtain have the following bound with probability at least 1 − δ / k ,
Revised: [ k ] we obtain have the following bound with probability at least 1 − δ / k ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Pick any y ∈ [ k ] and let u = e y in (cid:104) u, (cid:101)
Revised: Pick any y ∈ [ k ] and let u = e y in (cid:104) u, (cid:101)

Original: L t (cid:105) .
Revised: L t (cid:105) .

Original: By construction the following deﬁnes a martingale difference sequence,
Revised: By construction the following deﬁnes a martingale difference sequence,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In particular note that i t is uniformly sampled.
Revised: In particular note that i t is uniformly sampled.

Original: To apply the Bernstein’s type inequality we just need to bound | A t | and E (cid:2)
Revised: To apply the Bernstein’s type inequality we just need to bound | A t | and E (cid:2)

Original: A 2 t | q t , θ t (cid:3) .
Revised: A 2 t | q t , θ t (cid:3) .

Original: For | A t | we can crudely bound it as,
Revised: For | A t | we can crudely bound it as,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: To bound the variance observe that we have the following:
Revised: To bound the variance observe that we have the following:

Original: [Equation]
Revised: [Equation]

Original: (with i ∼unif( |N y (cid:48) |
Revised: (with i ∼

Original: [Equation]
Revised: [Equation]

Original: ∀ y (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: equation 11)

Original: It follows that E (cid:2)
Revised: It follows that E (cid:2)

Original: A 2 t | q t , θ t (cid:3) ≤ k / γ
Revised: A 2 t | q t , θ t (cid:3) ≤ k / γ

Original: Invoking Lemma 3 we get the following bound with probability at least 1 − δ / k , 
Revised: Invoking Lemma 3 we get the following bound with probability at least 1 − δ / k , [Equation]

Original: [Equation]
Revised: [Equation]

Original: By bounding the ﬁrst term on the right hand side with Lemma 2 and taking γ = 1 / 2 we obtain,
Revised: By bounding the ﬁrst term on the right hand side with Lemma 2 and taking γ = 1 / 2 we obtain,

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: [Equation]
Revised: [Equation]

Original: We are now ready to state the main theorem.
Revised: We are now ready to state the main theorem.

Original: Theorem 1.
Revised: Theorem 1.

Original: If algorithm 1 is run on bounded rewards L y t ,i t ( θ t ) ∈ [0 , 1] ∀ t with step-size η = (cid:112) log k/ (4 kC ) , mixing parameter γ = 1 / 2 and the model satisﬁes Assumption 1 for which C ≥ k log k , then after T iterations with probability at least 1 − δ , max y ∈
Revised: If algorithm 1 is run on bounded rewards L y t ,i t ( θ t ) ∈ [0 , 1] ∀ t with step-size η = (cid:112) log k/ (4 kC ) , mixing parameter γ = 1 / 2 and the model satisﬁes Assumption 1, then after T iterations with probability at least 1 − δ , max y ∈

Original: [ k ]
Revised: [ k ]

Original: [Equation]
Revised: [Equation]

Original: for an ensemble of size n where t j iid ∼ unif( T ) for j ∈ [ n ] .
Revised: for an ensemble of size n where t j iid ∼ unif( T ) for j ∈ [ n ] .

Original: Proof.
Revised: Proof.

Original: If we ﬁx y and let t j iid ∼ unif( T ) , then the following is a martingale difference sequence,
Revised: If we ﬁx y and let t j iid ∼ unif( T ) , then the following is a martingale difference sequence,

Original: [Equation]
Revised: [Equation]

Original: for which it is easy to see that | A j | ≤ 2 and A 2 j ≤ 1 given boundedness of the loss.
Revised: for which it is easy to see that | A j | ≤ 2 and A 2 j ≤ 1 given boundedness of the loss.

Original: This readily let us apply Lemma 3 with high probability 1 − δ / 2 k .
Revised: This readily let us apply Lemma 3 with high probability 1 − δ / 2 k .

Original: Combining this with the bound of Lemma 4 with probability 1 − δ / 2 k by using a union bound completes the proof.
Revised: Combining this with the bound of Lemma 4 with probability 1 − δ / 2 k by using a union bound completes the proof.

Original: [Equation]
Revised: [Equation]

Original: B CV A R
Revised: B CV A R

Original: Conditional value at risk (CVaR) is the expected loss conditioned on being larger than the (1 − α ) - quantile.
Revised: Conditional value at risk (CVaR) is the expected loss conditioned on being larger than the (1 − α ) - quantile.

Original: This has a distributional robust interpretation which for a discrete distribution can be written as,
Revised: This has a distributional robust interpretation which for a discrete distribution can be written as,

Original: [Equation]
Revised: [Equation]

Original: The optimal p of the above problem places uniform mass on the tail.
Revised: The optimal p of the above problem places uniform mass on the tail.

Original: Practically we can compute this best response by sorting the losses { (cid:96) ( θ, x i ) } i in descending order and assigning 1 αm mass per index until saturation.
Revised: Practically we can compute this best response by sorting the losses { (cid:96) ( θ, x i ) } i in descending order and assigning 1 αm mass per index until saturation.

Original: Primal CVaR When m is large a stochastic variant is necessary.
Revised: Primal CVaR When m is large a stochastic variant is necessary.

Original: To obtain a stochastic subgradient for the model parameter, the naive approach is to compute a stochastic best response over a minibatch.
Revised: To obtain a stochastic subgradient for the model parameter, the naive approach is to compute a stochastic best response over a minibatch.

Original: This has been studied in detail in (Levy et al., 2020).
Revised: This has been studied in detail in (Levy et al., 2020).

Original: Dual CVaR
Revised: Dual CVaR

Original: An alternative formulation relies on strong duality of CVaR originally showed in (Rockafellar et al., 2000),
Revised: An alternative formulation relies on strong duality of CVaR originally showed in (Rockafellar et al., 2000),

Original: [Equation]
Revised: [Equation]

Original: [Equation] In practice, one approach is to jointly minimize λ and the model parameters by computing stochastic gradients as done in (Curi et al., 2019).
Revised: In practice, one approach is to jointly minimize λ and the model parameters by computing stochastic gradients as done in (Curi et al., 2019).

Original: Alternatively, we can ﬁnd a close form solution for λ under a mini-batch (see e.g. (Xu et al., 2020, Appendix I.1)).
Revised: Alternatively, we can ﬁnd a close form solution for λ under a mini-batch (see e.g. (Xu et al., 2020, Appendix I.1)).

Original: In comparison CFOL acts directly on the probability distribution similarly to primal CVaR.
Revised: In comparison CFOL acts directly on the probability distribution similarly to primal CVaR.

Original: However, instead of ﬁnding a best response on a uniformly sampled mini-batch it updates the weights iteratively and samples accordingly (see algorithm 1).
Revised: However, instead of ﬁnding a best response on a uniformly sampled mini-batch it updates the weights iteratively and samples accordingly (see algorithm 1).

Original: Despite this difference, it is interesting that a direct connection can be established between the uncertainty sets of the methods.
Revised: Despite this difference, it is interesting that a direct connection can be established between the uncertainty sets of the methods.

Original: The following section is dedicated to this.
Revised: The following section is dedicated to this.

Original: To be pedantic it is worth pointing out a minor discrepancy when applying CVaR in the context of deep learning.
Revised: To be pedantic it is worth pointing out a minor discrepancy when applying CVaR in the context of deep learning.

Original: Common architectures such as ResNets incorporates batch normalization which updates the running mean and variance on the forward pass.
Revised: Common architectures such as ResNets incorporates batch normalization which updates the running mean and variance on the forward pass.

Original: The implication is that the entire minibatch is used to update these statistics despite the gradient computation only relying on the worst subset.
Revised: The implication is that the entire minibatch is used to update these statistics despite the gradient computation only relying on the worst subset.

Original: This makes implementation in this setting slightly more convoluted.
Revised: This makes implementation in this setting slightly more convoluted.

Original: B. R ELATIONSHIP WITH CV A R
Revised: B. R ELATIONSHIP WITH CV A R

Original: Exp3 is run with a uniform mixing to enforce exploration.
Revised: Exp3 is run with a uniform mixing to enforce exploration.

Original: This turns out to imply the necessary and sufﬁcient condition for CVaR.
Revised: This turns out to imply the necessary and sufﬁcient condition for CVaR.

Original: We make this precise in the following lemma:
Revised: We make this precise in the following lemma:

Original: Lemma 5.
Revised: Lemma 5.

Original: Let p be a uniform mixing p = γ 1 m + (1 − γ ) q with an arbitrary distribution q ∈ ∆ m and (cid:15) ∈ [0 , 1] such that the distribution has a lower bound on each element p i ≥ γ 1 m .
Revised: Let p be a uniform mixing p = γ 1 m + (1 − γ ) q with an arbitrary distribution q ∈ ∆ m and (cid:15) ∈ [0 , 1] such that the distribution has a lower bound on each element p i ≥ γ 1 m .

Original: Then an upper bound is implicitly implied on each element, (cid:107) p (cid:107) ∞ := max i =[ m ] p i ≤ 1 − ( m − 1) γ m .
Revised: Then an upper bound is implicitly implied on each element, (cid:107) p (cid:107) ∞ := max i =[ m ] p i ≤ 1 − ( m − 1) γ m .

Original: Proof.
Revised: Proof.

Original: Consider p i for any i .
Revised: Consider p i for any i .

Original: At least ( m − 1) γm of the mass must be on other components so p i ≤ 1 − ( m − 1) γ m .
Revised: At least ( m − 1) γm of the mass must be on other components so p i ≤ 1 − ( m − 1) γ m .

Original: The result follows.
Revised: The result follows.

Original: [Equation]
Revised: [Equation]

Original: Corollary 1.
Revised: Corollary 1.

Original: Consider the uncertainty set of Exp3,
Revised: Consider the uncertainty set of Exp3,

Original: [Equation]
Revised: [Equation]

Original: Given that the above lower bound is only introduced for practical reasons, we might as well consider an instantiation of CVaR which turns out to be a proper relaxation,
Revised: Given that the above lower bound is only introduced for practical reasons, we might as well consider an instantiation of CVaR which turns out to be a proper relaxation,

Original: [Equation]
Revised: [Equation]

Original: with α =
Revised: with α =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: γ .
Revised: γ .

Original: This leaves us with the following primal formulation,
Revised: This leaves us with the following primal formulation,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Lemma 5 and since CVaR requires (cid:107) p (cid:107) ∞ ≤ 1 αm we have,
Revised: From Lemma 5 and since CVaR requires (cid:107) p (cid:107) ∞ ≤ 1 αm we have,

Original: [Equation]
Revised: [Equation]

Original: By simple algebra we have,
Revised: By simple algebra we have,

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: [Equation]
Revised: [Equation]

Original: So if the starting point of the uncertainty set is the simplex and the uniform mixing in Exp3 is therefore only for tractability reasons, then we might as well minimize the CVaR objective instead.
Revised: So if the starting point of the uncertainty set is the simplex and the uniform mixing in Exp3 is therefore only for tractability reasons, then we might as well minimize the CVaR objective instead.

Original: This could even potentially lead to a more robust solution as the uncertainty set is larger (since the upper bound in U CVaR does not imply the lower bound in U Exp3 ).
Revised: This could even potentially lead to a more robust solution as the uncertainty set is larger (since the upper bound in U CVaR does not imply the lower bound in U Exp3 ).

Original: There are two things to keep in mind though.
Revised: There are two things to keep in mind though.

Original: First, α should not be too small since the optimization problem gets harder.
Revised: First, α should not be too small since the optimization problem gets harder.

Original: It is informative to consider the case where γ = 1 / 2 such that α = 2 m +1 .
Revised: It is informative to consider the case where γ = 1 / 2 such that α = 2 m +1 .

Original: From this it becomes clear that the recasting as CVaR only works for small m .
Revised: From this it becomes clear that the recasting as CVaR only works for small m .

Original: Secondly, despite the uncertainty sets being related, the training dynamics, and thus the obtained solution, can be drastically different, as we also observe experimentally in Section 5.
Revised: Secondly, despite the uncertainty sets being related, the training dynamics, and thus the obtained solution, can be drastically different, as we also observe experimentally in Section 5.

Original: For instance CVaR is known
Revised: For instance CVaR is known

Original: The discrete distribution p either governs the distribution over all N samples or over the depending
Revised: The discrete distribution p either governs the distribution over all N samples or over the depending

Original: Uncertainty set
Revised: Uncertainty set

Original: [Table]
Revised: [Table]

Original: for having high variance (Curi et al., 2019) while the uniform mixing in CFOL prevents this.
Revised: for having high variance (Curi et al., 2019) while the uniform mixing in CFOL prevents this.

Original: It is worth noting that despite this uniform mixing we are still able to show convergence for CFOL in terms of the worst class in Section 3.1.
Revised: It is worth noting that despite this uniform mixing we are still able to show convergence for CFOL in terms of the worst class in Section 3.1.

Original: In Table 4 we provide an overview of the different methods induced by the choice of uncertainty set. C E XPERIMENTS
Revised: In Table 4 we provide an overview of the different methods induced by the choice of uncertainty set

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: Classes 0. R o b u s t t e s t a cc u r a c y ERM Classes 0.
Revised: C E XPERIMENTS

Original: R o b u s t t e s t a cc u r a c y

Original: CFOL a i r p l a n e b i r d c a r c a t d e e r d o g h

Original: o r s e m o n k e y s h

Original: [Figure]
Revised: [Table]

Original: p

Original: [Figure]
Revised: [Figure]

Original: u[Figure] [Figure] R o b u s t t e s t acc u
Revised: u s t t e s t a

Original: [Figure]
Revised: [Table]

Original: y

Original: ERM a i r p l a n e b i r d c a [Figure] c a t d e e r d o g h o r
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: m o n k e

Original: y s h i p t [Figure] u c
Revised: [Table]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: R

Original: [Figure] b u s t t e s t a cc u [Figure]
Revised: [Table] [Figure]

Original: c y
Revised: [Table] [Figure] [Table] [Figure]

Original: CFOL
Revised: [Table] [Figure] [Table] [Table]

Original: ERM
Revised: [Figure] [Figure] [Table] [Table]

Original: CFOL
Revised: [Table] and that the immediate equivalence we get is only in expectation. We test the reweighted variant of CFOL-AT on CIFAR10 and observe similar results as for CFOL-AT. [Table]

Original: FOL
Revised: CFOL-AT (reweighted)

Original: [Table]
Revised: [Table]

Original: C
Revised: C. H YPERPARAMETERS

Original: H YPERPARAMETERS

Original: In this section we provide additional details to the hyperparameters speciﬁed in Section 5.
Revised: In this section we provide additional details to the hyperparameters speciﬁed in Section 5.

Original: For all experiment we use data augmentation in the form of random cropping, random horizontal ﬂip, color jitter and 2 degrees random rotations.
Revised: For all experiment we use data augmentation in the form of random cropping, random horizontal ﬂip, color jitter and 2 degrees random rotations.

Original: For temporal ensembling (TE) we use the parameters in (Dong et al., 2021) and set the momentum to 0 .
Revised: For temporal ensembling (TE) we use the parameters in (Dong et al., 2021) and set the momentum to 0 .

Original: 9 , balancing weight parameter to 30 along a Gaussian ramp-up curve until epoch 50 .
Revised: 9 , balancing weight parameter to 30 along a Gaussian ramp-up curve until epoch 50 .

Revised: See Appendix C.3 for precise deﬁnition of the regularization technique.

Original: For CFOL we used the adversarial step-size η = 5 · 10 − 6 across all datasets and the same parameters as for ERM described in Section 5.
Revised: For CFOL-AT we used the adversarial step-size η = 5 · 10 − 6 across all datasets and the same parameters as for ERM-AT described in Section 5.

Original: CFOL seems to be reasonable robust to step-size choice as seen in Table 7.
Revised: CFOL-AT seems to be reasonable robust to stepsize choice as seen in Table 11.

Original: Similarly for FOL we use the same parameters as for ERM and set η = 5 · 10 − 7 after optimizing based on the worst class robust accuracy.
Revised: Similarly for FOL-AT we use the same parameters as for ERM-AT and set η = 5 · 10 − 7 after optimizing based on the worst class robust accuracy.

Original: With both methods we make an exception with STL10, where, due to the fewer iterations induced by the 5 times smaller dataset, we scale the adversarial step-size linearly by the dataset ratio 5 , such that an equally strong attack can be obtained.
Revised: With both methods we make an exception with STL10, where, due to the fewer iterations induced by the 5 times smaller dataset, we scale the adversarial step-size linearly by the dataset ratio 5 , such that an equally strong attack can be obtained.

Original: The  step-size picked for both FOL and CFOL is smaller in practice than theory suggests.
Revised: The adversarial step-size picked for both FOL-AT and CFOL-AT is smaller in practice than theory suggests.

Original: This suggests that the mistake bound in Assumption 1 is not satisﬁed for any sequence.
Revised: This suggests that the mistake bound in Assumption 1 is not satisﬁed for any sequence.

Original: Instead we rely on the sampling process to be only mildly adversarial initially as implicitly enforced by the small adversarial step-size.
Revised: Instead we rely on the sampling process to be only mildly adversarial initially as implicitly enforced by the small adversarial step-size.

Original: It is an interesting future direction to incorporate this implicit tempering directly into the mixing parameter γ instead.
Revised: It is an interesting future direction to incorporate this implicit tempering directly into the mixing parameter γ instead.

Original: For LCVaR we ﬁrst optimized over the size of the uncertainty set by adjusting the parameter α .
Revised: For LCVaR-AT we ﬁrst optimized over the size of the uncertainty set by adjusting the parameter α .

Original: Despite getting reasonable performance for some run with α = 0 .
Revised: Despite getting reasonable performance for some run with α = 0 .

Original: 2 the method turned out to have large variance.
Revised: 2 the method turned out to have large variance.

Original: For this reason we explored reducing the model learning rate τ for which we found that τ = 0 .
Revised: For this reason we explored reducing the model learning rate τ for which we found that τ = 0 .

Original: 05 gave better performance while having much smaller variance.
Revised: 05 gave better performance while having much smaller variance.

Original: It is not surprising that the method requires smaller step-sizes given how CVaR methods assign higher weight to a fraction of the batch at every iteration as explained in Appendix B.
Revised: It is not surprising that the method requires smaller step-sizes given how CVaR methods assign higher weight to a fraction of the batch at every iteration as explained in Appendix B.

Original: Maybe unnecessarily meticulous, we also ensure that temporal ensembling indeed improves model performance for LCVaR by also running the method without the regularization.
Revised: Maybe unnecessarily meticulous, we also ensure that temporal ensembling indeed improves model performance for LCVaR-AT by also running the method without the regularization.

Original: Our instantiation of LCVaR used for comparison thus uses α = 0 .
Revised: Our instantiation of LCVaR-AT used for comparison thus uses α = 0 .

Original: 2 and τ = 0 .
Revised: 2 and τ = 0 .

Original: The hyperparameter exploration can be found in Table 7.
Revised: The hyperparameter exploration can be found in Table [Table]

Original: [Table]
Revised: [Table]

Original: C. E XPERIMENTAL SETUP
Revised: C. E XPERIMENTAL SETUP

Original: In this section we provide additional details for the experimental setup speciﬁed in Section 5.
Revised: In this section we provide additional details for the experimental setup speciﬁed in Section 5.

Original: The experiments are conducted on the following three datasets:
Revised: The experiments are conducted on the following three datasets:

Original: CIFAR10 50 , 000 training examples of 32 × 32 dimensional images and 10 classes.
Revised: CIFAR10 50 , 000 training examples of 32 × 32 dimensional images and 10 classes.

Original: CIFAR100 50 , 000 training examples of 32 × 32 dimensional images and 100 classes.
Revised: CIFAR100 50 , 000 training examples of 32 × 32 dimensional images and 100 classes.

Original: STL10 5000 training examples of 96 × 96 dimensional images and 10 classes.
Revised: STL10 5000 training examples of 96 × 96 dimensional images and 10 classes.

Original: As noted in Section 5 we use the average robust accuracy to early stop the model.
Revised: As noted in Section 5 we use the average robust accuracy to early stop the model.

Original: In contrast with common practice though, we use a validation set instead of the test set to avoid overﬁtting to the test set.
Revised: In contrast with common practice though, we use a validation set instead of the test set to avoid overﬁtting to the test set.

Original: The class accuracies across the remaining two datasets, CIFAR100 and STL10 , can be found in Figure 5.
Revised: The class accuracies across the remaining two datasets, CIFAR100 and STL10, can be found in Figure 6.

Original: We also include results when the model is early stopped based on the worst class accuracy on the validation set in Table 6.
Revised: We also include results when the model is early stopped based on the worst class accuracy on the validation set in Table 7.

Original: The mean and standard deviation in all tables and ﬁgures are computed over 5 runs.
Revised: The mean and standard deviation in all tables and ﬁgures are computed over 5 runs.

Original: However, for CIFAR100 we conduct 6 experiments and pick the top 5 to compute the statistics since all methods had one outlier with signiﬁcantly worst performance.
Revised: However, for CIFAR100 we conduct 6 experiments and pick the top 5 to compute the statistics since all methods had one outlier with signiﬁcantly worst performance.

Revised: A radius of 8 / 255 is used for the (cid:96) ∞ -constraint attack unless otherwise noted.

Revised: For training we usesteps of PGD and a stepsize of 2 / 255 .

Revised: At test time we use 20 steps of PGD with a stepsize of 2 . 5 × 8 / 25520 .

Revised: For 12 / 255 -bounded attacks we scale the training stepsize and test stepsize proportionally.

Original: C. I MPLEMENTATION
Revised: C. T EMPORAL ENSEMBLING

Revised: Let f θ : R d → R k be a model parameterized by θ that assigns a probability distribution over the k classes for any x ∈ R d .

Revised: Temporal ensembling (Laine & Aila, 2016; Dong et al., 2021) computes a so called ensemble prediction for a given data example x i by maintaining an average over past predictions, v i ← τ · v i + (1 − τ ) · f θ ( x i ) , where τ is the momentum term.

Revised: Let (cid:96) ( f θ ( x + δ ) , y ) be the training objective.

Revised: A regularizing term is added to the training objective that encourages predictions to stay close to v i ,

Revised: [Equation]

Revised: [Equation]

Original: We provide pytorch pseudo code for CFOL in Listing 1.
Revised: where ω ( t ) is the balancing weight parameter. Usually ω ( t ) interpolates from 0 to some constant ω along a Gaussian rampup curve. That is, ω ( t ) = ω · exp (cid:0) − 5(1 − t ) 2 (cid:1) where t = min { epochmaxepoch , 1 } .

Original: We also re-implemented FOL, which is similar in structure, but additionally requires wrapping the dataset in IndexWrapper(dataset) .
Revised: We provide pytorch pseudo code for how CFOL can be integrated into existing training setups in Listing 1. FOL is similar in structure, but additionally requires associating a unique index with each training example.

Original: This allows the sampling method to re-weight each example individually.
Revised: This allows the sampling method to re-weight each example individually.

Original: For LCVaR we use the implementation of (Xu et al., 2020), which uses the dual CVaR formulation described in Appendix B. More speciﬁcally, LCVaR uses the variant which ﬁnds a closed form solution for λ , since this was observed to be both faster and more stable in their work.
Revised: For LCVaR we use the implementation of (Xu et al., 2020), which uses the dual CVaR formulation described in Appendix B. More speciﬁcally, LCVaR uses the variant which ﬁnds a closed form solution for λ , since this was observed to be both faster and more stable in their work.

Original: from torch.utils.data import DataLoader sampler = ClassSampler(dataset, gamma=0.5) dataloader =
Revised: from torch.utils.data import DataLoader sampler = ClassSampler(dataset, gamma=0.5) dataloader =

Original: DataLoader(dataset, ..., sampler=sampler) ...
Revised: DataLoader(dataset, ..., sampler=sampler) ...

Original: # Training loop: for img,y in iter(dataloader): # attack img # compute gradients and update model # compute logits adv_loss = logits.argmax(dim=-1) !
Revised: # Training loop: for img,y in iter(dataloader): # attack img # compute gradients and update model # compute logits adv_loss = logits.argmax(dim=-1) !

Original: =
Revised: =

Original: y sampler.batch_update(y, eta * adv_loss)
Revised: y sampler.batch_update(y, eta * adv_loss)

Original: Listing 1:{+
 :+} Pseudocode for CFOL.
Revised: Listing {+
 :+} Pseudo CFOL.