Adversarial Training with Complementary Labels: On the Beneﬁt of Gradually Informative Attacks
Abstract
Adversarial training (AT) with imperfect supervision is signiﬁcant but receiveslimited attention.
To push AT towards more practical scenarios, we explore anew setting, i.e., AT with complementary labels (CLs), which specify a class thata data sample does not belong to.
However, the direct combination of AT withexisting methods for CLs results in consistent failure, but not on a simple baselineof two-stage training which consists of a complementary learning phase and an ATphase separately.
In this paper, we further explore the phenomenon and identifythe underlying challenges of AT with CLs as intractable adversarial optimizationand low-quality adversarial examples .
To address the above problems, we proposea new learning strategy using gradually informative attacks, which consists of twocritical components: 1) Warm-up Attack (Warm-up) gently raises the adversarialperturbation budgets to ease the adversarial optimization with CLs; 2) PseudoLabel Attack (PLA) incorporates the progressively informative model predictionsinto a corrected complementary loss.
Extensive experiments are conducted todemonstrate the effectiveness of our method on a range of benchmarked datasets.
Introduction
Deep neural networks are vulnerable to adversarial examples [13], which motivates the developmentof various defensive methods [26, 18, 21, 5] to mitigate the arisen security issue.
As one of the mosteffective and practical defensive methods, adversarial training (AT) [21] has been widely studied [34,3, 28].
Speciﬁcally, it formulates the problem as min-max optimization which generates adversarialdata during training and minimizes the training loss of the generated adversarial variants.
In thisway, the models are equipped with adversarial robustness.
Although AT with perfect supervision(e.g., ordinary labels) has been thoroughly studied by previous works [4, 29, 25, 24, 18, 12], the morecommon learning scenarios with imperfect supervision [1, 22] only has received limited attention.
To push AT towards more practical scenarios, we explore a new setting in this work, i.e., AT [21] withcomplementary labels (CLs) [14].
Due to the low acquisition cost and privacy preservability, CLs isone of those practical imperfect supervision that has been adopted in online learning [16] and medicalimage segmentation [27], which conveys partial label information by identifying a class that a datasample does not belong to.
Several works [14, 10, 30, 11] focused on learning with CLs, where themodel trained only with CLs through their designed losses (e.g., complementary loss ¯ ` ) is expectedto predict ordinary labels accurately during inference.
To conduct AT with CLs, a straightforwardattempt is to regard those loss functions as the training objectives of the min-max optimization.
However, the direct combination of AT with existing methods [14, 33, 15, 7, 10, 30, 11] for learningwith CLs results in consistent failure, but not on a simple baseline of two-stage training (as shown in
Figure 1).
To be speciﬁc, we mainly consider the single complementary label (SCL) with the uniform assumption [14, 15, 7], where labels other than the true label are chosen as a CL with the same
[Figure]
probability.
In complementary learning, the complementary
¯ are obtained by certain
techniques [14, 33], which guarantee the training with ¯ ` on complementary datasets achieves similarperformance to training with the ordinary loss ` on ordinary datasets.
In contrast, when conducting
AT with CLs, most experiments (as shown in Figure 1 with different ¯ ` and optimizers) consistentlyshow failed training curves except for a simple two-stage baseline, which consists of a normalcomplementary learning phase and an AT phase using the predicted labels of the
-stage model.
The above fact motivated us to further investigate AT with CLs, and we identiﬁed its unique challengesas intractable adversarial optimization and low-quality adversarial examples (as veriﬁed in Figure2).
Speciﬁcally, we ﬁrst theoretically analyze that conducting AT with the complementary risk isstatistically consistent to that with the ordinary risk when training with the general backward correctedloss [15].
Although the guarantee holds with sufﬁcient data, it is not preserved any more with alimited number of CLs given in practice [7] (see Eq.
(8) in Section 4.1).
Empirically, we analyze thetraining dynamics of AT with CLs (e.g., gradient directions and norms), and observe that adversarialoptimization with ¯ ` suffers from large gradient variance, with the gradient vanishing issue at the earlystage of training.
Moreover, we also ﬁnd that the complementary loss as the objective of adversarialoptimization fails to generate the high-quality adversarial examples due to the less informative CLsand the implicit way (i.e., attacking
P
[Equation]
p ✓
[Equation]
| x ) ) of attacking ordinary labels.
To mitigate the problems, we propose a new learning strategy using gradually informative attacks(in Algorithm 1) for AT with CLs, which consists of two critical components, i.e., Warm-up Attacks(Warm-up) and Pseudo-Label Attack (PLA) .
We introduce the Warm-up to ease the intractableoptimization with CLs, which is gradually transferred from natural training to AT and controlled by asuitable adversarial budget.
We design the PLA to improve the low-quality adversarial generation,which incorporates the informative predictions from the progressively discriminative model itself.
Overall, we propose a ﬂexible learning scheduler to adjust the two components during training.
Our contributions are summarized as follows.
1) Setting level: we study a new setting, namely, ATwith CLs, which is practical yet thus far unexplored.
2) Challenge level: we discover the substantialexperimental failures in the direct combination of AT with existing methods for learning with CLs (in
Figure 1), and provide the underlying insights of the observed phenomenon from both theoreticaland empirical perspectives (in Sections 4.1 and 4.2).
Speciﬁcally, we identify two challenges asintractable adversarial optimization and low-quality adversarial examples.
3) Methodology level: weaccordingly propose Warm-up and PLA with a controllable scheduler to mitigate the problems (in
Sections 4.3).
Experimental level: we conduct various experiments to verify the effectiveness ofour method, and to understand the beneﬁts of two critical components (in Section 5).
Related Work
Complementary Learning.
Learning with CLs was introduced by [14], where the SCL withthe uniform assumption was ﬁrstly considered.
An unbiased risk estimator (URE) was derivedby modifying a speciﬁc loss function (e.g., one-versus-all or pairwise comparison) that satisﬁes a symmetric condition.
Continuing with this work, [15] derived a general URE for arbitrary lossfunctions and models, and further proposed non-negative correction and gradient ascent methods tocope with the overﬁtting issue of URE in practice.
Although URE has good statistical properties,it has inferior empirical performance due to huge gradient variance.
[7] mitigated this problemby proposing a surrogate complementary loss framework, which is a biased risk estimator tradingzero bias with reduced variance.
Other than the uniform assumption, [33] considered the biased
CLs, where labels other than the true label are chosen as a CL with different probabilities due tothe annotator’s bias.
In addition to the SCL setting, [10] derived an URE of the ordinary risk formultiple complementary labels, and further improved it by minimizing properly chosen upper bounds.
Moreover, there are also several works less related to the loss correction in complementary learning,such as introducing regularization [30] or reweighting [11] during the training process.
Adversarial Training.
As one of the most effective defensive methods, AT has been widely studiedto improve the robustness of deep learning models [6, 5].
The standard AT [21] formulated theproblem as min-max optimization.
[2] proposed CAT to mitigate the catastrophic forgetting andthe generalization issues of AT.
[34] decomposed the prediction error into the natural and boundaryerrors, and proposed TRADES to balance the classiﬁcation performance between the natural andadversarial examples.
[32] incorporated the idea of weight perturbation to AT and proposed AWP,which explicitly regularizes the ﬂatness of the loss landscape by adversarially perturbing both inputsand weights.
[31] explored the inﬂuence of the misclassiﬁed examples on the adversarial robustness,and proposed MART to boost the robustness by emphasizing the misclassiﬁed examples.
In contrastto previous works, only limited effort has been made to explore AT on imperfect labels [22].
Preliminaries
In this section, we provide some preliminaries for ordinary learning, complementary learning and AT.
For complementary learning, we consider the SCL setting with the uniform assumption [14, 15, 7].
For AT, we focus on the standard method proposed in [21].
3.1 Ordinary Learning
Let X ⇢ R d be the input space, and Y
[ K ] := { 1 , 2 , . . .
, K } ( K > 2) be the label space.
The data{ ( x i , y i ) } ni =1 is sampled independently and identically from the joint distribution D with density 101 p ( X, Y ) , which could be further decomposed into p ( X ) p ( Y | X ) .
We deﬁne a class-probabilityfunction ⌘ i ( x )
=
p
( Y = i
| X = x ) .
The decision and loss functions are deﬁned as g : X !
R K and` : Y ⇥ R K !
R + , respectively.
The goal of ordinary learning is to train a classiﬁer f ( x ) :
X !
Y , 104 which is expected to predict the correct label by arg max i g ( x ) i .
The ordinary risk is deﬁned as
[Equation]
3.2 Complementary Learning
In complementary learning, the label space is transformed into ¯ Y , where each ordinary label y isﬂipped into ¯ y 2 ¯ Y with a class-dependent probability p
(¯ y | y ) .
Then, the data is sampled from adifferent joint distribution ¯ D with density p ( X, ¯ Y ) .
We assume the transition matrix Q 2 R K ⇥ K isinvertible, where Q ij = p ( ¯ Y =
j | Y = i ) .
With the uniform assumption, it takes 0 on diagonals and1 K  1 on non-diagonals.
The class-probability function is then written as
[Equation]
The complementary risk is deﬁned as ¯ R ( g ; ¯ ` ) =
E ( x, ¯ y )
⇠ ¯ D [¯ ` (¯ y, g ( x ))] , where ¯ ` is a complementaryloss function.
Typically, it can be derived by the loss correction, which is a technique that ensures riskor classiﬁer consistency in the weakly supervised setting [22, 14].
The backward correction is a riskconsistent algorithm that ensures ¯ R is a statistically consistent risk estimator of the ordinary risk R ,while the forward correction is a classiﬁer consistent algorithm that guarantees the classiﬁer learnedfrom ¯ D using ¯ ` converges to the optimal one learned from D using ` .
In complementary learning,[14, 15, 10] focused on the backward correction that multiplies the loss by Q  1 , while [33] proposeda forward correction method via multiplying the model prediction by Q .
Other methods (e.g., a bounded loss) can also be used to derive complementary losses [7, 10].
The detailed complementaryloss functions are provided in Table 2 of Appendix B.
General Backward Correction.
Among this line of research, a notable work [15] derived an UREof the ordinary risk, without any restriction on the used loss and model.
Given the notations borrowedfrom [15, 7]: ` ( g ( x )) =
[ ` (1 , g ( x )) , ` (2 , g ( x )) , . . .
, ` ( K, g ( x ))] and e i is the one-hot column vectorwith one on the i th entry, the URE is derived as follows:
Proposition 1.
The ordinary risk can be expressed in terms of CLs as follows,
[Equation]
when ¯ ` is rewritten as
[Equation]
With the uniform assumption, ¯ ` can be further rewritten as
[Equation]
With this expression, we can obtain an URE of the ordinary risk only from CLs.
3.3 Adversarial Training
AT has been a commonly used defensive technique that equips deep learning models with adversarialrobustness against imperceptible perturbations within a small neighborhood of input.
Here, weconsider conducting AT with ordinary labels (e.g., ` is the cross-entropy loss).
Generally, it can beformulated into a min-max optimization problem:
[Equation]
(6) where B ✏ [ x ] = { ˜ x : k x  ˜ x
k p  ✏ } is the closed ball centered at input x with radius ✏ > 0 under 135 l p -norm threat models, and ˜ x is the adversarial data found within B ✏ [ x ] .
Empirically, the solution ofinner maximization is approximated by projected gradient descent (PGD) as follows:
[Equation]
where ↵ is the step size, ⇧ is the projection operator that projects the adversarial data back to theepsilon ball, x ( t ) is the adversarial data found at step t , and x (0) is initialized by natural data ornatural data with a small Gaussian or uniformly random perturbation.
The adversarial data is updatediteratively in the direction of loss maximization until a stop criterion is satisﬁed.
Adversarial Training with Complementary Labels
In this section, we ﬁrst provide some theoretical insights for AT with CLs.
Then, we empiricallyanalyze the experimental failure and identify the critical challenges as intractable adversarial optimization and low-quality adversarial examples when conducting AT with corrected complementarylosses.
Accordingly, we propose a new learning strategy with gradually informative attacks.
4.1 Theoretical Analysis
Without loss of generality, below we analyze the general backward correction [15] theoretically, andother popular complementary loss functions on the SCL setting.
Proposition 2.
For the general backward correction, conducting AT on the complementary risk isequivalent to that on the ordinary risk.
However, this is not the case on their empirical risks:
[Equation]
min ✓ n
[Equation]
[Figure]
The detailed proof is provided in Appendix A.2.
In brief, when considering the general backwardcorrected loss, the maximization induced by AT keeps the statistical properties of the complementaryrisk, and it still yields a consistent risk estimator.
In theory, the adversarial optimization of thecomplementary risk with the general backward corrected loss is statistically consistent to that ofthe ordinary risk with the ordinary loss.
But in practice, we can only obtain limited CLs (e.g., thesample size of E ¯ y ⇠ p (¯ Y | X = x ) is one on the SCL setting), which causes the inconsistency between 157 their empirical risks (i.e., the solutions to their inner maximization are different), and hence leads to adifﬁcult adversarial optimization (as veriﬁed in Appendix D.2).
The solution to inner maximizationof the complementary risk is equivalent to that of the ordinary risk if and only if maximizing theweighted loss over all candidate CLs, that is E ¯ y ⇠ p (¯ Y | X = x ) [¯ ` (¯ y, g (˜ x ))] .
From the perspective of AT, we expect to ﬁnd the adversarial examples ˜ x within the neighborhoodof input x that make the model predictions on the ordinary label p ✓ ( y | ˜ x ) as low as possible.
Whenwe have ordinary labels, it is quite straightforward to achieve this objective when we maximizingthe cross-entropy ` =  log p ✓ ( y | ˜ x ) .
However, if we further consider recent proposed methods inaddition to the general backward correction in complementary learning, most of ¯ ` mathematically tryto maximize
P j 2 Y\{ ¯ y } p ✓ ( j | ˜ x ) (or equivalently minimize p ✓ (¯ y | ˜ x ) ) as shown in Table 2.
Maximizing¯ ` leads to the minimization of
P j 2 Y\{ ¯ y } p ✓ ( j | ˜ x ) , which cannot guarantee the minimization ofp ✓ ( y | ˜ x ) .
In such a case, the inner maximization may serve as a random attacker without priorknowledge.
Hence, corrected complementary losses fail to achieve the objective of AT, and lowquality adversarial examples may be generated, which is detailedly discussed in the following part.
4.2 Empirical Analysis
We conduct the experiment on Kuzushiji to justify the previous theoretical insights.
For the upperpanels of Figure 2, we focus on the issue of intractable adversarial optimization by adversariallytraining the model using several loss functions separately, with three random trials.
For the lowerpanels of Figure 2, we focus on the issue of low-quality adversarial examples by measuring thedifferent adversarial examples generated by various loss functions, given the same optimizationmodel (i.e., the oracle model, which is trained using AT with ordinary labels).
We show the average results over different seeds and training samples, respectively.
Without loss of generality, here weonly show the results of one complementary loss (i.e., LOG [10]), and our proposed method (i.e.,
Warm-up+PLA in Section 4.3).
The detailed experiment settings and more comprehensive empiricalresults with other complementary losses and datasets are provided in Appendix C.
Intractable Adversarial Optimization.
In the outer minimization of AT, we ﬁrst collect thegradient direction (with respect to the model parameters) of each data in one mini-batch (i.e., 256),and then compute the trace sum of their covariance matrix (the average result for each epoch is shownin the upper-left panel of Figure 2).
A larger value reﬂects a group of gradient directions with a biggervariance.
Note that the result of LOG nearly reaches the maximum trace sum of covariance matrixof the gradient directions, which results in its consistent failure at the early stage of training.
Wealso compute the average l 2 -norm of stochastic gradients of the last and ﬁrst layer as shown in theupper-middle and upper-right panels of Figure 2, respectively.
Although some gradient signals existfor LOG in the last layer, the gradients back-propagated to the ﬁrst layer is so small that gradientvanishing problem occurs at the early stage of training.
This phenomenon is similar to that observedin the standard AT, which is conjectured as the prevalence of dead neurons caused by a large radiusof epsilon ball [20].
Moreover, considering AT with CLs, it is expected that the gradient norm tendsto be small since the observed variance of gradient directions in one mini-batch is extremely large.
The above observations indicate the adversarial optimization with CLs is difﬁcult, and it is rational touse the method (i.e., our Warm-up+PLA) to ease the optimization.
Low-quality Adversarial Examples.
In the inner maximization of AT, given the same optimizationmodel, we measure the quality of generated adversarial examples by computing the cosine similarityof gradient directions (with respect to the same data) and the l 1 -distance of constructed adversarialexamples, with the oracle examples generated by ordinary label (in the lower-left and lower-middlepanels of Figure 2).
A larger cosine similarity and a lower l 1 -distance indicate a stronger resemblanceto the oracle, but not necessarily a stronger attack (or high-quality adversarial examples).
Hence,we further compute the difference of model predictions of natural data and adversarial data on theordinary label during training (in the lower-right panel of Figure 2).
A bigger difference explicitlydemonstrates a stronger attack.
The results show that the complementary loss as the objective of ATwith CLs fails to generate high-quality adversarial examples as our analysis in Section 4.1.
4.3 Methodology
To address the previous problems, we accordingly propose a new strategy using gradually informativeattacks to ease the intractable adversarial optimization and improve the quality of adversarial examples,respectively.
Our algorithm is concretely presented in Algorithm 1.
Speciﬁcally, it consists of Warmup Attack and Pseudo-Label Attack, whose dynamics are controlled by a ﬂexible scheduler:
[Equation]
where a is the maximum value of the scheduled variable (i.e., ✏ max = ✏ in Warm-up Attack and max = 1 in Pseudo-Label Attack), e is the epoch index, and E s is the duration of the scheduler.
Boththe proposed attacks can be controlled by a ﬂexible scheduler with the increasing and decreasingtrends respectively (e.g., see the example in Figure 3).
Below, we describe each component in detail.
Warm-up Attack.
Adversarial optimization is harder compared with ordinary optimization [20],and CLs further strengthen the degree of hardness as observed in our empirical analysis.
The majordifference of AT with CLs from the standard AT is the labels used in the adversarial generation.
Asshown in Figure 2, training with the adversarial examples generated by different labels results in thedifferent gradient variance.
Since the adversarial generation on CLs induces large gradient variancewhich causes the difﬁcult adversarial optimization, we can adjust the adversarial generation to controlthe variance introduced into the optimization.
To be speciﬁc, we adopt a dynamic adversarial budgetto gradually enhance the degree of adversarial generation.
Typically, adversarial budgets consist ofthe radius of epsilon ball ✏ , number of attack steps k , and step size ↵ .
Here, we mainly focus on thedynamics of ✏ in Eq.
The radius of the epsilon ball is increased from 0 to ✏ within E s epochs,and is kept constant until the end of adversarial optimization.
The number of attack steps is constant
Algorithm
AT with CLs Using Gradually Informative Attacks
Input: number of epochs: E , number of batches: B , number of class: K , radius of epsilon ball: ✏ , number of attack steps: k , step size: ↵ , hyperparameter for exponential moving average of cashed model predictions:  ; hyperparameter for label weights:  ;
Output: robust model ✓ R ; 1: p c 1 K  1 ( 1  e ¯ y ) T for all data 2: for e
= 1 , . .
, E do 3:  ,
Update hyperparameters 4: ✏ e , ↵ e , k e
Adversarial budget scheduler .
Warm-up Attack 5: for b = 1 , . . .
, B do 6: x, ¯ y
Sample a batch of data 7: p ✓ ( x )
Softmax ( ✓ ( x )) , p c ( x )  p c ( x ) + (1   ) p ✓ ( x ) , p c (¯ y | x ) 8: ˆ y arg max j 6 =
¯
y p c ( j | x ) 9: ˜ x
Approximate the solutions of max ¯ ` ( x, ¯ y ; ✓ ) (Eq. 10) by PGD( ✏ e , ↵ e , k e ) 10:
L =  ( K  1) log(
P j 6 =¯ y p
✓ ( j | ˜ x ) + (1   ) p ✓ (ˆ y | ˜ x )) .
Pseudo-Label Attack 11: ✓
SGD ( ✓ , r ✓ L ) 12: end for 13:
Evaluate( ✏ , k, ↵ ) 14: end for ( k e = k ), and the step size is increased proportionally to ✏ ( ↵ e = ✏ e ✏ ↵ ).
In this way, the graduallyincreased adversarial budget can ease the difﬁculty of adversarial optimization.
Pseudo-label Attack.
Due to the existence of an adversarial budget scheduler, at the early stageof optimization, simple patterns may be prevalent in adversarial data found within an extremelysmall epsilon ball, which is helpful for the self-evolvement of a discriminative model.
The modeltends to assign high conﬁdence to the potential ordinary label, while low conﬁdence to others.
Thisinformative model prediction is a strong supplementary information that is promising for improvingthe quality of the adversarial generation.
Therefore, we propose to optimize the objective consistingof the convex combination outputs of partial labels (labels other than CLs) and predicted ordinarylabel.
This transforms the whole process into an optimization that starts with learning with CLs usingcertain correction techniques, and gradually moves to learning with predicted labels regarding themas ordinary labels.
The loss function can be deﬁned as follows:
[Equation]
where  2 [0 , 1] is a hyperparameter that controls the transition of learning.
Recent studies foundthat adversarial training can degrade the generalization, and a trade-off between natural accuracyand adversarial robustness exists [28].
To avoid the effect of such degradation and stabilize theoptimization (e.g., a drastic shift of model predictions may occur due to the increasing adversarialbudget), we use a simple exponential moving average to collect the dynamics of model predictions,and choose the label with the maximum probability as a predicted ordinary label.
Speciﬁcally, giventhe model predictions p ✓ ( x ) of the current epoch, the cashed model predictions p c ( x ) is updated asp c ( x ) =
p c ( x ) + (1   ) p ✓ ( x ) , where p c ( x ) is initialized with ( 1  e ¯ y ) T K  1 , p c (¯ y | x ) = 0 , and  = 0 .
Experiments
In this section, we empirically verify the effectiveness of our method on MNIST [19], Kuzushiji [8],
CIFAR10 [17] and SVHN [23] datasets, and conduct ablation studies to understand the two attacks.
Setups.
1) General Setups : for MNIST and Kuzushiji, we train a small convolutional neuralnetwork, as used in [34], for 100 epochs 1 with batch size of 256.
For CIFAR-10 and SVHN, ResNet18 is trained for 120 epochs with batch size of 128.
The training sets of CIFAR-10 and SVHN areaugmented with random cropping and horizontal ﬂipping. 2)
Adversarial Training Setups : we focuson the L 1 threat model.
For MNIST and Kuzushiji, stochastic gradient descent (SGD) is used with
[Figure]
Baseline. 1).
Two-stage method: it ﬁrst trains a model for 50 epochs with the methods of complementary learning (i.e., LOG, which shows better performance and stability compared to others [10]).
Then we re-label the complementary training set using the checkpoint with the best validation performance, and adversarially train another model using the re-labeled training set afterwards.
Conducting adversarial training directly with complementary losses, including FORWARD [33],
FREE, NN [15], SCL_NL, SCL_EXP [7], EXP and LOG [10].
We evaluate the adversarial robustnessusing PGD20, CW30 and AutoAttack (AA) [9], and report the results of checkpoint with the best
PGD20 test accuracy.
All experiments are conducted with multiple random seeds (i.e., 1-3).
5.1 Performance Evaluation
The detailed results are reported in Table 1, where Oracle refers to the standard AT with ordinarylabels, and it is just given for measuring the performance gap between AT with CLs and ordinary labels.
For easy datasets (e.g., MNIST), the complementary losses are possible to obtain satisfying results,while most of them fail on complicated datasets.
We would like to mention that the performanceof the two-stage and our methods slightly outperform that of the oracle on SVHN.
We assume thiscase may be caused by the fact that noisy datasets (with a small portion of noisy data) are not alwaysdetrimental to AT, and can even further improve the adversarial robustness as mentioned in [29].
5.2 Ablation Study
In this part, we conduct the ablation study to demonstrate the importance of the two components, i.e.,
Warm-up Attack and Pseudo-Label Attack in our method with the critical hyperparameter.
Two Critical Components.
We investigate the roles of two components in our method by removingthem separately.
We report the results within three trials on Kuzushiji and CIFAR10 datasets as shownin the left-middle and right-middle panels of Figure 3.
In summary, warm-up attack could stabilizethe optimization especially when the model capacity are not sufﬁcient large, and pseudo-label attackis crucial for the achieving high adversarial robustness.
Speciﬁcally, the warm-up attack achievesquite good performance on Kuzushiji, while limited improvements on CIFAR10.
This is expecteddue to the difﬁculty of CIFAR10 without further informative attacks.
Pseudo-label attack solvesthis problem by incorporating the informative model predictions, and hence achieves quite goodperformance on CIFAR10.
However, it has an unstable result on Kusushiji, which is attributed to thelimited model capacity.
It is of great importance to achieve adversarial robustness given limited modelcapacity.
Therefore, warm-up attack is still a crucial component for stabilizing the optimization.
Hyperparameter.
We investigate the performance sensitivity to the key hyperparameter E s , whichcontrols the growing speed of informative attacks.
We report the results within three trials on Kuzushiji
[Table]
as shown in the right panels of Figure 3.
The result shows a similar performance across a wide rangeof E s , which demonstrates that our method is not sensitive to the tuning of key hyperparameter.
Conclusion
To the best of our knowledge, this is the ﬁrst work to study adversarial training (AT) with complementary labels (CLs), and to analyze its challenges from both theoretical and empirical perspectives.
Tosolve them, we proposed a new learning strategy using gradually informative attacks, narrowing theperformance gap between AT with CLs and ordinary labels.
Our work shed light on the applicationsof AT to a more practical scenario (e.g., with imperfect supervision).
In this paper, we focused onthe standard AT algorithm, which formulates the problem as min-max optimization.
We leave moreadvanced algorithms (e.g., TRADES) and other complementary settings (e.g., multiple CLs) to thefuture work.
Moreover, CLs can be viewed as an extremely case of noisy labels with 100% noise rate,or partial labels where the labels other than the CL are the candidate set of labels.
Hence, we hope ourwork could
AT with various imperfect supervision in community