Revisiting Adversarial Robustness of Classiﬁers With a Reject Option
Abstract
Adversarial training of deep neural networks (DNNs) is an important defense mechanism that allows a DNN to be robust to input perturbations, that can otherwise result in predictions errors.
Recently, there is a growing interest in learning a classiﬁer with a reject (abstain) option that can be more robust to adversarial perturbations by choosing to not return a prediction on inputs where the classiﬁer may be incorrect.
A challenge faced with robust learning of a classiﬁer with reject option is that existing works do not have a mechanism to ensure that (very) small perturbations of the input are not rejected, when they can in fact be accepted and correctly classiﬁed.
We ﬁrst propose a novel metric – robust error with rejection – that extends the standard deﬁnition of robust error to include the rejection of small perturbations.
The proposed metric has natural connections to the standard robust error (without rejection), as well as the robust error with rejection proposed in a recent work.
Motivated by this metric, we propose novel loss functions and a robust training method – stratiﬁed adversarial training with rejection (SATR) – for a classiﬁer with reject option, where the goal is to accept and correctly-classify small input perturbations, while allowing the rejection of larger input perturbations that cannot be correctly classiﬁed.
Experiments on well-known image classiﬁcation DNNs using strong adaptive attack methods validate that SATR can signiﬁcantly im- prove the robustness of a classiﬁer with rejection compared to standard adversarial training (with conﬁdence-based rejection) as well as a recently-proposed baseline.
Introduction
Training robust classiﬁers in the presence of adversarial inputs is an important problem from the standpoint of designing secure and reliable machine learning systems (Biggio and Roli 2018).
Adversarial training and its variations are the most effective methods for learning robust DNN classiﬁers (Madry et al. 2018;
Zhang et al. 2019)
However, adversarial training may still not be very effective against adaptive adversarial attacks, or even standard attacks with conﬁgurations not observed during training (Athalye, Carlini, and
Wagner 2018; Tram`er et al. 2020).
Given this limitation, it is important to design classiﬁers that learn when to reject or abstain from predicting on hard-to-classify inputs.
This can be especially crucial when it comes to real-world, safetycritical systems such as self-driving cars, where abstaining from prediction is often a much safer alternative to making an incorrect decision.
We focus on the problem of learning a robust classiﬁer with a reject option in the presence of adversarial inputs.
The related problem of learning a (non-robust) classiﬁer with a reject option has been studied extensively in the literature (Tax and Duin 2008; Guan et al.
; Cortes, DeSalvo, and Mohri 2016; Geifman and El-Yaniv 2019; Charoenphakdee et al.
Recently, a number of works have also addressed the problem of adversarial robustness for a classiﬁer equipped with a reject option (Laidlaw and Feizi 2019; Stutz, Hein, and Schiele 2020; Sheikholeslami, Lotﬁ, and Kolter 2021;
Pang et al. 2021b; Tram`er 2021; Kato, Cui, and Fukuhara 2020).
These approaches extend the standard deﬁnition of adversarial robustness (robust error) to the setting where the classiﬁer can also reject inputs.
In this setting, rejection of a perturbed input is considered to be a valid decision that does not count towards the robust error.
However, rejection of a clean input still counts towards the robust error (Tram ` er 2021).
A key limitation with this view of the robust error with rejection is that it treats equally the rejection of very small perturbations as well as large perturbations of an input.
However, many practical applications ( e.g. , object detection) may require that small perturbations of an input be handled accurately by the classiﬁer without resorting to rejection.
In other words, there could be a higher cost for rejecting small input perturbations, when in-fact the classiﬁer can accept and classify them correctly.
Existing methods for training a robust classiﬁer with rejection, such as conﬁdence-calibrated adversarial training (CCAT) (Stutz, Hein, and Schiele 2020), achieve a high robustness by simply rejecting a large fraction of the perturbed inputs (since rejecting perturbed inputs does not contribute to the robust error, no matter the perturbation size).
As we validate experimentally, CCAT often has a high rejection rate on even small perturbations of clean inputs, which may not be acceptable in many practical applications.
Motivated by these limitations in existing works, we revisit the problem of adversarial robustness of a classiﬁer with reject option, and make the following contributions:
• We propose a novel metric – robust error with rejection – that can provide a ﬁne-grained evaluation of the robust-
ness of a classiﬁer with reject option across a range of perturbation sizes.
• We provide a theoretical analysis of this problem, which motivates the need for learning a robust classiﬁer with rejection that can accept and correctly classify small input perturbations.
• We propose novel loss functions and a robust training method SATR for jointly learning a classiﬁer-detector system ( i
e. , a classiﬁer with rejection) that are designed to achieve the goal of accepting and correctly classifying small input perturbations, while also selectively rejecting larger input perturbations.
Related Work
Adversarial robustness of deep learning models has received signiﬁcant attention in recent years.
Many defenses have been proposed and most of them have been broken by strong adaptive attacks (Athalye, Carlini, and Wagner 2018;
Tram`er et al. 2020).
The most effective approach for improving adversarial robustness is adversarial training (Madry et al. 2018;
Zhang et al. 2019).
However, adversarial training still cannot achieve very good robustness on complex datasets, and often there is a large generalization gap in the robustness (Tsipras et al. 2019; Stutz, Hein, and Schiele 2019).
For example, on CIFAR-10, state-of-the-art adversarial training has only about 50% robustness under the strongest adaptive attacks.
One approach to break this robustness bottleneck is to allow rejection of adversarial examples instead of trying to correctly classify all of them.
Recently, there has been a great interest in exploring adversarial training of a classiﬁer with a reject option (Laidlaw and Feizi 2019; Stutz, Hein, and Schiele 2020; Sheikholeslami, Lotﬁ, and Kolter 2021; Pang et al.
2021b; Tram ` er 2021).
Stutz, Hein, and Schiele proposed to adversarially train conﬁdence-calibrated models so that they can generalize to unseen adversarial attacks.
Sheikholeslami, Lotﬁ, and Kolter modiﬁed existing certiﬁed-defense mechanisms to allow the classiﬁer to either robustly classify or detect adversarial attacks, and showed that it can lead to better certiﬁed robustness, especially for large perturbation sizes.
Laidlaw and Feizi proposed a method called Combined Abstention Robustness Learning (CARL) for jointly learning a classiﬁer and the region of the input space on which it should abstain, and showed that training with CARL can result in a more accurate and robust classiﬁer.
Problem Setup
Let X ⊆
R
d denote the space of inputs x and Y :
= { 1 , · · · , k } denote the space of outputs y .
Let Y :
= Y ∪ {⊥} be the extended output space where ⊥ denotes the abstain or rejection option.
Let ∆ k denote the set of output probabilities over Y ( i.e. , the simplex in k -dimensions).
Let d ( x , x (cid:48) ) be a norm-induced distance on X ( e.g. , the (cid:96) p -distance for some p > 1 ), and let N ( x , r ) :
= { x (cid:48) ∈
X : d ( x (cid:48) , x ) ≤ r } denote the neighborhood of x with distance r .
Let ∧ and ∨ denote the boolean AND and OR operations respectively.
Let 1 { c } deﬁne the binary indicator function which takes value 1 ( 0 ) when the condition c is true (false).
We denote vectors and matrices using boldface symbols.
Given samples from a distribution D over X × Y , our goal is to learn a classiﬁer with rejection option, f : X → Y , that can correctly classify adversarial examples with small perturbations, and can either correctly classify or reject those with large perturbations.
The standard robust error at adversarial budget (cid:15) > 0 is deﬁned as (Carlini and Wagner 2017)
[Equation]
which does not allow any rejection.
A few recent works (e.g. (Tram`er 2021)) have proposed a robust error with rejection at adversarial budget (cid:15) > 0 as
[Equation]
which allows the rejection of input perturbations within an (cid:15) -neighborhood without incurring an error.
Neither of these metrics for robust error is well-suited to our needs.
We therefore propose a new metric for evaluating a robust classiﬁer with reject option – the robust error with rejection at budgets (cid:15) 0 ∈ [0 , (cid:15) ] and (cid:15) ≥ 0 : [Equation] 0 ,(cid:15) (
[Equation]
The motivation for this metric is as follows.
For small perturbations of a clean input within a neighborhood of radius (cid:15) 0 , both an incorrect prediction and rejection are considered to be an error.
For larger perturbations outside the (cid:15) 0 -neighborhood, rejection is not considered to be an error, i.e. , the classiﬁer can either predict the correct class or reject larger perturbations.
Proposition 1.
The robust error with rejection can be equivalently deﬁned as
[Equation]
We ﬁrst note that
[Equation]
The maximum over the (cid:15) 0 -neighborhood can be expressed as
[Equation]
Finally
[Equation]
the second term in the RHS of the above expression can be combined with the second term inside the expectation of Eq.
(1), i
[Equation]
e. ,
[Equation]
, ⊥}} , which shows the equivalence of (1) and (2).
Our new metric has the following natural connections with existing metrics in the literature:
When (cid:15)
= (cid:15) , our metric R rej (cid:15) 0 ,(cid:15) ( f ) reduces to the standard robust error (without rejection) R (cid:15) ( f ) at budget (cid:15) .
When (cid:15)
= 0 , our metric reduces to the robust error with rejection at budget (cid:15) , R rej (cid:15) ( f ) proposed e g. , in (Tram`er 2021).
For this special case, rejection is considered to be an error only for clean inputs ( i e. , no perturbation).
• For any classiﬁer g : X → Y that does not allow rejection and any (cid:15) 0 ∈ [0 , (cid:15) ] , it is easily veriﬁed that our metric reduces to the standard robust error at budget (cid:15) , i.e. , R rej (cid:15) 0 ,(cid:15) ( g ) =
R (cid:15) ( g ) .
{+In our experiments, we evaluate R rej (cid:15) ,(cid:15) ( f ) over a range of (cid:15) 0 values by setting (cid:15) 
 = α (cid:15), α ∈ [0 , 1] .+}
This produces a curve with R rej α(cid:15)
(cid:15) ( f ) on the y-axis as a function of α on the x-axis.
The curve shows the robust error with rejection
of the classiﬁer
for a
range
of small-perturbation neighborhoods,
with R rej
(cid:15)
( f ) at the left end ( α = 0 ) and the
standard
robust error R (cid:15) ( f ) at the right end ( α = 1 ).
Theoretical Analysis
Our goal is to correctly classify small perturbations of the input and allow rejection of large perturbations when the classiﬁer is not conﬁdent.
Two fundamental questions arise:
Why not allow rejection of both small and large perturbations?
This is done in most existing studies on robust classiﬁcation with rejection.
On the other hand, many practical applications would like to handle small perturbations, and rejecting them can have severe costs.
The question is whether it is possible to correctly classify small perturbations without hurting the robustness i.e. , whether it is possible to achieve a small R rej (cid:15) 0 ,(cid:15) ( f ) .
Why not try to correctly classify both small and large perturbations?
This is done in traditional adversarial robustness, typically by adversarial training.
The question is essentially about the beneﬁt of allowing rejection.
To answer these questions, we will show that under mild conditions, there exists a classiﬁer f with rejection with a small R rej (cid:15) 0 ,(cid:15) ( f ) .
So it is possible to correctly classify small perturbations without rejecting them, answering the ﬁrst question.
Moreover, under the same conditions, all classiﬁers g : X → Y without rejection must have at least as large errors, i.e. , R rej (cid:15) 0 ,(cid:15) ( g ) =
R (cid:15) ( g ) ≥ R rej (cid:15) 0 ,(cid:15) ( f ) .
In fact, the robust error of g may be much larger than that of f .
This shows the beneﬁt of allowing rejection, answering the second question.
Theorem 1.
Consider binary classiﬁcation.
Let g ( x ) be any decision boundary ( i.e. , any classiﬁer without a rejection option).
For any 0 ≤ (cid:15) 0 ≤ (cid:15) , there exists a classiﬁer f with a rejection option such that
[Equation]
Moreover, the bound is tight: there exist simple data distributions and g such that any f must have R rej (cid:15) [Equation] ( f ) ≥
R ( (cid:15)
+ (cid:15) ) /
2 ( g ) .
The proof for Theorem 1 is given in the Appendix.
Intuitively, the theorem states that if the data allows a small robust error at adversarial budget ( (cid:15)
+ (cid:15) ) / 2 , then there exists a classiﬁer with small robust error with rejection at budget ( (cid:15) 0 , (cid:15) ) .
For example, if the two classes can be separated with a margin ( (cid:15)
+ (cid:15) ) / 2 , then there exists an f with 0 robust error with rejection, even considering perturbations as large as (cid:15) which can be signiﬁcantly larger than ( (cid:15)
+ (cid:15) ) / 2 .
Therefore, under mild conditions, it is possible to classify correctly small perturbations while rejecting large perturbations, answering our ﬁrst question.
On the other hand, under the same conditions, if we do not allow rejection and consider any classiﬁer g without rejection, then robust error of g at the same adversarial budget is
R rej (cid:15) 0 , [Equation] g )
[Equation] g ) [-≥ R ( (cid:15) 
-][Equation] [Equation]
+
[Equation]
[Equation] g ) ≥ R [Equation] (cid:15) 0 ,(cid:15) ( f ) .
In fact, there can be a big gap between R (cid:15) ( g ) and R ( (cid:15)
+ (cid:15) ) /
2 ( g ) , e.g. , when a large fraction of inputs have distances in (( (cid:15)
+ (cid:15) ) / 2 , (cid:15) ) to the decision boundary of g .
In this case, allowing rejection can bring signiﬁcant beneﬁt, answering our second question.
Note that if we set (cid:15) 0 = 0 , then the theorem reduces to
Theorem 5 in (Tram ` er 2021).
Proposed Method
Consider a classiﬁer without rejection g ( x ; θ c ) , g : X
(cid:55)→ Y realized by a DNN with parameters θ c .
The output of the
DNN is the predicted probability of each class
h ( x ; θ c ) =
[ h 1 ( x ; θ c ) , · · · , h k ( x ; θ c )] ∈ ∆ k .
We deﬁne the logits or the vector of un-normalized predictions as (cid:101) h
( x ; θ c ) =
[ (cid:101)
h 1 ( x ; θ c ) , · · · , (cid:101) h k ( x ; θ c )]
∈ R k .
The output of the DNN is obtained by applying the softmax function to the logits.
The class corresponding to the maximum predicted probability is returned by g , i.e. , g ( x ; θ c ) :
= argmax y ∈Y h y ( x ; θ c ) .
The corresponding maximum probability is referred to as the prediction conﬁdence h max ( x ; θ c ) :
= max y ∈Y h y ( x ; θ c ) .
The prediction conﬁdence has been used in prior works for determining when the classiﬁer should abstain from prediction (Wu et al. 2018; Stutz, Hein, and Schiele 2020).
In this work, we focus on the robust training of a classiﬁer with a conﬁdence-based reject option.
Unlike many prior works, the conﬁdence is not simply used at test time for rejection, but is included in our robust training objective.
We deﬁne a general classiﬁer with a conﬁdence-based reject option f : X (cid:55)→ Y as follows
[Equation]
where h ⊥ ( x ; θ ) ∈ [0 , 1] is the predicted probability of rejection and η ∈ [0 , 1] is a suitably-chosen threshold.
We can view h ⊥ ( x ; θ ) as a detector that either accepts or rejects an input based on the classiﬁer’s prediction, as shown in
Fig.1.
The detector is deﬁned as a general parametric function of the classiﬁer’s un-normalized prediction
h ⊥ ( x ; θ ) :
= u ( (cid:101) h ( x ; θ c ) ; θ d ) , u : R k (cid:55)→ [0 , 1] , with detector-speciﬁc parameters θ d 1 .
Here, we denote the combined parameter vector of the classiﬁer and detector by θ T :=
[ θ Tc θ Td ] .
[Figure]
Probability Model.
The class-posterior probability model of the classiﬁer with reject option f is deﬁned as follows:
[Equation]
An input x is accepted with probability 1 − h ⊥
( x ; θ ) and predicted into one of the classes y ∈ Y with probability h y
( x ; θ c ) ; otherwise x is rejected with probability h ⊥
( x ; θ ) and the class ⊥ is returned with probability 1 .
Loss Functions
Consider the robust error with rejection deﬁned in Eq.
We would like to design smooth surrogate loss functions to replace the 0 − 1 error functions in order the minimize the robust error with rejection.
Accept & Classify Correctly.
First, consider the 0 − 1 error corresponding to small perturbations in the (cid:15) 0neighborhood, 1 { f ( x (cid:48) )
(cid:54) =
y } .
We would like the corresponding surrogate loss to take a small value when the predicted probability for the true class y is high and the predicted probability of rejection is low.
The predicted probability of f can be viewed as a k + 1 dimensional probability vector over the k classes and the reject class Y :
[(1 − h ⊥ ( x (cid:48) ; θ ))
h 1 ( x (cid:48) ; θ c ) , · · · , (1 − h ⊥
( x (cid:48) ; θ ))
h k ( x (cid:48) ; θ c ) , h ⊥ ( x (cid:48) ; θ )] .
Note that the ﬁnal term corresponds to the probability of rejection, and the k +probabilities sum to 1 .
For an input ( x (cid:48) , y ) to be accepted and predicted into class y , the target k + 1 dimensional one-hot probability vector has a 1 corresponding to class y and zeros elsewhere.
We propose to use the cross-entropy loss between this target one-hot probability and the predicted probability of f , given by
[Equation]
We observe that the above loss function approaches 0 when the probability of rejection is close to 0 and the predicted probability of class y is close to 1 ; the loss function takes a large value in all other cases.
We also apply this cross-entropy loss for clean inputs since we expect the classiﬁer to accept and correctly classify them.
Accept & Classify Correctly or Reject.
Consider the 0 −error corresponding to perturbations in the (cid:15) -neighborhood, 1 { f ( x (cid:48) )
/ ∈ { y, ⊥}} .
We would like the corresponding surrogate loss to take a small value when the predicted probability for the true class is high, or when the probability of rejection is high.
To motivate the cross-entropy loss for this case, consider k meta-classes deﬁned as follows: { 1 , · · · , y ∨ ⊥ , y + 1 , · · · , k } , i.e. , the reject option is merged only with the true class y .
The predicted probability of f over these meta-classes is given by: [(1 − h ⊥
( x (cid:48) ; θ ))
h 1 ( x (cid:48) ; θ c ) , · · · , (1 − h ⊥ ( x (cid:48) ; θ )) h y ( x (cid:48) ; θ c ) + h ⊥ ( x (cid:48) ; θ ) , · · · , (1 − h ⊥
( x (cid:48) ; θ )) h k ( x (cid:48) ; θ c )] 2 .
For an input ( x (cid:48) , y ) to be either rejected or accepted and predicted into class y , the target k -dimensional one-hot probability vector has a 1 corresponding to the meta-class y ∨ ⊥ , and zeros elsewhere.
We propose to use the cross-entropy loss between this target one-hot probability and the predicted probability of f , given by
[Equation]
This loss function approaches 0 when i) the probability of rejection is close to 1 , or ii) the probability of rejection is close to 0 and the predicted probability of class y is close to 1 .
Note that both the loss functions have a range [0 , ∞ ) .
Robust Training Objective
Given clean labeled samples ( x , y ) from a data distribution
D , a perturbation budget for robust classiﬁcation (cid:15) > 0 , and a smaller perturbation budget (cid:15) 0 ∈ [0 , (cid:15) ] , we propose the following training objective for learning a robust classiﬁer with a reject option:
[Equation]
The ﬁrst term corresponds to the standard cross-entropy loss on clean inputs from the data distribution.
The second term corresponds to the robust loss for small perturbations in the (cid:15)
-neighborhood that we would like the classiﬁer to accept and correctly classify .
The third term corresponds to the robust loss for large perturbations in the (cid:15) -neighborhood that we would like the classiﬁer to either reject or accept and correctly classify .
The classiﬁer parameters θ c and the detector parameters θ d are jointly learned by minimizing
L ( θ ) .
The hyper-parameters β ≥ 0 and γ ≥ 0 control the trade-off between the natural error and robust error terms of the classiﬁer.
We use standard PGD attack (Madry et al. 2018) to solve the inner maximization in our training objective.
Comments.
Suppose we choose the detector to always accept inputs, i.e. , h ⊥ ( x ; θ ) = 0 , ∀ x , and ﬁx (cid:15)
= (cid:15) , β = 1 , γ = 0 , then the training objective specializes to standard adversarial training.
The proposed training objective (8) differs from adversarial training by allowing large perturbations of an input to be rejected, when the classiﬁer is likely to predict them incorrectly.
As we show experimentally, the proposed method of robust training with rejection typically has higher robustness on unseen adversarial attacks that have a larger perturbation budget (cid:15) than that used in training, whereas the robustness of standard adversarial training drops signiﬁcantly on those unseen adversarial attacks.
Choice of the Detector
Recall that we deﬁned the detector as a general parametric function of the classiﬁer’s un-normalized prediction, that outputs the probability of rejection h ⊥ ( x ; θ ) =
u ( (cid:101) h ( x ; θ c ) ; θ d ) .
We explored a few approaches for deﬁning u ( · ; θ d ) based on smooth approximations of the prediction conﬁdence max y ∈Y h y ( x ; θ c ) .
For instance, we used a temperature-scaled log-sum-exponential approximation to the max function, followed by an afﬁne transformation and the Sigmoid function (in order to get a probabilistic output).
We also explored a multilayer fully-connected neural network to model the detector, which takes the prediction logits as its input and predicts the probability of rejection.
We found the neural network-based model of the detector to have consistently better performance compared to the simple conﬁdence-based approaches.
Therefore, we adopt this model of the detector in our experiments.
Design of Adaptive Attacks
We design strong adaptive attacks to evaluate the robustness with rejection of our method.
To compute robustness with rejection at budgets (cid:15) 0 and (cid:15) , we need to generate two adversarial examples x
(cid:48) ∈ N ( x , (cid:15) 0 ) and x (cid:48)(cid:48) ∈
N ( x , (cid:15) ) for each clean input ( x , y ) .
We generate the adversarial example x (cid:48) within the (cid:15)
-ball N ( x , (cid:15) 0 ) using the following objective:
[Equation]
The goal of the adversary is to make the detector reject the adversarial input by pushing the probability of rejection close to 1 3 .
We generate the adversarial example x (cid:48)(cid:48) within the larger (cid:15)
-ball N ( x , (cid:15) ) via the following objective:
[Equation]
By solving this objective, the adversary attempts to push both the probability of rejection h ⊥ ( x (cid:48)(cid:48) ; θ ) and the predicted probability of the true class h y ( x (cid:48)(cid:48) ; θ c ) close to 0 .
Thus, the goal of the adversary is to make the classiﬁer-detector accept and incorrectly classify the adversarial input.
We use the Projected Gradient Descent (PGD) method with Backtracking proposed by (Stutz, Hein, and Schiele 2020) to solve the attack objectives.
The hyperparameters for PGD with backtracking are speciﬁed in the experiment section.
Adaptive attacks for evaluating the baseline methods are discussed in the Appendix.
Experiments
In this section, we perform experiments to evaluate the proposed method (SATR) and compare it to the baseline methods.
Our main ﬁndings are summarized as follows:
1) SATR achieves higher robustness with rejection compared to adversarial training (with and without conﬁdence-based rejection) and CCAT (Stutz, Hein, and Schiele 2019).
On small perturbations, SATR has a much lower rejection rate compared to CCAT, which often rejects a large fraction of the perturbed inputs; 3) Our method outperforms both CCAT and adversarial training under unseen attacks;
We next provide details on the experimental setup, datasets and DNN architectures, baseline methods, and the performance metric.
Setup
We describe the important experimental settings in this section, and provide additional details about our method and the baselines in the appendix.
Datasets.
We perform experiments on the MNIST (LeCun 1998) and CIFAR-10 (Krizhevsky, Hinton et al.
2009) image datasets.
MNIST contains 50,000 training images and 10,000 test images from 10 classes corresponding to handwritten digits.
CIFAR-10 contains 50,000 training images and 10,000 test images from 10 classes corresponding to object categories.
Following the setup in (Stutz, Hein, and Schiele 2020), we compute the accuracy of the models on the ﬁrst 9,000 images of the test set and compute the robustness of the models on the ﬁrst ,000 images of the test set.
We use the last 000 images of the test set as a validation dataset for selecting the rejection threshold of the methods.
Baseline Methods.
We compare the performance of SATR with the following three baselines: (1) AT: standard adversarial training without rejection (i.e. accepting every input) (Madry et al.
; (2) AT + Rejection: adversarial training with rejection based on the prediction conﬁdence; (3) CCAT:
conﬁdence-calibrated adversarial training (Stutz,
Hein, and Schiele 2020).
DNN Architectures.
On MNIST, we use the LeNet network architecture (LeCun et al. 1989) for the classiﬁer θ c , and use a three-layer fully-connected neural network with width 256 and ReLU activation function for the detector θ d .
On
CIFAR-10, we use the ResNet-20 network architecture (He et al. 2016) for the classiﬁer θ c , and use a seven-layer fullyconnected neural network with width 1024, ReLU activation function and a batch normalization layer for the detector θ d .
Training Details.
On both MNIST and CIFAR-10, we train the model for 100 epochs with a batch size of 128.
We use standard stochastic gradient descent (SGD) starting with a learning rate of 0.1.
The learning rate is multiplied by 0.
after each epoch.
We use a momentum of 0.9 and do not use weight decay for SGD.
We split each training batch into two sub-batches of equal size, and use the ﬁrst subbatch for the ﬁrst and the third loss terms in our training objective (Eq. (8)), and use the second sub-batch for the second loss term in our training objective. This is similar to the 50% adversarial training strategy used e.g. , in (Stutz, Hein, and Schiele 2019).
We set the hyper-parameters β = 1 and γ = 1 in our training objective without tuning. On
MNIST, we train the model from scratch, while on CIFAR10 we use an adversarially-trained model to initialize the classiﬁer parameters θ c .
On CIFAR-10, we also augment the training images using random crop and random horizontal ﬂip.
We use the standard PGD attack (Madry et al. 2018) to generate adversarial training examples.
On MNIST, we use the PGD attack with a step size of 0.01, 40 steps and a random start.
On CIFAR-10, we use the PGD attack with a step size of 2 / 255 , 10 steps and a random start.
In the training objective, by default, we set (cid:15) = 0 .
3 and (cid:15)
= 0 .
1 for MNIST, and set (cid:15) = 8 / 255 and (cid:15)
= 2 / 255 for CIFAR-10.
Performance Metric.
We use the robustness with rejection at budgets (cid:15) 0 and (cid:15) , deﬁned as 1 − R rej (cid:15) 0 ,(cid:15) ( f ) , as the evaluation metric.
For a ﬁxed (cid:15) , we vary (cid:15)
from 0 to (cid:15) over a given number of values.
Note that the (cid:15) 0 in this performance metric is different from the ﬁxed (cid:15)
that is used in the training objective of the proposed method.
For convenience, we deﬁne the factor α :=
(cid:15) 0 / (cid:15) ∈ [0 , 1] , and calculate the robustness with rejection metric for each of the α values from the set { 0 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 1 . 0 } .
Note that each α value corresponds to an (cid:15) 0 value equal to α (cid:15) .
We plot a robustness curve for each method, where the α value is plotted on the x-axis and the corresponding robustness with rejection metric is plotted on the y-axis.
A larger value of robustness corresponds to better performance.
Referring to
Fig.
3, we note that at the right end of this curve ( (cid:15)
= (cid:15) ), the robustness 1 − R rej (cid:15),(cid:15) ( f ) corresponds to the standard deﬁnition of adversarial robustness without rejection (Madry et al.
At the left end of this curve ( (cid:15)
= 0 ), the robustness 1 − R rej 0 ,(cid:15) ( f ) corresponds to the robustness with rejection deﬁned by (Tram`er 2021).
In practice, we are mainly interested only in the robustness for small values of α , where the radius of perturbations (to be accepted) is small.
Evaluation.
We use the same approach to set the rejection threshold for all the methods.
Speciﬁcally, on MNIST, we set the threshold such that only 1% of clean validation inputs are rejected.
On CIFAR-10, we set the threshold such that only 5% of clean validation inputs are rejected.
We consider (cid:96) ∞ -norm bounded attacks and generate adversarial examples to compute the robustness with rejection metric via the PGD attack with backtracking (Stutz, Hein, and Schiele 2020).
We use a base learning rate of 0.05, momentum factor of 0.9, a learning rate factor of 1 . 25 , 200 iterations, and 10 random restarts for generating adversarial examples x (cid:48) within the (cid:15)
-ball N ( x , (cid:15) 0 ) .
For generating adversarial examples x (cid:48)(cid:48) within the larger (cid:15) -ball N ( x , (cid:15) ) , we use a base learning rate of 0.001, a momentum factor of 0.9, a learning rate factor of 1 . 1 , 1000 iterations, and 10 random restarts.
Results
We discuss the performance of the proposed method and the baselines on the CIFAR-10 and MNIST datasets.
Evaluation under seen attacks.
Figure 3 compares the robustness with rejection of the methods as a function of α for the scenario where the adaptive attacks used for evaluation use the same (cid:15) budget that was used for training the methods.
For the proposed SATR, the (cid:15) 0 value used for training is indicated (with the corresponding α value) using the vertical dashed line.
We observe that CCAT has comparable robustness to AT only for α = 0 , but its robustness quickly drops for larger α .
This suggests that CCAT rejects a large fraction of small input perturbations based on its conﬁdence thresholding method.
AT with conﬁdence-based rejection has higher robustness compared to standard AT on both datasets, which suggests that including even a simple rejection mechanism can help improve the robustness.
On CIFAR-10, the proposed
SATR has signiﬁcantly higher robustness with rejection for small to moderate α , and its robustness drops only for large α values (which are not likely to be of practical interest).
On
MNIST, the robustness of SATR is slightly better than or comparable to AT + Rejection for small to moderate α .
This suggests that SATR is successful at accepting and correctly classifying a majority of adversarial attacks of small size.
Evaluation under unseen attacks.
Figure 4 compares the robustness with rejection of the methods as a function of α for the unseen-adaptive-attack scenario, wherein a larger (cid:15) (compared to training) is used for evaluation.
AT, both with and without rejection, performs poorly in this setting, suggesting that it does not generalize well to unseen (stronger) attacks.
CCAT has relatively high robustness for α = 0 ; however, its robustness sharply drops for larger α values.
The signiﬁcantly higher robustness of SATR for a range of small to moderate α values suggests that the proposed training method learns to reject larger input perturbations, even if the attack is unseen.
Ablation study.
We performed an ablation experiment to study the effect of the hyper-parameter (cid:15) 0 used by SATR [Figure] i t h R e j e c t
i o n [Figure] [Figure] c
[Figure]
i
[Figure]
during training.
The result of this experiment is shown in
Figure 5 for a few different (cid:15) 0 values.
Clearly, the choice (cid:15)
= 0 leads to poor robustness with rejection, suggesting that a small non-zero value of (cid:15) 0 is required for training to ensure that SATR does not reject too many small adversarial perturbations.
We also observe that a larger (cid:15) 0 during training typically leads to a higher robustness for large α values.
However, this may come at the expense of lower robustness for small α , as observed on CIFAR-10 for (cid:15)
= 4 / 255 .
Conclusion
We explored the problem of learning an adversarially-robust classiﬁer with a reject option.
We conducted a careful theoretical analysis of the problem and motivate the need for not rejecting small perturbations of the input.
We proposed a novel metric for evaluating the robustness of a classiﬁer with reject option that subsumes prior deﬁnitions of robustness, and provides a more ﬁne-grained analysis of the radius (size) of perturbations rejected by a given method.
We proposed a novel training objective for learning a robust classiﬁer with rejection that encourages small input perturbations to be accepted and classiﬁed correctly, while allowing larger input perturbations to be rejected when the classiﬁer’s prediction may be incorrect.
Experimental evaluations using strong adaptive attacks demonstrate signiﬁcant improvement in the adversarial robustness with rejection of the proposed method, including the setting where unseen attacks with a larger (cid:15) budget are present during evaluation.
Appendix Proof for Theorem
Theorem 2 (Restatement of Theorem 1) .
Consider binary classiﬁcation.
Let g ( x ) be any decision boundary ( i.e. , any classiﬁer without a rejection option).
For any 0 ≤ (cid:15) 0 ≤ (cid:15) , there exists a classiﬁer f with a rejection option such that
[Equation]
Moreover, the bound is tight: there exist simple data distributions and g such that any f must have R rej
[Equation]
≥
[Equation]
Proof.
For any r > 0 , let N ( g, r ) denote the region within distance r to the decision boundary of g :
[Equation]
Consider a parameter δ ∈ [0 , (cid:15) ] and construct a classiﬁer f δ with rejection as follows:
[Equation]
We will show that any sample ( x , y ) contributing error to
R rej (cid:15) 0 ,(cid:15) ( f δ ) must also contribute error to R (cid:15) (cid:48) ( g ) , where (cid:15) (cid:48)
= max { (cid:15)
+ δ, (cid:15) − δ } .
This will prove that R rej (cid:15) 0 ,(cid:15) ( f δ ) ≤ R (cid:15) (cid:48) ( g ) , which specializes to R rej (cid:15) 0 ,(cid:15) ( f δ ) ≤ R ( (cid:15)
+ (cid:15) ) /
2 ( g ) for the choice δ = ( (cid:15) − (cid:15) 0 ) / 2 .
Consider the following two cases:
• Consider the ﬁrst type of error in R rej (cid:15) 0 ,(cid:15) ( f δ ) : max x (cid:48) ∈N ( x ,(cid:15) 0 )
[ f δ ( x (cid:48) )
(cid:54) =
y ]
=
This implies
that
there exists x (cid:48) ∈
N ( x , (cid:15) 0 )
such that f δ ( x (cid:48) ) (cid:54)
=
y
So
there are two subcases to consider:
(1) x (cid:48) ∈
N ( g, δ ) : in this case x ∈ N ( g, δ + (cid:15) 0 ) .
(2) g
( x (cid:48) )
(cid:54) =
y : in this case either g ( x ) (cid:54) =
y , or g ( x ) =
y (cid:54) =
g ( x (cid:48) ) and thus x ∈ N ( g, (cid:15) 0 ) .
In summary, either g
( x ) (cid:54) =
y
or
x
∈ N ( g,
(cid:15)
+
δ )
• Next consider the second type of error in R rej (cid:15) 0 ,(cid:15) ( f δ ) : max
x (cid:48)(cid:48) ∈N (
x ,(cid:15) ) 1 [ f δ ( x (cid:48)(cid:48) )
(cid:54)∈ { y, ⊥} ] =
This means there
exists an x (cid:48)(cid:48) ∈
N ( x , (cid:15) ) such that f δ ( x (cid:48)(cid:48) )
(cid:54)∈ { y, ⊥} ,
i
e. , x (cid:48)(cid:48) (cid:54)∈ N ( g, δ ) and g ( x (cid:48)(cid:48) )
(cid:54)
=
y .
This implies that all z ∈ N ( x (cid:48)(cid:48) , δ ) should have g ( z ) =
g ( x (cid:48)(cid:48) )
(cid:54) =
y .
In particular, there exists z ∈ N ( x (cid:48)(cid:48) , δ ) with d ( z , x ) ≤ (cid:15) − δ and g ( z ) (cid:54) =
y .
It can be veriﬁed that z = δ(cid:15) x
+ (cid:15) − δ(cid:15)
x (cid:48)(cid:48) , which is a point on the line joining x and x (cid:48)(cid:48) , satisﬁes the above condition.
In summary, either g ( x ) (cid:54) =
y , or g ( x ) =
y (cid:54) =
g ( z ) and thus x ∈ N ( g, (cid:15) − δ ) .
Overall, a sample ( x , y ) contributing error to R rej (cid:15) 0 ,(cid:15) ( f δ ) must satisfy either g ( x ) (cid:54) =
y or x ∈ N ( g, (cid:15) (cid:48) ) , where (cid:15) (cid:48)
= max { (cid:15)
+ δ, (cid:15) − δ } .
Clearly, such a sample also contributes an error to R (cid:15) (cid:48) ( g ) .
Therefore, we have
[Equation]
which leads to the desired bound when δ =
( (cid:15) − (cid:15) 0 ) / 2 and (cid:15) (cid:48)
=
( (cid:15) + (cid:15) 0 ) / 2 .
To show that the bound is tight, consider the following data distribution.
Let x ∈ R and y ∈ {− 1 , +1 } , 0 < (cid:15) 0 < (cid:15) , and let α ∈ (0 , 1 / 2) be some constant: ( x , y ) is ( − 4 (cid:15), − 1) with probability (1 − α ) / 2 , ( − (cid:15) 0 / 4 , − 1) with probability α/ 2 , ( (cid:15) 0 / 4 , +1) with probability α/ 2 , and (4 (cid:15), +1) with probability (1 − α ) / 2 .
Let g ( x ) :
= sign ( x + (cid:15) ) .
It is clear that R ( (cid:15)
+ (cid:15) ) /
2 ( g ) = α/ 2 .
It is also clear that any f must have
R rej (cid:15) 0 ,(cid:15) ( f ) ≥ α/ 2 since the points x = − (cid:15) 0 / 4 and x = (cid:15) 0 /have distance only (cid:15) 0 / 2 but have different labels.
Experimental Details General Setup
Software and Hardware.
We ran all our experiments with PyTorch and NVDIA GeForce RTX 2080Ti GPUs.
Number of Evaluation Runs.
We ran all experiments once with ﬁxed random seeds.
Dataset.
MNIST (LeCun 1998) is a large dataset of handwritten digits.
Each digit has 5,500 training images and 1,000 test images.
Each image is 28 × 28 grayscale.
CIFAR10 (Krizhevsky, Hinton et al.
2009) is a dataset of 32 ×color images with ten classes, each consisting of 5,000 training images and 1,000 test images.
The classes correspond to different objects such as dogs, frogs, ships, trucks, etc.
We normalize the range of pixel values to [0,1].
Multiple restart PGD Attacks.
We use PGD attacks with multiple restarts for evaluating the robustness.
Following (Stutz, Hein, and Schiele 2020), we initialize the perturbation δ uniformly over directions and norm:
[Equation]
where δ (cid:48) is sampled from the standard Gaussian and u ∈ [0 , 1] from a uniform distribution.
We also consider zero initialization, i.e., δ = 0 .
We allocate one restart for zero initialization, and multiple restarts for the random initialization.
We ﬁnally select the perturbation corresponding to the best objective value obtained throughout the optimization.
Baselines
We consdier three baselines: (1) AT: adversarial training without rejection (i.e. accepting every input); (2) AT+Rejection: adversarial training with conﬁdence-based rejection; (3)
CCAT: conﬁdence-calibrated adversarial training.
We provide their training details below.
AT and AT+Rejection.
We consider the standard adversarial training proposed in (Madry et al. 2018) and use the following training objective:
[Equation]
We train on 50% clean and 50% adversarial examples per batch.
On MNIST, we use the LetNet network architecture (LeCun et al. 1989) and train the network for 100 epochs with a batch size of 128.
We use standard stochastic gradient descent (SGD) starting with a learning rate of 0.1.
The learning rate is multiplied by 0.95 after each epoch.
We use a momentum of 0.9 and do not use weight decay for SGD.
We use the PGD attack to generate adversarial training examples with (cid:15) = 0 .
3 , a step size of 0.01, 40 steps and a random start.
On CIFAR-10, we use the ResNet-20 network architecture (He et al. 2016) and train the network following the suggestions in (Pang et al. 2021a).
Speciﬁcally, we train the network for 110 epochs with a batch size of 128 using stochastic gradient decent (SGD) with Nesterov momentum and learning rate schedule.
We set momentum 0.9 and (cid:96)weight decay with a coefﬁcient of 5 × 10 − 4 .
The initial learning rate is 0.1 and it decreases by 0.1 at 100 and 105 epoch respectively.
We augment the training images using random crop and random horizontal ﬂip.
We use the PGD attack to generate adversarial training examples with (cid:15) = 8 / 255 , a step size of 2 / 255 , 10 steps and a random start.
CCAT.
We follow the original training settings for CCAT in (Stutz, Hein, and Schiele 2020) and train models on
MNIST and CIFAR-10 using standard SGD.
On MNIST, we use the LetNet network architecture (LeCun et al. 1989) and train the network for 100 epochs with a batch size of 100 and a learning rate of 0.1.
On CIFAR-10, we use the ResNet-20 network architecture (He et al. 2016) and train the network for 200 epochs with a batch size of 100 and a learning rate of 0.075.
We augment the training images using random crop and random horizontal ﬂip on CIFAR-10.
On both MNIST and CIFAR-10, we use learning rate schedule and the learning rate is multiplied by 0.95 after each epoch.
We use a momentum of 0.9 and do not use weight decay for SGD.
We use the PGD attack with backtracking to generate adversarial training examples: we use a learning rate of 0.005, a momentum of 0.9, a learning rate factor of 1.5, 40 steps and a random start.
We randomly switch between the random initialization and zero initialization.
We train on 50% clean and 50% adversarial examples per batch.
Adaptive Attacks for Conﬁdence-based Detectors
We design adaptive attacks to evaluate the robustness with rejection of classiﬁers using conﬁdence (or maximum softmax score) to reject adversarial inputs (e.g. AT+Rejection and CCAT).
To compute robustness with rejection at budgets (cid:15) 0 and (cid:15) , we need to generate two adversarial examples x
(cid:48) ∈ N ( x , (cid:15) 0 ) and x (cid:48)(cid:48) ∈
N ( x , (cid:15) ) for each clean input ( x , y ) .
We generate the adversarial example x (cid:48) within the (cid:15) 0 -ball
N ( x , (cid:15) 0 ) using the following objective:
[Equation]
The goal of the adversary is to make the detector reject the adversarial input by pushing the softmax output of the network close to uniform.
We generate the adversarial example x (cid:48)(cid:48) within the larger (cid:15)
-ball N ( x , (cid:15) ) via the following objective:
[Equation]
By solving this objective, the adversary attempts to ﬁnd misclassiﬁed adversarial examples with high conﬁdence.
Thus, the goal of the adversary is to make the classiﬁer-detector system accept and incorrectly classify the adversarial input.
We use the Projected Gradient Descent (PGD) method with Backtracking proposed by (Stutz, Hein, and Schiele 2020) to solve the attack objectives.
The hyperparameters for PGD with backtracking are speciﬁed in the experiment section of the main paper.
Additional Results
Evaluation on clean test inputs.
We evaluate our method and the baselines on clean test inputs.
We calculate the accuracy with rejection deﬁned as the accuracy on accepted test inputs, and the rejection rate deﬁned as the fraction of rejected test inputs.
The results given in Table 1 show that our method SATR has comparable performance to the baselines on clean test inputs.
Dataset
[Table]