Accelerating Certified Robustness Training via Knowledge Transfer
Abstract
Training deep neural network classifiers that are certifiably robust against adversarial attacks is critical to ensuring the security and reliability of AI-controlled systems.
Although numerous state-of-the-art certified training methods have been developed, they are computationally expensive and scale poorly with respect to both dataset and network complexity.
Widespread usage of certified training is further hindered by the fact that periodic retraining is necessary to incorporate new data and network improvements.
In this paper, we propose Certified Robustness
Transfer (CRT), a general-purpose framework for reducing the computational overhead of any certifiably robust training method through knowledge transfer.
Given a robust teacher, our framework uses a novel training loss to transfer the teacher’s robustness to the student.
We provide theoretical and empirical validation of CRT.
Our experiments on CIFAR-10 show that CRT speeds up certified robustness training by 8 × on average across three different architecture generations while achieving comparable robustness to state-of-the-art methods.
We also show that CRT can scale to large-scale datasets like ImageNet.
Introduction
Deep Neural Networks (DNNs) are susceptible to adversarial evasion attacks [31, 9], that add a small amount of carefully crafted imperceptible noise to an input to reliably trigger misclassification.
As a defense, numerous training methods have been proposed [25, 40, 35] to grant a classifier empirical robustness.
But in the absence of any provable guarantees for this robustness, these defenses were frequently broken [1, 32].
These failures have motivated the development of training methods that grant certifiable/provable robustness to a classifier, hence safeguarding them against all attacks (known or unknown) within a pre-determined threat model.
Such methods are broadly categorized as either deterministic or probabilistic [23].
Deterministic robustness training methods [12, 26, 33, 34, 28, 10, 41, 30] rely on computing provable bounds on the output neurons of a classifier for a given perturbation budget in the input space.
However, the deterministic robustness guarantees provided by these methods come at a high computational cost.
Probabilistic robustness training methods address this limitation by providing highly probable ( e.g., with 0.99 probability) robustness guarantees at a greatly reduced computational cost.
Within this category, randomized smoothingbased methods [19, 3, 29, 22, 20, 7, 37, 39, 16, 15] are considered the state-of-the-art for certifiable robustness in the ℓ 2 -space.
Even so, these training methods remain an order of magnitude slower than standard training.
In commercial applications where constant model re-deployment occurs to provide improvements (see Figure 1), re-training using computationally expensive methods is burdensome.
[Figure]
T opA cc u r a cy ( % )
[Figure]
N u m be r o f P a r a m s (
M )
In this work, reduce the training overhead of smoothing-based robustness training methods with minimal impact on the robustness achieved.
We propose Certified Robustness Transfer (CRT), a knowledge transfer framework that significantly speeds up the process of training ℓcertifiably robust image classifiers.
Given a pre-trained classifier that is certifiably robust ( i.e., teacher ), CRT trains a new classifier ( i.e., student ) that has comparable levels of robustness in a fraction of the time required by state-of-the-art methods.
CRT brings down the cost of training certifiably robust image classifiers to be comparable to standard training while preserving stateof-the-art robustness.
On CIFAR-10, CRT speeds up training by an average of 8 × across three different architecture generations compared to a state-of-the-art robustness training method [15].
Furthermore, we show that state-of-the-art robustness training is only necessary to train the initial classifier.
Afterward, CRT can be continuously reused to transfer robustness in order to expedite future model re-deployments and greatly reduce costs associated with computational resources.
Our contributions can be summarized as follows:
• We present Certified Robustness Transfer (CRT), the first framework, to our knowledge, that can transfer the robustness of a certifiably robust teacher classifier to a new student classifier.
CRT greatly reduces the time required to train certifiably robust image classifiers relative to existing state-of-the-art methods while achieving comparable or better robustness.
• We provide a theoretical understanding of CRT, showing how our approach of matching outputs enables robustness transfer between the student and teacher irrespective of the certified robustness training method used to train the teacher.
• On CIFAR-
, we show that CRT trains certifiably robust classifiers on average 8 × faster than a state-of-the-art method while having comparable or better Average Certified Radius (by 8% in the best case).
Furthermore, CRT reduces the cumulative computational cost of training three classifiers by 87 .
• We also show that CRT can be reused in a recursive manner, thus supporting a continuous redeployment scenario ( e.g., in commercial applications).
Finally, we show that CRT remains effective on a large-scale dataset, ImageNet.
Background
In this section, we briefly introduce certified robustness and discuss notable existing methods for training certifiably robust image classifiers using randomized smoothing.
2.1 Preliminaries
Problem Setup.
Consider a neural network classifier f parameterized by θ (denoted f θ ) trained to map a given input x ∈ R d to a set of discrete labels Y using a set of i.i.d.
samples
S = { ( x 1 , y 1 ) , ( x 1 , y 1 ) , · · · , ( x n , y n ) } drawn from a data distribution D .
The output of the classifier can be written as f θ ( x ) =
arg max c ∈Y z cθ ( x ) .
Here z θ ( x ) is the softmax output of the classifier and z cθ ( x ) denotes the probability that image x belongs to class c .
Certified Robustness via Randomized Smoothing.
The robustness of the classifier f θ for a given input pair ( x, y ) is defined using the radius of the largest ℓ 2 ball centered at x within which f θ has a constant output y .
This radius is referred to as robust radius and it can mathematically be expressed as:
[Equation]
, when f
[Equation]
=
y , when f θ
[Equation]
x
[Equation]
̸ = y
[Equation]
Within this ℓ 2 -neighborhood of x , f θ is considered to be certifiably robust .
Therefore, to improve the robustness of a classifier, one needs to maximize this robust radius corresponding to any point sampled from the given data distribution.
Directly maximizing the robust radius of a DNN classifier is an NP-hard problem [17].
Therefore, several prior works attempt to derive a lower bound for the robust radius [21, 19, 3].
This lower bound, often termed as the certified radius , satisfies the following condition: 0 ≤ CR ( f θ ; x, y ) ≤ R ( f θ ; x, y ) , for any f θ , ( x , y ).
In this paper, we utilize the certified robustness framework derived by Cohen et al.
[3] using randomized smoothing.
Given a classifier f θ , they first define the smooth classifier g θ as:
Definition 2.1.
For a given (base) classifier f θ and σ > 0 , the smooth classifier g θ corresponding to f θ is defined as follows:
[Equation]
Simply put, g θ returns the class c , which has the highest probability mass under the Gaussian distribution N ( x, σ 2 I ) .
Using Theorem 2.2, they proved that if the smooth classifier correctly classifies a given input x , it is certifiably robust at x .
They also provided an analytical form of the ℓcertified radius at x .
Theorem 2.2.
Let f θ : R d (cid:55)→ Y be a classifier and g
θ be its smoothed version (as defined in
Definition 2.1).
For a given input x ∈ R d and corresponding ground truth y ∈ Y , if g θ correctly classifies x
as y , i.e.,
[Equation]
then g θ is provably robust at x within the certified radius R given by:
[Equation]
where Φ is the c.d.f.
of the standard Gaussian distribution.
This certified radius is a tight lower bound of the robust radius defined in Equation 1, i.e., it is impossible to certify g θ at x for a radius larger than CR .
2.2 Training Methods for Maximizing Certified Radius
In addition to the theoretical framework discussed above, Cohen et al.
[3] also propose a simple yet effective method for training the base classifier in a way that maximizes the ℓ 2 certified radius of the smooth classifier, as expressed in Equation 4.
We include an evaluation of their method in
Appendix B.6.
Following their work, several other works build upon the randomized smoothing framework and propose training methods that better maximize the ℓ 2 certified radius of the smooth classifier.
Salman et al.
[29] proposed combining adversarial training [25] with randomized smoothing (called SmoothAdv).
They adapted the vanilla PGD attack to target the smooth classifier g θ instead of the base classifier f θ .
Zhai et al.
[39] proposed a new robustness loss, a hinge loss that enforces maximization of the soft approximation of the certified radius.
Their method (called MACER) is faster than SmoothAdv as it does not use adversarial training.
More recently, Jeong et al.
[15] proposed training with a convex combination of samples along the direction of adversarial perturbation for each input to regularize over-confident predictions.
Their method (called SmoothMix) is the current state-of-the-art in the domain of ℓ 2 certified robust image classifiers.
Finally, we note the
Consistency regularization method proposed by Jeong et al.
[16], which adds a regularization loss to existing methods that helps better maximize the certified radius.
[Table]
Maximizing Certified Radius via Knowledge Transfer
Although prior works have proposed methods for increasing the certified radius of the smooth classifier, their training overhead is significant, making them much slower than standard training.
As we show in Table 1, training a certifiably robust ResNet110 classifier to convergence using
SmoothAdv, MACER, and SmoothMix is 46 .
20 × , 20 . 86 × , and 4 . 97 × slower, respectively, compared to training a non-robust classifier with standard training.
Given constant innovations in architecture design (Figure 1) and the influx of new data, which may result in various tweaks to deployed networks that elicit retraining, the large overhead of state-of-theart robustness training methods makes preserving certified robustness across model re-deployment difficult.
Therefore, we propose Certified Robustness Transfer (CRT), a training method that improves the usability of certified robustness training methods by dramatically reducing their training overhead while preserving the certified robustness.
Given the base classifier of a pre-trained certifiably robust smooth classifier , we leverage the knowledge transfer framework to guide the training of a new base classifier (and its associated robust smooth classifier).
In this section, we describe our method and provide theoretical justification for its effectiveness.
3.1 Transferring Certified Robustness
From Equation 4, it follows that training the base classifier to maximize P η
( f θ ( x + η ) =
y ) for any given input x will result in the maximization of the certified radius associated with the smooth classifier, provided Equation 3 is satisfied.
Thus, for the base classifier f θ ( x ) , our goal is to maximize the following quantity over the training set:
[Equation]
In the above equation, like prior works [3, 29, 39], we leverage the fact that the softmax output of a classifier can be treated as a continuous and differentiable approximation of its arg max output.
Methods like SmoothAdv [29], MACER [39] and SmoothMix [15] that target ℓ 2 certifiable robustness
propose
training objectives that maximize this term.
Now, suppose we have a pre-trained base classifier f ϕ .
It follows that E η [ z yϕ ( x + η )] ≥ 0 .
Through straightforward algebraic manipulations (see Appendix A), we derive the following lower bound:
[Equation]
That is to say that, for a given input x i , if we minimize the difference between the softmax outputs of the teacher and the student ( f ϕ and f θ ) corresponding to the correct label
y
i , we will maximize
Equation 5 for the student.
However, to ensure that the student has a non-trivial certified radius, we must also ensure that Equation 3 is satisfied.
If we assume that Equation 3 holds true for the teacher ( i.e., the base classifier of a certifiably robust smooth classifier), this condition can also be achieved for the student by matching the overall softmax output of the student to that of the teacher.
3.2 Certified Robustness Transfer (CRT)
Based on the previous discussion, we now describe our method for training a certifiably robust classifier through knowledge transfer.
First, we obtain a pre-trained base classifier f ϕ , which has been trained using a randomized smoothing based robustness training method as this maximizes
1: Input: Training data distribution D , certifiably robust teacher base classifier f ϕ , noise level σ , total training iterations T , learning rate α 2: Output: Certifiably robust student base classifier f θ 3: θ ← random initialization 4: i ←5: while i < T do 6:
From D , sample a batch of inputs { x 1 , x 2 , · · · , x n } .
From N (0 , σ 2 I ) , generate a batch of Gaussian noise samples { η 1 , η 2 , · · · , η n } .
8: l i ← 1 n (cid:80) nj =1 ∥
z ϕ ( x j + η j ) − z θ ( x j + η j ) ∥9: θ ← θ − α · ∇ θ l i 10: i ← i +11: end while E η [ z yϕ ( x + η )] .
Next, we use f ϕ as a teacher to train a new student base classifier f θ .
The student is trained to match the output of the teacher.
In doing so, we can maximize the certified radius of the associated smooth classifier g θ (Equation 6), as well as ensure that Equation 3 is satisfied.
We describe our implementation in Algorithm 1.
Given a batch of inputs, we first perturb them with additive Gaussian noise. Next, we compute the ℓ 2 distance between the student and the teacher’s outputs for these Gaussian perturbed inputs.
This distance serves as our loss function, and we update the parameters of the student to minimize this loss.
At test time, the classifier f θ is converted to its smooth version g θ following Definition 2.1.
3.3 Prior Works on Robustness Transfer
Several prior works have examined transferring adversarial robustness between classifiers, but these works have been limited to transferring empirical rather than certified robustness [2, 8, 14, 42, 43].
Of note is the work by Goldblum et al.
[8] in which they combine adversarial training [25] with knowledge distillation [13].
They show that distilling knowledge from a large network to a small network improves its empirical robustness as compared to training the small network on its own, but their method makes no effort to improve the computational cost of adversarial training.
Evaluation
Our goal is to improve the usability of randomized smoothing based robustness training methods.
In this section, we demonstrate how CRT enables the reuse of an existing certifiably robust classifier to train new certifiably robust classifiers at significantly reduced training cost compared to prior methods.
In our first experiment, we train a ResNet110 classifier with a state-of-the-art method, i.e.,
SmoothMix [15]), and use CRT to transfer its robustness to train several newer generation classifiers.
In a second experiment, we recursively use CRT to train a newer generation classifier using the previous generation classifier that was also trained using CRT.
In each experiment, we compare the certified robustness of classifier trained using CRT against a classifier trained using SmoothMix (Section 4.1).
We find that classifiers trained using CRT are similarly robust as when trained using
SmoothMix but only require a fraction of training time (Section 4.2).
Our main results are generated using the CIFAR-10 dataset [18], but we also demonstrate the effectiveness of CRT on ImageNet [5] (Section 5.3).
Both these datasets are open-source and free for non-commercial use.
Architectures.
We use several popular DNN architectures that were proposed to either improve upon the visual recognition performance of the previous generation architectures or preserve performance while requiring significantly fewer parameters (or both).
For the CIFAR-10 experiments, we use
ResNet110 [11], ResNeXt29-2x64d [36], DLA [38], and RegNetX_200MF [27].
Training details.
All SmoothMix classifiers were trained using the code made available by the authors 3 and the hyperparameters reported by them [15].
All CRT classifiers were trained using
Stochastic Gradient Descent till convergence (200 epochs), with a batch size of 128.
Further hyperparameter details are available in Appendix C.1.
Unless specified, we report results for noise level σ = 0 .
25 in the main paper.
Additional results for higher noise levels σ = 0 .
5 and 1 . 0 are reported in Appendix B.1.
Evaluation Metrics.
We report our results using two metrics.
First, as done in prior work, we measure the certified robustness of a classifier based on (1) the certified test accuracy at ℓ 2 radius r 4 , which is defined as the fraction of test set inputs that the smooth classifier classifies correctly within an ℓball of radius r centered at each input, and (2) average certified radius (ACR), which is the average of the certified radius across all inputs in the test set:
[Equation]
On CIFAR-10, we compute these metrics using the entire test set.
Second, we measure training time of a classifier based on the per-epoch time and total training time .
The total training time is computed once the model’s loss has converged.
All classifiers were trained on the same machine with a single
Nvidia Titan V GPU.
4.1 Certified Robustness Comparison
Standard CRT Training.
Given a ResNet110 classifier trained using SmoothMix, we transfer its robustness to several newer generation classifiers.
We compare the certified robustness of these classifiers with their SmoothMix trained versions.
The results are summarized in Table 2.
We observe that using CRT does not reduce the certified robustness of the trained classifier compared to training with SmoothMix.
In fact, interestingly, CRT trained classifiers exhibit higher certified robustness compared to their SmoothMix baseline.
Not only do CRT trained classifiers have higher ACR (improvement of 8 . 1% in the best case), they also exhibit higher certified accuracy at different ℓ 2 radii.
Furthermore, CRT remains effective even as the generation gap between the student and the teacher increases.
This implies that the same teacher can potentially be reused indefinitely, amortizing the teacher’s training cost to a constant.
These results empirically validate our theoretical justification of
CRT.
Finally, we note that in Table 2, the accuracy on clean inputs and ACR of CRT trained classifiers follow the same trend as in Figure 1, thus motivating the need for periodic model re-deployment to incorporate architectural improvements.
[Table]
Recursive CRT Training.
We now explore the effectiveness of CRT if it is used recursively, i [Table] e., the newest generation is trained using a CRT trained classifier from the previous generation as the teacher.
We begin with a ResNet110 trained using SmoothMix.
Then, all subsequent classifiers are trained using CRT recursively and report the results in Table 3.
The chain length measures the number of times
CRT was used.
For example, the DLA network, with a chain length of 2, is the result of using CRT [Table] twice: once to transfer the SmoothMix trained ResNet110 network’s performance to the ResNeXt292x64d network and once to transfer the CRT trained ResNeXt29-2x64d network’s performance to the
DLA network.
We observe that the certified robustness of the resulting classifiers remains high even with recursive use of CRT.
The empirical results are to be expected given our theoretical understanding of CRT:
In order to train a robust student, we only require that the teacher is already robust ( i.e., satisfies the condition of Theorem 2.2) irrespective of the training method used to achieve robustness .
Thus, we expect CRT to remain effective even at longer chain lengths.
In Section 5.2, we will highlight the relationship between the teacher’s training method and the robustness of a CRT trained student.
4.2 Training Time Comparison
Having established that CRT effectively transfers certified robustness between classifiers, we now evaluate its training overhead.
For comparison, we also evaluate the training overhead of SmoothMix.
In Table 4, we report the per-epoch time and total time of training different architectures with each method.
For brevity, we only compare the training time for standard CRT.
5 We observe that the perepoch time of CRT is significantly lower than SmoothMix.
Similarly, when trained until convergence, the total training time of CRT is significantly lower.
Across the three architectures that we run our experiments on, CRT achieves an average epoch time speedup of 10 .
75 × .
Comparing overall training times, CRT speeds up training by, on average, 8 . 06 × .
If we consider the real-world scenario where the model has to be periodically redeployed to incorporate architectural improvements, the cumulative training time using SmoothMix is 96.21 hours as each new architecture is trained from scratch.
With CRT, the cumulative time is reduced to 11.70 hours representing a 87 .
84% savings in costs associated with computational resources.
Teacher’s availability.
So far, we assumed the availability of a certifiably robust teacher (ResNet110).
We argue that this is a reasonable assumption as the amortized cost associated with the one-time training of a robust teacher is negligible across many generations of the model.
Regardless, in
Section 5.1, we examine a scenario where the teacher is unavailable.
Under this scenario, we demonstrate how CRT can be used to speedup the training of ResNet110 for use as teacher.
[Table]
Discussion
In this section, we address the standout concerns about CRT.
The section layout is as follows: in
Section 5.1, we discuss the scenario in which a certifiably robust teacher is not readily available and demonstrate how CRT can still speed up robustness training; in Section 5.2, we examine how the method used to train the teacher affects the robustness of the student; in Section 5.3, we study the scalability of CRT using the ImageNet dataset; in Section 5.4, we compare CRT with a closely related prior work on fast certified robustness training, i.e., Consistency regularization [16]; in Section 5.5, we discuss the limitations of CRT; in Section 5.6, we address the broader impact of CRT.
5.1 Teacher Not Available
We’ve designed CRT under the assumption that a certifiably robust teacher is already available.
However, even if a certifiably robust teacher is not available, CRT can still speed up training.
Given a certifiable robust training method and a large network architecture, we can reduce the training overhead by robustly training a comparatively smaller network first.
Then, we can use CRT to transfer the robustness of the small network to a larger network.
In Table 5, we present results for such a process.
First, we trained a ResNet20 network using SmoothMix, then we used CRT to train a ResNet110 network.
We compare the robustness of a ResNet110 trained using this process with one trained using SmoothMix.
As we can see, the CRT ResNet110 network has comparable robustness with the SmoothMix ResNet110 network.
However, even when adding the teacher and student training times, CRT still speeds up training by approximately 2 × relative to SmoothMix.
while achieving a ∼ 2 × speedup in total training time.
[Table]
5.2 Teacher Training Method
We train a ResNet20 classifier using MACER [39], SmoothAdv [29], and SmoothMix [15].
For
MACER and SmoothAdv training, we use the code made available by the authors 67 and the hyperparameters reported by them.
Using CRT, we transfer the robustness of each teacher to a ResNet110 classifier.
The results are reported in Table 6.
For reference, we also report robustness of a ResNet110 network trained independently using the chosen robustness training methods.
Overall, we observe a slight variation in the robustness of the CRT trained networks depending on the teachers training method.
Based on Equation 6, this is expected as maximizing the teacher’s performance will in turn maximize the performance of the student.
Our empirical results align with this reasoning: the MACER teacher was the least robust of the three methods, and its student is similarly the least robust of the students.
However, in all cases, the CRT trained network obtained certified robustness comparable to its teacher.
5.3 Scalability
Here, we study the effectiveness of CRT on a large-scale dataset, i.e., ImageNet.
For this purpose, we train ResNet18 classifiers using three certified robustness training methods (MACER, SmoothAdv, and SmoothMix).
Next, we transfer their robustness to a ResNet50 classifier.
The results were generated on a 500 sample test set (following prior works [29, 39, 15]) and are summarized in Table 7.
For reference, we also report robustness of a ResNet50 network trained independently using the chosen robustness training methods.
In all cases, we observe that students achieve certified robustness comparable to their respective teachers.
Therefore, CRT remains effective even on a more complex dataset.
[Table]
[Table] 5.4 Comparison with Consistency Regularization
In Section
[Table]
we compared CRT against SmoothMix as it has state-of-the-art ACR.
However, another closely related work was recently published by Jeong & Shin [16], which shows state-of-the-art
ACR and potential training time improvements.
They proposed a consistency regularization loss that improves the certified robustness of smooth classifiers by enforcing the base classifier’s soft outputs to be consistent across multiple noisy copies of a given input.
Therefore, their additional computational overhead scales linearly with the number of noisy samples used to compute the consistency loss.
With respect to computational overhead, CRT adds only one forward pass, i.e., the pass through the teacher.
When paired with Gaussian data augmentation training, their regularization loss significantly improves the certified robustness of a smooth classifier.
By applying their regularization loss over only two noisy copies of the input, they can achieve better certified robustness than prior state-of-theart robustness training methods like MACER [39] and SmoothAdv [29].
The key difference between CRT and consistency regularization is in the training overhead when combined with other state-of-the-art certified training methods.
Consistency regularization augments classifier training with an additional loss term.
Therefore, their training overhead is dominated by the training method selected.
In their experiments, they focused on Gaussian data augmentation, which adds little to no training overhead relative to standard training.
However, if a more computationally intensive method was selected ( e.g., MACER), they remark their training overhead would dramatically increase.
With respect to CRT, if a teacher is available ( i.e., a previous generation model), the overhead of CRT is agnostic to the training method.
If it is not available, we demonstrated in Section 5.1, that CRT can still greatly reduce training overhead.
For interested readers, we include results for transferring robustness from a teacher trained using Consistency regularization in Appendix B.4.
5.5 Limitations
In this paper, we use probabilistic certified robustness methods as they rely on Theorem 2.2 and, thus, are designed to maximize the certified radius (Equation 4).
We found that deterministic methods ( e.g.,
CROWN-IBP [41]) impose a stricter training requirement on the base teacher classifier.
For a given input, deterministic training methods require the base classifier to be correct for all inputs within the ℓ 2 -norm ball, rather than simply be likely to correctly classify inputs within the ℓ 2 -norm ball.
This restriction lowers the potential ACR of the smooth teacher classifier, which also lowers the ACR of the student trained using CRT.
For example, when using CROWN-IBP [41] to train a ResNeXt base classifier, the ACR for the corresponding smooth classifier is only 0.064.
When transferring the robustness of this ResNeXt classifier to a WideResNet34-10 student, we get an ACR of 0.065.
Additionally, we note that the classifier architectures we present in the paper are restricted to CNNs.
Recently, a new class of transformer-based image classifiers [6, 24, 4] have been proposed that show improved performance over CNN classifiers.
We briefly studied the effectiveness of CRT when transferring robustness between CNN and transformer architectures using ViT [6] and present the results in Appendix B.2, but further exploration is needed.
Finally, CRT has only been studied using the ℓ 2 norm and image data due to the limitations of current certified robustness training methods.
5.6 Broader Impacts
As we have shown, our work improves the efficiency of training certifiably robust classifiers, in an effort to improve the security of AI-powered systems.
Beyond the broad negative societal impacts of machine learning, we are not aware of any impacts specific to our work.
Conclusion
In this paper, we proposed the first general-purpose framework to speed up the training of certifiably robust classifiers using knowledge transfer and randomized smoothing.
Our proposed method,
Certified Robustness Transfer (CRT) enables transferring the certified robustness of a classifier to another classifier at a cost comparable to standard training.
We provided a theoretical understanding of CRT and provided empirical evidence of its effectiveness.
On CIFAR-10, we showed that across several generations of classifier architectures, CRT trained classifiers 8 × faster than when using a state-of-the-art training method, while achieving comparable or better certified robustness.
Furthermore, CRT can reduce the training overhead of certified robustness training methods even when an initial robust classifier is not present.
The use of machine learning in security and safety critical environments motivates a need for models with certifiably robust performance, but the training overhead of existing certified robustness training methods inhibits their usability.
Our work addresses this issue, especially for commercial applications where periodical model re-deployment is inevitable
Acknowledgement
This work was supported by the Office of Naval Research under grants N00014-20-1-2858 and
N00014-22-1-2001, Air Force Research Lab under grant FA9550-22-1- and NVIDIA
GPU Grant.
Any opinions, findings, or conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
A Justification for CRT Training Objective
Following from the discussion in Section 3.1, we want to maximize E η [ z yθ ( x + η )] .
Given a nonnegative random variable t = E η
[ z yϕ ( x + η )] , we have
[Equation]
Here z ϕ is the soft output associated with classifier f ϕ that has been trained independently of f θ .
Therefore, we have
[Equation]
This implies that maximizing E η [ z yθ ( x + η )] is equivalent to minimizing E η [ z yϕ ( x + η ) − z yθ ( x + η )] .
B Additional Results B.1 Higher Noise Level
In the main paper, we conduct experiments on CIFAR-10 using noise level σ = 0 .
25 only.
Here, we report our main set of results on CIFAR-10 (Table 3) using higher σ values.
In Table 8, we report results using σ = 0 .
5 and in Table 9, we report results using σ = 1 . 0 .
B.2 Using ViT [6]
In the main paper, we used Convolutional Neural Network (CNN) based architectures.
However, there is another recently developed class of architectures that use Transformers for the task of image classification [6, 24, 4].
In Table 10, we present results measuring the effectiveness of CRT in transferring robustness from ResNet110 (a CNN-based classifier) to ViT [6] (a Transformer-based classifier).
For comparison, we also report results obtained on training ViT with SmoothMix.
CRT trained ViT classifiers perform comparable or better than their SmoothMix counterparts.
B.3 Training SmoothMix Classifiers with CRT Hyperparameters
For generating results using prior methods, we strictly adhere to the hyperparameters reported by them.
However, the hyperparameters that we use for CRT training is different than the ones used by prior methods (see Table 16).
In Table 11, we report results obtained on using CRT training hyperparameters with SmoothMix.
We note that there is not a significant difference in the robustness achieved using the two sets of hyperparameters.
B.4 Training Teacher with Consistency Regularization [16]
For our main set of results on CIFAR-10, we focused on SmoothMix.
In Section 5.4, we discussed
Consistency Regularization recently proposed by Jeong & Shin [16] as another method to attain certifiably robust classifiers with at a cost comparable to standard training depending on the setting.
Here, we show results when Consistency Regularization is used to train the teacher classifier in
Tables 12 and 13.
For comparison, in Table 12, we also report the results for training the classifiers using Consistency Regularization.
As with other training methods, CRT is effective in transferring the robustness of the teacher classifier irrespective of the training method.
B.5 ImageNet Results
In Table 14, we present an additional ImageNet result using a different student-teacher pair.
We note that, as in Table 7, CRT remains effective.
B.6 Gaussian Data Augmentation Baseline
Along with the theoretical framework for creating certifiably robust image classifiers using randomized smoothing, Cohen et al.
[3] also proposed a simple yet effective method for training classifiers with high certified robustness within this framework.
This method involves training the base classifier with Gaussian data augmentation.
To date, this method remains the fastest way to train classifiers with non-trivial certified robustness using the randomized smoothing framework.
However, this method is not as sophisticated as the more recently proposed methods, and so yields much poorer certified robustness than them.
Since our work is focused at accelerating certified robustness training, in
Table 15 we include the certified robustness of training time results for Gaussian data augmentation baseline for a more thorough comparison.
[Table]
- using SmoothMix [15] and CRT.
We use CRT to transfer the robustness of a ResNet110 trained using SmoothMix.
We report certified test accuracy at different values of ℓ 2 radius and the Average
Certified Radius (ACR).
The noise level σ is set to 1 . 0 .
[Table]
C Training Details
In this section, we provide all these details required to reproduce the results presented in the paper.
We begin by reporting the hyperparameters in Appendix C.1, followed by code links and other necessary instructions in Appendix C.2.
C.1 Hyperparamters
First, we provide details regarding training hyperparameters for our experiments in Table 16.
For all training, we perform regularization using a weight decay factor of 1 e − 4 .
Also for all training, learning rate is decayed by a factor of 0 .
1 at two pre-determined epochs (see column ‘LR Decay’ in
Table 16).
Next, we report method-specific hyperparameters in Table 17.
Note that CRT does NOT introduce any new hyperparameters.
C.2 Reproducing Results From This Paper
For reproducing results using SmoothAdv 8 , MACER 9 , SmoothMix 10 , and Consistency 11 , we follow instructions provided by the authors and use their respective codes.
For CRT, all necessary instructions and code required to reproduce the results are available at https://github .
com/
Ethos-lab/crt-neurips22 .
[Table]
accuracy at different values of ℓ 2 radius and the Average Certified Radius (
The architectures are sorted chronologically based on published date.
The noise level σ is set to 0 .
[Table]
[Table] [Table] [Table]
[Table] [Table]
The noise level σ is set to 0 .
[Table]
[Table] [Table]