Synergy-of-Experts: Collaborate to Improve Adversarial Robustness
Abstract
Learning adversarially robust models requires invariant predictions to a small neighborhood of its natural inputs, often encountering insufficient model capacity .
There is research showing that learning multiple sub-models in an ensemble could mitigate this insufficiency, further improving the generalization and the robustness.
However, the ensemble’s voting-based strategy excludes the possibility that the true predictions remain with the minority .
Therefore, this paper further improves the ensemble through a collaboration scheme—Synergy-of-Experts (SoE).
Compared with the voting-based strategy, the SoE enables the possibility of correct predictions even if there exists a single correct sub-model.
In SoE, every sub-model fits its specific vulnerability area and reserves the rest of the sub-models to fit other vulnerability areas, which effectively optimizes the utilization of the model capacity.
Empirical experiments verify that SoE outperforms various ensemble methods against white-box and transfer-based adversarial attacks.
The source codes are available at https://github.com/cuis15/synergy-of-experts .
Introduction
Deep models have been widely applied in various real-world applications including high-stakes scenarios (such as in healthcare, finance, and autonomous driving).
An increasing concern is whether these models make adversarially robust decisions [1, 2].
Recently, there are research revealing that an adversarially robust method requires invariant predictions to a small neighborhood of its natural inputs, thus often encountering insufficient model capacity [3, 4].
This limits the further improvement of robustness and has the undesirable degradation of generalization [5].
Learning multiple sub-models in an ensemble [6, 7] can mitigate this insufficiency.
Remarkably, there are research [8, 9, 10] proposing to minimize the vulnerability overlaps between each pair of sub-models and improving both robustness and generalization over a single model.
However, the voting-based ensemble may waste the limited capacity of multiple models.
In the example of three sub-models (see Figure 1(b)), the adversarial input that lies in the black areas can fool the ensemble successfully, i.e., more than half of sub-models must correctly classify the adversarial input.
Therefore, the ensemble’s voting-based strategy excludes the possibility that true [Figure] predictions remain with the minority .
In other words, learning an ensemble requires more than half of the sub-models to fit the same vulnerability areas, which leaves the following question unanswered whether we could only leverage a single sub-model to fit a vulnerability area and reserve the rest of the sub-models to fit other vulnerability areas.
Inspired by mixture-of-experts (MoE) [11], we propose to learn a collaboration among multiple submodels to optimally utilize the limited capacity.
As shown in Figure 1(c), the adversarial input that lies in the vulnerability overlaps of all sub-models can undoubtedly fool the collaboration.
Compared with the ensemble in Figure 1(b)), collaboration enables the possibility of correct predictions even if there exists a single correct sub-model.
Besides, learning a collaboration could enable every sub-model to fit its vulnerability area, which could collectively fix broader vulnerability areas.
Then, sub-models could collaboratively choose a trustworthy one to make the final predictions.
Classic MoE methods assume that the problem space is separable, and the separability is irrelevant to the learned model [11].
However, the non-i.i.d adversarial inputs [12] depend on the learned models and are hard to be classified by a learned gate in MoE. To tackle the above challenge, we propose Synergy-of-Experts (SoE), which explicitly builds the relationship between learned models and adversarial inputs.
Specifically, each sub-model has dual heads: one outputs a vector of predicted probability f θ ( · ) ; another outputs a scalar that measures the confidence of the prediction.
In the adversarial training phase, given an adversarial input x , each sub-model chooses an easy one(s) to feed itself.
The other head is meanwhile updated by comparing the predicted probability on the true label— f yθ ( · ) (a scalar).
In the inference phase, given an input, SoE chooses a sub-model with the largest confidence as the representative to output the overall prediction.
We highlight our key contributions as follows.
• We provide a new perspective on learning multiple sub-models for defending against adversarial attacks.
We show that the collaboration could make better decisions than the ensemble (Proposition 1), which implies collaboration may fix broader vulnerability areas.
• We propose a novel collaboration framework—SoE (see Section 3.2).
In the training phase, SoE minimizes the vulnerability overlap of all sub-models;
In the inference phase, SoE could effectively choose a representative sub-model to make correct predictions.
We also provide a comprehensive analysis illustrating the rationale of SoE in Appendix.
• Empirical experiments corroborate the SoE outperforms various ensemble methods [10, 8, 9, 13] against white-box and transfer attacks.
Related Work
Adversarial attack.
Adversarial attacks aim to craft the human-imperceptible adversarial input to fool the deep models.
Adversarial attacks could be roughly divided into white-box attacks in which the adversary is fully aware of the model’s structures [1, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26] and black-box attacks in which the deep models are treated as black boxes to the adversary [27, 28, 29, 30, 31, 32, 33, 34, 35, 36].
This paper focuses on building effective defense and select both white-box and black-box attack methods as our robustness evaluation metrics.
Adversarial defense.
Defending adversarial attacks is a challenging task and researchers have proposed various solutions.
Certified defense tries to learn provably robust deep models against normbounded (e.g., ℓ 2 and ℓ ∞ ) perturbations [37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49].
Empirical defense.
leverages adversarial data to build effective defense such as adversary detection [50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68] and adversarial training (AT), in which AT stands out as the most effective defense.
Researchers have investigated various aspects of AT, such as improving AT’s robustness or generalization [5, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 3], fixing AT’s undesirable robust overfitting [90, 91, 92], improving AT’s training efficiency [93, 94, 95, 96, 97, 98], understanding/interpreting AT’s unique traits [99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 49, 109, 110], applying AT into applications [111, 112], etc.
Besides, researchers have also actively investigated robust-structured models [113, 114, 115, 116, 117, 118, 119].
Nevertheless, the above research thoroughly investigated a single model; this paper focuses on the collaboration among multiple models for adversarial defense.
Ensemble methods for adversarial robustness.
The most relevant studies are the ensemble methods.
Ensemble methods such as bagging [6] and boosting [7] have been investigated for significantly improving the model’s generalization.
Motivated by the benefits of ensemble methods in improving generalization, researchers introduced an ensemble to improve the model robustness [10, 9, 8, 120].
Tramèr et al.
[120] proposed to reduce the adversarial transferability by training a single model with adversarial examples from multiple pretrained sub-models.
Pang et al.
[8] introduce a regularization method—ADP—to encourage high diversity in the non-maximal predictions of sub-models.
Kariyappa et al.
[9] improved the ensemble diversity by maximizing the introduced cosine distance between the gradients of sub-models with respect to the input.
Yang et al.
[10] proposed to distill non-robust features in the input and diversify the adversarial vulnerability.
These methods reduced overlaps of vulnerability areas between sub-models [10].
Compared with voting strategy in ensemble, mixture-of-experts (MoE) assumes that the problem space can be divided into multiple sub-problems through a gate module [11, 121].
However, in adversarial training, the adversarial samples, which depend on the learned models, are not i.i.d.
A vanilla MoE is hard to identify the best performing sub-models for each adversarial sample without the information about the learned models.
Collaboration to Defend Against Adversarial Attacks 3.1 Superiority of Collaboration
This section shows a collaboration , in theory, could make better decisions than an ensemble .
Ensemble.
Suppose that there are M learned sub-models { f θ 1 , f θ 2 , ..., f θ M } , given an input x , M sub-models make predictions (cid:8) f θ 1 ( x ) , f θ 2 ( x ) , ..., f θ M ( x ) (cid:9) .
The ensemble outputs a final prediction ensemble( x, f θ 1 , ..., f θ M ) by the voting-based strategy:
[Equation]
where 1 is the indicator function and K denotes the number of classes.
Note that the ensemble outputs the predicted label y that agrees with the majority predictions of the sub-models.
Definition 1 (best-performing sub-model) .
Given an input x and its label y , the best-performing sub-model achieves the lowest objective loss on the data ( x, y ) among all M sub-models:
[Equation]
Note that the best-performing sub-model is w.r.t.
the input data ( x, y ) , i.e., different input data correspond to different best-performing sub-models.
Collaboration.
Suppose that there are M learned sub-models { f θ 1 , f θ 2 , ..., f θ M } .
Given an input x , sub-models make predictions (cid:8) f θ 1 ( x ) , f θ 2 ( x ) , ..., f θ M ( x ) (cid:9) .
The collaboration tries to output a final prediction collaboration( x, f θ 1 , ..., f θ M ) by the best-performing sub-model:
[Equation]
[Table]
x
[Table] [Table]
+a24M9J53DE8ux9OHFpiP0mS4tVeL1/QSpoc6jQop6CAdNS8/DM66MODiQgOGFxwwjZ0hPS1UYYGn7guYuICQpaMMyTIkTaiLEYZOrFjmoe0a6esS3vhGUq1SafY9AekVLFLGo/yAsLiNFXGI+ks2N+8Y+kp7jah1Ui9HGI5RsT+pfvM/K9O1MIxwJGswaKafMmI6szUJZKvIm6ufqmKk4NPnMB9igeETan8fGdVakJZu3hbXcZfZaZgxd5McyO8iVtSg8s/2zkNGvulcqV0cFYpVI/TVmexjR0UqZ+HqOIUNdTJ+xoPeMSTcqHcKLfK3Ueqkkk1m/g2lPt3Z3Waew==</latexit>
[Figure]
x <latexit sha1_base64="Pbc+1s8Sc82XeqeLL9Ole7+uVlw=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LIoiMsW7AO0SJJO69DJg8lELEV/wK1+m/gH+hfeGVNQi+iEJGfOvefM3Hv9RPBUOc5rwZqbX1hcKi6XVlbX1jfKm1vtNM5kwFpBLGLZ9b2UCR6xluJKsG4imRf6gnX80amOd26ZTHkcXahxwnqhN4z4gAeeIqp5d12uOFXHLHsWuDmoIF+NuPyCK/QRI0CGEAwRFGEBDyk9l3DhICGuhwlxkhA3cYZ7lEibURajDI/YEX2HtLvM2Yj22jM16oBOEfRKUtrYI01MeZKwPs028cw4a
[Table]
+6j8z/6mQtAl1sqRo41RQqRlbn5C6J6oq8uf6lKkEOIXESdygeEXaU8qPPutLEqnbZW0vFX1SmZOXeyXMTvMpb0oDNn+McBM31ilmtbOxXy7XtfNQFLGIJqzTPTdSwizoa5H2FezzgUfO0a+1Gu31P1YZyzQK+Le3uDeEfoAI=</latexit> f ✓ M (˜ x ✓ M ) <latexit sha1_base64="AGt4k7VJmkQM5psYLcza9FRQ7aE=">AAAC63icjVHLSsNAFD3GV31XXboJFkU3JZGKLotu3AgVrAqthiSdtoN5kUzEEvoH7tyJW3/Arf6H+Af6F94ZU1CL6IQkZ86958zce53I44kwjNcRbXRsfGKyMDU9Mzs3v1BcXDpJwjR2Wd0NvTA+c+yEeTxgdcGFx86imNm+47FT53Jfxk+vWJzwMDgWvYid+3Yn4G3u2oIoq7jetrKm6DJhW9lhv7/RFNxrsey6f/GV3rSKJaNsqKUPAzMHJeSrFhZf0EQLIVyk8MEQQBD2YCOhpwETBiLizpERFxPiKs7QxzRpU8pilGETe0nfDu0aORvQXnomSu3SKR69MSl1rJEmpLyYsDxNV/FUOUv2N+9Mecq79ejv5F4+sQJdYv/SDTL/q5O1CLSxq2rgVFOkGFmdm7ukqivy5vqXqgQ5RMRJ3KJ4TNhVykGfdaVJVO2yt7aKv6lMycq9m+emeJe3pAGbP8c5DE62ymalvH1UKVX38lEXsIJVbNA8d1DFAWqok/cNHvGEZ83XbrU77f4zVRvJNcv4trSHDyaGoB4=</latexit> ` ( f ✓ best (˜ x ✓ 1 ) ,y )
[Table]
[Table] [Table]
<
[Table]
+60oSqdtlbU8XfVKZk5d5OciO8y1vSgAs/xzkNqvv5QjF/cFbMlY6TUaexjR3s0TwPUcIpyqiQ9w0e8YRn7VK71e60+89ULZVoNvFtaQ8fRn6a2Q==</latexit>
PGD (a2) collaboration training
g  ( x ) <latexit sha1_base64="7ctncwgFuMdaXoGwpIQz8wNf/kc=">AAACznicjVHLTsJAFD3UF+ILdemmkZjghrQGo0uiG5eYyCMBQtoywMTSNtMpkRDi1h9wq59l/AP9C++MJVGJ0Wnanjn3nDtz73Ujn8fSsl4zxtLyyupadj23sbm1vZPf3avHYSI8VvNCPxRN14mZzwNWk1z6rBkJ5oxcnzXc20sVb4yZiHkY3MhJxDojZxDwPvccSVRr0J22oyGfFe+Ou/mCVbL0MheBnYIC0lUN8y9oo4cQHhKMwBBAEvbhIKanBRsWIuI6mBInCHEdZ5ghR96EVIwUDrG39B3QrpWyAe1Vzli7PTrFp1eQ08QReULSCcLqNFPHE51Zsb/lnuqc6m4T+rtprhGxEkNi//LNlf/1qVok+jjXNXCqKdKMqs5LsyS6K+rm5peqJGWIiFO4R3FB2NPOeZ9N7Yl17aq3jo6/aaVi1d5LtQne1S1pwPbPcS6C+knJLpdOr8uFykU66iwOcIgizfMMFVyhipru+COe8GxUjbExM+4/pUYm9ezj2zIePgC26JOV</latexit> Proposition 1. Given M learned sub-models, the predicted accuracy of the collaboration is upperbounded that of the ensemble, i.e., [Equation]
Proof.
Given an ( x, y ) ∈ D , if the ensemble’s prediction is correct, at least one sub-model makes correct prediction, i.e., 1 f θ best ( x )=
y holds; therefore, the collaboration’ prediction is correct.
If the collaboration’s prediction is correct, there exists a case that the majority of sub-models make consistent but wrong predictions, while a single sub-model’s prediction is correct; then, ensemble’s prediction is wrong.
Therefore, Proposition 1 holds.
From Proposition 1, a collaboration could achieve an equal or higher performance than an ensemble.
Compared with the ensemble, the collaboration requires the identification of the best-performing sub-models using label information.
Next, we will introduce a realization of our collaboration framework.
3.2 Realization of collaboration for defending against adversarial attacks
Notation.
We firstly introduce the needed notations.
Suppose X and Y denote input space and output space, where Y = { 1 , ..., K } for a K -class classification problem.
There are N samples in the dataset D = { ( x, y ) } , where x ∈ X and y ∈ Y .
Let d inf ( x, x ′ ) =
∥ x − x ′ ∥ ∞ denotes the infinity distance metric, and B ϵ [ x ] = { x ′
∈ X | d inf ( x, x ′ ) ≤ ϵ } is the closed ball of of radius ϵ >centered at x .
To search for adversarial data within norm ball B ϵ [ x ] , [5] proposed a projected gradient descent (PGD) method that iteratively searches for adversarial data ˜ x
( x refers to natural data).
f θ ( x ) outputs a K-dimensional predicted probability, i.e., ˆ p ( x ) =
[ˆ p 1 ( x ) , ..., ˆ p K ( x )] .
Goal of collaboration.
1) ensure the correct prediction of the best-performing sub-model for a given input, and 2) select the best-performing sub-model among all sub-models to make predictions.
First, intuitively, every sub-model in a collaboration should maximize its expertise to fit its areas and leave the remaining areas fitted by others.
As a result, the collaboration can minimize the vulnerability overlaps of all sub-models.
Section 4 shows “minimizing the vulnerability overlap of all sub-models”
Minimize BCE loss ℓ ϕ = BCE( g ϕ j (˜ x θ i ) ˆ p y
(˜ θ i )) to update the g
Collect sub
’s entropy loss data ˜ θ i : ℓ (cid:0) f θ j (cid:0) ˜
x θ i (cid:1) , y (cid:1) ; 8:
Calculate surrogate data ˜ i : ˆ ℓ m = − σ ln (cid:80) Mj =1 exp (cid:18) − ℓ ( f θj ( ˜ x θi ) ,y ) σ (cid:19) ; 10: is “minimizing the objective loss of the best-performing sub-models”.
Therefore, during the training phase, the given data should always be allocated to the sub-model that has the lowest objective loss.
In other words, the sub-models always choose the easiest data to learn.
In the example of Figure 2(a), i ) Data 3 ⃝ is misclassified by both sub-models.
The blue sub-model is near Data 3 ⃝ and has the lowest objective loss.
We assign the blue sub-model to fit Data 3 ⃝ .
ii ) Data 2 ⃝ is correctly classified by the pink model but wrongly classified by the blue model; for ease of effort, we assign the pink model to fit Data 2 ⃝ , because the collaboration can correctly be classified Data 2 ⃝ by selecting the pink model as the representative.
iii ) Data 1 ⃝ is correctly classified by both models.
The blue model is far from
Data 1 ⃝ and takes the lowest effort on fitting it; therefore, we assign the blue model to fit Data 1 ⃝ .
Second, to select the best-performing sub-model, we construct dual-head structured sub-models .
As shown in Figure 2(b), our sub-model has dual heads: 1) predictor (P) head: f θ predicts the label probability f θ ( x )
=
ˆ p ( x ) =
[ˆ p 1 ( x ) , ..., ˆ p K ( x )] (a vector); 2) evaluator (E) head: g ϕ approximates the confidence of the prediction ˆ p ( x ) .
Note that we use g ϕ to approximate the true label probability ˆ p y ( x ) , which denotes the true confidence of a given prediction ˆ p ( x ) .
Meanwhile, the largest confidence corresponds to the lowest objective loss, and vice versa (see theoretical proof in Proposition 2).
Therefore, the best-performing sub-models could be identified using the approximated confidence ( g ϕ ( x ) ) by the E head. [Figure] the learned SoE
To defend against adversarial attacks, the collaboration needs to learn from adversarial data.
Algorithm 1,2 along with Figurearticulates how to learn such the collaboration.
In particular, the training of our framework SoE shown in Figure 4 is as follows:
the sub-model training: the sub-models fit adversarial samples from their own or other sub-models; 2.
the collaboration training: the sub-models fit adversarial samples from the collaboration.
The sub-model training.
As shown in Figure 3(a1), given a natural training data x and its label y , we use the PGD method to obtain M adversarial variants { ˜ x θ i } Mi =1 of M sub-models as existing baselines do.
For each adversarial variant ˜ x
θ i , we assign the best-performing sub-model to learn and update its feature extractor and label head.
This process corresponds to Lines 1–8 and Lines 9–10 in Algorithm 1.
Note that in Line 9, we use a surrogate loss to approximate the sub-model assignment process (reasons see Eq.
5 and 6).
As shown in Figure 3(b), given an adversarial variant ˜ x θ i , we propose to use the binary-cross-entropy (BCE) loss between Input
} Mi { g ϕ i } Mi =1 , f θ i outputs the label prediction and g ϕ approximated confidence training dataset D hyperparameter σ
Obtain ˜ x by perturbing x to worsen the prediction; // generate the adversarial samples of the collaboration 8: Minimize BCE loss ℓ x to update the g all sub models; 9: minimizing surrogate ; 10: end for 11: the predicted label probability on the true label (i.e., ˆ p y (˜ x θ i ) ) and the approximated confidence (i.e., g ϕ (˜ x θ i ) ) to update each sub-model’s E head.
This process corresponds to Lines 5–6 in Algorithm 1.
The collaboration training.
During the sub-model training, we learn the most adversarial data from each sub-model using the sub-models performing best.
The most adversarial samples cannot attack the collaboration successfully.
However, there may exist harmful samples (which may not be the most adversarial for any sub-model) that are unexplored and can attack all sub-models.
Therefore, the submodel training may converge without a full exploration of the adversarial samples.
(The experimental verification about this could be found in Section D.1.)
To defend these potential adversarial samples, we propose the collaboration training shown in Figure 3 (a2) and Algorithm 2.
Firstly, we propose to generate the adversarial samples that can worsen the outputs of the collaboration.
In particular, for each data sample x , we output the prediction ˆ p ( x ) whose confidence g ϕ ( x ) is the highest.
We perturb x to fool the prediction using PGD method.
Then we minimize a surrogate loss to fit this adversarial data ˜ x .
This process is detailly shown in Algorithm 2.
We use Algorithm 1 and Algorithm 2 in sequence to train our collaboration.
Algorithm 2 is proposed to explore the adversarial samples which could be not the most adversarial samples of any sub-model, but could fool all sub-models.
In Algorithm 2, we attack the collaboration iteratively to obtain the adversarial samples.
However, in each iteration we update the adversarial sample ˜ x , the best-performing sub-model could be different.
For example, in the first iteration, given the input x , the best-performing sub-model is f θ 1 .
We obtain an adversarial sample ˜ x ′ by attacking f θ 1 .
However, in the second iteration, given the input ˜ x ′ , the best-performing sub-model is another sub-model (e.g., f θ 2 ).
We attack the best-performing submodel f θ 2 to obtain the adversarial sample ˜ x ′′ .
Therefore, by attacking the collaboration following Algorithm 2, we could obtain the adversarial sample ˜ x ′′ which is not the most adversarial samples but could fool all sub-models and is unseen in Algorithm 1.
During the inference phase shown in Algorithm 3 in Appendix, once M sub-models are properly learned, SoE chooses a representative sub-model whose confidence is the highest among all submodels, and then outputs this sub-model’s prediction.
3.3 Analyses of SoE
Optimizing the best-performing sub-models.
We firstly show that minimizing the vulnerability overlap of all sub-models is equal to minimizing the objective loss of the best-performing sub-models.
For ease of optimization of the best-performing sub-model, we provide a surrogate loss.
The vulnerability overlap of all sub-models refers to the set of adversarial data (˜ x, y ) that are misclassified by all sub-models, i.e., all sub-models’ objective loss is higher than a certain degree δ : min θ i ∈{ θ 1 ,...,θ M } ℓ ( f θ i (˜ x, y ))
>
δ, ˜ x ∈ ˜ D , where ˜ D denotes the vulnerability overlap of all sub-models.
To reduce the vulnerability overlap of all sub-models, we only need to reduce objective loss of a single model, which is equal to minimizing the loss of the best-performing sub-model, i.e.,
[Equation]
where ℓ best (˜
[Equation]
where ˜ x θ i is the adversarial data generated by the sub-model f θ i .
While directly performing the outer minimization in Eq.(5) may cause a trivial solution (e.g., there is only one optimized sub-model), for ease of the optimization of Eq.(5)
(Corresponding to Lines 9– in Algorithm 1), we provide a surrogate objective as follows.
[Equation]
where ˆ ℓ m (˜ x θ i , y ) = − σ ln (cid:80) Mj =1 exp [Equation] and σ > 0 is a pre-defined hyper-parameter.
In Eq.(6), we approximate the objective min θ j ∈{ θ 1 ,θ 2 ,...,θ M } ℓ [Equation] f θ j [Equation]
˜ x θ i ) , y (cid:1) using a smooth surro- gated maximum function due to
[Equation]
The proof of Eq.(7) is in Appendix.
The best-performing sub-model has the highest confidence.
We show that a sub-model with the highest confidence achieves the minimum of the objective loss among all sub-models, i.e., the best-performing sub-model.
Proposition 2.
Given an input x , the sub-model that has the highest confidence corresponds to the best-performing sub-model, i.e.,
[Equation]
where confidence corresponds to the probability of the true label in a given ˆ p ( x ) .
To learn the E head to approximate the confidence of ˆ p ( x ) , we could simply compare its output with the predicted probability on the true label (i.e., ˆ p y ( x ) ), then update the E head by gradient descent (corresponds to
Lines 6–7 in Algorithm 1).
Given an input to the collaboration, our dual-head structured sub-models can collaboratively decide the best-performing sub-model by comparing the values of the E head (corresponds to Algorithm 3 in Appendix).
Note that the E head may be susceptible to adversarial attacks in the white-box setting.
In our implementation, we use a simple linear structure to regress the confidence.
Experimental results on adaptive attacks demonstrate the reliability of our framework.
Experiments
In this section, we conduct experiments on a benchmark dataset to verify the effectiveness of our method in defending against white-box and transfer attacks.
Then we provide ablation studies to demonstrate the significance of the collaboration training described in Algorithm 2.
The experimental results about black-box attacks and more discussions could be found in Appendix.
4.1 Experimental Setup
Following the work in [10], we compare our method with various related methods, including ADP [8],
GAL [9], DVERGE [10] and MoRE [13].
We use ResNet-20 [122] as sub-models in all methods for fair comparisons, and we use CIFAR10 as the data set, a classical image dataset [123] that has 50, training images and 10,000 test images.
4.2 Performance on White-box attack
As there are mainly two threat modes in the adversarial attack setting: white-box attack and black-box attack.
White-box attack refers to that attackers know all the information about the models, including training data, model architectures, and parameters, while black-box attackers have no access to the information about the model’s structures and parameters and rely on surrogate models to generate transferable adversarial examples.
We compare our method with 4 baselines on defending against white-box attacks using a subset of CIFAR10 .
We use 50-step PGD with five random starts and the step size of ϵ/ 5 to attack all methods as in [10].
We learn the models in adversarial training and evaluate the robustness under various attacks with the same ϵ .
For example, to evaluate the robustness under white-box attack with ϵ = 0 .
01 , we first learn the models in adversarial training with ϵ = 0 .
We evaluate all methods following the setting in [10].
In particular, we randomly select 1000 samples under different ϵ .
For the PGD attack, we select the cross-entropy loss to update the perturbations to search for adversarial samples.
In addition to the robustness, we also report the performance of all methods on clean data with the adversarial training under different ϵ
[Table]
[Table]
From Table 1, SoE achieves a better robustness performance under white-box attack.
The results verify that collaboration significantly improves the utilization of the limited model capacity.
Therefore,
SoE can fit more adversarial data and have a relatively smaller vulnerable area.
Performance on Adaptive Attacks.
We follow the suggestions in [124] and conduct two different adaptive attacks to fool the dual heads simultaneously.
For the first adaptive attack, we attack the E head to minimize the confidence of the best-performing sub-model.
In particular, we maximize l 1 = BCE( g ϕ j ( x ) , 1) , where j = arg max i ∈
[ M ] g ϕ i ( x ) means the j -th sub-model is identified as the best-performing sub-model.
For the second adaptive attack, we try to achieve a mismatch between the correct predictions and the highest confidence.
Specifically, we maximize l 2 = − log (cid:104)
f θ j ( x ) y ∗ g ϕ j
( x ) + 1 − g ϕ j
( x ) (cid:105) , where j = arg max i ∈
[ M ] − log (cid:104)
f θ i ( x ) y ∗ g
ϕ
i
( x ) + 1 − g ϕ
i ( x ) (cid:105) .
We conduct experiments by maximizing the weighted loss ℓ adp= ℓ ( f θ ( x ) , y ) + λ · ℓ 1 and ℓ adp= ℓ ( f θ ( x ) , y ) + λ · ℓ 2 with varying λ .
The robustness with respect to λ on the stronger adaptive attack (i.e., the attack achieves a higher success rate) is shown in Figure 7.
Though both adaptive attacks could attack the predictor and the evaluator
[Table] [Table]
simultaneously, the evaluator in our method is robust to the adversarial samples because of its sample structure.
As seen in Figure 7 and Table 1, our method is slightly degraded by adaptive attacks and still outperforms baseline models.
4.3 Visualization of the our Collaboration Scheme
To intuitively understand the collaboration mechanism of our proposed SoE, we show the decision boundaries of the ensemble and the collaboration in Figure 5.
In particular, we learn the ensemble and the collaboration with 3 ResNet-20 sub-models on CIFAR10 dataset with adversarial training.
The same color areas in Figure 5 denotes the same predicted label.
The black regions means that the models predict correctly while other color regions means that the models are fooled and predict [Figure] incorrectly.
The ensemble defending against attacks requires more than half of the sub-models fit the same vulnerability areas.
From the top of Figure 5, all sub-models fit similar vulnerability areas and there is a boarder vulnerability overlap between sub-models.
Our proposed collaboration proposes to minimize the overlap of all sub-models.
From the bottem of Figure 5, different sub-models defend against different vulnerability areas collaboratively and our collaboration achieves a smaller vulnerability areas.
4.4 Performance on Transfer Attack
Due to the transferability of adversarial examples, transfer adversaries can craft adversarial examples based on surrogate models and perform an attack on the target model.
In our experiments, we follow the transfer attack setting in [10] and select 1000 test samples randomly.
We use hold-out baseline ensembles with three ResNet-20 sub-models as the surrogate models to generate adversarial samples.
In particular, we use three attack methodologies: PGD with momentum [35], SGM [29] which adds weight to the gradient through the skip connections of the model, and M-FGSM [36] which randomly augments the input images in each step.
For each sample, three adversarial variants are using the three attack methods.
Only when the model can classify all kinds of adversarial variants can the model successfully defend against adversarial attacks.
We show the results of all methods in Table 2.
Furthermore, we also use a more challenging setting following the work in [10].
We use hold-out baseline models with 3, 5, and 8 ResNet-20 sub-models as the surrogate models.
Meanwhile, we generate adversarial samples with cross-entropy loss and CW loss [15].
For each sample, we generate 30 adversarial variants, and only if the model classifies all the 30 variants can the model defend the transfer attack successfully.
The results are shown in Table 3.
In our experiments, GAL is hard to learn stably in adversarial training.
In Table 2, when 0 .
01 ≤ ϵ ,
SoE achieves a better performance compared with baselines.
With the increase of ϵ , the volume of ϵ -ball increases exponentially.
The performances of all methods get worse significantly because of insufficient model capacity.
Similar results can also be found in Table 3.
Since SoE addresses more adversarial data using a collaboration mechanism, it achieves a relatively better robustness performance as ϵ increases.
4.5 The Robustness of SoE under Different Number of Sub-models
To explore the robustness of the collaboration under different ϵ with different number of sub-models, we conduct experiments under transfer attacks with different number of sub-models ( 1 ≤ N ≤ 5 ).
The detailed information could be found in Section D.3 in Appendix.
In summary, we have the following findings.
For different ϵ , more sub-models could achieve a more significant robustness improvement with the increase of ϵ .
For different number of sub-models, more sub-models are more likely to achieve a higher robustness, but the margin gain decreases with more sub-models.
Conclusion
In this paper, we study an essential question in the field of adversarial attacks that when we should collaborate.
( i )
If a single model can handle everything, there is no need for multiple models.
( ii )
If a single model can only handle a part of the whole, collaboration among multiple models makes sense.
Adversarial defense is a typical task that falls into the circumstance ( ii ) because a single model hardly fits adversarial data.
We provided a collaboration framework—SoE—as the defense strategy over ensemble methods, and empirical experiments indeed verified the efficacy of SoE. Future work includes applying our collaboration framework to other areas such as kernel methods, fairness, and federated model, etc.
Acknowledgments
We would like to thank the anonymous reviewers of NeurIPS 2022 for their constructive comments.
Sen Cui and Changshui Zhang would like to acknowledge the funding by the Natural Science Fundation of China(NSFC. No. 62176132 ). Bo Han was supported by the RGC ECS No.
22200720, NSFC YSF No. 62006202, Guangdong Basic and Applied Basic Research Foundation
No. 2022A1515011652, and RIKEN Collaborative Research Fund.
Jingfeng Zhang was supported by JST ACT-X Grant Number JPMJAX21AF and JSPS KAKENHI Grant Number 22K17955, Japan. Masashi Sugiyama was supported by JST AIP Acceleration Research Grant Number JPMJCR20U and the Institute for AI and Beyond, UTokyo, Japan.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] We discuss it in Appendix (b) Did you describe the limitations of your work?
[Yes] The limitations are in Appendix (c) Did you discuss any potential negative societal impacts of your work?
[Yes] The societal impacts are in Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] The proofs are in Appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The source codes are made publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The training details are in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] The devices of computing are in Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[Yes] We discuss our used data Section
Appendix.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Theoretical Proofs
The proof of Eq.(7) is as follows.
Proof.
Since we have
[Equation]
9) considering that − δ ln
( x ) monotonically decreases w.r.t x , we have − δ ln
[Equation]
−
[Equation]
σ
[Equation]
10) where − δ ln (cid:18) M · max j exp (cid:18)
[Equation]
(cid:19)(cid:19) =
min θ
[Equation]
θ
[Equation]
[Equation]
θ
[Equation]
f
[Equation]
˜
[Equation]
y
[Equation] − δ · ln( M ) and − [Equation] (cid:1) , so Eq.(7) holds.
The proof of Proposition 2 is as follows.
Proof.
The objective loss depends on the label probability (e.g., cross-entropy loss ℓ = − ln p
y ( x )) , and the loss monotonically decreases with respect to p y ( x ) , so the best-performing sub-model has the highest label probability.
Combined the fact that confidence( p ( x )) monotonically increases w.r.t.
p y ( x ) , the best-performing sub-model corresponds to the highest confidence; therefore, Eq.(8) holds.
B Algorithms
Algorithm 3 SoE (inference phase)
Input: the learned sub-models
f θ i } Mi = with { g ϕ i } Mi =1 , test input x . do 2: end
Return
The whole pipeline of our method during inference is summarized in Algorithm 3.
For an input natural (or adversarial) sample, each sub-model outputs the prediction using P head and the confidence using E head. Then the collaboration predicts the label as the prediction with the highest confidence. C More Discussions about SoE
[Figure]
C.1 the Behavior of SoE on Synthetic Data
To demonstrate the behavior of collaboration, we apply our collaboration framework on the well-known XOR problem as an example.
In XOR problem, there is a binary training set D
= { x i , y i ) } ni =1 , where y i ∈ {± 1 } .
For the samples with the label y i
= 1 , the input feature x is independently sampled from the two Gaussian distributions x ∈ N ( µ = [1 , 1] , σ = 0 .
1 · I 2 ) and x ∈ N ( µ = [ − 1 , − 1] , σ = 0 .
1 · I 2 ) , where N ( µ = [1 , 1] , σ = 0 .
1 · I 2 ) denotes 2-d Gaussian distribution with the mean vector µ = [1 , 1] and covariance matrix σ = 0 .
1 · I 2 .
For the samples with the label y i = − 1 , we sample the input feature independently x i ∈ N ( µ = [ − 1 , 1] , σ = 0 .
1 · I 2 ) and x i ∈ N ( µ = [1 , − 1] , σ = 0 .
1 · I 2 ) .
Suppose there are two linear sub-models f 1 ( x ) = a 1 · x + b 1 and f 2 ( x ) = a 2 · x + b 2 .
Ensemble methods output a prediction ˆ p ( x ) by a voting-based strategy e.g., averaging the predictions as the output, i.e., ˆ p ( x ) = 1 M · (cid:80) Mi
=1 f θ i ( x ) .
Collaboration aims to 1).
minimize the mean square error (MSE) of the best-performing sub-models; 2).
optimize the module g 1 and g 2 to measure the quality of the multiple predictions during inference.
From Figure 6(a), learning multiple linear sub-models and averaging the predictions (ensemble) is still a linear model, so it cannot tackle XOR problem.
Collaboration can address XOR which classifies each sample by learning to specify the sub-tasks to different sub-models.
From Figure 6(b), the samples are assigned to two sub-models, in which the blue samples are assigned to f 1 and the red samples are assigned to f 2 .
Finally, it output the prediction by identifying the best-performing sub-models.
C.2 Optimization Efficiency
The training cost is a notable issue.
We compare the training cost of all methods from the two aspects; 1).
parameters and GFLOPs: all methods have the same model architecture (ResNet20), so all methods have a similar number of parameters and GFLOps.
Compared with baselines, our method has an additional head (E head), which is a one-layer MLP with 128 parameters and has a negligible computation cost; 2).
training manner; all methods except DVERGE achieve adversarial training by generating adversarial samples using PGD attack.
The time consumption of all methods using the device Geforce 2080Ti (100 epochs) is in Table 4.
[Table]
DVERGE distills non-robust features by computing transferable adversarial samples, which have an O( N 2 ) time complexity in which N is the number of sub-models, so it has a relatively large time consumption.
Our method outperforms baselines by training an additional E head and it could cause an additional small-time consumption as shown in the above table.
C.3 a Single Big Model or a Collaboration
advantages of the SoE Compared with SoE, a single big model may be difficult to fit the adversarial data.
For a big single model with a deeper structure, it may face gradient vanish or gradient explosion during optimization.
Without a well-designed optimization method, it could be more vulnerable to an imperceptible perturbation compared to a simple model.
The SoE alleviates this problem by learning multiple relatively small models.
disadvantages of the SoE
A single big model may be a direct solution for addressing insufficient capacity in adversarial training.
Compared with learning a single big model, SoE needs to design a complicated mechanism to improve the utilization of the model capacity.
C.4 SoE in Adversarial Training Can Avoid a Trivial Optimization
As our proposed collaboration mechanism enhances the performance of the best-performing sub-model.
One may wonder whether it obtains a trivial case, e.g., only one sub-model is properly trained.
In fact, our SoE does not bring such a trivial case.
Fitting the adversarial data consumes a tremendous model capacity so the model usually cannot fit all adversarial samples.
For a learned sub-model, there still exist adversarial samples that cannot handle.
From our proposed collaboration mechanism, these adversarial samples generated from a learned sub-model are more likely to be assigned to other sub-models which perform better.
Therefore, in our SoE, all sub-models will be trained.
[Figure]
To experimentally verify this claim we present the accuracy on clean data of all three sub-models with adversarial training ( ϵ = 0 .
02 ) in the Table [Figure]
From the Table all sub-models have a similar performance on the clean data.
D More Experimental Results and Analysis D.1 Ablation Studies
To make full use of the limited modal capacity of an ensemble [Figure] SoE proposes the sub-model training (shown in Algorithm 1), which makes the adversarial samples from sub-models be learned by the best-performing ones.
The sub-model training enables the most adversarial attacks of sub-models could be successfully defended.
However, learning from the most adversarial samples of all sub-models could converge without a full exploration of the vulnerable area.
Therefore, we propose the collaboration training (shown in Algorithm 2), which fits the adversarial samples unexplored before.
To verify the significance of the collaboration training, we conduct ablation studies under two transfer attacks.
In particular, we train two kinds of models to defend against the attacks: 1).
SoE ; 2).
SoE without the collaboration training.
The two transfer attacks are generated as stated in Section 4.4, 1).
attacks with 3 adversarial variants; 2).
attacks with 30 adversarial variants.
We show the robustness of two methods in Figure 8.
From Figure 8(a) and 8(b), when 0 .
01 ≤ ϵ ≤ 0 .
SoE without the collaboration training achieves a similar robustness compared with SoE. It means that the sub-model owns relatively sufficient model capacity and the adversarial samples could be largely explored by the sub-model training.
However, when ϵ ≥ 0 .
04 , the robustness of SoE without the collaboration training degrades significantly.
In this case, the need for model capacity increases dramatically.
The most adversarial samples identified by the sub-model training cannot cover all cases, and our proposed the collaboration training mitigates it effectively.
D.2 Performance on Black-box Attack
In addition to transfer attack, query attack can also craft adversarial samples based on the predicted scores of the model.
To evaluate the robustness of SoE under query attack, we use Square Attack method [125] to attack all methods.
Square Attack selects localized square-shaped updates at random positions in each step [125].
In particular, we set 0 .
01 ≤ ϵ ≤ 0 .
07 and learn the models for each method with adversarial training.
Then we randomly select 1000 samples from the test set and evaluate the robustness of each method under Square Attack with 5000 iterations.
We repeated the experiments under 5 different random seeds and the results are presented in Table 6.
[Table]
[Table]
[Table] different
D.3 Analysis about the Number of Sub-models
To explore the effect of the number of sub-models, we conduct experiments under transfer attacks with different number of sub-models ( 1 ≤ N ≤ 5 ).
The results are shown in the following Figure 9 and Figure 10.
From Figure 9, as ϵ = 0 . 01 , multiple sub-models achieve a similar adversarial robustness compared with a single model.
In this case, a single has enough model capacity and the collaboration is not necessary.
When 0 .
02 ≤ ϵ ≤ 0 .
03 , SoE achieves a better performance as the number of sub-models increases.
If ϵ > 0 .
SoE with multiple sub-models still outperforms a single model.
However, SoE having N = 3 sub-models achieves a comparable robustness compared with SoE having N = 5 .
The volume of the sample surrounding is exponentially ( (1 + ϵ ) |X| ) large with the input X [3].
With the increase of ϵ , the required model capacity has a sharp rise.
We guess that given a large ϵ (e.g., ϵ = 0 . 07 ), more sub-models (e.g., N = 5 ) may be insufficient to improve the performance further compared with learning SoE with N = 3 .
Similar phenomenon could also be found in Figure 10 when there are 30 generated adversarial variants.
D.4 Analysis about the Model Capacity of SoE
A model with sufficient capacity to cover all cases does not need to collaborate with others.
To verify this claim, we conduct experiments using the ResNet model with different depths and show the clean accuracy ( % ) of single/multiple models in the following table.
[Table]
From the Table 7, with depth = 2 , the model has the insufficient model capacity to learn the feature extractor, SoE can have a relatively large improvement (2.0).
As the depth of the model is 20, the model has sufficient model capacity to fit all data samples.
SoEachieves a slight improvement compared with a single model (0.6).
Compared with standard training, adversarial data are adaptively changed based on the current model to smooth the natural data’s local neighborhoods.
The volume of these surroundings is exponentially large.
The model often encounters insufficient model capacity especially when there is a relatively large ϵ ball.
Therefore, it is urgent to improve the utilization of the capacity for adversarial training.
E Implementation Details
the design of the E head In our experiments, we use ResNet20 as our backbone model.
For the additional
E head, we propose to use a one-layer MLP to model the relationship between the prediction and its confidence.
Our design of the E head is based on the following aspects: 1).
convenient optimization; a simple model structure can be optimized easily and will not bring a significant computation cost; 2).
robustness; a complex E head may be vulnerable to the adversarial attack.
Therefore, in our implementation, we choose to learn a simple E head for each sub-model to improve its robustness.
Training devices We run our experiments on a local Linux server. We use Pytorch framework to implement our model and train all models on GeForce RTX 2080 Ti GPUs.