Enhance the Visual Representation via Discrete Adversarial Training
s
Abstract
Adversarial Training (AT), which is commonly accepted as one of the most effective approaches defending against adversarial examples, can largely harm thestandard performance, thus has limited usefulness on industrial-scale productionand applications.
Surprisingly, this phenomenon is totally opposite in Natural
Language Processing (NLP) task, where AT can even benefit for generalization.
We notice the merit of AT in NLP tasks could derive from the discrete and symbolic input space.
For borrowing the advantage from NLP-style AT, we propose
Discrete Adversarial Training (DAT).
DAT leverages VQGAN to reform the imagedata to discrete text-like inputs, i.e. visual words.
Then it minimizes the maximal risk on such discrete images with symbolic adversarial perturbations.
Wefurther give an explanation from the perspective of distribution to demonstratethe effectiveness of DAT.
As a plug-and-play technique for enhancing the visualrepresentation, DAT achieves significant improvement on multiple tasks includingimage classification, object detection and self-supervised learning.
Especially, themodel pre-trained with Masked Auto-Encoding (MAE) and fine-tuned by our DATwithout extra data can get 31.40 mCE on ImageNet-C and 32.77% top-1 accuracyon Stylized-ImageNet, building the new state-of-the-art.
Introduction
Nowadays, Deep Neural Networks (DNNs) has achieved excellent performance surpassing humansin most computer vision tasks.
Although remarkable progress has been made, the success of DNNs isactually a false sense when i.i.d hypothesis is not satisfied in wild.
Researchers have shown that deepmodels fail in most circumstances including adversarial perturbations [1], common corruptions [2],colors or textures changing [3, 4] etc.
There is still a long way to make DNNs closer to the robusthuman perception.
A possible way towards robust machine perception can be Adversarial Training (AT) [5], whichautomatically finds failure input cases of DNNs and augment online with these cases for fixing "bugs".
With online augmentation of adversarial examples, AT greatly enhances the adversarial robustness,and helps for learning perceptually-aligned representations [6] with good interpretability [7, 8]and transferability [9].
However, AT is double-edged, which meanwhile degrades the standardperformance caused by problematic regularization [10].
Such problematic regularization makes thedecision boundaries over-smoothed and enlarges indecisive regions.
Surprisingly, previous works [11, 12] observe a strange phenomenon that AT behaves converselyin Natural Language Processing (NLP) tasks.
By automatically finding adversarial textual inputs,
AT will not hurt the accuracy and even benefit for both generalization and robustness of languagemodels.
This phenomenon motivates us considering whether the merit of NLP-style AT can betransferred to vision tasks.
We notice such merit could derive from the unique data organizingform of language models.
To be specific, an adversarial image perturbed in continuous pixel spaceactually differs with the truly "hard" examples appeared in real world.
Contrarily, text space isdiscrete and symbolic, where adversarial text is practically existing when a typo is made by humans.
Learning on such adversarial text will obviously improve the generalization on other more texts withconfusing typos.
Therefore, we borrow the symbolic nature of languages, and apply it on CV tasks bydiscretizing continuous images into a more meaningful symbolic space.
Afterwards, AT is conductedfor minimizing the maximal risk on such text-like inputs with symbolic adversarial perturbations.
In this paper, we propose Discrete Adversarial Training (DAT), a new type of adversarial trainingwhich aims to improve both robustness and generalization of vision models.
DAT leverages VQGAN [13] to learn a vocabulary of visual words, also known as image codebook.
For a continuousimage input, each encoded patch embedding is replaced with its closest visual word in the codebook,and represented as a corresponding index.
Then the image is transformed to a sequence of symbolicindices similar with language input.
For generating adversarial examples based on such symbolicsequence, direct use of optimization methods in NLP like combinatorial optimization [14, 15, 16] orsynonym substitutions [14, 16] can be challenging.
The reason lies in: 1) the large search space ofimages and 2) the non-existence of synonym in visual codebook.
To make it more efficient, DATadopts a gradient-based method which assumes the backward adversarial gradient goes straightthrough the complex discretization process, thus gradients on discretized image can be copied tooriginal input.
Then it use one-step search along the direction of estimated gradient such that thediscrete representation will altered adversarially during the discretization process, resulting a discreteadversarial examples.
Finally the discrete adversarial examples is fed into models for training.
Different from AT which always adds l p bound on augmented adversarial examples, DAT affects thediscretization process to produce diverse adversarial inputs beyond l p bound for training.
We show inablation experiment that DAT not only enhances the robustness on l p bounded attacks, but also ispartly beneficial in defense of unrestricted semantic attacks [17, 18].
The overall pipeline of discreteadversarial training is shown in Figure 1.
We further give an analysis to explain the effectiveness of our DAT from the perspective of distribution.
By comparing the distributional difference of training examples in AT and DAT with clean images,we find the discrete adversarial examples in our DAT are much closer to the clean distribution.
Such ability of generating "in-distribution" adversarial examples makes DAT can improve the visualrepresentation learning on multiple vision models and tasks, with no sacrificing of clean accuracy.
Our contributions are summarized below:
• To the best of our knowledge, we appear to be the first to transfer the merit of NLPstyle adversarial training to vision models, for improving robustness and generalizationsimultaneously.•
We propose Discrete Adversarial Training (DAT), where images are presented as discretevisual words, and the model is training on example which has the adversarially altereddiscrete visual representation.• DAT achieves significant improvement on multiple tasks including image classification,object detection and self-supervised learning.
Especially, it establishes a new state-of-the-artfor robust image classification.
By combining MAE [19] pre-training and DAT fine-tuning,
our ViT-Huge [20] achieves 31.40 mCE on ImageNet-C [2] and 32.77% top-1 accuracy on
Stylized-ImageNet [3].
Related Work
Adversarial Training Adversarial Training (AT) [5] is first proposed to improve robustnessby training models with adversarial examples.
As one of the most effective defense, existing works [21, 22] have suggested a trade-off between adversarial and clean accuracy in AT.
Despite greatefforts [23, 24, 25] have been made for mitigating this trade-off, the bad generalization of AT stillcannot be fully remedied till now.
In opposite way, some other works [26, 27] use AT to improve theclean accuracy rather than adversarial robustness.
The most close work to ours is the AdvProp [26],which splits batch norms to prevent the mixed statistics of clean and adversarial examples, thuslearns better adversarial feature for generalization.
Pyramid AT [28] makes AT specific to ViTs bycrafting pyramid perturbation with Dropout enabling, yielding imporved performance.
However,these methods are only applicable under specific models or tasks.
When models do not use batchnorm and Dropout, both AdvProp [26] and Pyramid AT [28] are not applicable anymore.
Adversarial Augmentation
By borrowing the idea of AT, some previous works [29, 30, 31]propose to search augmentations adversarially for improving the hardness of training examples.
Theonly differences of these methods is the way of searching adversarial augmentations.
Adversarial
AutoAugment [29] uses augmentation policy network to produce hard augmentation policies on apre-defined policy space.
AugMax [30] mixes multiple randomly sampled augmentation operatorslike AugMix [32], by using adversarially learned mixing factors.
AdA [31] optimizes the parametersof image-to-image models to generate adversarially corrupted augmented images.
MaxUp [33] usesthe worst augmented data of each data point in a set of random perturbations or transforms for training.
However, these methods create adversarial inputs indirectly and rely on pre-defined augmentations ortranslation models.
They cannot perturb the images locally.
Instead, our DAT directly modifies theimage with no need of pre-defined transforms, and can craft local perturbations on images, which ismore elaborate.
Discrete Visual Representation Learning Early technique [34, 35] of Bag-Of-Visual-Words(BOVW) model has shown great power of discrete representation in visual understanding.
VQVAE [36] uses DNNs to learn neural discrete representations, also known as visual codebook, bygenerative modeling the image distribution.
Recently, the idea of discrete representations learning hasbeen widely emerged in many vision tasks.
In most Masked Image Modeling (MIM) methods [37, 38],visual codebook is needed for BERT-like self-supervised pretraining.
For image classification, discreterepresentations strengthen the robustness by preserving the global structure of an object and ignoringlocal details [39].
For image synthesis, adversarial and perceptual objective can be added to VQ-VAEfor learning perceptually-aligned visual codebook [13].
With growing power of generative models,now a well trained VQGAN can produce vivid images with 0.58 FID [40].
In this work, we use
VQGAN for the discretization process in our DAT.
As a powerful generative model, VQGAN cangreatly reduce the information loss in reconstruction process and ensure the high-quality of thegenerated discrete adversarial examples.
Method
3.1 Traditional Adversarial Training
We take the typical image classification task as an example to show the formulation of Adversarial
Training (AT).
Consider the training image and label set D
= { x i , y i } ni =1 , and a classifier Fwith learnable parameters θ , the classification objective is always a cross-entropy loss L ( x, y, θ ) .
Adversarial Training (AT) finds the optimal θ by solving a minimax optimization problem:
[Equation]
where the inner optimization finds the perturbations δ on per-pixel values for maximizing the loss,and the outer minimization update θ to improve the worst-case performance of the network w.r.t.
theperturbation.
∥ · ∥ p constraints the p -norm of δ to a small value ϵ .
A problem is that AT finds thefailure case i.e. x + δ in continuous pixel space.
However, human does not create or recognize imagesfrom complex pixel values, but from discrete semantic concepts.
Although adversarial examplescan successfully fool the models, they are still different from the real "hard" examples appeared inpractice.
Find the worst 𝜹 by
𝐦𝐚𝐱 𝜹 𝓛
( 𝓠 𝒙 + 𝜹 , 𝒚 , 𝜽 )
Update model parameters 𝜽 by 𝐦𝐢𝐧 𝜽
𝓛 ( 𝓠 𝒙 + 𝜹 , 𝒚 , 𝜽 )
Enc 2 0 10 1 3
Dec
Codebook 𝓩
N … … 0 1 22 3 10 1 3
Visual Tasks
Self-Supervised Learning 00.20.40.60.81 cat tiger lion dog
Classification
Object Detection ∇ !"
ℒ 𝛿
ForwardDataFlow
Backward Gradient Flow Straight-Through Gradient ⨁
Straight-Through Gradient Flow 𝑥 + 𝛿 𝑥 𝑥
#$% =
𝒬 ( 𝑥 + 𝛿 ) * 𝑥 = 𝒬 ( 𝑥 )
3.2 Discrete Adversarial Training
3.2.1 Image Discretization by Visual Codebook
For discrete adversarial training, it is desirable first to learn an expressive visual codebook and represent the training image set in discrete space.
We utilize VQGAN [13] for image discretization.
Moreprecisely, consider a continuous image x ∈ R H
× W × 3 , VQGAN learns an encoder Enc ϕ ( · ) , decoder
Dec ψ ( · ) and quantization q Z ( · ) .
Enc ϕ is a convolutional model which maps x to intermediate latentvectors v =
Enc ϕ ( x ) ∈ R
( h × w ) × d , where h, w is the height, width of the intermediate feature mapand d is the latent dimension.
Subsequently, q Z ( · ) learn a codebook Z
= { z k | z k ∈ R d } Kk =1 , suchthat each latent vector v ij ∈ R d can be quantized onto its closest codebook entry z k as:
[Equation]
where i, j present each location in feature map.
Then the decoder Dec ψ ( · ) outputs the reconstructionimage ˆ x from the quantized vectors v q by:
[Equation]
VQGAN is trained by minimizing the reconstructed difference between ˆ x and x .
More details can bereferred to Appendix A. So far, given a continuous image x , we can get its corresponding discretereconstruction
ˆ x .
For simplicity, we use Q to stand for the above image discretization process, andthen we have ˆ x = Q ( x ) .
3.2.2 Discrete Adversarial Training
Based on the definition in Sec 3.2.1, we can generate discrete adversarial examples in inner maximization step of AT.
By slightly modifying the Eq 1, the objective of DAT is formulated as:
[Equation]
where Q transforms the continuous pixel space to discrete input space.
We delete the constraint termsince there is no need to bound the per-pixel values of δ .
Suppose that Q is an ideal discretizer withno information loss in discretization process.
The problem lies how to find the worst δ for maximizingthe classification loss.
Similar with traditional AT, we can use gradient-based methods to approximateδ by:
[Equation]
where α determines the magnitude of the perturbations along the gradient direction.
We set α = 0 .
1by default in DAT.
To expand ∇ x L ( Q ( x ) , y, θ ) by chain rule, we have four partial derivative termsas follows:
[Equation]
∇ adversarial perturbations x adv ← //Generate discrete adversarial examples 6: classification w .t. network parameter arg min θ adv θ
Through analysing the feasibility of each term, we find only ∂v q ∂v is hard to solve as the nondifferentiable nature of Eq 2.
Fortunately, as proposed in previous work, a straight-through gradientestimator can be used by copying the gradients from v q to v .
By replacing ∂ ˆ x ∂v q · ∂v q ∂v with ∂ ˆ x∂v , we cansimplify the Eq 6 to ∇ x L ( Q ( x ) , y, θ ) =
∂
L ∂ ˆ x · ∂ ˆ x ∂v · ∂v ∂x , which has derivative everywhere.
Although thesolution seems workable theoretically, the huge cost makes it impractical on large-scale vision tasks.
The bottleneck mainly lies on that ∂ ˆ x∂v and ∂v∂x require the adversarial gradients backward through Enc and Dec .
Actually, a generator capable of producing high-quality images always has a largeamount of parameters.
Compared with original adversarial training which only needs F for gradientcalculation, it requires more than tripled GPU memory and computation cost.
[Figure]
To solve this problem, we propose an efficientalternative solution.
Since ˆ x ≃ x established foran ideal discretizer Q , we can also use a straightthrough estimator between ˆ x and x , which isgiven by
[Equation]
Finally, we can solve the worst δ by∇ ˆ x L
(ˆ x, y, θ ) .
By this way, the computationcost of DAT has been largely reduced.
Compared with original adversarial training, it onlyhas extra computation cost on VQGAN forward,which is relatively controllable.
For clarity, let us restate the pipeline of our DAT.
For each training image x , DAT first use VQGANto get discrete reconstruction ˆ x .
By feeding ˆ x to classifier F , a worst-case perturbation δ can beestimated by computing the gradient of ˆ x towards maximizing the classification loss.
The perturbedimage thus can be created by adding δ on original x .
Finally, x + δ is discretized by VQGAN againand acts as the adversarial input, on which F is trained by minimizing the classificaiton loss.
Thedetails of our DAT is summarized in Algorithm 1.
Explaining the Effectiveness of DAT from the Perspective of Distribution We give an empiricalanalysis to explain why DAT can improve the robustness and generalization without sacrificing cleanaccuracy.
Previous work has pointed out that the underlying distributions of adversarial examplesare different from clean images.
Training on both clean and adversarial images will force the Batch
Normalization (BN) [41] to estimate an inaccurate mixture statistics of feature distribution, and thusimpact the standard performance.
We study this effect by sampling 1000 mini-batches in ImageNetvalidation set, and generate corresponding adversarial images for AT and our DAT.
For each batchwe calculate the mean and variance statistics of last BN of ResNet50.
Then the Pearson correlationcoefficient (PCC) between the statistics on clean and adversarial examples is computed for showing the distributional difference.
Figure 2 shows the frequency histogram of the distributional differenceon 1000 mini-batches.
For training samples of DAT, the peak of the histogram is at 0.95, which isgreater than AT.
It suggests that DAT generates discrete adversarial examples much closer with theclean distributions.
Therefore, training on these examples will reduce the shift of clean distribution in
AT, yielding both the robustness and generalization improvement.
Experiments
To demonstrate the versatility of our method, we experiment Discrete Adversarial Training (DAT) onmultiple tasks including image classification, object detection and self-supervised learning.
4.1 Image Classification
Implementation We implement DAT on two representative architectures: ResNet50 [42] and
ViTs [20].
For ResNet50, we first experiment DAT with vanilla training recipes using robustness 1 .
Then we combine DAT with other orthogonal robust training techniques: DeepAugment [43] and AugMix [32].
Only cross entropy loss is used for generating discrete adversarial examples.
The
JSD loss in AugMix is optimized merely on clean samples.
For ViTs, we adopt ViT-B/16 asbaseline models, which is trained by the recipes in AugReg [44].
Besides, we use DAT to conductsupervised finetuning on downstream ImageNet classification task based on a self-supervised ViTHuge pretrained by MAE [19].
By default, we refer ViT to ViT-B/16 in all tables and figures.
Benchmarks The trained model is evaluated in three aspects: 1) in-distribution performance on
ImageNet-Validation set; 2) adversarial robustness on white-box FGSM [45] and transfer-basedblack-box attack dataset DamageNet [46]; 3) out-of-distribution robustness on ImageNet(IN)-A,
IN-C, IN-V2, IN-R, IN-Sketch and Stylized IN.
Each of them represents a type of out-of-distributionscenario where the classifier is prone to make mistakes.
IN-A [47] places the ImageNet objects inhard contexts; IN-C [48] applies a series of noise, blur, digital and weather corruptions; IN-R [43]collects online images with artificial creation, e.g., cartoons, graphics, video game renditions, etc;
IN-Sketch [49] contains images described by sketches; Stylized IN [3] destroys the texture butmaintains the shape feature by conducting style transfer on ImageNet images.
Except for IN-C whichis measured by mCE, we report the top-1 accuracy on all above datasets.
Results We report all results in Table 1.
For fair comparison, we add DAT on base methods withoutmodifying the original training hyper-parameters, such that improvement is entirely attributed to the
DAT.
For ResNet50, DAT achieves significant improvement on both clean accuracy, adversarial andout-of-distribution robustness.
The improvement seems greater when combining DAT with DeepAugment and AugMix.
For ViTs, we find DAT is compatible with other complex augmentations suchas MixUp [50], CutMix [51] or RandAugment [52] used in AugReg, yielding greater improvement.
Compared with plain ViT, AugReg-ViT and ViT with discrete representation called DrViT [39], DATwith AugReg achieves better performance.
The best result is from ViT-Huge pretrained by MAE [19]and finetuned by DAT, which suggests DAT is also effective in downstream fine-tuning tasks.
Additionally, we also compare DAT with other robust training strategies in Table 2.
AugReg-ViT isadopted as the baseline model.
Most strategies, e.g., AdvProp [26] and Debiased [53] are proposedfor ResNet with auxiliary BatchNorm.
We show these methods cannot work properly on ViTs withonly LayerNorm.
Compared with Pyramid AT [28], our DAT has lower clean accuracy but yieldsstronger robustness.
More results of strategies comparison on ResNet50 refer to Appendix B.
4.2 Self-Supervised Learning
Implementation We experiment DAT on three self-supervised methods: SimCLR [54], SimSiam [55] and recently proposed MoCov3 [56].
The discrete adversarial training is only conducted [Table] DAT on image classification.
Bold number indicates the better performance.
[Table]
during pre-training stage, and the learned representation is evaluated on downstream tasks by standardpipeline [57]. We craft adversarial examples based on RoCL [58], which attacks the pre-trainingobjective by maximizing the contrastive loss. For preventing the cost explosion, we pre-train 200epochs for SimCLR, and 100 epochs for both SimSiam and MoCov3.
We use the implementations on Github 2 , and the only modified argument is batch size, which is set as 1024, 512, 2048 for SimCLR, SimSiam, MoCov3 respectively.
Benchmarks For self-supervised learning, we adopt ImageNet-1K for both training and indistribution testing.
Beyond that, in order to give more comprehensive assessment, we build threedownstream tasks to metric the transferability of the learned visual representation.
Specifically,the linear evaluation reports the top-1 accuracy on five classification datasets: Flowers, CIFAR10,
Caltech101, Cars and DTD.
For downstream evaluation of object detection, we present mAP, AP50,
AP75 on Pascal VOC2007 [59].
ADE20K [60] is used for semantic segmentation task, and both themean intersection over union (mIoU) and accuracy are reported.
Results As shown in Table 3, DAT can enhance the learned representation on ImageNet andget 0.97%, 0.58% and 0.25% improvement on MoCov3, SimCLR, SimSiam respectively.
Forexcluding that DAT is not just over-fitting on ImageNet, we transfer the representations on downstreamrecognition, object detection, semantic segmentation tasks.
The results suggest that DAT enhances theself-supervised representations with better transferability.
In particular, MoCov3 with DAT achievessignificant improvement in all downstream tasks.
This conclusion echoed with previous works [9],which find adversarially robust models often perform better on transfer learning.
4.3 Object Detection
Implementation We implement DAT on two popular detectors: EfficientDet [61] and YOLOv3 [62].
Object detection generally has two sub-tasks: classification and localization.
Det-AdvProp [63]proposes to select more vulnerable sub-task to generate adversarial images, and use auxiliary BN fortraining.
Instead, DAT regards the two sub-tasks as a whole, and attacks the overall detection loss toproduce adversarial images.
Besides, DAT does not modify BN in detectors.
As the memory andcomputation cost of VQGAN increased quadratic with the input resolution, DAT become unaffordable [Table] when input size is large.
Therefore, we only experiment DAT on lightweight version of EfficientDetand YOLOv3 with input size smaller than 512.
Benchmarks We train the detectors using the COCO 2017 object detection dataset [64] and evaluatethem on COCO’s validation set.
COCO-C [65] is used for testing the robustness to natural corruptions.
We report mAP on COCO and COCO-C as the clean and robust performance respectively. rPC [65]is used to to measure relative performance degradation under corruption.
[Table]
Results We compare the detectors with DAT, Det-AdvProp [63] and vanillatraining in Table 4.
Although DetAdvProp has been shown effective on
EfficientDet with size larger than D0,we find on smaller detectors, the promotion of Det-AdvProp is subtle.
On
EfficientDet-Lite0 and YOLOv3-320,it even gets worse on COCO mAPthan vanilla training.
While our DATachieves better result on both clean andcorrupted input.
EfficientDet-Lite0 with
DAT get 27.3 and 17.89 mAP on COCOand COCO-C, resulting in 65.53% Relative rPC.
By comparison, vanilla YOLOv3 models have lower Relative rPC, showing it is morevulnerable than EfficientDet at same clean mAP.
After equipped with DAT, YOLOv3-320 can get20.55 mAP on COCO-C, leading to 7.75% improvement on rPC.
4.4 Ablations
Impact of the Codebook in DAT A general understanding is that the codebook with larger Kshould have stronger representation power.
As suggested in Table 5, when the K is reduced from16384 to 256, the FID of generated images increases 0.35.
We show DAT on larger codebook size canachieve better generalization.
However, the clean accuracy seems no improvement with increasing of
K .
We think the reason is that the reconstruction quality has already met the demand of DAT.
Andthe improvement on clean accuracy is saturated when K further increases.
Different Types of Discretizer Q
Our work is based on the hypothesis of an ideal Q with no lossin discretization process.
However, such perfect Q is not existing in practice.
So we study if and howthe ability of Q affects the performance of our DAT in Table 5.
The results show DAT performs betterby using discretizer with higher FID.
We find clean accuracy declines 1.3% on DALL-E, showing itis important for choosing a powerful discretizer.
Types of pre-training datasets have little impact.
Wefind the discretizer pre-trained on OpenImages [66] and transferred to discretize ImageNet images,can also achieve comparable results.
The Performance on Different Magnitude α
We present the DAT results on ResNet50 and ViTwith different magnitude α in Figure 3.
α = 0 means the model is only trained on images augmented Models Datasets f K d
FID
ImageNet FGSM A C ↓
R
Stylized
VQGAN [40]
OpenImages [66]   1.
10.80 VQGAN [40]
OpenImages [66]   5.
10.76 VQGAN [40]
OpenImages [66]   1.
10.01 VQGAN [13]
ImageNet [67]   4.
9.58 DALL-E [68]
Private   32. by VQGAN with no adversarial training process.
We show DAT with α = 0 makes the models havehigh clean accuracy but the lowest robustness.
With the increase of α , there is still a robustness andaccuracy trade-off.
DAT with α = 0 .
4 on ResNet50 achieves the best adversarial robustness butgeneralization is getting worse.
By contrast, we surprisingly find ViT has the lower sensibility on α .
Even if ViT is trained on DAT with α = 0 . 4 , there is still no great drop of clean and OOD accuracy.
We suspect that ViT is more suitable for training on discrete examples, and the strong modelingability makes ViT greater than ResNet50.
��� ��� ��� ��� ��� ��������� ���� ���� ���� ���� ���� � � � � � � � �� � � � � � �������� ���
[Figure]
��� ��� ��� ��� ��� ��������� ���� ���� ���� ���� ���� � � � � � � � � � � � � � � � � � � � � �� �������� ���
[Figure]
Adversarial Robustness ��� ��� ��� ��� ��� ��������� ���� ���� ���� ���� ���� �� � � � � � � � � � � �� �������� ���
[Figure]
Results on Stronger Attacks In the main experiment, only one-step FGSM is used for examiningthe robustness under white-box adversarial attacks.
To give more comprehensive evaluation, weadditionally test our DAT under stronger AutoAttack [69], and two unrestricted attacks named AdvDrop [17] and PerC-Adversarial [18].
The result is shown in Figure 4.
DAT brings the improvementof robustness under all three attackers.
ViT trained with DAT achieved extremely high robust accuracyon PerC-Adversarial, which suggests DAT also effects well in defense against unrestricted adversarialattacks.
More details can be referred to Appendix C.��� ��� ��� ��� ��� ���� ��� ���� ���� ���� ���� � � � � � � � � �� � � � � � �������� ������������ ��� ������� [Figure] ���� ���� ���� ���� ���� ������������������ ��� ���� ���� ���� ���� � � � � � � � � �� � � � � � �������� ������������ [Figure]
�������
[Figure]
���� [Figure]  Conclusion
In this paper, we find transferring NLP-style adversarial training to vision models can enhance thelearned visual representation effectively.
The proposed Discrete Adversarial Training (DAT) does notneed to modify the model architecture and works for both ResNet50 and ViT across multiple tasks.
DAT reports the state-of-the-art robustness on ImageNet-C and Stylized-ImageNet, exhibiting stronggeneralization.
This work only gives a empirical exploration of visual adversarial training in discretespace.
We hope our work can encourage more researchers to investigate this direction. Also we arelooking for more deeper theoretical explanation of the effectiveness of DAT in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
No There are
no negative societal impacts needed to discuss
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The experiment is expensive and need long time, so
we
couldn’t run multiple times
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See in Appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
We do not release new assets
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c
) Did
you
include the estimated hourly
wage
paid to participants and the total amount
spent on participant compensation?
[N/A