Enhance the Visual Representation via Discrete Adversarial Training
Anonymous Author(s) AfﬁliationAddress email
Abstract
Adversarial Training (AT), which is commonly accepted as one of the most effective approaches defending against adversarial examples, can largely harm thestandard performance, thus has limited usefulness on industrial-scale productionand applications.
Surprisingly, this phenomenon is totally opposite in Natural
Language Processing (NLP) task, where AT can even beneﬁt for generalization.
We notice the merit of AT in NLP tasks could derive from the discrete and symbolic input space.
For borrowing the advantage from NLP-style AT, we propose
Discrete Adversarial Training (DAT).
DAT leverages VQGAN to reform the imagedata to discrete text-like inputs, i.e. visual words.
Then it minimizes the maximal risk on such discrete images with symbolic adversarial perturbations.
Wefurther give an explanation from the perspective of distribution to demonstratethe effectiveness of DAT.
As a plug-and-play technique for enhancing the visualrepresentation, DAT achieves signiﬁcant improvement on multiple tasks includingimage classiﬁcation, object detection and self-supervised learning.
Especially, themodel pre-trained with Masked Auto-Encoding (MAE) and ﬁne-tuned by our DATwithout extra data can get 31.40 mCE on ImageNet-C and 32.77% top-1 accuracyon Stylized-ImageNet, building the new state-of-the-art.
Introduction
Nowadays, Deep Neural Networks (DNNs) has achieved excellent performance surpassing humansin most computer vision tasks.
Although remarkable progress has been made, the success of DNNs isactually a false sense when i.i.d hypothesis is not satisﬁed in wild.
Researchers have shown that deepmodels fail in most circumstances including adversarial perturbations [1], common corruptions [2],colors or textures changing [3, 4] etc.
There is still a long way to make DNNs closer to the robusthuman perception.
A possible way towards robust machine perception can be Adversarial Training (AT) [5], whichautomatically ﬁnds failure input cases of DNNs and augment online with these cases for ﬁxing "bugs".
With online augmentation of adversarial examples, AT greatly enhances the adversarial robustness,and helps for learning perceptually-aligned representations [6] with good interpretability [7, 8]and transferability [9].
However, AT is double-edged, which meanwhile degrades the standardperformance caused by problematic regularization [10].
Such problematic regularization makes thedecision boundaries over-smoothed and enlarges indecisive regions.
Surprisingly, previous works [11, 12] observe a strange phenomenon that AT behaves converselyin Natural Language Processing (NLP) tasks.
By automatically ﬁnding adversarial textual inputs,
AT will not hurt the accuracy and even beneﬁt for both generalization and robustness of languagemodels.
This phenomenon motivates us considering whether the merit of NLP-style AT can betransferred to vision tasks.
We notice such merit could derive from the unique data organizingform of language models.
To be speciﬁc, an adversarial image perturbed in continuous pixel spaceactually differs with the truly "hard" examples appeared in real world.
Contrarily, text space isdiscrete and symbolic, where adversarial text is practically existing when a typo is made by humans.
Learning on such adversarial text will obviously improve the generalization on other more texts withconfusing typos.
Therefore, we borrow the symbolic nature of languages, and apply it on CV tasks bydiscretizing continuous images into a more meaningful symbolic space.
Afterwards, AT is conductedfor minimizing the maximal risk on such text-like inputs with symbolic adversarial perturbations.
In this paper, we propose Discrete Adversarial Training (DAT), a new type of adversarial trainingwhich aims to improve both robustness and generalization of vision models.
DAT leverages VQGAN [13] to learn a vocabulary of visual words, also known as image codebook.
For a continuousimage input, each encoded patch embedding is replaced with its closest visual word in the codebook,and represented as a corresponding index.
Then the image is transformed to a sequence of symbolicindices similar with language input.
For generating adversarial examples based on such symbolicsequence, direct use of optimization methods in NLP like combinatorial optimization [14, 15, 16] orsynonym substitutions [14, 16] can be challenging.
The reason lies in: 1) the large search space ofimages and 2) the non-existence of synonym in visual codebook.
To make it more efﬁcient, DATadopts a gradient-based method which assumes the backward adversarial gradient goes straightthrough the complex discretization process, thus gradients on discretized image can be copied tooriginal input.
Then it use one-step search along the direction of estimated gradient such that thediscrete representation will altered adversarially during the discretization process, resulting a discreteadversarial examples.
Finally the discrete adversarial examples is fed into models for training.
Different from AT which always adds l p bound on augmented adversarial examples, DAT affects thediscretization process to produce diverse adversarial inputs beyond l p bound for training.
We show inablation experiment that DAT not only enhances the robustness on l p bounded attacks, but also ispartly beneﬁcial in defense of unrestricted semantic attacks [17, 18].
The overall pipeline of discreteadversarial training is shown in Figure 1.
We further give an analysis to explain the effectiveness of our DAT from the perspective of distribution.
By comparing the distributional difference of training examples in AT and DAT with clean images,we ﬁnd the discrete adversarial examples in our DAT are much closer to the clean distribution.
Such ability of generating "in-distribution" adversarial examples makes DAT can improve the visualrepresentation learning on multiple vision models and tasks, with no sacriﬁcing of clean accuracy.
Our contributions are summarized below:
• To the best of our knowledge, we appear to be the ﬁrst to transfer the merit of NLPstyle adversarial training to vision models, for improving robustness and generalizationsimultaneously.
• We propose Discrete Adversarial Training (DAT), where images are presented as discretevisual words, and the model is training on example which has the adversarially altereddiscrete visual representation.• DAT achieves signiﬁcant improvement on multiple tasks including image classiﬁcation,object detection and self-supervised learning.
Especially, it establishes a new state-of-the-art for robust image classiﬁcation.
By combining MAE [19] pre-training and DAT ﬁne-tuning,our ViT-Huge [20] achieves 31.40 mCE on ImageNet-C [2] and 32.77% top-1 accuracy onStylized-ImageNet [3].
Related Work
Adversarial Training Adversarial Training (AT) [5] is ﬁrst proposed to improve robustnessby training models with adversarial examples.
As one of the most effective defense, existing works [21, 22] have suggested a trade-off between adversarial and clean accuracy in AT.
Despitegreat efforts [23, 24, 25] have been made for mitigating this trade-off, the bad generalization of AT still cannot be fully remedied till now.
In opposite way, some other works [26, 27] use ATto improve the clean accuracy rather than adversarial robustness.
The most close work to ours isthe AdvProp [26], which splits batch norms to prevent the mixed statistics of clean and adversarialexamples, thus learns better adversarial feature for generalization.
Pyramid AT [28] makes AT speciﬁcto ViTs by crafting pyramid perturbation with Dropout enabling, yielding imporved performance.
However, these methods are only applicable under speciﬁc models or tasks.
VILLA [29] is also arepresentation enhancement technique using AT. But contrary to us, it applies vision-style AT only forvision-and-language representation learning. AGAT [30] is another kind of AT beyond pixel space,which perturbs images along attributes, however it has strict requirement of attributes annotation.
Adversarial Augmentation
By borrowing the idea of AT, some previous works [31, 32, 33] proposeto search augmentations adversarially for improving the hardness of training examples.
Adversarial
AutoAugment [31] uses augmentation policy network to produce hard augmentation policies on apre-deﬁned policy space.
AugMax [32] mixes multiple randomly sampled augmentation operatorslike AugMix [34], by using adversarially learned mixing factors.
AdA [33] optimizes the parametersof image-to-image models to generate adversarially corrupted augmented images.
MaxUp [35] usesthe worst augmented data of each data point in a set of random perturbations or transforms for training.
However, these methods create adversarial inputs indirectly and rely on pre-deﬁned augmentations ortranslation models.
They cannot perturb the images locally.
Instead, our DAT directly modiﬁes theimage with no need of pre-deﬁned transforms, and can craft local perturbations on images, which ismore elaborate.
Discrete Visual Representation Learning Early technique [36, 37] of Bag-Of-Visual-Words(BOVW) model has shown great power of discrete representation in visual understanding.
VQVAE [38] uses DNNs to learn neural discrete representations, also known as visual codebook, bygenerative modeling the image distribution.
Recently, the idea of discrete representations learning hasbeen widely emerged in many vision tasks.
In most Masked Image Modeling (MIM) methods [39, 40],visual codebook is needed for BERT-like self-supervised pretraining.
For image classiﬁcation, discreterepresentations strengthen the robustness by preserving the global structure of an object and ignoringlocal details [41].
For image synthesis, adversarial and perceptual objective can be added to VQ-VAEfor learning perceptually-aligned visual codebook [13].
With growing power of generative models,now a well trained VQGAN can produce vivid images with 0.58 FID [42].
In this work, we use
VQGAN for the discretization process in our DAT.
As a powerful generative model, VQGAN cangreatly reduce the information loss in reconstruction process and ensure the high-quality of thegenerated discrete adversarial examples.
Method
3.1 Traditional Adversarial Training
We take the typical image classiﬁcation task as an example to show the formulation of Adversarial
Training (AT).
Consider the training image and label set D
= { x i , y i } ni =1 , and a classiﬁer Fwith learnable parameters ✓ , the classiﬁcation objective is always a cross-entropy loss L ( x, y, ✓ ) .
Adversarial Training (AT) ﬁnds the optimal ✓ by solving a minimax optimization problem:
[Equation]
where the inner optimization ﬁnds the perturbations  on per-pixel values for maximizing the loss,and the outer minimization update ✓ to improve the worst-case performance of the network w.r.t.
theperturbation.
k · k p constraints the p -norm of  to a small value ✏ .
A problem is that AT ﬁnds thefailure case i.e. x +  in continuous pixel space.
However, human does not create or recognize imagesfrom complex pixel values, but from discrete semantic concepts.
Although adversarial examplescan successfully fool the models, they are still different from the real "hard" examples appeared inpractice.
Find the worst (cid:1) by (cid:2)(cid:3)(cid:4) (cid:1) (cid:5) ( (cid:6) (cid:7) + (cid:1) , (cid:8) , (cid:9) )
Update model parameters (cid:9) by (cid:2)(cid:10)(cid:11) (cid:2) (cid:5) ( (cid:6) (cid:7)
+ (cid:1) , (cid:8) , (cid:9) )
Enc 2 0 10 1 3
Dec
Codebook (cid:1)
N … … 0 1 22 
 [Figure] [Figure]
Visual Tasks
Self-Supervised Learning 00.20.40.60.81 cat tiger lion dog
Classification
Object Detection (cid:1) !"
(cid:2) (cid:3)
ForwardDataFlow
Backward Gradient Flow Straight-Through Gradient (cid:12)
Straight-Through Gradient Flow (cid:4) + (cid:3) (cid:4) (cid:4) #$% = (cid:5) ( (cid:4) + (cid:3) )
* (cid:4) = (cid:5) ( (cid:4) ) [Figure]
3.2.1 Image Discretization by Visual Codebook
For discrete adversarial training, it is desirable ﬁrst to learn an expressive visual codebook and represent the training image set in discrete space.
We utilize VQGAN [13] for image discretization.
Moreprecisely, consider a continuous image x 2 R H ⇥ W ⇥ 3 , VQGAN learns an encoder Enc  ( · ) , decoder
Dec ( · ) and quantization q Z ( · ) .
Enc  is a convolutional model which maps x to intermediate latentvectors v = Enc  ( x ) 2 R ( h ⇥ w ) ⇥ d , where h, w is the height, width of the intermediate feature mapand d is the latent dimension.
Subsequently, q Z ( · ) learn a codebook Z
= { z k | z k 2 R d } Kk =1 , suchthat each latent vector v ij 2 R d can be quantized onto its closest codebook entry z k as:
[Equation]
where i, j present each location in feature map.
Then the decoder Dec ( · ) outputs the reconstructionimage ˆ x from the quantized vectors v q by:
[Equation]
VQGAN is trained by minimizing the reconstructed difference between ˆ x and x .
More details can bereferred to Appendix A. So far, given a continuous image x , we can get its corresponding discretereconstruction
ˆ x .
For simplicity, we use Q to stand for the above image discretization process, andthen we have ˆ x = Q ( x ) .
3.2.2 Discrete Adversarial Training
Based on the deﬁnition in Sec 3.2.1, we can generate discrete adversarial examples in inner maximization step of AT.
By slightly modifying the Eq 1, the objective of DAT is formulated as:
[Equation]
where Q transforms the continuous pixel space to discrete input space.
We delete the constraint termsince there is no need to bound the per-pixel values of  .
Suppose that Q is an ideal discretizer withno information loss in discretization process.
The problem lies how to ﬁnd the worst  for maximizingthe classiﬁcation loss.
Similar with traditional AT, we can use gradient-based methods to approximate by:
[Equation]
where ↵ determines the magnitude of the perturbations along the gradient direction.
We set ↵ = 0 .
1by default in DAT.
To expand r x L ( Q ( x ) , y, ✓ ) by chain rule, we have four partial derivative termsas follows:
[Equation]
Through analysing the feasibility of each term, we ﬁnd only @ v q @ v is hard to solve as the nondifferentiable nature of Eq 2.
Fortunately, as proposed in previous work, a straight-through gradientestimator [43, 44] can be used by copying the gradients from v q to v .
By replacing @ ˆ x @ v q · @ v q @ v with@ ˆ x @ v , we can simplify the Eq 6 to r x L ( Q ( x ) , y, ✓ ) = @
L @ ˆ x · @ ˆ x @ v · @ v @ x , which has derivative everywhere.
Although the solution seems workable theoretically, the huge cost makes it impractical on large-scalevision tasks.
The bottleneck mainly lies on that @ ˆ x @ v and @ v @ x require the adversarial gradients backwardthrough Enc and Dec .
Actually, a generator capable of producing high-quality images always has alarge amount of parameters.
Compared with original adversarial training which only needs F forgradient calculation, it requires more than tripled GPU memory and computation cost.
[Figure]
images.
Larger PCC value means smaller distributional difference with clean images.
(a), (b) present the difference on mean and variance statistics respectively.
To solve this problem, we propose an efﬁcientalternative solution.
Since ˆ x ' x established foran ideal discretizer Q , we can also use a straightthrough estimator between ˆ x and x , which isgiven by
[Equation]
Finally, we can solve the worst  byr ˆ x L (ˆ x, y, ✓ ) .
By this way, the computationcost of DAT has been largely reduced.
Compared with original adversarial training, it onlyhas extra computation cost on VQGAN forward, which is relatively controllable.
For clarity, let us restate the pipeline of our DAT.
For each training image x , DAT ﬁrst use VQGANto get discrete reconstruction ˆ x .
By feeding ˆ x to classiﬁer F , a worst-case perturbation  can beestimated by computing the gradient of ˆ x towards maximizing the classiﬁcation loss.
The perturbedimage thus can be created by adding  on original x .
Finally, x +  is discretized by VQGAN againand acts as the adversarial input, on which F is trained by minimizing the classiﬁcaiton loss.
Thedetails of our DAT is summarized in Algorithm 1.
Explaining the Effectiveness of DAT from the Perspective of Distribution We give an empiricalanalysis to explain why DAT can improve the robustness and generalization without sacriﬁcing cleanaccuracy.
Previous work [26] has pointed out that the underlying distributions of adversarial examplesare different from clean images.
Training on both clean and adversarial images will force the Batch
Normalization (BN) [45] to estimate an inaccurate mixture statistics of feature distribution, and thusimpact the standard performance.
We study this effect by sampling 1000 mini-batches in ImageNetvalidation set, and generate corresponding adversarial images for AT and our DAT.
For each batchwe calculate the mean and variance statistics of last BN of ResNet50.
Then the Pearson correlationcoefﬁcient (PCC) between the statistics on clean and adversarial examples is computed for showingthe distributional difference.
Figure 2 shows the frequency histogram of the distributional differenceon 1000 mini-batches.
For training samples of DAT, the peak of the histogram is at 0.95, which isgreater than AT.
It suggests that DAT generates discrete adversarial examples much closer with the clean distributions.
Therefore, training on these examples will reduce the shift of clean distribution in
AT, yielding both the robustness and generalization improvement.
Experiments
To demonstrate the versatility of our method, we experiment Discrete Adversarial Training (DAT) onmultiple tasks including image classiﬁcation, object detection and self-supervised learning.
4.1 Image Classiﬁcation
Implementation We implement DAT on two representative architectures: ResNet50 [46] and
ViTs [20].
For ResNet50, we ﬁrst experiment DAT with vanilla training recipes using "robustness"library 1 .
Then we combine DAT with other orthogonal robust training techniques: DeepAugment [47]and AugMix [34].
Only cross entropy loss is used for generating discrete adversarial examples.
The
JSD loss in AugMix is optimized merely on clean samples.
For ViTs, we adopt ViT-B/16 asbaseline models, which is trained by the recipes in AugReg [48].
Besides, we use DAT to conductsupervised ﬁnetuning on downstream ImageNet classiﬁcation task based on a self-supervised ViTHuge pretrained by MAE [19].
By default, we refer ViT to ViT-B/16 in all tables and ﬁgures.
Benchmarks The trained model is evaluated in three aspects: 1) in-distribution performance on
ImageNet-Validation set; 2) adversarial robustness on white-box FGSM [49] and transfer-basedblack-box attack dataset DamageNet [50]; 3) out-of-distribution robustness on ImageNet(IN)-A,
IN-C, IN-V2, IN-R, IN-Sketch and Stylized IN.
Each of them represents a type of out-of-distributionscenario where the classiﬁer is prone to make mistakes.
IN-A [51] places the ImageNet objects inhard contexts; IN-C [52] applies a series of noise, blur, digital and weather corruptions; IN-R [47]collects online images with artiﬁcial creation, e.g., cartoons, graphics, video game renditions, etc;
IN-Sketch [53] contains images described by sketches; Stylized IN [3] destroys the texture butmaintains the shape feature by conducting style transfer on ImageNet images.
Except for IN-C whichis measured by mCE, we report the top-1 accuracy on all above datasets.
Results We report all results in Table 1.
For fair comparison, we add DAT on base methods withoutmodifying the original training hyper-parameters, such that improvement is entirely attributed to the
DAT.
For ResNet50, DAT achieves signiﬁcant improvement on both clean accuracy, adversarial andout-of-distribution robustness.
The improvement seems greater when combining DAT with DeepAugment and AugMix.
For ViTs, we ﬁnd DAT is compatible with other complex augmentations suchas MixUp [54], CutMix [55] or RandAugment [56] used in AugReg, yielding greater improvement.
Compared with plain ViT, AugReg-ViT and ViT with discrete representation called DrViT [41], DATwith AugReg achieves better performance.
The best result is from ViT-Huge pretrained by MAE [19]and ﬁnetuned by DAT, which suggests DAT is also effective in downstream ﬁne-tuning tasks.
Additionally, we also compare DAT with other robust training strategies in Table 2.
AugReg-ViT isadopted as the baseline model.
Most strategies, e.g., AdvProp [26] and Debiased [57] are proposedfor ResNet with auxiliary BatchNorm.
We show these methods cannot work properly on ViTs withonly LayerNorm.
Compared with Pyramid AT [28], our DAT has lower clean accuracy but yieldsstronger robustness.
More results of strategies comparison on ResNet50 refer to Appendix B.
4.2 Self-Supervised Learning
Implementation We experiment DAT on three self-supervised methods: SimCLR [58], SimSiam [59] and recently proposed MoCov3 [60].
The discrete adversarial training is only conductedduring pre-training stage, and the learned representation is evaluated on downstream tasks by standardpipeline [61].
We craft adversarial examples based on RoCL [62], which attacks the pre-trainingobjective by maximizing the contrastive loss.
For preventing the cost explosion, we pre-train
[Table]
[Table]
Benchmarks For self-supervised learning, we adopt ImageNet-1K for both training and indistribution testing.
Beyond that, in order to give more comprehensive assessment, we build threedownstream tasks to metric the transferability of the learned visual representation.
Speciﬁcally,the linear evaluation reports the top-1 accuracy on ﬁve classiﬁcation datasets: Flowers, CIFAR10,
Caltech101, Cars and DTD.
For downstream evaluation of object detection, we present mAP, AP50,
AP75 on Pascal VOC2007 [63].
ADE20K [64] is used for semantic segmentation task, and both themean intersection over union (mIoU) and accuracy are reported.
Results As shown in Table 3, DAT can enhance the learned representation on ImageNet andget 0.97%, 0.58% and 0.25% improvement on MoCov3, SimCLR, SimSiam respectively.
Forexcluding that DAT is not just over-ﬁtting on ImageNet, we transfer the representations on downstreamrecognition, object detection, semantic segmentation tasks.
The results suggest that DAT enhances theself-supervised representations with better transferability.
In particular, MoCov3 with DAT achievessigniﬁcant improvement in all downstream tasks.
This conclusion echoed with previous works [9],which ﬁnd adversarially robust models often perform better on transfer learning.
4.3 Object Detection
Implementation We implement DAT on two popular detectors: EfﬁcientDet [65] and YOLOv3 [66].
Object detection generally has two sub-tasks: classiﬁcation and localization.
Det-AdvProp [67]proposes to select more vulnerable sub-task to generate adversarial images, and use auxiliary BN fortraining.
Instead, DAT regards the two sub-tasks as a whole, and attacks the overall detection loss toproduce adversarial images.
Besides, DAT does not modify BN in detectors.
As the memory andcomputation cost of VQGAN increased quadratic with the input resolution, DAT become unaffordablewhen input size is large.
Therefore, we only experiment DAT on lightweight version of EfﬁcientDetand YOLOv3 with input size smaller than 512.
Benchmarks We train the detectors using the COCO 2017 object detection dataset [68] and evaluatethem on COCO’s validation set.
COCO-C [69] is used for testing the robustness to natural corruptions.
[Table]
[Table]
[Table]
Results We compare the detectorswith DAT, Det-AdvProp [67] and vanillatraining in Table 4.
Although DetAdvProp has been shown effective on
EfﬁcientDet with size larger than D0,we ﬁnd on smaller detectors, the promotion of Det-AdvProp is subtle.
On
EfﬁcientDet-Lite0 and YOLOv3-320,it even gets worse on COCO mAPthan vanilla training.
While our DATachieves better result on both clean andcorrupted input.
EfﬁcientDet-Lite0 with
DAT get 27.3 and 17.89 mAP on COCOand COCO-C, resulting in 65.53% Relative rPC.
By comparison, vanilla YOLOv3 models have lower Relative rPC, showing it is morevulnerable than EfﬁcientDet at same clean mAP.
After equipped with DAT, YOLOv3-320 can get20.55 mAP on COCO-C, leading to 7.75% improvement on rPC.
4.4 Ablations
Impact of the Codebook in DAT A general understanding is that the codebook with larger Kshould have stronger representation power.
As suggested in Table 5, when the K is reduced from16384 to 256, the FID of generated images increases 0.35.
We show DAT on larger codebook size canachieve better generalization.
However, the clean accuracy seems no improvement with increasing of
K .
We think the reason is that the reconstruction quality has already met the demand of DAT.
Andthe improvement on clean accuracy is saturated when K further increases.
Different Types of Discretizer Q
Our work is based on the hypothesis of an ideal Q with no lossin discretization process.
However, such perfect Q is not existing in practice.
So we study if and howthe ability of Q affects the performance of our DAT in Table 5.
The results show DAT performs betterby using discretizer with higher FID.
We ﬁnd clean accuracy declines 1.3% on DALL-E, showing itis important for choosing a powerful discretizer.
Types of pre-training datasets have little impact.
Weﬁnd the discretizer pre-trained on OpenImages [70] and transferred to discretize ImageNet images,can also achieve comparable results.
The Performance on Different Magnitude ↵
We present the DAT results on ResNet50 and ViTwith different magnitude ↵ in Figure 3.
↵ = 0 means the model is only trained on images augmentedby VQGAN with no adversarial training process.
We show DAT with ↵ = 0 makes the models havehigh clean accuracy but the lowest robustness.
With the increase of ↵ , there is still a robustness andaccuracy trade-off.
DAT with ↵ = 0 .
4 on ResNet50 achieves the best adversarial robustness butgeneralization is getting worse.
By contrast, we surprisingly ﬁnd ViT has the lower sensibility on ↵ .
Even if ViT is trained on DAT with ↵ = 0 . 4 , there is still no great drop of clean and OOD accuracy.
We suspect that ViT is more suitable for training on discrete examples, and the strong modelingability makes ViT greater than ResNet50.
[Figure]
Results on Stronger Attacks In the main experiment, only one-step FGSM is used for examiningthe robustness under white-box adversarial attacks.
To give more comprehensive evaluation, weadditionally test our DAT under stronger AutoAttack [73], and two unrestricted attacks named AdvDrop [17] and PerC-Adversarial [18].
The result is shown in Figure 4.
DAT brings the improvementof robustness under all three attackers.
ViT trained with DAT achieved extremely high robust accuracyon PerC-Adversarial, which suggests DAT also effects well in defense against unrestricted adversarialattacks.
More details can be referred to Appendix C.
[Figure]
Limitations and Conclusions
In this paper, we ﬁnd transferring NLP-style adversarial training to vision models can enhance thelearned visual representation effectively.
We propose Discrete Adversarial Training (DAT), whereimages are presented as discrete visual words by VQGAN, and the model is training on exampleswhich have the adversarially altered discrete visual representation. DAT needs not to modify themodel architecture and works for both CNNs and ViTs across multiple tasks.
DAT reports the state-ofthe-art robustness on ImageNet-C and Stylized-ImageNet, exhibiting strong generalization.
However,
DAT still costs increased training time, this limitation also holds for any adversarial training.
Thestrict assumption of an ideal discretizer is another potential limitation, despite DAT has used powerful
VQGAN model to approach this assumption.
Overall, this work is an empirical exploration of visualadversarial training in discrete space.
All the above limitations are remained as the future optimizationdirection, and we will further seek for more deeper theoretical explanation of the effectiveness ofdiscrete representation in the next work.