STRATA: B UILDING R OBUSTNESS WITH A S IMPLE M ETHOD FOR G ENERATING B LACK - BOX A DVERSARIAL A TTACKS FOR M ODELS OF C ODE
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial examples are imperceptible perturbations in the input to a neural model that result in misclassiﬁcation. Generating adversarial examples for source code poses an additional challenge compared to the domains of images and natural language, because source code perturbations must adhere to strict semantic guidelines so the resulting programs retain the functional meaning of the code.
We propose a simple and efﬁcient black-box method for generating state-of-the-art adversarial examples on models of code.
Our method generates untargeted and targeted attacks, and empirically outperforms competing gradient-based methods with less information and less computational effort.
We also use adversarial training to construct a model robust to these attacks; our attack reduces the F1 score of CODE 2 SEQ by 42% .
Adversarial training brings the F1 score on adversarial examples up to 99% of baseline.
I NTRODUCTION
Although machine learning has been shown to be effective at a wide variety of tasks across computing, statistical models are susceptible to adversarial examples .
Adversarial examples, ﬁrst identiﬁed in the continuous domain by Szegedy et al.
(2014), are imperceptible perturbations to input that result in misclassiﬁcation.
Researchers have developed effective techniques for adversarial example generation in the image domain (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2017; Papernot et al., 2016a) and in the natural language domain (Alzantot et al., 2018;
Belinkov & Bisk, 2018;
Cheng et al., 2020; Ebrahimi et al., 2018; Michel et al., 2019; Papernot et al., 2016b), although work in the source code domain is less extensive (see Related Work).
The development of adversarial examples for deep learning models has progressed in tandem with the development of methods to make models which are robust to such adversarial attacks, though much is still being learned about model robustness (Goodfellow et al., 2015;
Madry et al., 2018; Shafahi et al., 2019; Wong et al., 2019).
The threat of adversarial examples poses severe risks for ML-based malware defenses (Al-Dujaili et al., 2018; Grosse et al., 2016; Kaur & Kaur, 2015; Kolosnjaji et al., 2018; Kreuk et al., 2019;
Suciu et al., 2019), and introduces the ability of malicious actors to trick ML-based code-suggestion tools to suggest bugs to an unknowing developer (Schuster et al., 2020).
Generating adversarial examples for models of code poses a challenge compared to the image and natural language domain, since the input data is discrete and textual and adversarial perturbations must abide by strict syntactical rules and semantic requirements.
The CODE 2 SEQ model is a state-of-the-art model of code that has been used to explore adversarial example design and robustness methods on models of code (Rabin & Alipour, 2020; Ramakrishnan et al., 2020).
In this work, we propose the Simple TRAined Token Attack ( STRATA ), a novel and effective method for generating black-box and white-box adversarial attacks against CODE 2 SEQ .
Our method replaces local variable names with high impact candidates that are identiﬁed by dataset statistics.
It can also be used effectively for targeted attacks, where the perturbation targets a speciﬁc (altered) output classiﬁcation.
Further, we demonstrate that adversarial training, that is, injecting adversarial examples into CODE 2 SEQ ’s training set, improves the robustness of CODE 2 SEQ to adversarial attacks.
We evaluate STRATA on CODE 2 SEQ , though we hypothesize that the method can be applied to other models.
The principles underlying STRATA apply not only to models of source code, but also to natural language models in contexts where the vocabulary is large and there is limited training data.
STRATA has a number of advantages compared to previously proposed adversarial attack strategies:
STRATA constructs state-of-the-art adversarial examples that outperform gradient-based methods; 2.
STRATA utilizes white-box or black-box adversarial example generation, with black-box attack performance close to that of the white-box; 3.
STRATA does not require the use of a GPU and can be executed more quickly than competing gradient-based attacks (however, we leave the discussion of this result to the Appendix); 4.
STRATA is the only available method (known to the authors at present) which performs targeted attacks on CODE 2 SEQ .
M OTIVATION
CODE 2 SEQ , developed by Alon et al. (2019a), is an encoder-decoder model inspired by SEQ 2 SEQ (Sutskever et al., 2014); it operates on code rather than natural language.
The model is tasked to predict method names from the source code body of a Java method, using a corpus of approximately 16 million Java methods from top-starred public GitHub repositories for training data.
The model considers both the structure of an input program’s
Abstract Syntax Trees (ASTs) as well as the tokens corresponding to identiﬁers such as variable names, types, and invoked method names.
To reduce the vocabulary size, identiﬁer tokens are split into subtokens by commonly used delimiters such as camelCase and under_scores.
In this example, subtokens would include “camel” and “case” and “under” and “scores”.
CODE 2 SEQ encodes subtokens into distributed embedding vectors.
These subtoken embedding vectors are trained to capture semantic structure, so nearby embedding vectors should correspond to semantically similar subtokens (Bengio et al., 2003).
In this paper, we distinguish between subtoken embedding vectors and token embedding vectors.
Subtoken embedding vectors are trained model parameters.
Token embedding vectors are computed as a sum of the embedding vectors of the constituent subtokens.
If the token contains more than ﬁve subtokens, only the ﬁrst ﬁve are summed, as per the CODE 2 SEQ architecture.
The full description and architecture of the CODE 2 SEQ model is given in the original paper by Alon et al. (2019a).
The CODE 2 SEQ model only updates a subtoken embedding as frequently as that subtoken appears during training, which is proportional to its representation in the training dataset.
However, the training datasets have very large vocabularies consisting not only of standard programming language keywords, but also a huge quantity of neologisms.
The frequency at which subtokens appear in the CODE 2 SEQ java-large training set varies over many orders of magnitude, with the least common subtokens appearing fewer than 150 times, and the most common over 10 8 times.
Thus, subtoken embedding vectors corresponding with infrequently-appearing subtokens will be modiﬁed by the training procedure much less often than common subtokens.
Figure 1a demonstrates this phenomenon, showing a disparity between L2 norms of frequent and infrequently-appearing subtoken embedding vectors.
We conﬁrm this empirically.
When we initialized embedding vectors uniformly at random and then trained the model as normal, as per Alon et al. (2019a), we found that the vast majority of ﬁnal, i.e., post-training, embedding vectors change very little from their initialization value.
In fact, 90% of embedding tokens had an L2 distance of less than 0.05 between the initial vector and ﬁnal, post-training vector.
About 10% of subtokens had a large L2 distance between the initial embedding and ﬁnal embedding; these subtokens were more frequent in the training dataset and had embedding vectors with a notably larger ﬁnal L2 magnitude (Figure 1).
The observation that high-L2-norm embedding vectors are associated with subtokens that appear sufﬁciently frequently in the dataset motivates the core intuitions of our attack 1 .
We show in this
[Figure] [Figure]
[Figure]
paper that subtokens with high-L2-norm embedding vectors can be used for effective adversarial examples, which are constructed as follows:
To maximize adversarial effectiveness in a white-box setting, we should use tokens with high L2 norm embedding vectors as local variable name replacements.
We conﬁrm this empirically in the Experiments section.
In the absence of information about the L2 norms of embedding vectors, we can isolate high-L2-norm subtokens for local variable name replacement by selecting tokens which appear in the training dataset often enough to be well trained.
This is empirically conﬁrmed by the large intersection of high-L2-norm subtokens and subtokens with high frequency.
M ETHODS 3. D ATASET DEFINITIONS AND CONSIDERATIONS
Alon et al.
(2019a) presents three datasets that were used for training different CODE 2 SEQ models: javasmall (700k examples), java-medium (4M examples), and java-large (16M examples).
Unsurprisingly,
CODE 2 SEQ performs best on java-large (Alon et al., 2019a).
We also used these three datasets to train three different models: we disambiguate the trained CODE 2 SEQ models from the datasets by denoting them CODE 2 SEQ - SM , CODE 2 SEQ - MD , CODE 2 SEQ - LG for models trained on java-small, -medium, and -large, respectively.
Many of our experiments are evaluated on all three models; however, experiments that require adversarial training are only evaluated on CODE 2 SEQ - SM , for computational feasibility.
3. T HE ATTACK
Traditional adversarial attacks on discrete spaces involve searching the discrete space for semantically similar perturbations that yield a misclassiﬁcation.
Searching the space of all possible valid discrete changes in source code is often intractable or even impossible (Rice, 1953).
However, there are strategies to reduce the search space.
For example, perturbations may be limited to a small number of predeﬁned operations which are known to be semantically equivalent, such as inserting deadcode, replacing local variable names, or replacing expressions with known equivalent expressions.
Gradientbased attacks on the embedding space may also be used in order to optimize the search of the space itself (Ramakrishnan et al., 2020; Yefet et al., 2020).
Gradient-based attacks rely on knowledge of the exact parameters of the model.
We propose STRATA , which replaces local variable names with high-impact subtokens to generate adversarial examples.
STRATA leverages the fact that only a relatively small number of subtoken embeddings are critical for the classiﬁcation task performed by the CODE 2 SEQ model.
In Section 2, we presented two ways to identify high-impact subtokens.
STRATA will use these to replace local variable names.
Recall that the model composes these subtokens into tokens by summing the ﬁrst ﬁve constituent subtoken embedding vectors.
We wish to maximize the L2 norm of the resulting token, while minimizing the semantic change.
We propose three strategies:
single: pick a single subtoken as the token; 2. 5-diff: pick ﬁve different (not necessarily unique) subtokens and concatenate them, which will have a higher expected L2 norm than single ; 3. 5-same: pick a single subtoken, and repeat the subtoken ﬁve times to form a token, which will have the largest expected L2 norm, by the triangle inequality 2 .
We subjectively propose that single is the smallest semantic change, 5-same is the largest change, and 5-diff represents a medium.
For a given Java method, STRATA generates an untargeted perturbation as follows:
Select one random local variable v ; 2.
Choose an adversarial token v ∗ appropriately, using the chosen concatenation strategy ( single , 5-diff , or 5-same ).
For white-box attacks, choose each subtoken from a high-L2norm vocabulary (top- n by L2 norm).
For black-box attacks, choose each subtoken with sufﬁciently high frequency (top- n by frequency).
We discuss the optimal cutoff values ( n ) for L2 and frequency in Section 3.5. 3.
Replace v with v ∗ .
Similarly, to perform targeted attacks in which we want the output to include a particular subtoken t , we perform the same steps as the untargeted attack, and choose v ∗ to be a 5-same concatenation of t .
3. D ATASET - AGNOSTIC BLACK - BOX ATTACKS
The black-box version of STRATA , as proposed, utilizes the subtoken distribution in the training dataset of the targeted model to determine high-impact subtoken candidates.
This technique thus relies on knowledge of the original dataset, which may not always be known or available.
Nonetheless,
STRATA can generate effective adversarial attacks even without the training dataset.
Rather, we can determine subtoken frequency statistics from a different (potentially non-overlapping)
Java dataset.
We empirically conﬁrm that STRATA can use non-overlapping datasets (java-small, java-medium, and java-large) in this way to attack CODE 2 SEQ models that have been trained on a different dataset.
We conclude that the subtoken distributions of non-overlapping Java datasets are sufﬁciently similar for STRATA to be effective.
This conﬁrms that STRATA can be applied in a completely black-box setting without knowledge of the model parameters or training dataset.
We leave the results of this experiment to the appendix.
3. A DVERSARIAL TRAINING
We perform standard adversarial training in order to make CODE 2 SEQ most robust to adversarial attacks.
To test the robustness of an adversarially-trained CODE 2 SEQ model, we perform the following experiment:
We train a CODE 2 SEQ - SM model.
We generate STRATA perturbations of the java-small training dataset using a vocabulary of the top- 1000 subtokens by L2 norm of the embedding.
For each training example, we generate ﬁve adversarial perturbations for each concatenation strategy, single , 5-diff , and 5-same .
We ﬁne-tune the CODE 2 SEQ - SM model with the adversarial examples for a ﬁnal epoch.
3. I MPLEMENTATION
We assayed the effectiveness of the method by applying a simple local variable token replacement attack.
We select a single locally-declared variable at random and rename it with an adversarial token which does not conﬂict with another name.
Because the initial variable is locally declared, we know that changing the name will have no effect elsewhere in the program, and will have no behavioral effect.
The token replacement, however, can effectively attack CODE 2 SEQ ; example shown in the
Appendix.
The CODE 2 SEQ test set consists of ~57,000 examples; we exclude from our testing dataset all methods that cannot be attacked by our method, i.e. all methods without local variables, leaving ~40, examples.
We then create several vocabularies of subtokens from which to choose adversarial substitutions:
All: Contains all subtokens.
Note that the number of subtokens varies by dataset (Table 1); 2.
Top n by L2 norm contains subtokens for which their L2 norm embedding vectors are the n highest; 3.
Top n by frequency contains only the n subtokens which occur in the training data with highest frequency.
To obtain optimal thresholds of n , we swept the range of possibilities to ﬁnd the n that minimizes F score, i.e., generates the best performing adversarial examples (Figure 2).
We present the ﬁnal values of n in Table 1.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
[Figure]
corresponds a more
E XPERIMENTS AND D ISCUSSION 4.
[Figure]
NTARGETED ATTACKS
Table 2 compares the three proposed concatenation strategies on each of the three proposed vocabularies.
To perform comparisons, we measure the F1 score of the model on the java-small testing dataset, with each adversarial perturbation.
Lower F1 scores correspond with better attacks.
We see that the performance of CODE 2 SEQ drops when a local variable is replaced with a token composed of a random selection from all subtokens, using the 5-diff and 5-same concatenation strategies.
However, we observe a larger drop in F1 score when we make variable replacements from subtokens selected from the top- n subtokens by L2 norm or by frequency, using values of n speciﬁed in Table 1.
When we replaced local variables with random tokens with high L2 norm, the F1 score dropped substantially, conﬁrming our hypothesis that we can improve the effectiveness of adversarial examples by selecting replacement tokens such that their subtoken embedding vectors have high L2 norm.
Similarly, adversarial examples that replace a local variable with subtokens of high frequency in the training dataset are highly effective, suggesting that the black-box method of choosing adversarial subtokens based on frequency alone can approximate the white-box attack.
Surprisingly, the F1 score of CODE 2 SEQ - SM increased for random and frequency-based adversarial perturbations constructed with the single concatenation strategy, suggesting that CODE 2 SEQ - SM relies less on variable names for classiﬁcation than CODE 2 SEQ - MD or CODE 2 SEQ - LG .
[Table] [Table]
As proposed in Section
[Table] [Table]
the effectiveness of the adversarial attack is optimized when we replace local variables with a token that is constructed with the 5-same strategy.
Strikingly, the black-box attack (top- n by frequency) is nearly as effective as the white-box attack (top- n by L2).
4. T ARGETED ATTACKS
We perform targeted adversarial attacks that aim to inject a chosen subtoken into the output by performing a STRATA attack using the targeted subtoken for replacement.
See the Appendix for an example.
To assay the effectiveness of targeted attacks, we perform targeted attacks that target three different subtoken vocabularies: (1) all valid subtokens, (2) the optimized L2 vocabulary, and (3) the optimized frequency vocabulary, where vocabularies are optimized for CODE 2 SEQ - SM , CODE 2 SEQ - MD , or CODE 2 SEQ - LG appropriately.
We determine that a particular attack is successful if the selected subtoken is included in the output.
We measure the percent of successful attacks, thus computing an aggregate effectiveness of targeted attacks (Table 3)
[Table]
Table 3 reveals that CODE 2 SEQ is especially vulnerable to targeted attacks performed on high-impact subtokens.
The black-box (frequency) attack performs similarly to the white-box (L2) attack.
4. A DVERSARIAL TRAINING
[Table]
Adversarial training signiﬁcantly improves robustness to the STRATA (Table 4).
Unexpectedly, training with adversarial examples slightly improves the ability of the model to classify clean data (improvement from F1 of 0.369 to 0.371), suggesting that the added robustness forces the model to learn a better representation of the source code.
This adversarial ﬁne-tuning was not more computationally intensive than standard training and represents a simple way to make this model of code more robust.
4. T RANSFERABILITY
In this section we show that gradient-based adversarial training as proposed by Ramakrishnan et al.
(2020) is not effective at improving robustness to STRATA attacks.
We test a CODE 2 SEQ model that has been trained to be robust to the gradient-based adversarial examples proposed by Ramakrishnan et al.
(2020), and ﬁnd that the model is indeed robust to the gradient-based token-replacement perturbations.
However, neither the original nor the robust model are impervious to perturbations produced by
STRATA (Table 5).
This result conﬁrms that STRATA can effectively target models that are robust to some gradient-based perturbations; therefore it is a useful tool when hardening models of code, even when gradient-based perturbations are also being used.
[Table]
STRATA OUTPERFORMS SIMILAR ATTACKS
At the time of writing there are two other works that address adversarial attacks targeting CODE 2 SEQ : a class of source code transformations by Rabin & Alipour (2020) and gradient-based attacks by
Ramakrishnan et al.
We greatly outperform the transformations considered in Rabin &
Alipour (2020).
These results are presented in the Appendix.
We compare our work to gradient-based adversarial perturbations proposed by Ramakrishnan et al.
We consider the variable replacement, print statement insertion, try-catch statement insertion, and worst-case single transformation attacks for our comparison.
We ﬁnd that STRATA outperforms all attacks performed by Ramakrishnan et al.
(2020), including greatly outperforming the gradientbased local variable replacement attack and performing similarly to the worst-case transformation, despite the fact that STRATA generates smaller perturbations with less computational effort
[Table]
and with less information about the model (Table 6).
These results indicate that STRATA attacks are state-of-the-art on CODE 2 SEQ .
[Table]
R ELATED W ORK
Adversarial examples for models of code Allamanis et al.
(2018) provide a comprehensive survey of prior research on models of code.
Several papers develop techniques for generating adversarial examples on models of source code: Quiring et al.
(2019) perform adversarial attacks on source code by applying a Monte-Carlo tree search over possible transformations, and Zhang et al.
(2020) apply Metropolis-Hastings sampling to perform identiﬁer replacement to create adversarial examples.
Bielik & Vechev (2020) improve on the adversarial robustness of models of code by developing a model that can learn to abstain if uncertain.
Adversarial examples for CODE 2 VEC and CODE 2 SEQ
Yefet et al.
(2020) generate both gradientbased targeted and untargeted adversarial examples for CODE 2 VEC Alon et al.
(2019b).
Most directly related to our paper, Ramakrishnan et al.
(2020) perform gradient-based adversarial attacks and adversarial training on CODE 2 SEQ .
Rabin & Alipour (2020) evaluate the robustness of CODE 2 SEQ to semantic-preserving transformations.
Targeted attacks on models of code As previously noted, Yefet et al.
(2020) propose a method for targeted adversarial examples for CODE 2 VEC (Alon et al., 2019b).
To our knowledge, at the time of writing, no other paper performs targeted attacks on CODE 2 SEQ .
Adversarial training for robustness
Many recent papers have examined the robustness of neural network models to adversarial perturbations.
Szegedy et al.
(2014) demonstrates that neural networks are vulnerable to adversarial perturbations, but that training on these perturbations, i.e., adversarial training, can increase robustness.
Further papers explore faster and more effective methods of adversarial training to improve robustness, though mostly in the continuous domain (Madry et al., 2018; Shafahi et al., 2019; Wong et al., 2019).
Ramakrishnan et al.
(2020) perform adversarial training on CODE 2 SEQ to improve robustness.
Yefet et al.
(2020) propose multiple methods to improve robustness of models of source code, including adversarial training, outlier detection, and excluding variable names from model input.
C ONCLUSION
In this work, we presented STRATA , a simple method for generating adversarial examples in discrete space that can be used to help build robustness in models of code.
Because the L2 norm of an embedding vector can be approximated by the frequency in the training data, STRATA can be used to generate both white- and black-box attacks.
We presented effective attacks using this method, including targeted attacks, and also showed that adversarial ﬁne-tuning using STRATA examples can lead to increased robustness, even when the ﬁne-tuned model is only being tested on clean data.
Our work does have some limitations.
In the continuous domain, many adversarial attacks may not even be detected by humans; in the discrete domain, this is not possible, but some attacks in the discrete space are more realistic (harder to spot) than others.
The most powerful attack we propose, 5-same , is also the least realistic; single is a smaller perturbation but it has less effective attacks.
An exciting area of future inquiry would be the application of this attack to other models, including natural language models.
The magnitude differences in subtoken frequency in code is similar to the magnitude differences in word frequency in natural language (Piantadosi, 2014).
In fact, tokens in software follow the same distribution as words in natural language (Zhang, 2008).
We theorize that
STRATA could be used to good effect in models of natural language as well as models of code, and that the basic intuitions underlying the attack can extend even to domains where the actual training data is not available, as long as a suitably similar dataset is available.
To summarize, STRATA is a high-impact, low-cost attack strategy that can be used to bolster robustness; we have made its code open-source so that those who wish to use it may do so.
A F ULL COMPARISON WITH OTHER CODE 2 SEQ ATTACKS
We have shown that STRATA works well to attack the CODE 2 SEQ model and can outperform the attacks by Ramakrishnan et al.
Here, we present a comparison by the transformations proposed by Rabin & Alipour (2020).
In order to be able to compare the same metrics, we calculate the percent prediction change, which is the percent of the time that an adversarial change resulted in a change in prediction.
A higher percent prediction change indicates a better attack.
In Table we compare the performance of our attack to the performance of the transformations generated by Rabin & Alipour (2020)
We ﬁnd that the transformations performed by Rabin & Alipour (2020) result in fewer prediction changes than STRATA .
As above, the most effective strategy is our 5-same attack.
Adversarial attack % prediction change
Variable renaming (Rabin et al.)
[Table]
B C ROSS - DATASET ATTACKS
We present two fully black-box attacks that do not require any information about the targeted
CODE 2 SEQ model or dataset:
As a surrogate model, we train a CODE 2 SEQ model on any available dataset for the targeted programming language.
To obtain adversarial examples from the surrogate, we identify optimal L2 and frequency cutoffs for this model.
Using these cutoffs
[Table]
we construct a vocabulary of the optimal top- n by frequency or by L2 norm.
We show that these adversarial examples can be transferred to other models.
We present the results of the cross-dataset transfer attack proposed in Section 3.3.
In particular, we generate both frequency and L2 STRATA adversarial examples.
We use the L2 norm of the embeddings of CODE 2 SEQ - SM , CODE 2 SEQ - MD , and CODE 2 SEQ - LG , and the subtoken frequencies of java-small, java-medium
[Table]
and java-large to construct six different collections of adversarial examples, of which each collection is a perturbation of the java-small test set.
We test each dataset on each model.
Tables 8 and 9 show the results of the experiments, revealing that while the white-box and known-dataset attacks (the diagonals of the tables) outperform the cross-dataset attacks, the cross-dataset attacks are nonetheless effective.
Furthermore, we note that L2-based cross-dataset attacks are more effective than frequency-based cross-dataset attacks, conﬁrming that L2 norms can effectively identify subtokens that are high impact in other models.
We conclude that STRATA can be performed in a true black-box setting with no information about the model parameters nor the training dataset.
The cross-dataset attack is likely effective due to similar distributions of the Java datasets.
Similar to word frequencies in natural language corpora, we expect that most Java datasets should have a similar subtoken distributions, and thus STRATA should transfer across models trained on different datasets.
[Table]
All adversarial subtokens are concatenated with
[Table]
C E XAMPLES OF TARGETED ATTACKS
To illustrate the effectiveness of STRATA targeted attacks more concretely
[Table]
we target particular arbitrarily-picked subtokens and measure the success rate over the entire testing set (Table 10) and ﬁnd that though effectiveness can vary across different targets, the average effectiveness is quite high
[Table]
We present an example of an attack targeting the token identify in Figure 3.
[Table]
C. C OMPUTATIONALLY INEXPENSIVE
Current alternative methods for attacking models of source code with comparable results involve either an extensive search for optimal transformations or gradient-based optimization for token replacement, or a combination of the two (Rabin & Alipour, 2020; Ramakrishnan et al., 2020; Yefet et al., 2020).
Extensive searches are inherently computationally expensive and, in the case of gradientbased optimization, oftentimes require a GPU for efﬁcient implementation.
STRATA , however, can be implemented to run quickly on even CPU-only machines.
After an initial pre-processing step to mark local variable names for easy replacement which took less than ﬁve minutes on our 24-core CPU-only machine, we were able to construct adversarial examples using STRATA on a dataset of 20, within seconds.
The analogous gradient-based method proposed by Ramakrishnan et al.
(2020) took multiple hours on the same machine.
The combined speed and effectiveness of STRATA will allow researchers to quickly harden their models against adversarial attacks with efﬁcient large-scale adversarial training