Original: Demystifying the Adversarial Robustness of Random Transformation Defenses
Revised: Demystifying the Adversarial Robustness of Random Transformation Defenses

Original: Authors

Original: Abstract
Revised: Abstract

Original: Current machine learning models suffer from evasion attacks (i.e., adversarial examples) raising concerns inmany securitysensitive settings such as autonomous vehicles.
Revised: Current machine learning models suffer from evasion at- tacks (i.e., adversarial examples) raising concerns in securitysensitive settings such as autonomous vehicles.

Original: While many countermeasures have shown promising results, only a few withstand rigorous evaluation from more recent attacks.
Revised: While many countermeasures may look promising, only a few withstand rigorous evaluation.

Original: Re- cently, the use of random transformations (RT) has shownan impressive result, particularly BaRT (Raff et al. 2019) on Ima- geNet.
Revised: Recently, defenses using random trans- formations (RT) have shown impressive results, particularly BaRT (Raff et al. 2019) on ImageNet.

Original: However, this type of defense has not been rigorously evaluated, and its robustness propertiesare poorly understood.
Revised: However, this type of defense has not been rigorously evaluated, leaving its robustness properties poorly understood.

Original: These modelsare also stochastic in nature, making evaluation more challenging and many proposed attacks on deterministic models inapplicable.
Revised: The stochasticity of these models also makes evaluation more challenging and many proposed attacks on deterministic models inapplicable.

Revised: First, we show that the BPDA attack (Athalye, Carlini, and

Revised: Wagner 2018) used in BaRT’s evaluation is ineffective and likely overestimates its robustness.

Original: In this paper, we attempt to construct the strongest possible RT defense through the informed selection of transformations andthe use of Bayesian optimization to tune their parameters.
Revised: We then attempt to construct the strongest possible RT defense through the informed selection of transformations and Bayesian optimization for tuning their parameters.

Original: Furthermore, we attempt to identify the strongest possible attack to evaluate our RT defense.
Revised: Furthermore, we create the strongest possible attack to evaluate our RT defense.

Original: Our new attack vastly outperforms the naive attack, reducing the accu- racy by 83%, while the baseline EoT attack can only achieve 19% reduction, a 4 . 3 × improvement.
Revised: Our new attack vastly outperforms the baseline, reducing the accuracy by 83% compared to the 19% reduction by the commonly used EoT attack ( 4 . 3 × improvement).

Original: This indicates that the RT defense on Imagenette dataset (ten-class subset of Ima- geNet) is not robust against adversarial examples.
Revised: Our result indicates that the RT defense on Imagenette dataset (ten-class subset of ImageNet) is not robust against adversarial examples.

Original: Extending the study further, we use our new attack to adversarially train RT defense (called AdvRT) with an intuition that a stronger attack used during adversarial training will lead to more robust models.
Revised: Extending the study further, we use our new attack to adversarially train RT defense (called AdvRT).

Original: However, the attack is still not sufficiently strong, and thus, the AdvRT model is no more robust than its RT counter- part.
Revised: However, the attack is still not sufficiently strong, and thus, the AdvRT model is no more robust than its RT counterpart.

Original: The outcomes are slightly different for CIFAR-10 dataset where both RT and AdvRT models show some level robust- ness, but they are still outperformed by robust deterministic models.

Original: In the process of formulating our defense and attack, we perform several ablation studies and uncover insights that we hope will broadly benefit scientific communities that study stochastic neural networks and their robustness properties.
Revised: In the process of formulating our defense and attack, we perform several ablation studies and uncover insights that we hope will broadly benefit scientific communities studying stochastic neural networks and their robustness properties.

Original:  Introduction
Revised:  Introduction

Original: Today, deep neural networks are widely deployed in safetycritical settings such as autonomous driving and cybersecurity.
Revised: Today, deep neural networks are widely deployed in safetycritical settings such as autonomous driving and cybersecurity.

Original: Despite their effectiveness at solving a wide-range of challenging problems, they are known to have a major vulnerability.
Revised: Despite their effectiveness at solving a wide-range of challenging problems, they are known to have a major vulnerability.

Original: Tiny crafted perturbations added to inputs (so called adversarial examples ) can arbitrarily manipulate the outputs of these large models, posing a threat to the safety and privacy of the millions of people who rely on existing ML systems.
Revised: Tiny crafted perturbations added to inputs (so called adversarial examples ) can arbitrarily manipulate the outputs of these large models, posing a threat to the safety and privacy of the millions of people who rely on existing ML systems.

Original: The importance of this problem has drawn substantial attention, and yet we have not devised a concrete countermeasure as a research community.
Revised: The importance of this problem has drawn substantial atten- tion, and yet we have not devised a concrete countermeasure as a research community.

Original: Adversarial training (Madry et al. 2018) has been the foremost approach for defending against adversarial examples.
Revised: Adversarial training (Madry et al. 2018) has been the foremost approach for defending against adversarial examples.

Original: While adversarial training provides increased robustness, it results in a loss of accuracy on benign inputs.
Revised: While adversarial training provides increased robustness, it results in a loss of accuracy on benign inputs.

Original: Recently, a promising line of defenses against adversarial examples has emerged.
Revised: Recently, a promising line of defenses against adversarial examples has emerged.

Original: These defenses randomize either the model parameters or the inputs themselves (Lecuyer et al.
Revised: These defenses randomize either the model parameters or the inputs themselves (Lecuyer et al.

Original: 2019; He, Rakin, and Fan 2019;
Revised: 2019; He, Rakin, and Fan 2019;

Original: Raff et al.
Revised: Raff et al.

Original: Liu et al.
Revised: Liu et al.

Original: Xie et al.
Revised: Xie et al.

Original: ; Zhang and Liang 2019; Bender et al. 2020; Liu et al.
Revised: ; Zhang and Liang 2019; Bender et al. 2020; Liu et al.

Original: ; Cohen, Rosenfeld, and Kolter 2019; Dhillon et al. 2018;
Revised: ; Cohen, Rosenfeld, and Kolter 2019; Dhillon et al. 2018;

Original: Guo et al. 2018).
Revised: Guo et al. 2018).

Original: Introducing randomness into the model can be thought of as a form of smoothing that removes sinuous portions of the decision boundary where adversarial examples frequently lie (He, Li, and Song 2018).
Revised: Introducing randomness into the model can be thought of as a form of smoothing that removes sinuous portions of the decision boundary where adversarial examples frequently lie (He, Li, and Song 2018).

Original: Among these randomization approaches, Raff et al.
Revised: Among these randomization approaches, Raff et al.

Original: (2019) propose Barrage of Random Transforms (BaRT), a new defense which applies a large set of random image transformations to classifier inputs.
Revised: (2019) propose Barrage of Random Transforms (BaRT), a new defense which applies a large set of random image transformations to classifier inputs.

Original: They report a 24 × increase in robust accuracy over previously proposed defenses.
Revised: They report a 24 × increase in robust accuracy over previously proposed defenses.

Original: Despite these promising results, researchers still lack a clear understanding of how to properly evaluate random defenses.
Revised: Despite these promising results, researchers still lack a clear understanding of how to properly evaluate random defenses.

Original: This is concerning as a defense can falsely appear more robust than it actually is when evaluated using suboptimal attacks (Athalye, Carlini, and Wagner 2018; Tramer et al. 2020).
Revised: This is concerning as a defense can falsely appear more robust than it actually is when evaluated using suboptimal attacks (Athalye, Carlini, and Wagner 2018; Tramer et al. 2020).

Original: Therefore, in this work, we improve existing attacks on randomized defenses, and use them to rigorously evaluate BaRT and more generally, random transformation (RT) defenses.
Revised: Therefore, in this work, we improve existing attacks on randomized defenses, and use them to rigorously evaluate BaRT and more generally, random transformation (RT) defenses.

Original: We find that sub-optimal attacks have led to an overly optimistic view of these RT defenses.
Revised: We find that sub-optimal attacks have led to an overly optimistic view of these RT defenses.

Original: Notably, we show that even our best RT defense is much less secure than previously thought, formulating a new attack that reduces its security (from 70% adversarial accuracy found by the baseline attack to only 6% on Imagenette).
Revised: Notably, we show that even our best RT defense is much less secure than previously thought, formulating a new attack that reduces its security (from 70% adversarial accuracy found by the baseline attack to only 6% on Imagenette).

Original: We also take the investigation further and combine RT defense with adversarial training.
Revised: We also take the investigation further and combine RT defense with adversarial training.

Original: Nevertheless, this turns out to be ineffective as the attack is not sufficiently strong and only generates weak adversarial examples for the model to train with.
Revised: Nevertheless, this turns out to be ineffective as the attack is not sufficiently strong and only generates weak adversarial examples for the model to train with.

Original: The outcomes appear more promising for CIFAR10, but it still lacks behind deterministic defense such as
Revised: The outcomes appear more promising for CIFAR10, but it still lacks behind deterministic defense such as

Original: Madry et al.
Revised: Madry et al.

Original: (2018) and Zhang et al.
Revised: (2018) and Zhang et al.

Original: We believe that stronger and more efficient attacks on RT-based models will be necessary not only for accurate evaluation of the stochastic defenses but also for improving the effectiveness of adversarial training for such models.
Revised: We believe that stronger and more efficient attacks on RT-based models will be necessary not only for accurate evaluation of the stochastic defenses but also for improving the effectiveness of adversarial training for such models

Original: To summarize, we make the following contributions:
Revised: To summarize, we make the following contributions:

Original: • We show that non-differentiable transforms impede optimization during an attack and even an adaptive technique for circumventing non-differentiability (i e., BPDA (Athalye, Carlini, and Wagner 2018)) is not sufficiently effective.
Revised: • We show that non-differentiable transforms impede optimization during an attack and even an adaptive technique for circumventing non-differentiability (i.e., BPDA (Athalye, Carlini, and Wagner 2018)) is not sufficiently effective.

Original: This reveals that existing RT defenses are likely non-robust.
Revised: This reveals that existing RT defenses are likely non-robust.

Original: • To this end, we suggest that an RT defense should only use differentiable transformations for reliable evaluations and compatibility with adversarial training.
Revised: • To this end, we suggest that an RT defense should only use differentiable transformations for reliable evaluations and compatibility with adversarial training.

Original: • We propose a new state-of-the-art attack for RT defense that improves over EoT (Athalye et al. 2018) in terms of both the loss function and the optimizer.
Revised: • We propose a new state-of-the-art attack for RT defense that improves over EoT (Athalye et al. 2018) in terms of both the loss function and the optimizer.

Original: We explain the success of our attack through the variance of the gradients.
Revised: We explain the success of our attack through the variance of the gradients.

Original: • Improve the RT scheme by using Bayesian optimization for hyperparameter tuning and combining it with adversarial training which uses our new attack method instead of the baseline EoT.
Revised: • Improve the RT scheme by using Bayesian optimization for hyperparameter tuning and combining it with adversarial training which uses our new attack method instead of the baseline EoT.

Original:  Background and Related Works 2. Adversarial Examples
Revised:  Background and Related Works 2. Adversarial Examples

Original: Adversarial examples are carefully perturbed inputs designed to fool a machine learning model (Szegedy et al. 2014; Biggio et al. 2013; Goodfellow, Shlens, and Szegedy 2015).
Revised: Adversarial examples are carefully perturbed inputs designed to fool a machine learning model (Szegedy et al. 2014; Biggio et al. 2013; Goodfellow, Shlens, and Szegedy 2015).

Original: An adversarial perturbation δ is typically constrained to be within some ℓ p -norm ball with a radius of ϵ .
Revised: An adversarial perturbation δ is typically constrained to be within some ℓ p -norm ball with a radius of ϵ .

Original: The ℓ p -norm ball is a proxy to the “imperceptibility” of δ and can be thought of as the adversary’s budget.
Revised: The ℓ p -norm ball is a proxy to the “imperceptibility” of δ and can be thought of as the adversary’s budget.

Original: In this work, we primarily use p = ∞ and only consider adaptive white-box adversary.
Revised: In this work, we primarily use p = ∞ and only consider adaptive white-box adversary.

Original: Finding the worst-case perturbation δ ∗ requires solving the following optimization problem:
Revised: Finding the worst-case perturbation δ ∗ requires solving the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where L : R d × R C → R is the loss function of the target model which, in our case, is a classifier which makes predictions among C classes.
Revised: where L : R d × R C → R is the loss function of the target model which, in our case, is a classifier which makes predictions among C classes.

Original: Projected gradient descent (PGD) is often used to solve the optimization problem in Eqn. 1.
Revised: Projected gradient descent (PGD) is often used to solve the optimization problem in Eqn. 1.

Original: 2. Randomization Defenses
Revised: 2. Randomization Defenses

Original: A number of recent papers have proposed defenses against adversarial examples which utilize inference-time randomization.
Revised: A number of recent papers have proposed defenses against adversarial examples which utilize inference-time randomization.

Original: One common approach is to sample weights of the network from some probability distribution (Liu et al.
Revised: One common approach is to sample weights of the network from some probability distribution (Liu et al.

Original: He, Rakin, and Fan 2019; Liu et al. 2019; Bender et al. 2020).
Revised: He, Rakin, and Fan 2019; Liu et al. 2019; Bender et al. 2020).

Original: In this paper, we instead focus on defenses that apply random transforms to the input (Raff et al. 2019;
Revised: In this paper, we instead focus on defenses that apply random transforms to the input (Raff et al. 2019;

Original: Xie et al.
Revised: Xie et al.

Original: Zhang and Liang 2019; Cohen, Rosenfeld, and Kolter 2019). Many of these defenses claim to achieve state-of-the-art robustness.
Revised: Zhang and Liang 2019; Cohen, Rosenfeld, and Kolter 2019), many of which claim to achieve state-of-the-art robustness.

Original: Unlike prior evaluations, we test these defenses using a wide range of white-box attacks as well as a novel stronger attack.
Revised: Unlike prior evaluations, we test these defenses using a wide range of white-box attacks as well as a novel stronger attack.

Original: A key issue when evaluating these schemes is that PGD attacks require gradients through the entire model pipeline, but many defenses use non-differentiable transforms.
Revised: A key issue when evaluating these schemes is that PGD attacks require gradients through the entire model pipeline, but many defenses use non-differentiable transforms.

Original: later
Revised: As we show later, this can cause evaluation results to be misleading.

Original: Different works have tried applying different random transformations to their inputs.
Revised: Different works have tried applying different random transformations to their inputs.

Original: Xie et al. randomly resize and pad images (Xie et al. 2018).
Revised: et al randomly resize and pad images ( et this defense ranked second in

Original: While this defense ranked second in the NeurIPS 2017 adversarial robustness competition, their security evaluation did not consider adaptive attacks where the adversary has full knowledge of the transformations.
Revised: NeurIPS 2017 adversarial robustness competition their security evaluation did not consider adaptive attacks where the adversary has full knowledge of the transformations.

Original: Zhang et al.
Revised: Zhang et al.

Original: (Zhang and Liang 2019) add Gaussian noise to the input and then quantize it.
Revised: (Zhang and Liang 2019) add Gaussian noise to the input and then quantize it.

Original: They report that this defense outperforms all of the NeurIPS 2017 submissions.
Revised: They report that this defense outperforms all of the NeurIPS 2017 submissions.

Original: For their attack, Zhang et al.
Revised: For their attack, Zhang et al.

Original: approximate the gradient of the transform, which could lead to a sub-optimal attack.
Revised: approximate the gradient of the transform, which could lead to a sub-optimal attack.

Original: In this paper, we use the exact gradients for all transformations when available.
Revised: In this paper, we use the exact gradients for all transformations when available.

Original: More recently, Raff et al.
Revised: More recently, Raff et al.

Original: (Raff et al. 2019) claim to achieve a state-of-the-art robust accuracy 24 × better than adversarial training using a random transformation defense known as Barrage of Random Transforms (BaRT).
Revised: (Raff et al. 2019) claim to achieve a state-of-the-art robust accuracy 24 × better than adversarial training using a random transformation defense known as Barrage of Random Transforms (BaRT).

Original: BaRT involves randomly sampling a large set of image transformations and applying them to the input in a random order.
Revised: BaRT involves randomly sampling a large set of image transformations and applying them to the input in a random order.

Original: Because many transformations are non-differentiable, BaRT evaluates their scheme using an attack that approximates the gradients of the transforms.
Revised: Because many transformations are non-differentiable, BaRT evaluates their scheme using an attack that approximates the gradients of the transforms.

Original: In Section 4, we show that this approximation is ineffective, giving overly optimistic impression of BaRT’s robustness, and we re-evaluate BaRT using a stronger attack which utilizes exact transform gradients.
Revised: In Section 4, we show that this approximation is ineffective, giving overly optimistic impression of BaRT’s robustness, and we re-evaluate BaRT using a stronger attack which utilizes exact transform gradients.

Original:  Random Transformation Defense
Revised:  Random Transformation Defense

Original: Here, we introduce notations and the design of our RT defense, formalizing the BaRT defense.
Revised: Here, we introduce notations and the design of our RT defense, formalizing the BaRT defense.

Original: 3. Decision Rules
Revised: 3. Decision Rules

Original: RT repeatedly applies a randomly chosen transform to the input, uses a neural network to make a prediction, and then averages the softmax prediction scores:
Revised: RT repeatedly applies a randomly chosen transform to the input, uses a neural network to make a prediction, and then averages the softmax prediction scores:

Original: [Equation]
Revised: [Equation]

Original: where σ ( · ) is the softmax function, f : R d → R C a neural network ( C is the number of classes), and the transformation t ( · ; θ ) :
Revised: where σ ( · ) is the softmax function, f : R d → R C a neural network ( C is the number of classes), and the transformation t ( · ; θ ) :

Original: R d → R d is parameterized by a random variable θ drawn from some distribution p ( θ ) .
Revised: R d → R d is parameterized by a random variable θ drawn from some distribution p ( θ ) .

Original: In practice, we approximate the expectation in Eqn. 2 with n Monte Carlo samples per one input x :
Revised: In practice, we approximate the expectation in Eqn. 2 with n Monte Carlo samples per one input x :

Original: [Equation]
Revised: [Equation]

Original: We then define the final prediction as the class with the largest softmax probability: ˆ y ( x ) =
Revised: We then define the final prediction as the class with the largest softmax probability: ˆ y ( x ) =

Original: arg max c ∈ [ C ] [ g n ( x )] c .
Revised: arg max c ∈ [ C ] [ g n ( x )] c .

Original: Note that this decision rule is different from most previous works that use a majority vote on hard labels, i.e., ˆ y maj ( x ) =
Revised: Note that this decision rule is different from most previous works that use a majority vote on hard labels, i.e., ˆ y maj ( x ) =

Original: arg max c ∈ [ 
Revised: arg max c ∈ [ C ]

Original: [Equation]
Revised: [Equation]

Original: et al. 2019; Cohen, Rosenfeld, and Kolter 2019).
Revised: et al. 2019; Cohen, Rosenfeld, and Kolter 2019).

Original: We later show in Appendix D.1 that our rule is empirically superior to the majority vote.
Revised: We later show in Appendix D.1 that our rule is empirically superior to the majority vote.

Original: From the Law of Large Numbers, as n increases, the approximation in Eqn.
Revised: From the Law of Large Numbers, as n increases, the approximation in Eqn.

Original: 3 converges to the expectation in Eqn.
Revised: 3 converges to the expectation in Eqn.

Original: Fig.
Revised: Fig.

Original: 1 illustrates the structure and the components of the RT architecture.
Revised: 1 illustrates the structure and the components of the RT architecture.

Original: 3. Parameterization of Transformations
Revised: 3. Parameterization of Transformations

Original: Here, t ( · ; θ ) represents a composition of S different image transformations where θ = { θ (1) , . .
Revised: Here, t ( · ; θ ) represents a composition of S different image transformations where θ = { θ (1) , . .

Original: , θ ( S ) } and θ ( s ) denotes the parameters for the s -th transformation, i.e., 
Revised: , θ ( S ) } and θ ( s ) denotes the parameters for the s -th transformation, i.e., [Equation] θ

Original: [Equation]
Revised: [Equation]

Original: Each θ ( s ) is a random variable comprised of three components, i.e., θ ( s ) =
Revised: Each θ ( s ) is a random variable comprised of three components, i.e., θ ( s ) =

Original: { τ ( s ) , β ( s ) , α ( s ) } , which dictate the properties of a transformation:
Revised: { τ ( s ) , β ( s ) , α ( s ) } , which dictate the properties of a transformation:

Original: Type τ of transformation to apply (e.g., rotation, JPEG compression), which is uniformly drawn, without replacement, from a pool of K transformation types: τ ∼ Cat ( K, 1 /K ) .
Revised: Type τ of transformation to apply (e.g., rotation, JPEG compression), which is uniformly drawn, without replacement, from a pool of K transformation types: τ ∼ Cat ( K, 1 /K ) .

Original: A boolean β indicating whether the transformation will be applied.
Revised: A boolean β indicating whether the transformation will be applied.

Original: This is a Bernoulli random variable with prob- ability
Revised: This is a Bernoulli random variable with probability

Original: p
Revised: p

Original: β :
Revised: β :

Original: β ∼ Bern ( p ).
Revised: β ∼ Bern ( p )

Original: Strength of the transformation (e.g., rotation angle, JPEG quality) denoted by α , sampled from a predefined distribution (either uniform or normal): α ∼ p ( a ) .
Revised: Strength of the transformation (e.g., rotation angle, JPEG quality) denoted by α , sampled from a predefined distribution (either uniform or normal): α ∼ p ( a ) .

Original: Specifically, for each of the n transformed samples, we sample a permutation of size S out of K transformation types in total, i.e. { τ (1) , . . .
Revised: Specifically, for each of the n transformed samples, we sample a permutation of size S out of K transformation types in total, i.e. { τ (1) , . . .

Original: , τ ( S ) } ∈ Perm( K, S ) .
Revised: , τ ( S ) } ∈ Perm( K, S ) .

Original: Then the boolean and the strength of the s -th transform are sampled: β ( s ) ∼ Bern ( p τ ( s ) ) and α ( s ) ∼ p ( a τ ( s ) ) .
Revised: Then the boolean and the strength of the s -th transform are sampled: β ( s ) ∼ Bern ( p τ ( s ) ) and α ( s ) ∼ p ( a τ ( s ) ) .

Original: We abbreviate this sampling process as θ ∼ p ( θ ) which is repeated for every transformed sample (out of n ) for a single input.
Revised: We abbreviate this sampling process as θ ∼ p ( θ ) which is repeated for every transformed sample (out of n ) for a single input.

Original: Assuming that the K transformation types are fixed, an RT defense introduces, at most, 2 K hyperparameters, { p 1 , . .
Revised: Assuming that the K transformation types are fixed, an RT defense introduces, at most, 2 K hyperparameters, { p 1 , . .

Original: . , p K } and { a 1 , . .
Revised: . , p K } and { a 1 , . .

Original: . , a K } , that can be tuned.
Revised: . , a K } , that can be tuned.

Original: It is also possible to tune by selecting K ′ out of K transformation types, but this is combinatorially large in K .
Revised: It is also possible to tune by selecting K ′ out of K transformation types, but this is combinatorially large in K .

Original: In Appendix C, we show a heuristic for “pruning” the transformation types through tuning p and a (e.g., setting p = 0 is equivalent to removing that transformation type).
Revised: In Appendix C, we show a heuristic for “pruning” the transformation types through tuning p and a (e.g., setting p = 0 is equivalent to removing that transformation type).

Original: 3. Choices of Transformations
Revised: 3. Choices of Transformations

Original: In this work, we use a pool of K = 33 different image transformations including 19 differentiable and 2 nondifferentiable transforms taken from the 30 BaRT transforms (Raff et al. 2019)
Revised: In this work, we use a pool of K = 33 different image transformations including 19 differentiable and 2 nondifferentiable transforms taken from the 30 BaRT transforms (Raff et al. 2019)

Original: (counting each type of noise injection as its own transform).
Revised: (counting each type of noise injection as its own transform).

Original: We replace non-differentiable transformations with a smooth differentiable alternative (Shin and
Revised: We replace non-differentiable transformations with a smooth differentiable alternative (Shin and

Original: Song 2017).
Revised: Song 2017).

Original: The transformations fall into seven groups: noise injection (7), blur filtering (4), color-space alteration (8), edge detection (2), lossy compression (3), geometric transformation (5), and stylization (4).
Revised: The transformations fall into seven groups: noise injection (7), blur filtering (4), color-space alteration (8), edge detection (2), lossy compression (3), geometric transformation (5), and stylization (4).

Original: All transforms are described in
Revised: All transforms are described in

Original: Appendix A
Revised: Appendix A.1.

Original: .  Evaluating Raff et al.(2019)’s BaRT
Revised:  Evaluating Raff et al.

Revised: [Equation] 2019)’s BaRT

Original: Backward-pass differentiable approximation (BPDA) was proposed as a heuristic for approximating gradients of nondifferentiable components in many defenses to make gradientbased attacks applicable (Athalye, Carlini, and Wagner 2018).
Revised: Backward-pass differentiable approximation (BPDA) was proposed as a heuristic for approximating gradients of nondifferentiable components in many defenses to make gradientbased attacks applicable (Athalye, Carlini, and Wagner 2018).

Original: It works by first approximating the function with a neural network and backpropagate through this network instead of the non-differentiable function.
Revised: It works by first approximating the function with a neural network and backpropagate through this network instead of the non-differentiable function.

Original: Evaluations of BaRT in Raff et al.
Revised: Evaluations of BaRT in Raff et al.

Original: (2019) have considered BPDA as some transformations are innately non-differentiable or have zero gradients almost everywhere (e.g., JPEG compression, precision reduction, etc.).
Revised: (2019) have considered BPDA as some transformations are innately non-differentiable or have zero gradients almost everywhere (e.g., JPEG compression, precision reduction, etc.).

Original: To approximate a transformation, we train a model ˜ t ϕ that minimizes the Euclidean distance between the transformed image and the model output:
Revised: To approximate a transformation, we train a model ˜ t ϕ that minimizes the Euclidean distance between the transformed image and the model output:

Original: [Equation]
Revised: [Equation]

Original: We evaluate the BPDA approximation below in a series of experiments that compare the effectiveness of the BPDA attack to an attack that uses exact gradients.
Revised: We evaluate the BPDA approximation below in a series of experiments that compare the effectiveness of the BPDA attack to an attack that uses exact gradients.

Original: 4. Experiment Setup
Revised: 4. Experiment Setup

Original: Our experiments use two datasets: CIFAR-10 and Imagenette (Howard 2021), a ten-class subset of ImageNet.
Revised: Our experiments use two datasets: CIFAR-10 and Imagenette (Howard 2021), a ten-class subset of ImageNet.

Original: While CIFAR-10 is the most common benchmark in the adversarial robustness domain, some image transformations work poorly on low-resolution images.
Revised: While CIFAR-10 is the most common benchmark in the adversarial robustness domain, some image transformations work poorly on low-resolution images.

Original: We choose Imagenette because BaRT was created on ImageNet, but we do not have resources to do thorough investigation on top of adversarial training on ImageNet.
Revised: We choose Imagenette because BaRT was created on ImageNet, but we do not have resources to do thorough investigation on top of adversarial training on ImageNet.

Original: Additionally, the large and realistic images from Imagenette more closely resemble real-world usage All Imagenette models are pre-trained on ImageNet to speed up training and boost performance.
Revised: Additionally, the large and realistic images from Imagenette more closely resemble real-world usage All Imagenette models are pre-trained on ImageNet to speed up training and boost performance.

Original: Since RT models are stochastic, we report their average accuracy together with the 95% confidence interval from 10 independent runs.
Revised: Since RT models are stochastic, we report their average accuracy together with the 95% confidence interval from 10 independent runs.

Original: Throughout this work, we consider the perturbation size ϵ of 16 / 255 for Imagenette and 8 / 255 for CIFAR-10.
Revised: Throughout this work, we consider the perturbation size ϵ of 16 / 255 for Imagenette and 8 / 255 for CIFAR-10.

Original: Appendix A.2 has more details on the experiments (network architecture, hyperparameters, etc.).
Revised: Appendix A.2 has more details on the experiments (network architecture, hyperparameters, etc.).

Original: 4. BPDA Attack is Not Sufficiently Strong
Revised: 4. BPDA Attack is Not Sufficiently Strong

Original: We re-implemented and trained a BaRT model on these datasets, and then evaluated the effectiveness of BPDA attacks against this model.
Revised: We re-implemented and trained a BaRT model on these datasets, and then evaluated the effectiveness of BPDA attacks against this model.

Original: First, we evaluate the full BaRT model in Table 1, comparing an attack that uses a BPDA approximation (as Raff et al.
Revised: First, we evaluate the full BaRT model in Table 1, comparing an attack that uses a BPDA approximation (as Raff et al.

Original: vs an attack that uses the exact gradient for differentiable transforms and
Revised: vs an attack that uses the exact gradient for differentiable transforms and

Original: Transforms used
Revised: Transforms used

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: BPDA for non-differentiable transforms, denoted “BPDA” and “Combo”, respectively.
Revised: BPDA for non-differentiable transforms, denoted “BPDA” and “Combo”, respectively.

Original: Empirically, we observe that attacks using BPDA are far weaker than the equivalent attack using exact gradient approximations.
Revised: Empirically, we observe that attacks using BPDA are far weaker than the equivalent attack using exact gradient approximations.

Original: Similarly, on a variant BaRT model that uses only the subset of differentiable transforms, the BDPA attack is worse than an attack that uses the exact gradient for all transforms.
Revised: Similarly, on a variant BaRT model that uses only the subset of differentiable transforms, the BDPA attack is worse than an attack that uses the exact gradient for all transforms.

Original: BPDA is surprisingly weaker than even a naive attack which approximates all transform gradients with the identity.
Revised: BPDA is surprisingly weaker than even a naive attack which approximates all transform gradients with the identity.

Original: There are a few possible explanations for the inability of BPDA to approximate transformation gradients well: 1.
Revised: There are a few possible explanations for the inability of BPDA to approximate transformation gradients well: 1.

Original: As Fig.
Revised: As Fig.

Original: 2 illustrates, BPDA struggles to approximate some transforms accurately.
Revised: 2 illustrates, BPDA struggles to approximate some transforms accurately.

Original: This might be partly because the architecture Raff et al.
Revised: This might be partly because the architecture Raff et al.

Original: (2019) used (and we use) to approximate each transform has limited functional expressivity: it consists of five convolutional layers with 5x5 kernel and one with 3x3 kernel (all strides are 1), so a single output pixel can only depend on the input pixels fewer thanspaces away in any direction ( 5 ·⌊ 52 ⌋ +1 ·
Revised: (2019) used (and we use) to approximate each transform has limited functional expressivity: it consists of five convolutional layers with 5x5 kernel and one with 3x3 kernel (all strides are 1), so a single output pixel can only depend on the input pixels fewer thanspaces away in any direction ( 5 ·⌊ 52 ⌋ +1 ·

Original: ⌊ 32 ⌋ = 11 ).
Revised: ⌊ 32 ⌋ = 11 ).

Original: Considering the inputs for Imagenette are of size 224 × 224 , some transforms like “crop” which require moving pixels much longer distances are impossible to approximate with such an architecture.
Revised: Considering the inputs for Imagenette are of size 224 × 224 , some transforms like “crop” which require moving pixels much longer distances are impossible to approximate with such an architecture.

Original: The BPDA network training process for solving Eqn.may only find a sub-optimal solution, yielding a poor approximation of the true transformation.
Revised: The BPDA network training process for solving Eqn.may only find a sub-optimal solution, yielding a poor approximation of the true transformation.

Original: During the attack, the trained BPDA networks are given partially transformed images, yet the BPDA networks are only trained with untransformed inputs.
Revised: During the attack, the trained BPDA networks are given partially transformed images, yet the BPDA networks are only trained with untransformed inputs.

Original: Since we are backpropagating through several transforms, one poor transform gradient approximation could ruin the overall gradient approximation.
Revised: Since we are backpropagating through several transforms, one poor transform gradient approximation could ruin the overall gradient approximation.

Original: Appendix A.3 has more details on these experiments.
Revised: Appendix A.3 has more details on these experiments.

Original: These results show that BaRT’s evaluation using BPDA was overly optimistic, and BaRT is not as robust as previously thought.
Revised: These results show that BaRT’s evaluation using BPDA was overly optimistic, and BaRT is not as robust as previously thought.

Original: Since BPDA is unreliable for approximating gradients of non-differentiable image transformations, we recommend that other ensuing RT-based defenses only use differentiable transformations.
Revised: Since BPDA is unreliable for approximating gradients of non-differentiable image transformations, we recommend that other ensuing RT-based defenses only use differentiable transformations.

Original: For the rest of this paper, we only study the robustness of RT defenses with differentiable transforms to isolate them from an orthogonal line of research on non-differentiable defenses (e.g., with approximate gradients or zero-th order attacks).
Revised: For the rest of this paper, we only study the robustness of RT defenses with differentiable transforms to isolate them from an orthogonal line of research on non-differentiable defenses (e.g., with approximate gradients or zero-th order attacks).

Original: Additionally, differentiable models can also boost their robustness further when combined with adversarial training.
Revised: Additionally, differentiable models can also boost their robustness further when combined with adversarial training.

Original: We explore this direction further in Section 7.
Revised: We explore this direction further in Section 7.

Original: Even without non-differentiable transforms, we still lack reliable evaluation on stochastic defenses apart from
Revised: Even without non-differentiable transforms, we still lack reliable evaluation on stochastic defenses apart from

Original: EoT. In the next section, we show that applying an EoT attack on RT defense results in a critically sub-optimal evaluation.
Revised: EoT. In the next section, we show that applying an EoT attack on RT defense results in a critically sub-optimal evaluation.

Original: After that, we propose a stronger attack.
Revised: After that, we propose a stronger attack.

Original:  Hyperparameter Tuning on RT Defenses
Revised:  Hyperparameter Tuning on RT Defenses

Original: Before investigating attacks, we want to ensure we evaluate on the most robust RT defense possible.
Revised: Before investigating attacks, we want to ensure we evaluate on the most robust RT defense possible.

Original: We found that
Revised: We found that

Original: BaRT is not robust, but it could be because of the chosen transformations and their hyperparameters which they do not provide any justification for.
Revised: BaRT is not robust, but it could be because of the chosen transformations and their hyperparameters which they do not provide any justification for.

Original: Finding the most robust
Revised: Finding the most robust

Original: RT defense is, however, challenging because it consists of numerous hyperparameters including the K transformation types, the number of transformations to apply ( S ), and their parameters ( a and p ).
Revised: RT defense is, however, challenging because it consists of numerous hyperparameters including the K transformation types, the number of transformations to apply ( S ), and their parameters ( a and p ).

Original: A typical grid search is intractable since we have 33 transformations, and trying to optimize the parameters directly with the reparameterization trick does not work as most transforms are not differentiable w.r.t.
Revised: A typical grid search is intractable since we have 33 transformations, and trying to optimize the parameters directly with the reparameterization trick does not work as most transforms are not differentiable w.r.t.

Original: their parameters.
Revised: their parameters.

Original: Wesystematically address this problem by using Bayesian optimization (BO) (Snoek, Larochelle, and Adams 2012), a well-known black-box optimization technique used for hyperparameter search, to fine-tune a and p .
Revised: Wesystematically address this problem by using Bayesian optimization (BO) (Snoek, Larochelle, and Adams 2012), a well-known black-box optimization technique used for hyperparameter search, to fine-tune a and p .

Original: In short, BO optimizes an objective function that takes in the hyperparameters ( a and p in our case) as inputs and outputs adversarial accuracy.
Revised: In short, BO optimizes an objective function that takes in the hyperparameters ( a and p in our case) as inputs and outputs adversarial accuracy.

Original: This process, which is equivalent to one iteration in BO, is computationally expensive as it involves training a neural network as a backbone for an RT defense and evaluating it with our new attack.
Revised: This process, which is equivalent to one iteration in BO, is computationally expensive as it involves training a neural network as a backbone for an RT defense and evaluating it with our new attack.

Original: Consequently, we have to scale down the problem by shortening the training, using fewer training/testing data samples, and evaluating with fewer attack steps.
Revised: Consequently, we have to scale down the problem by shortening the training, using fewer training/testing data samples, and evaluating with fewer attack steps.

Original: Essentially, we have to trade off precision of the search for efficiency.
Revised: Essentially, we have to trade off precision of the search for efficiency.

Original: Because BO does not natively support categorical or integral
Revised: Because BO does not natively support categorical or integral

Original: Datasets
Revised: Datasets

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 1 x adv ← x + u ∼ U
Revised: 1 x adv ← x + u ∼ U

Original: [ − ϵ, ϵ ] , { v b } Bb =1 ←2 for t ← 1 to T do { θ i } ni
Revised: [ − ϵ, ϵ ] , { v b } Bb =1 ←2 for t ← 1 to T do { θ i } ni

Original: =1 ∼ p ( θ ) //
Revised: =1 ∼ p ( θ ) //

Original: Compute a gradient estimate with linear loss on logits (Section 6.2) and with SGM (Section 6.3)
Revised: Compute a gradient estimate with linear loss on logits (Section 6.2) and with SGM (Section 6.3)

Original: [Table]
Revised: [Table]

Original: ∇L
Revised: ∇L

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: i )) , y (cid:1)
Revised: i )) , y (cid:1)

Original: [Table]
Revised: [Table]

Original: // Use signed gradients // Update velocities and x adv with
Revised: // Use signed gradients // Update velocities and x adv with

Original: AggMo (Section 6.4)
Revised: AggMo (Section 6.4)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: 9 return
Revised: 9 return

Original: [Table]
Revised: [Table]

Original: variables, we experiment with different choices for K and S without the use of BO.
Revised: variables, we experiment with different choices for K and S without the use of BO.

Original: The full details of this procedure are presented Appendix C.
Revised: The full details of this procedure are presented Appendix C.

Original:  State-of-the-Art Attack on RT Defenses
Revised:  State-of-the-Art Attack on RT Defenses

Original: We propose a new attack on differentiable RT defenses that leverages insights from previous literature on transfer attacks as well as recent stochastic optimization algorithms.
Revised: We propose a new attack on differentiable RT defenses that leverages insights from previous literature on transfer attacks as well as recent stochastic optimization algorithms.

Original: Our attack is immensely successful and shows that even the fine-tuned RT defense from Section 5 shows almost no adversarial robustness (Table 2).
Revised: Our attack is immensely successful and shows that even the fine-tuned RT defense from Section 5 shows almost no adversarial robustness (Table 2).

Original: We summarize our attack in
Revised: We summarize our attack in

Original: Algorithm 1 before describing the setup and investigating the three main design choices that make this attack successful and outperform the baseline from Athalye et al.
Revised: Algorithm 1 before describing the setup and investigating the three main design choices that make this attack successful and outperform the baseline from Athalye et al.

Original: (2018) by a large margin.
Revised: (2018) by a large margin.

Original: 6. Setup: Stochastic Gradient Method
Revised: 6. Setup: Stochastic Gradient Method

Original: First, we describe the setup and explain intuitions around variance of the gradient estimates.
Revised: First, we describe the setup and explain intuitions around variance of the gradient estimates.

Original: Finding adversarial examples on RT defenses can be formulated as the following stochastic optimization problem:
Revised: Finding adversarial examples on RT defenses can be formulated as the following stochastic optimization problem:

Original: [Equation]
Revised: [Equation]

Original: for some objective function L .
Revised: for some objective function L .

Original: Note that we drop dependence on ( x, y ) to declutter the notation.
Revised: Note that we drop dependence on ( x, y ) to declutter the notation.

Original: Since it is not possible to evaluate the expectation or its gradients exactly, the gradients are estimated by sampling { θ i } ni =1 similarly to how we obtain a prediction g n .
Revised: Since it is not possible to evaluate the expectation or its gradients exactly, the gradients are estimated by sampling { θ i } ni =1 similarly to how we obtain a prediction g n .

Original: Suppose that H is smooth and convex, and variance of the gradient estimates is bounded by σ 2 , i.e.,
Revised: Suppose that H is smooth and convex, and variance of the gradient estimates is bounded by σ 2 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: the error of SGD after T iterations is O (cid:16)
Revised: the error of SGD after T iterations is O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: for an appropriate step size (Ghadimi and Lan 2013).
Revised: for an appropriate step size (Ghadimi and Lan 2013).

Original: This result suggests that small σ or low-variance gradient speeds up convergence which is highly desirable for attackers and defenders alike.
Revised: This result suggests that small σ or low-variance gradient speeds up convergence which is highly desirable for attackers and defenders alike.

Original: Specifically, it leads to more efficient and more accurate evaluation as well as a stronger attack to use during adversarial training, which in turn, could yield a better defense (we explore this in Section 7).
Revised: Specifically, it leads to more efficient and more accurate evaluation as well as a stronger attack to use during adversarial training, which in turn, could yield a better defense (we explore this in Section 7).

Original: As a result, the analyses on our attack will be largely based on variance and two other measures of spread of the gradients.
Revised: As a result, the analyses on our attack will be largely based on variance and two other measures of spread of the gradients.

Original: Specifically, we measure (1) the dimension-averaged variance in Eqn. 8, (2) cosine similarity and (3) a percentage of matching signs between mean gradient and each gradient sample.
Revised: Specifically, we measure (1) the dimension-averaged variance in Eqn. 8, (2) cosine similarity and (3) a percentage of matching signs between mean gradient and each gradient sample.

Original: Since all three metrics appear to be highly correlated in theory and in practice, we only report the variance in the main paper.
Revised: Since all three metrics appear to be highly correlated in theory and in practice, we only report the variance in the main paper.

Original: For the other metrics and their mathematical definitions, please see Appendix B.3.
Revised: For the other metrics and their mathematical definitions, please see Appendix B.3.

Original: EoT Baseline.
Revised: EoT Baseline.

Original: We compare our attack to the baseline which is exactly taken from Athalye et al.
Revised: We compare our attack to the baseline which is exactly taken from Athalye et al.

Original: This attack takes on the same form as Eqn. 7 and its gradients are averaged over n gradient samples:
Revised: This attack takes on the same form as Eqn. 7 and its gradients are averaged over n gradient samples:

Original: [Equation]
Revised: [Equation]

Original: It is important to note that this approximation does not exactly match the decision rule of RT defenses as the expectation should be in front of f but behind the loss function (see
Revised: It is important to note that this approximation does not exactly match the decision rule of RT defenses as the expectation should be in front of f but behind the loss function (see

Original: Eqn. 2).
Revised: Eqn. 2).

Original: While the gradient estimates from Eqn.
Revised: While the gradient estimates from Eqn.

Original: 9 are unbiased, they may have high variance as each gradient sample is equivalent to computing the loss on g n with n = 1 .
Revised: 9 are unbiased, they may have high variance as each gradient sample is equivalent to computing the loss on g n with n = 1 .

Original: In the next section, we will compare other options for objective functions and decision rules and show that there are better alternatives to the original EoT.
Revised: In the next section, we will compare other options for objective functions and decision rules and show that there are better alternatives to the original EoT.

Original: Signed gradients.
Revised: Signed gradients.

Original: All of the attacks used in this study including ours and the baseline use signs of gradients instead of the gradients themselves.
Revised: All of the attacks used in this study including ours and the baseline use signs of gradients instead of the gradients themselves.

Original: This is a common practice for gradient-based ℓ ∞ -attacks, and we have also empirically confirm that it leads to much stronger attacks.
Revised: This is a common practice for gradient-based ℓ ∞ -attacks, and we have also empirically confirm that it leads to much stronger attacks.

Original: This is also the
Revised: This is also the

Original: [Figure]
Revised: [Figure]

Original: are too small to see with the markers so we report the numerical results in Table 4.
Revised: are too small to see with the markers so we report the numerical results in Table 4.

Original: “Baseline” refers to with CE reason that we measure sign matching as a measure of spread of the gradient estimates.
Revised: “Baseline” refers to with CE reason that we measure sign matching as a measure of spread of the gradient estimates.

Original: In addition to the ℓ ∞ -constraint, using signed gradients as well as signed momentum is also beneficial as it has been shown to reduce variance for neural network training and achieve even faster convergence than normal SGD in certain cases (Bernstein et al. 2018).
Revised: In addition to the ℓ ∞ -constraint, using signed gradients as well as signed momentum is also beneficial as it has been shown to reduce variance for neural network training and achieve even faster convergence than normal SGD in certain cases (Bernstein et al. 2018).

Original: 6. Adversarial Objectives and Decision Rules
Revised: 6. Adversarial Objectives and Decision Rules

Original: Here, we propose new decision rules and loss functions for the attacks as alternatives to EoT. Note that this need not be the same as the rule used for making prediction in Eqn. 2.
Revised: Here, we propose new decision rules and loss functions for the attacks as alternatives to EoT. Note that this need not be the same as the rule used for making prediction in Eqn. 2.

Original: First, we introduce softmax and logits rules:
Revised: First, we introduce softmax and logits rules:

Original: [Equation]
Revised: [Equation]

Original: H softmax , or loss of the expected softmax probability, is the same rule as the decision rule of RT defenses (Eqn. 2).
Revised: H softmax , or loss of the expected softmax probability, is the same rule as the decision rule of RT defenses (Eqn. 2).

Original: It was also used by Salman et al.
Revised: It was also used by Salman et al.

Original: (2019) where L is cross-entropy loss.
Revised: (2019) where L is cross-entropy loss.

Original: H logits or an expected logits, is similar to H softmax but without the softmax function to avoid potential vanishing gradients from softmax.
Revised: H logits or an expected logits, is similar to H softmax but without the softmax function to avoid potential vanishing gradients from softmax.

Original: In addition to the rules, we experiment with two choices of L commonly used for generating adversarial examples: cross-entropy loss (CE) and linear loss (Linear).
Revised: In addition to the rules, we experiment with two choices of L commonly used for generating adversarial examples: cross-entropy loss (CE) and linear loss (Linear).

Original: The linear loss is defined as the difference between the largest logit of the gradient estimates across ( the wrong class and logit of the correct class:
Revised: The linear loss is defined as the difference between the largest logit of the gradient estimates across ( the wrong class and logit of the correct class:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The advantage of the linear loss is that its gradient estimates are unbiased, similarly to EoT, meaning that the expectation can be moved in front of L due to linearity.
Revised: The advantage of the linear loss is that its gradient estimates are unbiased, similarly to EoT, meaning that the expectation can be moved in front of L due to linearity.

Original: However, this is not the case for CE loss.
Revised: However, this is not the case for CE loss.

Original: Attack evaluation and comparison.
Revised: Attack evaluation and comparison.

Original: We evaluate the attacks by their effectiveness in reducing the adversarial accuracy (lower means stronger attack) on the RT defense obtained from Section 5.
Revised: We evaluate the attacks by their effectiveness in reducing the adversarial accuracy (lower means stronger attack) on the RT defense obtained from Section 5.

Original: In our settting, the adversarial examples are generated once, and then they are used to compute the accuracy 10 times, each with a different random seed on the RT defense.
Revised: In our settting, the adversarial examples are generated once, and then they are used to compute the accuracy 10 times, each with a different random seed on the RT defense.

Original: We report the average accuracy over these 10 runs together with the 95%-confidence interval.
Revised: We report the average accuracy over these 10 runs together with the 95%-confidence interval.

Original: Alternatively, one can imagine a threat model that counts at least one misclassification among a certain number of trials as incorrect.
Revised: Alternatively, one can imagine a threat model that counts at least one misclassification among a certain number of trials as incorrect.

Original: This is an interesting and perhaps more realistic in some settings, but the optimal attack will be very different from EoT as we care a lot less about the expectation.
Revised: This is an interesting and perhaps more realistic in some settings, but the optimal attack will be very different from EoT as we care a lot less about the expectation.

Original: This, however, is outside of the scope of our work.
Revised: This, however, is outside of the scope of our work.

Original: In Fig.
Revised: In Fig.

Original: 3a, we compare the effectiveness of four attacks, each using a different pair of losses and decision rules with varying numbers of PGD steps and samples n .
Revised: 3a, we compare the effectiveness of four attacks, each using a different pair of losses and decision rules with varying numbers of PGD steps and samples n .

Original: The widely used EoT method performs the worst of the four.
Revised: The widely used EoT method performs the worst of the four.

Original: CE loss on mean softmax probability performs better than EoT, confirming the observation made by Salman et al .
Revised: CE loss on mean softmax probability performs better than EoT, confirming the observation made by Salman et al .

Original: (Salman et al.
Revised: (Salman et al.

Original: Linear loss and CE loss on average logits are even better and are consistently the strongest attacks, across all hyperparameters.
Revised: Linear loss and CE loss on average logits are even better and are consistently the strongest attacks, across all hyperparameters.

Original: For the rest of this paper, we adopt the linear loss with mean logits as the main objective function.
Revised: For the rest of this paper, we adopt the linear loss with mean logits as the main objective function.

Original: Connection to variance.
Revised: Connection to variance.

Original: As we predicted in Section 6.1, a stronger attack directly corresponds to lower variance.
Revised: As we predicted in Section 6.1, a stronger attack directly corresponds to lower variance.

Original: This hypothesis is confirmed by Fig.
Revised: This hypothesis is confirmed by Fig.

Original: For instance, the EoT baseline has the highest variance as well as the worst performance according to Fig.
Revised: For instance, the EoT baseline has the highest variance as well as the worst performance according to Fig.

Original: On the other hand, the linear loss (Lin) has the lowest variance among the three loss functions (blue) and hence, it performs the best.
Revised: On the other hand, the linear loss (Lin) has the lowest variance among the three loss functions (blue) and hence, it performs the best.

Original: The other three points in orange will be covered in the next section.
Revised: The other three points in orange will be covered in the next section.

Original: 6. Ensemble and Transfer Attacks
Revised: 6. Ensemble and Transfer Attacks

Original: RT can be regarded as an ensemble with each member sharing the same neural network parameters but applying different sets of transformations to the input (i.e., different θ ’s from random sampling).
Revised: RT can be regarded as an ensemble with each member sharing the same neural network parameters but applying different sets of transformations to the input (i.e., different θ ’s from random sampling).

Original: Consequently, we may view a white-box attack on RT defenses as a “partial” black-box attack on an ensemble of (infinitely) many models where the adversary wishes to “transfer” adversarial examples generated on some subset of the members to another unseen subset.
Revised: Consequently, we may view a white-box attack on RT defenses as a “partial” black-box attack on an ensemble of (infinitely) many models where the adversary wishes to “transfer” adversarial examples generated on some subset of the members to another unseen subset.

Original: Given this interpretation, we apply four techniques designed to enhance the transferability of adversarial examples to improve the attack success rate on RT defense.
Revised: Given this interpretation, we apply four techniques designed to enhance the transferability of adversarial examples to improve the attack success rate on RT defense.

Original: The techniques include momentum boosting (MB)
Revised: The techniques include momentum boosting (MB)

Original: (Dong et al.
Revised: (Dong et al.

Original: , modifying backward passes by ignoring non-linear activation (LinBP) (Guo, Li, and Chen 2020) or by emphasizing the gradient through skip connections of ResNets more than through the residual block (SGM) (Wu et al. 2020), and simply using a targeted attack with the linear loss function (TG) (Zhao,
Revised: , modifying backward passes by ignoring non-linear activation (LinBP) (Guo, Li, and Chen 2020) or by emphasizing the gradient through skip connections of ResNets more than through the residual block (SGM) (Wu et al. 2020), and simply using a targeted attack with the linear loss function (TG) (Zhao,

Original: Liu, and Larson 2021).
Revised: Liu, and Larson 2021).

Original: In Fig.
Revised: In Fig.

Original: 3b, we compare these techniques combined with the best performing loss and decision rule from Section 6.2 (i.e., the linear loss on logits).
Revised: 3b, we compare these techniques combined with the best performing loss and decision rule from Section 6.2 (i.e., the linear loss on logits).

Original: Only
Revised: Only

Original: SGM improves the attack success rate at all settings while the rest result in weaker attacks than the one without any of the techniques (denoted by “Linear (logits)” in Fig. 3a).
Revised: SGM improves the attack success rate at all settings while the rest result in weaker attacks than the one without any of the techniques (denoted by “Linear (logits)” in Fig. 3a).

Original: SGM essentially normalizes the gradients and scales ones from the residual blocks by some constant less than 1 (we use 0 . 5 ) to reduce its influence and prioritize the gradients from the skip connection.
Revised: SGM essentially normalizes the gradients and scales ones from the residual blocks by some constant less than 1 (we use 0 . 5 ) to reduce its influence and prioritize the gradients from the skip connection.

Original: Wu et al.
Revised: Wu et al.

Original: (2020) explain that SGM leads to better transferability because gradients through skip connections preserve “low-level information” which tends to transfer better.
Revised: (2020) explain that SGM leads to better transferability because gradients through skip connections preserve “low-level information” which tends to transfer better.

Original: Intuitively, this agrees with our variance explanation as the increased transferability implies a stronger agreement among gradient samples and hence, less spread or lower variance.
Revised: Intuitively, this agrees with our variance explanation as the increased transferability implies a stronger agreement among gradient samples and hence, less spread or lower variance.

Original: 6. Stochastic Optimization Algorithm
Revised: 6. Stochastic Optimization Algorithm

Original: While most attacks on deterministic models can use naive
Revised: While most attacks on deterministic models can use naive

Original: PGD to solve Eqn.
Revised: PGD to solve Eqn.

Original: 1 effectively, this is not the case for stochastic models like the RT defense.
Revised: 1 effectively, this is not the case for stochastic models like the RT defense.

Original: Here, the adversary only has access to noisy estimates of the gradients, making it a strictly more difficult problem, and techniques used in the deterministic case may no longer apply.
Revised: Here, the adversary only has access to noisy estimates of the gradients, making it a strictly more difficult problem, and techniques used in the deterministic case may no longer apply.

Original: As mentioned in Section 6.1, high-variance gradient estimates undermine the convergence rate of SGD.
Revised: As mentioned in Section 6.1, high-variance gradient estimates undermine the convergence rate of SGD.

Original: Thus, the attack should benefit from optimization techniques aimed at reducing the variance or speeding up the convergence of
Revised: Thus, the attack should benefit from optimization techniques aimed at reducing the variance or speeding up the convergence of

Original: SGD.
Revised: SGD.

Original: We first experiment with common optimizers such as
Revised: We first experiment with common optimizers such as

Original: SGD and Adam (Kingma and Ba 2015) with different hyperparameters, e.g., momentum, Nesterov acceleration, and learning rate schedules, to find the best setting for the linear loss with SGM.
Revised: SGD and Adam (Kingma and Ba 2015) with different hyperparameters, e.g., momentum, Nesterov acceleration, and learning rate schedules, to find the best setting for the linear loss with SGM.

Original: Based on this experiment, we found that a momentum term with an appropriate damping constant plays an important role in the attack success rate.
Revised: Based on this experiment, we found that a momentum term with an appropriate damping constant plays an important role in the attack success rate.

Original: Momentum is also well-known to accelerate and stabilize training of neural networks (Sutskever et al. 2013).
Revised: Momentum is also well-known to accelerate and stabilize training of neural networks (Sutskever et al. 2013).

Original: Fig.
Revised: Fig.

Original: 10a reports adversarial
Revised: 10a reports adversarial

Original: All but the baseline (CE loss with EoT) use the linear loss with SGM, and all but AggMo ( B = 6 ) use the default hyperparameters.
Revised: All but the baseline (CE loss with EoT) use the linear loss with SGM, and all but AggMo ( B = 6 ) use the default hyperparameters.

Original: AggMo with B = 6 outperforms the other algorithms in terms of both the convergence rate and the final adversarial accuracy obtained.
Revised: AggMo with B = 6 outperforms the other algorithms in terms of both the convergence rate and the final adversarial accuracy obtained.

Original: This result is not very sensitive to B as any sufficiently large value ( ≥ 4 ) yields the accuracy at varying attack iterations and indicates that higher momentum constant leads to faster convergence and a higher attack success rate.
Revised: This result is not very sensitive to B as any sufficiently large value ( ≥ 4 ) yields the accuracy at varying attack iterations and indicates that higher momentum constant leads to faster convergence and a higher attack success rate.

Original: However, the results seem highly sensitive to this momentum constant which also varies from one setting to another (e.g., number or types of transformations, dataset, etc.).
Revised: However, the results seem highly sensitive to this momentum constant which also varies from one setting to another (e.g., number or types of transformations, dataset, etc.).

Original: To mitigate this issue, we introduce another optimizer.
Revised: To mitigate this issue, we introduce another optimizer.

Original: AggMo is exactly designed to be less sensitive to choices of the damping coefficient by aggregating B momentum terms with different constants instead of one (Lucas et al.
Revised: AggMo is exactly designed to be less sensitive to choices of the damping coefficient by aggregating B momentum terms with different constants instead of one (Lucas et al.

Original: After only a few tries, we found a wide range of values of B where AggMo outperforms SGD with a finetuned momentum constant (see Fig. 10b).
Revised: After only a few tries, we found a wide range of values of B where AggMo outperforms SGD with a finetuned momentum constant (see Fig. 10b).

Original: Fig.
Revised: Fig.

Original: 5 compares the attacks using different choices of the optimizers to the baseline EoT attack.
Revised: 5 compares the attacks using different choices of the optimizers to the baseline EoT attack.

Original: Here, the baseline can only reduce the adversarial accuracy from 89% to 70% while our best attack manages to reach 6 % or over 4 .
Revised: Here, the baseline can only reduce the adversarial accuracy from 89% to 70% while our best attack manages to reach 6 % or over 4 .

Original: 3 × improvement.
Revised: 3 × improvement.

Original: This concludes that the optimizer plays a crucial role in the success of the attack, and the RT defense, even with a carefully and systematically chosen transformation hyperparameters, is not robust against adversarial examples.
Revised: This concludes that the optimizer plays a crucial role in the success of the attack, and the RT defense, even with a carefully and systematically chosen transformation hyperparameters, is not robust against adversarial examples.

Original: Furthermore, we note that without our loss function and only using AggMo, the accuracy only goes down to 23% at a much slower rate.
Revised: Furthermore, we note that without our loss function and only using AggMo, the accuracy only goes down to 23% at a much slower rate.

Original: Conversely, when the linear loss and SGM are used with SGD (no momentum), the accuracy drops to 51% .
Revised: Conversely, when the linear loss and SGM are used with SGD (no momentum), the accuracy drops to 51% .

Original: This signifies that all three techniques we deploy play important roles to the attack’s effectiveness.
Revised: This signifies that all three techniques we deploy play important roles to the attack’s effectiveness.

Original: 6. Comparison with AutoAttack
Revised: 6. Comparison with AutoAttack

Original: AutoAttack (Croce and Hein 2020) was proposed as a standardized benchmark for evaluating deterministic defenses against adversarial examples.
Revised: AutoAttack (Croce and Hein 2020) was proposed as a standardized benchmark for evaluating deterministic defenses against adversarial examples.

Original: It uses an ensemble of four different attacks that cover weaknesses of one another, one
Revised: It uses an ensemble of four different attacks that cover weaknesses of one another, one

Original: Defenses
Revised: Defenses

Original: [Table]
Revised: [Table]

Original: of which does not use gradients.
Revised: of which does not use gradients.

Original: AutoAttack has been proven to be one of the strongest attack currently and is capable of catching defenses with false robustness caused by gradient obfuscation (Athalye, Carlini, and Wagner 2018).
Revised: AutoAttack has been proven to be one of the strongest attack currently and is capable of catching defenses with false robustness caused by gradient obfuscation (Athalye, Carlini, and Wagner 2018).

Original: While not particularly designed for stochastic models, AutoAttack can be used to evaluate them when combined with
Revised: While not particularly designed for stochastic models, AutoAttack can be used to evaluate them when combined with

Original: EoT. We report the accuracy on adversarial examples generated on AutoAttack with all default hyperparameters in the “standard” mode and 10-sample EoT in Table 2.
Revised: EoT. We report the accuracy on adversarial examples generated on AutoAttack with all default hyperparameters in the “standard” mode and 10-sample EoT in Table 2.

Original: AutoAttack performs worse than the baseline EoT and our attack on both
Revised: AutoAttack performs worse than the baseline EoT and our attack on both

Original: Imagenette and CIFAR-10 by a large margin.
Revised: Imagenette and CIFAR-10 by a large margin.

Original: One of the reasons is that AutoAttack is optimized for efficiency and so each of its attacks is usually terminated once a misclassification occurs.
Revised: One of the reasons is that AutoAttack is optimized for efficiency and so each of its attacks is usually terminated once a misclassification occurs.

Original: This is applicable to deterministic models, but for stochastic ones such as an RT defense, the adversary is better off finding the adversarial examples that maximize the expected loss instead of ones that are misclassified once.
Revised: This is applicable to deterministic models, but for stochastic ones such as an RT defense, the adversary is better off finding the adversarial examples that maximize the expected loss instead of ones that are misclassified once.

Original: To take this property into account, we include the accuracy reported by AutoAttack that treats a sample as incorrect if it is misclassified at least once throughout the entire process.
Revised: To take this property into account, we include the accuracy reported by AutoAttack that treats a sample as incorrect if it is misclassified at least once throughout the entire process.

Original: For Imagenette, the accuracies after each of the four attacks (APGD-CE, APGD-T, FAB, and Square) is applied sequentially are 82 .
Revised: For Imagenette, the accuracies after each of the four attacks (APGD-CE, APGD-T, FAB, and Square) is applied sequentially are 82 .

Original: 03 , 78 . 81 , 78 . 03 , and 77 . 34 , respectively.
Revised: 03 , 78 . 81 , 78 . 03 , and 77 . 34 , respectively.

Original: Note that this is a one-time evaluation so there is no error bar here.
Revised: Note that this is a one-time evaluation so there is no error bar here.

Original: Needless to say, the adversarial accuracy computed this way is strictly lower than the one we reported in Table 2 and violates our threat model.
Revised: Needless to say, the adversarial accuracy computed this way is strictly lower than the one we reported in Table 2 and violates our threat model.

Original: However, it is still higher than that of the baseline EoT and our attack, suggesting that AutoAttack is ineffective against randomized models like RT defenses.
Revised: However, it is still higher than that of the baseline EoT and our attack, suggesting that AutoAttack is ineffective against randomized models like RT defenses.

Original: AutoAttack also comes with a “random” mode for randomized models which only use APGD-CE and APGD-DLR with 20-sample EoT. The adversarial accuracies obtained from this mode are 85 .
Revised: AutoAttack also comes with a “random” mode for randomized models which only use APGD-CE and APGD-DLR with 20-sample EoT. The adversarial accuracies obtained from this mode are 85 .

Original: 62 and 83 .
Revised: 62 and 83 .

Original: 83 or 88 .
Revised: 83 or 88 .

Original: 62 ± 0 .
Revised: 62 ± 0 .

Original: 46 for single-pass evaluation as in Table 2.
Revised: 46 for single-pass evaluation as in Table 2.

Original: This random mode performs worse than the standard version.
Revised: This random mode performs worse than the standard version.

Original:  Combining with Adversarial Training
Revised:  Combining with Adversarial Training

Original: To deepen our investigation, we explore the possibility of combining RT defense with adversarial training.
Revised: To deepen our investigation, we explore the possibility of combining RT defense with adversarial training.

Original: However, this is a challenging problem on its own.
Revised: However, this is a challenging problem on its own.

Original: For normal deterministic models, 10-step PGD is sufficient for reaching adversarial accuracy close to best known attack or the optimal adversarial accuracy.
Revised: For normal deterministic models, 10-step PGD is sufficient for reaching adversarial accuracy close to best known attack or the optimal adversarial accuracy.

Original: However, this is not the case for
Revised: However, this is not the case for

Original: RT defenses as even our new attack still requires more than one thousand iterations before the adversarial accuracy starts to plateau.
Revised: RT defenses as even our new attack still requires more than one thousand iterations before the adversarial accuracy starts to plateau.

Original: Ultimately, the robustness of adversarially trained models largely depends on the strength of the attack used to generate the adversarial examples, and using a weak attack means that the obtained model will not be robust.
Revised: Ultimately, the robustness of adversarially trained models largely depends on the strength of the attack used to generate the adversarial examples, and using a weak attack means that the obtained model will not be robust.

Original: A similar phenomenon is observed by Tram ` er et al.
Revised: A similar phenomenon is observed by Tram ` er et al.

Original: (2018) and Wong,
Revised: (2018) and Wong,

Original: Rice, and Kolter (2020) where an adversarially trained model overfits to the weak FGSM attacks but has shown to be nonrobust with the accurate evaluation.
Revised: Rice, and Kolter (2020) where an adversarially trained model overfits to the weak FGSM attacks but has shown to be nonrobust with the accurate evaluation.

Original: To test this hypothesis, we adversarially train the RT defense from Section 5 using our new attack with 50 iterations (already 5 ×
Revised: To test this hypothesis, we adversarially train the RT defense from Section 5 using our new attack with 50 iterations (already 5 ×

Original: the common number of steps) and call this defense ”AdvRT.”
Revised: the common number of steps) and call this defense ”AdvRT.”

Original: The attack step size is also adjusted accordingly to ϵ/ 8 .
Revised: The attack step size is also adjusted accordingly to ϵ/ 8 .

Original: In Table 3, we confirm that training AdvRT this way results in a model with virtually no robustness improvement over the normal RT on Imagenette.
Revised: In Table 3, we confirm that training AdvRT this way results in a model with virtually no robustness improvement over the normal RT on Imagenette.

Original: On the other hand, the
Revised: On the other hand, the

Original: AdvRT trained on CIFAR-10 proves to be more promising even though it is still not as robust as deterministic models trained with adversarial training or TRADES (Zhang et al.
Revised: AdvRT trained on CIFAR-10 proves to be more promising even though it is still not as robust as deterministic models trained with adversarial training or TRADES (Zhang et al.

Original: Based on this result, we conclude that a stronger attack on RT defenses that converge within a much fewer iterations will be necessary to make adversarial training successful.
Revised: Based on this result, we conclude that a stronger attack on RT defenses that converge within a much fewer iterations will be necessary to make adversarial training successful.

Original: In theory, it might be possible to achieve a robust
Revised: In theory, it might be possible to achieve a robust

Original: RT model with 1,000-step attack on Imagenette, but this is too computationally intensive for us to verify, and it will not to scale to any realistic setting.
Revised: RT model with 1,000-step attack on Imagenette, but this is too computationally intensive for us to verify, and it will not to scale to any realistic setting.

Original:  Conclusion
Revised:  Conclusion

Original: While recent papers report state-of-the-art robustness with
Revised: While recent papers report state-of-the-art robustness with

Original: RT defenses, our evaluations show that RT generally underperforms existing defenses like adversarial training when met with a stronger attack, even after fine-tuning the hyperparameters of the defense.
Revised: RT defenses, our evaluations show that RT generally underperforms existing defenses like adversarial training when met with a stronger attack, even after fine-tuning the hyperparameters of the defense.

Original: Through our experiments, we found that non-differentiability and high-variance gradients can seriously inhibit adversarial optimization, so we recommend using only differentiable transformations along with their exact gradients in the evaluation of future RT defenses.
Revised: Through our experiments, we found that non-differentiability and high-variance gradients can seriously inhibit adversarial optimization, so we recommend using only differentiable transformations along with their exact gradients in the evaluation of future RT defenses.

Original: In this setting, we propose a new state-of-the-art attack that improves significantly over the baseline (PGD with EoT) and show that RT defenses as well as their adversarially trained counterparts are not as robust to adversarial examples as they were previously believed to be.
Revised: In this setting, we propose a new state-of-the-art attack that improves significantly over the baseline (PGD with EoT) and show that RT defenses as well as their adversarially trained counterparts are not as robust to adversarial examples as they were previously believed to be.

Original: A Experiment Details A. Details on the Image Transformations
Revised: A Experiment Details A. Details on the Image Transformations

Original: The exact implementation of RT models and all the transformations will be released.
Revised: The exact implementation of RT models and all the transformations will be released.

Original: Here, we provide some details on each of the transformation types and groups.
Revised: Here, we provide some details on each of the transformation types and groups.

Original: Then, we describe how we approximate some non-differentiable functions with differentiable ones.
Revised: Then, we describe how we approximate some non-differentiable functions with differentiable ones.

Original: Noise
Revised: Noise

Original: injection
Revised: injection

Original: • Erase: Set the pixels in a box with random size and location to zero.
Revised: • Erase: Set the pixels in a box with random size and location to zero.

Original: • Gaussian noise: Add Gaussian noise to each pixel.
Revised: • Gaussian noise: Add Gaussian noise to each pixel.

Original: • Pepper:
Revised: • Pepper:

Original: Zero out pixels with some probability.
Revised: Zero out pixels with some probability.

Original: • Poisson noise: Add Poisson noise to each pixel.
Revised: • Poisson noise: Add Poisson noise to each pixel.

Original: Salt: Set pixels to one with some probability.
Revised: Salt: Set pixels to one with some probability.

Original: • Speckle noise: Add speckle noise to each pixel.
Revised: • Speckle noise: Add speckle noise to each pixel.

Original: • Uniform noise: Add uniform noise to each pixel.
Revised: • Uniform noise: Add uniform noise to each pixel.

Original: Blur filtering
Revised: Blur filtering

Original: • Box blur:
Revised: • Box blur:

Original: Blur with randomly sized mean filter.
Revised: Blur with randomly sized mean filter.

Original: • Gaussian blur: Blur with randomly sized Gaussian filter with randomly chosen variance.
Revised: • Gaussian blur: Blur with randomly sized Gaussian filter with randomly chosen variance.

Original: • Median blur: Blur with randomly sized median filter.
Revised: • Median blur: Blur with randomly sized median filter.

Original: • Motion blur: Blur with kernel for random motion angle and direction.
Revised: • Motion blur: Blur with kernel for random motion angle and direction.

Original: Color space
Revised: Color space

Original: alteration
Revised: alteration

Original: • HSV: Convert to HSV color-space, add uniform noise, then convert back.
Revised: • HSV: Convert to HSV color-space, add uniform noise, then convert back.

Original: • LAB: Convert to LAB color-space, add uniform noise, then convert back.
Revised: • LAB: Convert to LAB color-space, add uniform noise, then convert back.

Original: • Gray scale mix: Mix channels with random proportions.
Revised: • Gray scale mix: Mix channels with random proportions.

Original: • Gray scale partial mix: Mix channels with random proportions, then mix gray image with each channel with random proportions.
Revised: • Gray scale partial mix: Mix channels with random proportions, then mix gray image with each channel with random proportions.

Original: • Two channel gray scale mix: Mix two random channels with random proportions.
Revised: • Two channel gray scale mix: Mix two random channels with random proportions.

Original: • One channel partial gray: Mix two random channels with random proportions, then mix gray image with other channel.
Revised: • One channel partial gray: Mix two random channels with random proportions, then mix gray image with other channel.

Original: • XYZ: Convert to XYZ color-space, add uniform noise, then convert back.
Revised: • XYZ: Convert to XYZ color-space, add uniform noise, then convert back.

Original: • YUV: Convert to YUV color-space, add uniform noise, then convert back.
Revised: • YUV: Convert to YUV color-space, add uniform noise, then convert back.

Original: Edge detection
Revised: Edge detection

Original: • Laplacian: Apply Laplacian filter.
Revised: • Laplacian: Apply Laplacian filter.

Original: • Sobel: Apply the Sobel operator.
Revised: • Sobel: Apply the Sobel operator.

Original: Lossy compression
Revised: Lossy compression

Original: • JPEG compression: Compress image using JPEG to a random quality.
Revised: • JPEG compression: Compress image using JPEG to a random quality.

Original: • Color precision reduction: Reduce color precision to a random number of bins.
Revised: • Color precision reduction: Reduce color precision to a random number of bins.

Original: • FFT perturbation: Perform FFT on image and remove each component with some probability.
Revised: • FFT perturbation: Perform FFT on image and remove each component with some probability.

Original: • Affine: Perform random affine transformation on image.
Revised: • Affine: Perform random affine transformation on image.

Original: • Crop: Crop image randomly and resize to original shape.
Revised: • Crop: Crop image randomly and resize to original shape.

Original: • Horizontal flip: Flip image across the vertical.
Revised: • Horizontal flip: Flip image across the vertical.

Original: • Swirl: Swirl the pixels of an image with random radius and strength.
Revised: • Swirl: Swirl the pixels of an image with random radius and strength.

Original: • Vertical flip: Flip image across the horizontal.
Revised: • Vertical flip: Flip image across the horizontal.

Original: Stylization • Color jitter: Randomly alter the brightness, contrast, and saturation.
Revised: Stylization • Color jitter: Randomly alter the brightness, contrast, and saturation.

Original: • Gamma: Randomly alter gamma.
Revised: • Gamma: Randomly alter gamma.

Original: • Sharpen: Apply sharpness filter with random strength.
Revised: • Sharpen: Apply sharpness filter with random strength.

Original: • Solarize: Solarize the image.
Revised: • Solarize: Solarize the image.

Original: Non-differentiable (for BPDA Tests Only) • Adaptive histogram: Equalize histogram in patches of random kernel size.
Revised: Non-differentiable (for BPDA Tests Only) • Adaptive histogram: Equalize histogram in patches of random kernel size.

Original: • Chambolle denoise: Apply Chambolle’s total variation denoising algorithm with random weight (can be implemented differentiably but was not due to time constraints).
Revised: • Chambolle denoise: Apply Chambolle’s total variation denoising algorithm with random weight (can be implemented differentiably but was not due to time constraints).

Original: • Contrast stretching: Pick a random minimum and maximum pixel value to rescale intensities (can be implemented differentiably but was not due to time constraints).
Revised: • Contrast stretching: Pick a random minimum and maximum pixel value to rescale intensities (can be implemented differentiably but was not due to time constraints).

Original: • Histogram: Equalize histogram using a random number of bins.
Revised: • Histogram: Equalize histogram using a random number of bins.

Original: Unused transforms from BaRT
Revised: Unused transforms from BaRT

Original: • Seam carving: Algorithm used in Raff et al.
Revised: • Seam carving: Algorithm used in Raff et al.

Original: (2019) has been patented and is no longer available for open-source use.
Revised: (2019) has been patented and is no longer available for open-source use.

Original: • Wavelet denoising: The implementation in Raff et al.
Revised: • Wavelet denoising: The implementation in Raff et al.

Original: (2019) is incomplete.
Revised: (2019) is incomplete.

Original: • Salt & pepper: We have already used salt and pepper noise separately.
Revised: • Salt & pepper: We have already used salt and pepper noise separately.

Original: • Non-local means denoising: The implementation of NL means denoising in Raff et al.
Revised: • Non-local means denoising: The implementation of NL means denoising in Raff et al.

Original: (2019) is too slow.
Revised: (2019) is too slow.

Original: A. Experiment Details
Revised: A. Experiment Details

Original: All of the experiments are evaluated on 1000 randomly chosen test samples.
Revised: All of the experiments are evaluated on 1000 randomly chosen test samples.

Original: Since we choose the default n to befor inference and 10 for the attacks, the experiments are at least 10 times more expensive than usual, and we cannot afford enough computation to run a large number of experiments on the entire test set.
Revised: Since we choose the default n to befor inference and 10 for the attacks, the experiments are at least 10 times more expensive than usual, and we cannot afford enough computation to run a large number of experiments on the entire test set.

Original: The networks used in this paper are ResNet-34 (He et al. 2016a) for Imagenette and Preactivation ResNet-20 (He et al. 2016b) for CIFAR-10.
Revised: The networks used in this paper are ResNet-34 (He et al. 2016a) for Imagenette and Preactivation ResNet-20 (He et al. 2016b) for CIFAR-10.

Original: In all of the experiments, we use a learning rate of 0.05, batch size of 128, and weight decay of 0.0005.
Revised: In all of the experiments, we use a learning rate of 0.05, batch size of 128, and weight decay of 0.0005.

Original: We use cosine annealing schedule (Loshchilov and Hutter 2017) for the learning rate with a period of 10 epochs which also doubles after every period.
Revised: We use cosine annealing schedule (Loshchilov and Hutter 2017) for the learning rate with a period of 10 epochs which also doubles after every period.

Original: All models are trained for 70 epochs, and we save the weights with the highest accuracy on the held-out validation data (which does not overlap with the training or test set).
Revised: All models are trained for 70 epochs, and we save the weights with the highest accuracy on the held-out validation data (which does not overlap with the training or test set).

Original: For adversarially trained RT defenses, the cosine annealing step is set to 10 and the training lasts for 70 epochs to reduce the computation.
Revised: For adversarially trained RT defenses, the cosine annealing step is set to 10 and the training lasts for 70 epochs to reduce the computation.

Original: To help the training converge faster, we pre-train these RT models on clean data before turning on adversarial training as suggested by Gupta, Dube, and Verma (2020).
Revised: To help the training converge faster, we pre-train these RT models on clean data before turning on adversarial training as suggested by Gupta, Dube, and Verma (2020).

Original: The network has six convolutional layers.
Revised: The network has six convolutional layers.

Original: All layers have a stride of 1.
Revised: All layers have a stride of 1.

Original: The first five layers have kernel size ofand padding size of 2, and the last layer has a kernel size of 3 and padding size of 1.
Revised: The first five layers have kernel size ofand padding size of 2, and the last layer has a kernel size of 3 and padding size of 1.

Original: The input consists of more thanchannels, 3 of which are for the image RGB channels, 2 of which are CoordConv channels that include the coordinates of each pixel at that pixel’s location, and the remaining channels are the parameters for the transformation copied at each pixel location.
Revised: The input consists of more thanchannels, 3 of which are for the image RGB channels, 2 of which are CoordConv channels that include the coordinates of each pixel at that pixel’s location, and the remaining channels are the parameters for the transformation copied at each pixel location.

Original: The network contains a skip connection from the input to each layer except the final layer.
Revised: The network contains a skip connection from the input to each layer except the final layer.

Original: A. Details on BPDA Experiments
Revised: A. Details on BPDA Experiments

Original: We used the following setup for the differentiability related experiments conducted in Section 4.2:
Revised: We used the following setup for the differentiability related experiments conducted in Section 4.2:

Original: • Each accuracy is an average over 10 trials on the same set of 1000 Imagenette images.
Revised: • Each accuracy is an average over 10 trials on the same set of 1000 Imagenette images.

Original: • The defense samples S
Revised: • The defense samples S

Original: = 10 transforms from the full set of K transforms.
Revised: = 10 transforms from the full set of K transforms.

Original: • The image classifier uses a ResNet-50 architecture like in Raff et al.
Revised: • The image classifier uses a ResNet-50 architecture like in Raff et al.

Original: (2019) trained on transformed images forepochs.
Revised: (2019) trained on transformed images forepochs.

Original: • The attack uses 40 PGD steps of size 4 / 255 with an ϵ = 16 / 255 to minimize the EoT objective.
Revised: • The attack uses 40 PGD steps of size 4 / 255 with an ϵ = 16 / 255 to minimize the EoT objective.

Original: The BPDA network architecture is the same used by Raff et al.
Revised: The BPDA network architecture is the same used by Raff et al.

Original: and is outlined in Fig.
Revised: and is outlined in Fig.

Original: Here are more details on BPDA training:
Revised: Here are more details on BPDA training:

Original: • All BPDA networks were trained using Adam with a learning rate of 0 .
Revised: • All BPDA networks were trained using Adam with a learning rate of 0 .

Original: 01 for 10 epochs.
Revised: 01 for 10 epochs.

Original: • All networks achieve a per-pixel MSE below 0 .
Revised: • All networks achieve a per-pixel MSE below 0 .

Original: The outputs of the BPDA networks are compared to the true transform outputs for several different transform types in Fig.
Revised: The outputs of the BPDA networks are compared to the true transform outputs for several different transform types in Fig.

Original: The specific set of transforms used in each defense are the following:
Revised: The specific set of transforms used in each defense are the following:

Original: • BaRT (all): adaptive histogram, histogram, bilateral blur, box blur, Gaussian blur, median blur, contrast stretching, FFT, gray scale mix, gray scale partial mix, two channel gray scale mix, one channel gray scale mix, HSV, LAB, XYZ, YUV, JPEG compression, Gaussian noise, Poisson noise, salt, pepper, color precision reduction, swirl, Chambolle denoising, crop.
Revised: • BaRT (all): adaptive histogram, histogram, bilateral blur, box blur, Gaussian blur, median blur, contrast stretching, FFT, gray scale mix, gray scale partial mix, two channel gray scale mix, one channel gray scale mix, HSV, LAB, XYZ, YUV, JPEG compression, Gaussian noise, Poisson noise, salt, pepper, color precision reduction, swirl, Chambolle denoising, crop.

Original: • BaRT (only differentiable): all of the BaRT all transforms excluding adaptive histogram, histogram, contrast stretching, and Chambolle denoising.
Revised: • BaRT (only differentiable): all of the BaRT all transforms excluding adaptive histogram, histogram, contrast stretching, and Chambolle denoising.

Original: B Details of the Attacks B. Differentiable Approximation
Revised: B Details of the Attacks B. Differentiable Approximation

Original: Some of the transformations contain non-differentiable operations which can be easily approximated with differentiable functions.
Revised: Some of the transformations contain non-differentiable operations which can be easily approximated with differentiable functions.

Original: Specifically, we approximate the rounding function in JPEG compression and color precision reduction, and the modulo operator in all transformations that require conversion between RGB and HSV color-spaces (HSV alteration and color jitter).
Revised: Specifically, we approximate the rounding function in JPEG compression and color precision reduction, and the modulo operator in all transformations that require conversion between RGB and HSV color-spaces (HSV alteration and color jitter).

Original: Note that we are not using the non-differentiable transform on the forward pass and a differentiable approximation on the backward pass (like in BPDA).
Revised: Note that we are not using the non-differentiable transform on the forward pass and a differentiable approximation on the backward pass (like in BPDA).

Original: Instead, we are using the differentiable version both when performing the forward pass and when computing the gradient.
Revised: Instead, we are using the differentiable version both when performing the forward pass and when computing the gradient.

Original: We take the approximation of the rounding function from
Revised: We take the approximation of the rounding function from

Original: Shin and Song (2017) shown in Eqn.
Revised: Shin and Song (2017) shown in Eqn.

Original: [Equation]
Revised: [Equation]

Original: For the modulo or the remainder function, we approximate it using the above differentiable rounding function as a basis.
Revised: For the modulo or the remainder function, we approximate it using the above differentiable rounding function as a basis.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: To obtain a differentiable approximation, we can replace the rounding operator with its smooth version in Eqn.
Revised: To obtain a differentiable approximation, we can replace the rounding operator with its smooth version in Eqn.

Original: This function (approximately) returns decimal numbers or a fractional part of a given real number, and it can be scaled to approximate a modulo operator with any divisor.
Revised: This function (approximately) returns decimal numbers or a fractional part of a given real number, and it can be scaled to approximate a modulo operator with any divisor.

Original: Note that these operators are step functions and are differentiable almost everywhere, like ReLU.
Revised: Note that these operators are step functions and are differentiable almost everywhere, like ReLU.

Original: However, their derivatives are always zero (unlike ReLU), and so a first-order optimization algorithm would still fail on these functions.
Revised: However, their derivatives are always zero (unlike ReLU), and so a first-order optimization algorithm would still fail on these functions.

Original: B. Effect of the Permutation of the Transformations
Revised: B. Effect of the Permutation of the Transformations

Original: We mentioned in Section 3.2 that a permutation of the transforms { τ ( s ) } Ss =1 is randomly sampled for each of the n samples.
Revised: We mentioned in Section 3.2 that a permutation of the transforms { τ ( s ) } Ss =1 is randomly sampled for each of the n samples.

Original: However, we found that in practice, this leads to highvariance estimates of the gradients.
Revised: However, we found that in practice, this leads to highvariance estimates of the gradients.

Original: On the other hand, fixing the permutation across n samples in each attack iteration (i.e., τ is fixed but not α or β ) results in lower variance and hence, a stronger attack, even though the gradient estimates are biased as τ is fixed.
Revised: On the other hand, fixing the permutation across n samples in each attack iteration (i.e., τ is fixed but not α or β ) results in lower variance and hence, a stronger attack, even though the gradient estimates are biased as τ is fixed.

Original: For instance, with fixed permutation, adversarial accuracy achieved by EoT attack is 51 .
Revised: For instance, with fixed permutation, adversarial accuracy achieved by EoT attack is 51 .

Original: 44 where the baseline EoT with completely random permutation is 70 . 79 .
Revised: 44 where the baseline EoT with completely random permutation is 70 . 79 .

Original: The variance also reduces from 0 .
Revised: The variance also reduces from 0 .

Original: 97 to 0 .
Revised: 97 to 0 .

Original: Additionally, the fixed permutation reduces the computation time as all transformations can be applied in batch.
Revised: Additionally, the fixed permutation reduces the computation time as all transformations can be applied in batch.

Original: All of the attacks reported in this paper, apart from the baseline, use this fixed permutation.
Revised: All of the attacks reported in this paper, apart from the baseline, use this fixed permutation.

Original: B. Variance of Gradients
Revised: B. Variance of Gradients

Original: We have described how we compute the sample variance of the gradients in Section 6.1.
Revised: We have described how we compute the sample variance of the gradients in Section 6.1.

Original: Here, we provide detailed calculations of the other three metrics.
Revised: Here, we provide detailed calculations of the other three metrics.

Original: First, the unbiased variance is computed as normal with an additional normalization by dimension.
Revised: First, the unbiased variance is computed as normal with an additional normalization by dimension.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: (logits
Revised: (logits

Original: [Table]
Revised: [Table]

Original: LinBP
Revised: LinBP

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: “CE (softmax)”, and linear loss on logits “Lin
Revised: “CE (softmax)”, and linear loss on logits “Lin

Original: [Figure]
Revised: [Figure]

Original: where ˆ G 1 is the signed gradients where the loss is estimated with one sample as defined in Algorithm 1.
Revised: where ˆ G 1 is the signed gradients where the loss is estimated with one sample as defined in Algorithm 1.

Original: The cosine similarity is computed between the mean gradient and all n samples and then averaged.
Revised: The cosine similarity is computed between the mean gradient and all n samples and then averaged.

Original: [Equation]
Revised: [Equation]

Original: Lastly, the sign matching percentage is sign match
Revised: Lastly, the sign matching percentage is sign match

Original: [Equation]
Revised: [Equation]

Original: Fig. 8 and Fig.
Revised: Fig. 8 and Fig.

Original: 9 plot the cosine similarly and the sign matching for varying loss functions and varying transfer attacks, respectively.
Revised: 9 plot the cosine similarly and the sign matching for varying loss functions and varying transfer attacks, respectively.

Original: Similarly to Fig. 4, better attacks result in less spread of the gradient samples which corresponds to higher cosine similarity and sign matching percentage.
Revised: Similarly to Fig. 4, better attacks result in less spread of the gradient samples which corresponds to higher cosine similarity and sign matching percentage.

Original: C Details on Bayesian Optimization
Revised: C Details on Bayesian Optimization

Original: One major challenge in implementing an RT defense is selecting the defense hyperparameters which include the K transformation types, the number of transformations to apply ( S ), and their parameters ( a and p ).
Revised: One major challenge in implementing an RT defense is selecting the defense hyperparameters which include the K transformation types, the number of transformations to apply ( S ), and their parameters ( a and p ).

Original: To improve the robustness of RT defense, we use Bayesian optimization (BO), a well-known black-box optimization technique, to fine-tune a and p (Snoek, Larochelle, and Adams 2012).
Revised: To improve the robustness of RT defense, we use Bayesian optimization (BO), a well-known black-box optimization technique, to fine-tune a and p (Snoek, Larochelle, and Adams 2012).

Original: In this case,
Revised: In this case,

Original: BO models the hyperparameter tuning as a Gaussian process where the objective function takes in a and p , trains a neural network as a backbone for an RT defense, and outputs adversarial accuracy under some pre-defined ℓ ∞ -budget ϵ as the metric used for optimization.
Revised: BO models the hyperparameter tuning as a Gaussian process where the objective function takes in a and p , trains a neural network as a backbone for an RT defense, and outputs adversarial accuracy under some pre-defined ℓ ∞ -budget ϵ as the metric used for optimization.

Original: Since BO quickly becomes ineffective as we increase the dimensions of the search space, we choose to tune either a or p , never both, for each of the K transformation types.
Revised: Since BO quickly becomes ineffective as we increase the dimensions of the search space, we choose to tune either a or p , never both, for each of the K transformation types.

Original: For transformations that have a tunable a , we fix p = 1 (e.g., noise injection, affine transform).
Revised: For transformations that have a tunable a , we fix p = 1 (e.g., noise injection, affine transform).

Original: For the transformations without an adjustable strength a , we only tune p (e.g., Laplacian filter, horizontal flip).
Revised: For the transformations without an adjustable strength a , we only tune p (e.g., Laplacian filter, horizontal flip).

Original: Additionally, because BO does not natively support categorical or integral variables, we experiment with different choices for K and S without the use of BO.
Revised: Additionally, because BO does not natively support categorical or integral variables, we experiment with different choices for K and S without the use of BO.

Original: Therefore, our BO problem must optimize over K (up to 33 ) variables, far more than are typically present when doing model hyperparamter tuning using BO.
Revised: Therefore, our BO problem must optimize over K (up to 33 ) variables, far more than are typically present when doing model hyperparamter tuning using BO.

Original: [Figure]
Revised: [Figure]

Original: Mathematically, the objective function ψ is defined as
Revised: Mathematically, the objective function ψ is defined as

Original: [Equation]
Revised: [Equation]

Original: where the input is K real numbers between 0 and 1 , and R ∞ ,ϵ denotes the adversarial accuracy or the accuracy on x adv as defined in Eqn. 1.
Revised: where the input is K real numbers between 0 and 1 , and R ∞ ,ϵ denotes the adversarial accuracy or the accuracy on x adv as defined in Eqn. 1.

Original: Since ψ is very expensive to evaluate as it involves training and testing a large neural network, we employ the following strategies to reduce the computation: (1) only a subset of the training and validation set is used, (2) the network is trained for fewer epochs with a cosine annealing learning rate schedule to speed up convergence (Loshchilov and Hutter 2017), and (3) the attack used for computing R ∞ ,ϵ is weaker but faster.
Revised: Since ψ is very expensive to evaluate as it involves training and testing a large neural network, we employ the following strategies to reduce the computation: (1) only a subset of the training and validation set is used, (2) the network is trained for fewer epochs with a cosine annealing learning rate schedule to speed up convergence (Loshchilov and Hutter 2017), and (3) the attack used for computing R ∞ ,ϵ is weaker but faster.

Original: Even with these speedups, one BO run still takes approximately two days to complete on two GPUs (Nvidia GeForce GTX 1080 Ti).
Revised: Even with these speedups, one BO run still takes approximately two days to complete on two GPUs (Nvidia GeForce GTX 1080 Ti).

Original: We also experimented with
Revised: We also experimented with

Original: [Figure]
Revised: [Figure]

Original: // Starting Bayesian optimization
Revised: // Starting Bayesian optimization

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: // Best adversarial accuracy
Revised: // Best adversarial accuracy

Original: [Equation]
Revised: [Equation]

Original: other sophisticated hyperparameter-tuning algorithms based on Gaussian processes (Bergstra, Yamins, and Cox 2013;
Revised: other sophisticated hyperparameter-tuning algorithms based on Gaussian processes (Bergstra, Yamins, and Cox 2013;

Original: Kandasamy et al.
Revised: Kandasamy et al.

Original: Falkner, Klein, and Hutter 2018) but do not find them more effective.
Revised: Falkner, Klein, and Hutter 2018) but do not find them more effective.

Original: We summarize the main steps for tuning and training an RT defense in Algorithm 2.
Revised: We summarize the main steps for tuning and training an RT defense in Algorithm 2.

Original: We use the Ray Tune library for RT’s hyperparameter tuning in Python (Liaw et al. 2018).
Revised: We use the Ray Tune library for RT’s hyperparameter tuning in Python (Liaw et al. 2018).

Original: The Bayesian optimization tool is implemented by Nogueira (2014), following analyses and instructions by Snoek, Larochelle, and Adams (2012) and Brochu, Cora, and de Freitas (2010).
Revised: The Bayesian optimization tool is implemented by Nogueira (2014), following analyses and instructions by Snoek, Larochelle, and Adams (2012) and Brochu, Cora, and de Freitas (2010).

Original: As mentioned in
Revised: As mentioned in

Original: Section 5, we sub-sample the data to reduce computation for each BO trial.
Revised: Section 5, we sub-sample the data to reduce computation for each BO trial.

Original: Specifically, we use 20% and 10% of the training samples for Imagenette and CIFAR-10 respectively (Algorithm 2, line 1) as Imagenette has a much smaller number of samples in total.
Revised: Specifically, we use 20% and 10% of the training samples for Imagenette and CIFAR-10 respectively (Algorithm 2, line 1) as Imagenette has a much smaller number of samples in total.

Original: The models are trained with the same transformations and hyperparameters used during inference, and here, n is set to 1 during training, just as is done during standard data augmentation.
Revised: The models are trained with the same transformations and hyperparameters used during inference, and here, n is set to 1 during training, just as is done during standard data augmentation.

Original: We use 200 samples to evaluate each BO run in line 7 of Algorithm 2 with only 100 steps and n = 10 .
Revised: We use 200 samples to evaluate each BO run in line 7 of Algorithm 2 with only 100 steps and n = 10 .

Original: One BO experiment executes two BO’s in parallel.
Revised: One BO experiment executes two BO’s in parallel.

Original: The maximum number of BO runs is 160, but we terminate the experiment if no improvement has been made in the last 40 runs after a minimum of 80 runs have taken place.
Revised: The maximum number of BO runs is 160, but we terminate the experiment if no improvement has been made in the last 40 runs after a minimum of 80 runs have taken place.

Original: The runtime depends on S and the transformation types used.
Revised: The runtime depends on S and the transformation types used.

Original: In our typical case, when all 33 transformation types are used and S = 14 , one BO run takes almost an hour on an Nvidia
Revised: In our typical case, when all 33 transformation types are used and S = 14 , one BO run takes almost an hour on an Nvidia

Original: [Figure]
Revised: [Figure]

Original: C l ea n A cc u r ac y (
Revised: C l ea n A cc u r ac y (

Original: R )
Revised: R )

Original: [Figure]
Revised: [Figure]

Original: GeForce GTX 1080 Ti for Imagenette.
Revised: GeForce GTX 1080 Ti for Imagenette.

Original: One BO experiment then takes about two days to finish.
Revised: One BO experiment then takes about two days to finish.

Original: In line 13 and 14 of Algorithm 2, we now use the full training set and 1000 test samples as mentioned earlier.
Revised: In line 13 and 14 of Algorithm 2, we now use the full training set and 1000 test samples as mentioned earlier.

Original: During the full training, n is set to four which increases the training time by approximately four times.
Revised: During the full training, n is set to four which increases the training time by approximately four times.

Original: We find that using a larger n is beneficial to both the clean and the adversarial accuracy, but n larger than four does not make any significant difference.
Revised: We find that using a larger n is beneficial to both the clean and the adversarial accuracy, but n larger than four does not make any significant difference.

Original: C. Details on the Final RT Model
Revised: C. Details on the Final RT Model

Original: We run multiple BO experiments (Algorithm 2) on different subsets of transformation types to identify which transformations are most/least effective in order to reduce K as well as the number of hyperparameters our final run of BO has to tune.
Revised: We run multiple BO experiments (Algorithm 2) on different subsets of transformation types to identify which transformations are most/least effective in order to reduce K as well as the number of hyperparameters our final run of BO has to tune.

Original: We then repeat Algorithm 2 initialized with the inputoutput pairs from the prior runs of BO to obtain a new set of hyperparameters.
Revised: We then repeat Algorithm 2 initialized with the inputoutput pairs from the prior runs of BO to obtain a new set of hyperparameters.

Original: Finally, we remove the transformations whose p or a has been set to zero by the first run of BO, and we run BO once more with this filtered subset of transformations.
Revised: Finally, we remove the transformations whose p or a has been set to zero by the first run of BO, and we run BO once more with this filtered subset of transformations.

Original: At the end of this expensive procedure, we obtain the best and final RT model that we use in the experiments throughout this paper.
Revised: At the end of this expensive procedure, we obtain the best and final RT model that we use in the experiments throughout this paper.

Original: For Imagenette, the final set oftransformation types used in this model are color jitter, erase, gamma, affine, horizontal flip, vertical flip, Laplacian filter,
Revised: For Imagenette, the final set oftransformation types used in this model are color jitter, erase, gamma, affine, horizontal flip, vertical flip, Laplacian filter,

Original: Sobel filter, Gaussian blur, median blur, motion blur, Poisson noise, FFT, JPEG compression, color precision reduction, salt noise, sharpen, and solarize.
Revised: Sobel filter, Gaussian blur, median blur, motion blur, Poisson noise, FFT, JPEG compression, color precision reduction, salt noise, sharpen, and solarize.

Original: S is set to 14.
Revised: S is set to 14.

Original: D Additional Experiments on the RT Model D. Decision Rules and Number of Samples
Revised: D Additional Experiments on the RT Model D. Decision Rules and Number of Samples

Original: Fig.
Revised: Fig.

Original: 11 and Fig.
Revised: 11 and Fig.

Original: 12 compare three different decision rules that aggregate the n outputs of the RT model to produce the final prediction ˆ y ( x ) given an input x .
Revised: 12 compare three different decision rules that aggregate the n outputs of the RT model to produce the final prediction ˆ y ( x ) given an input x .

Original: We choose the average softmax probability rule for all of our RT models because it provides a good trade-off between the clean accuracy and the
Revised: We choose the average softmax probability rule for all of our RT models because it provides a good trade-off between the clean accuracy and the

Original: Number of Monte Carlo Samples ( n )
Revised: Number of Monte Carlo Samples ( n )

Original: A dv e r s a r i
Revised: A dv e r s a r i

Original: a l A cc u r ac
Revised: a l A cc u r ac

Original: y
Revised: y

Original: R ∞ , 16 /)
Revised: R ∞ , 16 /)

Original: Majority vote
Revised: Majority vote

Original: Avg.
Revised: Avg.

Original: softmax probs
Revised: softmax probs

Original: Avg.
Revised: Avg.

Original: logits
Revised: logits

Original: [Table]
Revised: [Table]

Original: robustness.
Revised: robustness.

Original: Majority vote has poor clean accuracy, and the average logits have poor robustness.
Revised: Majority vote has poor clean accuracy, and the average logits have poor robustness.

Original: D. Importance of the Transformation Groups
Revised: D. Importance of the Transformation Groups

Original: Choosing the best set of transformation types to use is a computationally expensive problem.
Revised: Choosing the best set of transformation types to use is a computationally expensive problem.

Original: There are many more transformations that can be applied outside of the 33 types we choose, and the number of possible combinations grows exponentially.
Revised: There are many more transformations that can be applied outside of the 33 types we choose, and the number of possible combinations grows exponentially.

Original: BO gives us an approximate solution but is by no means perfect.
Revised: BO gives us an approximate solution but is by no means perfect.

Original: Here, we take a step further to understand the importance of each transformation group.
Revised: Here, we take a step further to understand the importance of each transformation group.

Original: Tablegives an alternative way to gauge the contribution of each transformation group.
Revised: Tablegives an alternative way to gauge the contribution of each transformation group.

Original: According to this experiment, noise injection appears most robust followed by lossy compression and geometric transformations.
Revised: According to this experiment, noise injection appears most robust followed by lossy compression and geometric transformations.

Original: However, this result is not very informative as most of the groups have zero adversarial accuracy, and the rest are likely to also reduce to zero given more attack steps.
Revised: However, this result is not very informative as most of the groups have zero adversarial accuracy, and the rest are likely to also reduce to zero given more attack steps.

Original: This result also surprisingly follows the commonly observed robustness-accuracy trade-off (Tsipras et al. 2019).
Revised: This result also surprisingly follows the commonly observed robustness-accuracy trade-off (Tsipras et al. 2019).

Original: D. Number of Transformations
Revised: D. Number of Transformations

Original: We test the effect of the transform permutation size S on the clean and the robust accuracy of RT models (Fig. 13).
Revised: We test the effect of the transform permutation size S on the clean and the robust accuracy of RT models (Fig. 13).

Original: We run Bayesian optimization experiments for different values of S using all 33 transformation types, and all of the models are trained using the same procedure.
Revised: We run Bayesian optimization experiments for different values of S using all 33 transformation types and all of the models are trained using the same procedure.

Original: Fig.
Revised: Fig.

Original: 13 shows that generally more transformations (larger S ) increase robustness but lower accuracy on benign samples. 
Revised: 13 shows that generally more transformations (larger S ) increase robustness but lower accuracy on benign samples. Acknowledgements

Revised: The first author of this paper was supported by the Hewlett

Revised: Foundation through the Center for Long-Term Cybersecurity (CLTC), by the Berkeley Deep Drive project and by generous gifts from Open Philanthropy and

Revised: Google Cloud Research

Revised: Credits program (award GCP19980904)