Original: G ENERATING S EMANTIC A DVERSARIAL E XAMPLES WITH D IFFERENTIABLE R ENDERING
Revised: G ENERATING S EMANTIC A DVERSARIAL E XAMPLES WITH D IFFERENTIABLE R ENDERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine learning (ML) algorithms, especially deep neural networks, have demonstrated success in several domains.
Revised: Machine learning (ML) algorithms, especially deep neural networks, have demonstrated success in several domains.

Original: However, several types of attacks have raised concerns about deploying ML in safety-critical domains, such as autonomous driving and security.
Revised: However, several types of attacks have raised concerns about deploying ML in safety-critical domains, such as autonomous driving and security.

Original: An attacker perturbs a data point slightly in the pixel space  and causes the ML algorithm to misclassify (e.g. a perturbed stop sign is classiﬁed as a yield sign).
Revised: An attacker perturbs a data point slightly in the concrete feature space (e.g., pixel space) and causes the ML algorithm to produce incorrect output ( a perturbed stop sign is classiﬁed as a yield sign).

Original: These perturbed data points are called adversarial examples, and there are numerous algorithms in the literature for constructing adversarial examples and defending against them.
Revised: These perturbed data points are called adversarial examples, and there are numerous algorithms in the literature for constructing adversarial examples and defending against them.

Original: In this paper we explore semantic adversarial examples (SAEs) where an attacker creates perturbations in the semantic space.
Revised: In this paper we explore semantic adversarial examples (SAEs) where an attacker creates perturbations in the semantic space representing the environment that produces input for the ML model.

Original: For example, an attacker can change the background of the image to be cloudier to cause misclassiﬁcation.
Revised: For example, an attacker can change the background of the image to be cloudier to cause misclassiﬁcation.

Original: We present an algorithm for constructing SAEs that uses recent advances in differential rendering and inverse graphics.
Revised: We present an algorithm for constructing SAEs that uses recent advances in differential rendering and inverse graphics.

Revised: The generated SAEs are easily realizable in the physical world, and can be used to help the model adapt better to different domains.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning (ML) techniques, especially Deep Neural Networks (DNNs), have been successful in several domains, such as ﬁnance and healthcare.
Revised: Machine learning (ML) techniques, especially Deep Neural Networks (DNNs), have been successful in several domains, such as ﬁnance and healthcare.

Original: However, several test-time (Biggio et al., 2013;
Revised: However, several test-time (Biggio et al., 2013;

Original: Szegedy et al., 2014; Goodfellow et al., 2015; Kurakin et al., 2016) and training-time (Jagielski et al., 2018; Shafahi et al., 2018)
Revised: Szegedy et al., 2014; Goodfellow et al., 2015; Kurakin et al., 2016) and training-time (Jagielski et al., 2018; Shafahi et al., 2018)

Original: attacks have made their adoption in safety-critical applications, such as autonomous driving and security, problematic.
Revised: attacks have made their adoption in high-assurance applications, such as autonomous driving and security, problematic.

Original: ML techniques, such as generative models, have also been used for nefarious purposes such as generating “deepfakes” (Liu et al., 2017; Zhu et al., 2017).
Revised: ML techniques, such as generative models, have also been used for nefarious purposes such as generating “deepfakes” (Liu et al., 2017; Zhu et al., 2017).

Original: Our focus in this paper is on test-time attacks in which an adversary generates a slightly perturbed sample to fool a classiﬁer or an object-detector.
Revised: Our focus in this paper is on test-time attacks in which an adversary generates a slightly perturbed sample to fool a classiﬁer or an object-detector.

Original: Let X be the sample space and Y be the space of labels.
Revised: Let X be the sample space and Y be the space of labels.

Original: A classiﬁer F is a function from X to
Revised: A classiﬁer F is a function from X to

Original: Y .
Revised: Y .

Original: Given a sample x ∈ X , most attacks for constructing adversarial examples ﬁnd a perturbation δ with a small norm (typical norms that are used are l ∞ , l 0 , and l 2 ) such that x + δ has a different label than x , i.e. F ( x ) (cid:54) =
Revised: Given a sample x ∈ X , most attacks for constructing adversarial examples ﬁnd a perturbation δ with a small norm (typical norms that are used are l ∞ , l 0 , and l 2 ) such that x + δ has a different label than x , i.e. F ( x ) (cid:54) =

Original: F ( x + δ ) .
Revised: F ( x + δ ) .

Original: In this paper we consider the problem of generating semantic adversarial examples (SAEs) (Hosseini & Poovendran, 2018; Joshi et al., 2019; Qiu et al., 2019).
Revised: In this paper we consider the problem of generating semantic adversarial examples (SAEs) (Hosseini & Poovendran, 2018; Joshi et al., 2019; Qiu et al., 2019; Dreossi et al., 2018b).

Original: In these examples, there is a richer set of transformations T .
Revised: In these examples, there is a richer set of transformations T that capture semantically-meaningful changes to inputs to the ML model.

Original: We assume a norm on T (this norm is induced by various parameters corresponding to the transformations, such as angle of rotation and size of the translation).
Revised: We assume a norm on T (this norm is induced by various parameters corresponding to the transformations, such as angle of rotation and size of the translation).

Original: In our universe, an adversary is given a sample x and wishes to ﬁnd a transformation parameterized by θ ∈ Θ with small norm such that F ( τ ( x , θ ))
Revised: In our universe, an adversary is given a sample x and wishes to ﬁnd a transformation parameterized by θ ∈ Θ with small norm such that F ( τ ( x , θ ))

Original: (cid:54) =
Revised: (cid:54) =

Original: F ( x ) (we consider untargeted attacks, but our ideas extend to targeted attacks as well).
Revised: F ( x ) (we consider untargeted attacks, but our ideas extend to targeted attacks as well) 1 .

Original: SAEs can also be viewed as outcomes of perturbations in a “rich” semantic  space ( e.g. , texture of the image) rather than just the concrete (pixel) space.
Revised: SAEs can also be viewed as outcomes of perturbations in a “rich” semantic feature space ( e.g., , texture of the image) rather than just the concrete feature space ( e.g., , pixels).

Original: Consequently, SAEs are physically realizable, and it is easy to understand how the changes in semantics results in an adversarial example.
Revised: Consequently, SAEs are physically realizable, and it is easy to understand how the changes in semantics results in an adversarial example.

Original: SAEs have been considered in the literature (Xiao et al., 2018; Dreossi et al., 2018b; Huang et al., 2019), but prior works typically consider a small set of ﬁxed transformations ( e.g. rotation and translation, or modifying a single object’s texture).
Revised: SAEs have been considered in the literature (Xiao et al., 2018; Dreossi et al., 2018b; Huang et al., 2019), but prior works typically consider a small set of ﬁxed transformations ( e.g., rotation and translation, or modifying a single object’s texture).

Original: Our goal is to  support a richer set of transformations  ( e.g. making the background of the image cloudy orchanging the time of day from morning to evening).
Revised: Our goal is to ﬂexibly support a richer set of transformations implemented in a state of the art renderer ( e.g., changing the background of the image, weather conditions, or the time of day), as we specify in detail in § 2.

Original: There is evidence that SAEs can help with domain adaptation (Volpi et al., 2018) or making the control loop more robust (Dreossi et al., 2018b), further motivating our approach. 
Revised: There is evidence that SAEs can help with domain adaptation (Volpi et al., 2018) or making the control loop more robust (Dreossi et al., 2018b), further motivating our approach. We also plan to open source our code to advance research in this space.

Original: To summarize, the main contributions of this paper are the following:
Revised: To summarize, the main contributions of this paper are the following:

Original: • We present a new class of test-time attacks in the form of SAEs.
Revised: • We present a new class of test-time attacks in the form of SAEs.

Original: We demonstrate how to generate SAEs that support a rich set of transformations (refer § 3) using an inverse graphics framework (refer § 2).
Revised: We demonstrate how to generate SAEs that support a rich set of transformations (refer § 3) using an inverse graphics framework (refer § 2).

Original: Speciﬁcally, we show how one can systematically take techniques to perform attacks in the pixel space such as FGSM (Goodfellow et al., 2015) and PGD (Madry et al., 2017) and transform them to their semantic counterparts.
Revised: Speciﬁcally, we show how one can systematically take techniques to perform attacks in the pixel space such as FGSM (Goodfellow et al., 2015) and PGD (Madry et al., 2017) and transform them to their semantic counterparts.

Original: • We evaluate the generated SAEs on the popular object detector SqueezeDet (Wu et al., 2016).
Revised: • We evaluate the generated SAEs on the popular object detector SqueezeDet (Wu et al., 2016).

Original: By correctly choosing the semantic parameters, SAEs degrade performance (characterized by the mean average precision or mAP) by 28 percentage points (refer § 5.1).
Revised: By correctly choosing the semantic parameters, SAEs degrade performance (characterized by the mean average precision or mAP) by 28 percentage points (refer § 5.1).

Original: • Wealso show that by augmenting the dataset using SAEs, we can boost the robustness of SqueezeDet (characterized by mAP) by up to 15 percentage points (refer § 5.2).
Revised: • We show that by augmenting the dataset using SAEs, we can boost the robustness of SqueezeDet (characterized by mAP) by up to 15 percentage points (refer § 5.2).

Revised: • We show that SAEs generated using SqueezeDet do not transfer to YOLOv3 (Redmon & Farhadi, 2018), indicating a tight coupling between the model architecture and SAE generation (refer § 5.3).

Original:  While augmentation with SAEs improves robustness against SAEs, augmentation using traditional pixel-based perturbations does not produce the same effect (refer § 5.3).
Revised: • While augmentation with SAEs improves robustness against SAEs, augmentation using traditional pixel-based perturbations does not produce the same effect (refer § A.1).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Adversarial Examples and Robustness: There is extensive research for generating adversarial examples in the pixel space; we henceforth refer to these as pixel-perturbations .
Revised: Adversarial Examples and Robustness: There is extensive research for generating adversarial examples in the pixel space; we henceforth refer to these as pixel-perturbations .

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2015) propose the fast gradient sign method (FGSM) where inputs are modiﬁed in the direction of the gradients of the loss function with respect to input, causing a variety of models to misclassify their inputs.
Revised: (2015) propose the fast gradient sign method (FGSM) where inputs are modiﬁed in the direction of the gradients of the loss function with respect to input, causing a variety of models to misclassify their inputs.

Original: Madry et al.
Revised: Madry et al.

Original: (2017) generalize this approach and propose the projected gradient descent (PGD) approach working using the same intuition.
Revised: (2017) generalize this approach and propose the projected gradient descent (PGD) approach working using the same intuition.

Original: While these approaches suggest modiﬁcations to the raw pixel values, other methods of generating adversarial examples exist.
Revised: While these approaches suggest modiﬁcations to the raw pixel values, other methods of generating adversarial examples exist.

Original: Athalye et al.
Revised: Athalye et al.

Original: (2017) introduce an approach to generate 3D adversarial examples (over a chosen distribution of transformations).
Revised: (2017) introduce an approach to generate 3D adversarial examples (over a chosen distribution of transformations).

Original: Engstrom et al.
Revised: Engstrom et al.

Original: (2019) observe that modifying the spatial orientation of images results in misclassiﬁcations.
Revised: (2019) observe that modifying the spatial orientation of images results in misclassiﬁcations.

Original: Similarly, Geirhos et al.
Revised: Similarly, Geirhos et al.

Original: (2018) discovered that certain models are biased towards textural cues.
Revised: (2018) discovered that certain models are biased towards textural cues.

Original: To improve robustness, current approaches include adversarial training (Madry et al., 2017), smoothing-based approaches (Cohen et al., 2019; L ´ ecuyer et al., 2018), or through speciﬁc regularization (Raghunathan et al., 2018).
Revised: To improve robustness, current approaches include adversarial training (Madry et al., 2017), smoothing-based approaches (Cohen et al., 2019; L ´ ecuyer et al., 2018), or through speciﬁc regularization (Raghunathan et al., 2018).

Original: An alternative approach, utilizing some notion of semantics, is advocated in the work of Guo et al.
Revised: An alternative approach, utilizing some notion of semantics, is advocated in the work of Guo et al.

Original: The authors augment the training set with transformed versions of training images, utilizing basic image transformations ( e.g. , scale and re-cropping) and total variance minimization, and demonstrate an improvement in robustness.
Revised: The authors augment the training set with transformed versions of training images, utilizing basic image transformations ( e.g., , scale and re-cropping) and total variance minimization, and demonstrate an improvement in robustness.

Original: Dreossi et al.
Revised: Dreossi et al.

Original: (2018a) improve the robustness of SqueezeDet (Wu et al., 2016) through counterexample guided data augmentation; these counterexamples are synthetically generated by sampling from a space of transformations and applying them to original training images.
Revised: (2018a) improve the robustness of SqueezeDet (Wu et al., 2016) through counterexample guided data augmentation; these counterexamples are synthetically generated by sampling from a space of transformations and applying them to original training images.

Original: Inverse Graphics:
Revised: Inverse Graphics:

Original: The process of ﬁnding 3D scene parameters (geometric, textural, lighting, etc.)
Revised: The process of ﬁnding 3D scene parameters (geometric, textural, lighting, etc.)

Original: given images is referred to as inverse graphics (Baumgart, 1974).
Revised: given images is referred to as inverse graphics (Baumgart, 1974).

Original: There is a history of using gradients to solve this problem (Blanz & Vetter, 2002; Shacked & Lischinski, 2001; Barron & Malik, 2015).
Revised: There is a history of using gradients to solve this problem (Blanz & Vetter, 2002; Shacked & Lischinski, 2001; Barron & Malik, 2015).

Original: Kulkarni et al.
Revised: Kulkarni et al.

Original: (2015) propose a model that learns interpretable representations of images (similar to image semantics), and show how these interpretations can be modiﬁed to produce changes in the input space.
Revised: (2015) propose a model that learns interpretable representations of images (similar to image semantics), and show how these interpretations can be modiﬁed to produce changes in the input space.

Original: Pipelines for general differential rendering were proposed by Loper &
Revised: Pipelines for general differential rendering were proposed by Loper &

Original: Black (2014) and Kato et al.
Revised: Black (2014) and Kato et al.

Original: Li et al.
Revised: Li et al.

Original: (2018) design a general-purpose differentiable ray tracer; gradients can be computed with respect to arbitrary semantic parameters such as camera pose, scene geometry, materials, and lighting parameters.
Revised: (2018) design a general-purpose differentiable ray tracer; gradients can be computed with respect to arbitrary semantic parameters such as camera pose, scene geometry, materials, and lighting parameters.

Original: Yao et al.
Revised: Yao et al.

Original: (2018) propose a pipeline that, through de-rendering obtains various forms of semantics, geometry, texture, and appearance, which can be rendered using a generative model. 
Revised: (2018) propose a pipeline that, through de-rendering obtains various forms of semantics, geometry, texture, and appearance, which can be rendered using a generative model. Semantic Adversarial Examples (SAEs) :

Revised: The notion of SAEs has been proposed in the literature in speciﬁc contexts.

Revised: For example, in the NLP domain, Lei et al.

Revised: (2018) propose a gradient guided greedy algorithm to make semantic changes to text documents.

Revised: Sharif et al.

Original: S EMANTIC A DVERSARIAL L EARNING
Revised: (2016) propose attacks against facial recognition systems that are both inconspicuous and realizable. Similarly, Evtimov et al. (2017) obtain patches that can be added to road signs, rendering object detectors useless. In both these works, however, the modiﬁcations to semantics are not found by searching the space of semantic modiﬁcations (and found instead by realizing changes in pixel intensities); such approaches will fail to generalize for other domains.

Revised: Dreossi et al.

Revised: (2018b) present an approach to generate SAEs based on systematic sampling of the semantic space coupled with veriﬁcation techniques for a closed-loop cyber-physical system containing the ML model, where they treat the ML model as a black box.

Revised: However, this sampling-based “black box” approach faces scalability issues when dealing with high-dimensional spaces and also fails to exploit structure of the ML model to perform a targeted search for SAEs.

Revised: Xiao et al.

Revised: (2019) utilize a differentiable rendering framework to introduce changes in shape and texture that are capable of fooling a variety of ML models for various tasks.

Revised: However, our approach takes into account advances in differentiable inverse graphics permitting us to change a richer set of semantic parameters.

Revised: An interesting contribution by Qiu et al.

Revised: (2019) suggests how a generative model can be used to introduce semantic modiﬁcations.

Revised: However, generative models are notoriously difﬁcult to train and operate, making them unusuable for a wide variety of tasks.

Revised: Similar issues occur in the work by Joshi et al.

Original: Consider a space Z of the form X × Y , where X is the sample space and Y is the set of labels.
Revised: Consider a space Z of the form X × Y , where X is the sample space and Y is the set of labels.

Original: From here on we will assume that X = (cid:60) n .
Revised: From here on we will assume that X = (cid:60) n .

Original: Let H be a hypothesis space ( e.g. , weights of a DNN).
Revised: Let H be a hypothesis space ( e.g., , weights of a DNN).

Original: We assume a loss function (cid:96) : H × Z (cid:55)→ R
Revised: We assume a loss function (cid:96) : H × Z (cid:55)→ R

Original: so that given a hypothesis w ∈ H and a labeled data point ( x
Revised: so that given a hypothesis w ∈ H and a labeled data point ( x

Original: , y ) ∈ Z , the loss is (cid:96) ( w, x, y ) .
Revised: , y ) ∈ Z , the loss is (cid:96) ( w, x, y ) .

Original: The output of the learning algorithm is a classiﬁer , which is a function from (cid:60) n to Y .
Revised: The output of the learning algorithm is a classiﬁer , which is a function from (cid:60) n to Y .

Original: To emphasize that a classiﬁer depends on a hypothesis w ∈ H , which is output of the learning algorithm, we will denote it as F w (if w is clear from the context, we will sometimes simply write F ).
Revised: To emphasize that a classiﬁer depends on a hypothesis w ∈ H , which is output of the learning algorithm, we will denote it as F w (if w is clear from the context, we will sometimes simply write F ).

Original: 3. T RADITIONAL A DVERSARIAL E XAMPLES
Revised: 3. T RADITIONAL A DVERSARIAL E XAMPLES

Original: We will focus our discussion on untargeted attacks, but our discussion also applies to targeted attacks.
Revised: We will focus our discussion on untargeted attacks, but our discussion also applies to targeted attacks.

Original: An adversary A ’s goal is to take any input vector x ∈ (cid:60) n and produce a minimally altered version of x , an adversarial example denoted by A ( x ) , that has the property of being misclassiﬁed by a classiﬁer F : (cid:60) n → Y .
Revised: An adversary A ’s goal is to take any input vector x ∈ (cid:60) n and produce a minimally altered version of x , an adversarial example denoted by A ( x ) , that has the property of being misclassiﬁed by a classiﬁer F : (cid:60) n → Y .

Original: The adversary wishes to solve the following optimization problem:min δ ∈(cid:60) n µ ( δ ) such that
Revised: The adversary wishes to solve the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original:  The various terms in the formulation are: µ is a norm on (cid:60) n ; µ can be l ∞ , l 0 , l 1 , or l p ( p ≥ 2 ).
Revised: such that [Equation] The various terms in the formulation are: µ is a norm on (cid:60) n ; µ can be l ∞ , l 0 , l 1 , or l p ( p ≥ 2 ).

Revised: While not representative, these norms are commonly used as an approximation of a human’s visual perception (Sen et al., 2019).

Original: If δ is the solution of the optimization problem given above, then the adversarial example A ( x ) = x + δ .
Revised: If δ is the solution of the optimization problem given above, then the adversarial example A ( x ) = x + δ .

Original: FGSM.
Revised: FGSM.

Original: The fast gradient sign method (FGSM) (Goodfellow et al., 2015) was one of the ﬁrst untargeted attacks developed in literature.
Revised: The fast gradient sign method (FGSM) (Goodfellow et al., 2015) was one of the ﬁrst untargeted attacks developed in literature.

Original: The adversary crafts an adversarial example for a given legitimate sample x by computing (and then adding) the following perturbation:
Revised: The adversary crafts an adversarial example for a given legitimate sample x by computing (and then adding) the following perturbation:

Original: [Equation]
Revised: [Equation]

Original: The function L F ( x ) is a shorthand for (cid:96) ( w, x , l ( x )) , where w is the hypothesis corresponding to the classiﬁer F , x is the data point and l ( x ) is the true label of x (essentially we evaluate the loss function at the hypothesis corresponding to the classiﬁer).
Revised: The function L F ( x ) is a shorthand for (cid:96) ( w, x , l ( x )) , where w is the hypothesis corresponding to the classiﬁer F , x is the data point and l ( x ) is the true label of x (essentially we evaluate the loss function at the hypothesis corresponding to the classiﬁer).

Original: The gradient of the function L F is computed with respect to x using sample x and label y
Revised: The gradient of the function L F is computed with respect to x using sample x and label y

Original: = l ( x ) as inputs.
Revised: = l ( x ) as inputs.

Original: Note that ∇ x L F ( x ) is an n -dimensional vector and sign( ∇ L F ( x )) is a n -dimensional vector whose i th element is the sign of the ∇ x L F ( x )[ i ] .
Revised: Note that ∇ x L F ( x ) is an n -dimensional vector and sign( ∇ L F ( x )) is a n -dimensional vector whose i th element is the sign of the ∇ x L F ( x )[ i ] .

Original: The value of the input variation parameter ε factoring the sign matrix controls the perturbation’s amplitude.
Revised: The value of the input variation parameter ε factoring the sign matrix controls the perturbation’s amplitude.

Original: Increasing its value increases the likelihood of A ( x ) being misclassiﬁed by the classiﬁer F but also makes adversarial examples easier to “detect” by humans.
Revised: Increasing its value increases the likelihood of A ( x ) being misclassiﬁed by the classiﬁer F but also makes adversarial examples easier to “detect” by humans.

Original: The key idea is that FGSM takes a step in the direction of the gradient of the loss function with respect to the input , thus attempting to maximize the loss function using its ﬁrst-order approximation.
Revised: The key idea is that FGSM takes a step in the direction of the gradient of the loss function with respect to the input , thus attempting to maximize the loss function using its ﬁrst-order approximation.

Original: Recall that stochastic gradient descent (SGD) takes a step in the direction that is on expectation opposite to the gradient of the loss function because it is trying to minimize the loss function.
Revised: Recall that stochastic gradient descent (SGD) takes a step in the direction that is on expectation opposite to the gradient of the loss function because it is trying to minimize the loss function.

Original: PGD.
Revised: PGD.

Original: In Projected Gradient Descent (PGD) (Madry et al., 2017), we ﬁnd a perturbation in an iterative manner.
Revised: In Projected Gradient Descent (PGD) (Madry et al., 2017), we ﬁnd a perturbation in an iterative manner.

Original: The PGD attack can be thought of an iterative version of FGSM.
Revised: The PGD attack can be thought of an iterative version of FGSM.

Original: Assume that we are using the l p norm.
Revised: Assume that we are using the l p norm.

Original: Assume x 0 is the original sample x .
Revised: Assume x 0 is the original sample x .

Original: [Equation]
Revised: [Equation]

Original: The operator Π B p ( x ,ε ) ( y ) is the projection operator, i.e. it takes as input a point y and outputs the closest point in the ε -ball (using the l p -norm) around x .
Revised: The operator Π B p ( x ,ε ) ( y ) is the projection operator, i.e. it takes as input a point y and outputs the closest point in the ε -ball (using the l p -norm) around x .

Original: The iteration stops after a certain number of steps (the exact number of steps is a hyperparameter).
Revised: The iteration stops after a certain number of steps (the exact number of steps is a hyperparameter).

Original: 3. S EMANTIC A DVERSARIAL E XAMPLES (SAE S )
Revised: 3. S EMANTIC A DVERSARIAL E XAMPLES (SAE S )

Original: Let T : ( (cid:60) n × Θ) →
Revised: Let T : ( (cid:60) n × Θ) →

Original: (cid:60) n be a set of transformations parameterized by a space Θ , and µ is a norm over Θ .
Revised: (cid:60) n be a set of transformations parameterized by a space Θ , and µ is a norm over Θ .

Original: The reader can think of Θ as parameters that control the transformations ( e.g. the angle of rotation).
Revised: The reader can think of Θ as parameters that control the transformations ( e.g., the angle of rotation).

Original: Given θ ∈ Θ , τ ( x, θ ) is the image transformed according to the parameters θ .
Revised: Given θ ∈ Θ , τ ( x, θ ) is the image transformed according to the parameters θ .

Original: We assume that there is a special identity element in Θ (which we call ⊥ ) such that τ ( x , ⊥ ) =
Revised: We assume that there is a special identity element in Θ (which we call ⊥ ) such that τ ( x , ⊥ ) =

Original: x .
Revised: x .

Original: An adversarial attack in this universe is characterized as follows:
Revised: An adversarial attack in this universe is characterized as follows:

Original: [Equation]
Revised: [Equation]

Original: such that F ( τ ( x , θ ))
Revised: such that F ( τ ( x , θ ))

Original: (cid:54) =
Revised: (cid:54) =

Original: F ( x )
Revised: F ( x )

Original: In other words, we  want to ﬁnd a “small perturbation” in the parameter space Θ that will misclassify the sample.
Revised: In other words, we ideally want to ﬁnd a “small perturbation” in the parameter space Θ that will misclassify the sample 2 .

Original: Consider the function L F ( τ ( x , θ )) .
Revised: Consider the function L F ( τ ( x , θ )) .

Original: The derivative with respect to θ is (cid:2) ∂τ∂θ (cid:3) (cid:62) (cid:12)(cid:12)(cid:12) x ∇ z L F ( z )
Revised: The derivative with respect to θ is (cid:2) ∂τ∂θ (cid:3) (cid:62) (cid:12)(cid:12)(cid:12) x ∇ z L F ( z )

Original: | z
Revised: | z

Original: = τ ( x ,θ ) (the notation (cid:2) ∂τ∂θ (cid:3) (cid:62) (cid:12)(cid:12)(cid:12) x is the transposed Jacobian matrix of τ as a vector-valued function of θ , evaluated at x , and ∇ z L F ( z ) |
Revised: = τ ( x ,θ ) (the notation (cid:2) ∂τ∂θ (cid:3) (cid:62) (cid:12)(cid:12)(cid:12) x is the transposed Jacobian matrix of τ as a vector-valued function of θ , evaluated at x , and ∇ z L F ( z ) |

Original: z = τ ( x ,θ ) is the derivative evaluated at τ ( x , θ ) ).
Revised: z = τ ( x ,θ ) is the derivative evaluated at τ ( x , θ ) ).

Original: The semantic version of FGSM ( sFGSM ) will produce the following θ :
Revised: The semantic version of FGSM ( sFGSM ) will produce the following θ :

Original: [Equation]
Revised: [Equation]

Original: The adversarial example A ( x ) is τ ( x , θ (cid:63) ) .
Revised: The adversarial example A ( x ) is τ ( x , θ (cid:63) ) .

Original: Note that we do not assume any special properties about τ , such as linearity.
Revised: Note that we do not assume any special properties about τ , such as linearity.

Original: We only assume that τ is differentiable.
Revised: We only assume that τ is differentiable.

Original: In a similar manner a semantic version of the PGD attack ( sPGD ) can be constructed.
Revised: In a similar manner a semantic version of the PGD attack ( sPGD ) can be constructed.

Original: Let θ 0 = ⊥ and x 0 = x .
Revised: Let θ 0 = ⊥ and x 0 = x .

Original: The update steps correspond to the following two equations:
Revised: The update steps correspond to the following two equations:

Original: [Equation]
Revised: [Equation]

Original: Note that Π B µ ( · , · ) is the projection operator in the parameter space Θ .
Revised: Note that Π B µ ( · , · ) is the projection operator in the parameter space Θ .

Original: We also assume that the projection operator will keep the parameters in the feasible set, which depends on the image ( e.g. translation does not take the car off the road).
Revised: We also assume that the projection operator will keep the parameters in the feasible set, which depends on the image ( e.g., translation does not take the car off the road).

Original: The operator ⊕ is the aggregation operator (similar to addition in (cid:60) n ), but in the parameter space Θ .
Revised: The operator ⊕ is the aggregation operator (similar to addition in (cid:60) n ), but in the parameter space Θ .

Original: The precise axioms satisﬁed by ⊕ depends on Θ , but one axiom we require is:
Revised: The precise axioms satisﬁed by ⊕ depends on Θ , but one axiom we require is:

Original: [Equation]
Revised: [Equation]

Original: In fact, our recipe can be used to transform any attack algorithm such as Carlini & Wagner (2017) that adds a perturbation δ to its “semantic version” as follows:
Revised: In fact, our recipe can be used to transform any attack algorithm such as Carlini & Wagner (2017) that adds a perturbation δ to its “semantic version” as follows:

Original: • Replace δ with θ .
Revised: • Replace δ with θ .

Original: • Replace x + δ with τ ( x , θ ) .
Revised: • Replace x + δ with τ ( x , θ ) .

Original: • Use chain rule to compute the gradients of terms that involve τ ( x , θ 
Revised: • Use chain rule to compute the gradients of terms that involve τ ( x , θ ) .

Original: Differentiable rendering and inverse graphics.
Revised: Differentiable rendering and inverse graphics.

Original: We apply the above framework to images by employing a differentiable renderer/de-renderer in an inverse graphics setting.
Revised: We apply the above framework to images by employing a differentiable renderer/de-renderer in an inverse graphics setting.

Original: Such an inverse graphics setting can be thought of two transformations: (a) a de-renderer β : (cid:60) n → S , and (b) a renderer γ : S → (cid:60) n .
Revised: Such an inverse graphics setting can be thought of two transformations: (a) a de-renderer β : (cid:60) n → S , and (b) a renderer γ : S → (cid:60) n .

Original: Here, S is the intermediate representation (IR).
Revised: Here, S is the intermediate representation (IR).

Original: In the differentiable renderer/derenderer we utilize (Yao et al., 2018), the IR contains a semantic map, texture codes, and 3D attributes.
Revised: In the differentiable renderer/derenderer we utilize (Yao et al., 2018), the IR contains a semantic map, texture codes, and 3D attributes.

Original: Let Θ be the set of changes to the IR ( e.g. change to the texture code to make it more cloudy) and ⊥∈ Θ corresponds to the identity.
Revised: Let Θ be the set of changes to the IR ( e.g., change to the texture code to make it more cloudy) and ⊥∈ Θ corresponds to the identity.

Original: Suppose there is an operator α : ( S × Θ) → S that given a θ ∈ Θ transforms the IR, i.e. α ( s, θ )
Revised: Suppose there is an operator α : ( S × Θ) → S that given a θ ∈ Θ transforms the IR, i.e. α ( s, θ )

Original: =
Revised: =

Original: s (cid:48) for s, s (cid:48) ∈ S .
Revised: s (cid:48) for s, s (cid:48) ∈ S .

Original: In this case, the function τ ( x , θ ) is equal to γ ( α ( β ( x ) , θ )) .
Revised: In this case, the function τ ( x , θ ) is equal to γ ( α ( β ( x ) , θ )) .

Original: We use the fact that for differentiable renderers/de-renderers the functions β , γ , α are differentiable and hence attacks like sFGSM and sPGD can be implemented.
Revised: We use the fact that for differentiable renderers/de-renderers the functions β , γ , α are differentiable and hence attacks like sFGSM and sPGD can be implemented.

Original:  V ALIDATION
Revised:  V ALIDATION

Original: In this section, we describe the various components used in our implementation to generate SAEs, and describe experiments carried out to determine the impact of choice of semantic parameters towards generating effective SAEs.
Revised: In this section, we describe the various components used in our implementation to generate SAEs, and describe experiments carried out to determine the impact of choice of semantic parameters towards generating effective SAEs for object detectors.

Revised: We stress that the exact choice of components are not important; our contribution is our automated approach to combine these components to generate SAES in a manner that is general (see § 3.2 for more details) and would work with any inverse graphics framework.

Revised: We also believe that our methodology will work for other tasks such as classiﬁcation and semantic segmentation; these tasks require optimizing over a different cost function.

Revised: In other words, if there is an existing pixel attack, we should be able to easily transform it into a semantic attack (with the right inverse graphics framework).

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: The three main components required to successfully generate SAEs include: (a) a differentiable inverse graphics framework, (b) a victim model (which is also differentiable), and (c) an attack strategy.
Revised: The three main components required to successfully generate SAEs include: (a) a differentiable inverse graphics framework, (b) a victim model (which is also differentiable), and (c) an attack strategy.

Original: We describe each of these below.
Revised: We describe our choices for the proof-of-concept below.

Original: To obtain the semantics associated with our inputs and to generate the ﬁnal SAEs, we use the inverse graphics ( i.e. a combination of a semantic, textural and geometric de-rendering pipeline and a generative model for rendering) created by Yao et al.
Revised: To obtain the semantics associated with our inputs and to generate the ﬁnal SAEs, we use the inverse graphics framework( i.e. a combination of a semantic, textural and geometric de-rendering pipeline and a generative model for rendering) created by Yao et al.

Original: The models in this framework were trained entirely using the VKITTI dataset (Gaidon et al., 2016).
Revised: The models in this framework were trained entirely using the VKITTI dataset (Gaidon et al., 2016).

Original: These images comprise of simulations of cars in different road environments in virtual worlds.
Revised: These images comprise of simulations of cars in different road environments in virtual worlds.

Revised: The VKITTI dataset is constructed using a novel real-to-virtual cloning methodology, mirroring many elements that exist in a real world.

Original: The de-rendering pipeline is used to obtain the initial semantic features associated with input images.
Revised: The de-rendering pipeline is used to obtain the initial semantic features associated with input images.

Original: These semantic features include (a) color : the car’s texture codes, which change its color, (b) weather : the weather and time of day, (c) foliage : the surrounding foliage and scenery, (d) rotate : the car’s orientation, (e) translate : the car’s position in 2D, and (f) mesh : the 3D mesh which provides structure to the car. 
Revised: These semantic features include (a) color : the car’s texture codes, which change its color, (b) weather : the weather and time of day, (c) foliage : the surrounding foliage and scenery, (d) rotate : the car’s orientation, (e) translate : the car’s position in 2D, and (f) mesh : the 3D mesh which provides structure to the car. Building an inverse graphics framework for other tasks/datasets is a non-trivial process which is outside the scope of this paper.

Original: The ﬁnal SAEs were produced using the generative model.
Revised: The ﬁnal SAEs were produced using the generative model (which is part of the inverse graphics framework).

Original: Speciﬁc modiﬁcations were made to the differentiable graphics framework we used to ensure that gradients were easy to calculate.
Revised: Speciﬁc modiﬁcations were made to the differentiable graphics framework we used to ensure that gradients were easy to calculate.

Original: The codebase did not originally support end-to-end differentiation as each branch (semantic, geometric, textural) was trained separately.
Revised: The codebase did not originally support end-to-end differentiation as each branch (semantic, geometric, textural) was trained separately.

Original: In particular, several image manipulation operations (normalization, rescaling through nearest-neighbor and bilinear interpolation) were implemented in a nondifferentiable manner.
Revised: In particular, several image manipulation operations (normalization, rescaling through nearest-neighbor and bilinear interpolation) were implemented in a non-differentiable manner.

Original: We implemented the differentiable equivalents of these operations to allow backpropagation.
Revised: We implemented the differentiable equivalents of these operations to allow backpropagation.

Original: Furthermore, we implemented a weak perspective projection for vehicle objects, as well as an improved heuristic for inpainting of gaps in the segmentation map due to object translations/rotations, in order to improve the quality of the rendering.
Revised: Furthermore, we implemented a weak perspective projection for vehicle objects, as well as an improved heuristic for inpainting of gaps in the segmentation map due to object translations/rotations, in order to improve the quality of the rendering.

Revised: Additionally, the inverse graphics framework and the object detector (which we describe next) had contrasting dependencies (for libraries, tensorflow , cuDNN, and CUDA); resolving these dependencies involved signiﬁcant code rewriting.

Original: We use the popular and representative SqueezeDet object detector (Wu et al., 2016) as the victim model.
Revised: We use the popular and representative SqueezeDet object detector (Wu et al., 2016) as the victim model.

Original: This model was originally trained on the KITTI dataset (Geiger et al., 2013).
Revised: This model was originally trained on the KITTI dataset (Geiger et al., 2013).

Original: We perform transfer learning on this model using 6339 randomly chosen images from the VKITTI dataset; we wanted the object detector to better adapt to images outside the domain it was initially trained for.
Revised: We perform transfer learning on this model using 6339 randomly chosen images from the VKITTI dataset; we wanted the object detector to better adapt to images outside the domain it was initially trained for.

Original: However, images produced by the differentiable graphics framework contain artifacts ( i.e. distortions in the images); these artifacts could be mistaken for pixel perturbations and would impact our evaluation results.
Revised: However, images produced by the differentiable graphics framework contain artifacts ( i.e. distortions in the images); these artifacts could be mistaken for pixel perturbations and would impact our evaluation results.

Original: To deal with this issue, we retrain SqueezeDet using identity transform re-rendered images 1 produced by the generative model.
Revised: To deal with this issue, we retrain SqueezeDet using identity transform rerendered images 3 produced by the generative model.

Original: Finally, we utilize these gradients and the semantics associated with each input in crafting adversarial attacks using the iterative sFGSM (for 6 iterations).
Revised: Finally, we utilize these gradients and the semantics associated with each input in crafting adversarial attacks using the iterative sFGSM (for 6 iterations).

Revised: On average, for a CPU-only implementation, generating such a SAE requires 42.

Revised: seconds with about 85% of the time expended in the inverse graphics framework.

Original: We  stress that our choice of the number of iterations is restricted by our choice of the differentiable graphics framework.
Revised: We also stress that our choice of the number of iterations is restricted by our choice of the differentiable graphics framework.

Original: Using more iterations resulted in unintelligble outputs.
Revised: Using more iterations resulted in unintelligible outputs.

Original: We also stress that the exact choice of three components are irrelevant; our constructions are general (refer § 3.2 for more details).
Revised: We believe that the generation of SAEs is not inhibited by the complexity of the semantic parameter space; better engineering can substantially improve the outputs and the time it takes to generate them.

Original: 4. S ELECTING S EMANTIC P ARAMETERS
Revised: 4. S ELECTING S EMANTIC P ARAMETERS

Original: In the pixel perturbation setting, all pixels are equal i.e.any pixel can be perturbed.
Revised: In the pixel perturbation setting, all pixels are equal i.e. any pixel can be perturbed.

Original: Whether such homogeneity naturally exists in the semantic space is unclear.
Revised: Whether such homogeneity naturally exists in the semantic space is unclear.

Original: However, we have additional ﬂexibility; we can choose to modify any of the above listed semantic parameters independently without altering the others, i.e. perform single parameter modiﬁcations .
Revised: However, we have additional ﬂexibility; we can choose to modify any of the above listed semantic parameters independently without altering the others, i.e. perform single parameter modiﬁcations .

Original: Alternatively, we can modify any subset of the parameters in unison, i.e. perform multi-parameter modiﬁcations .
Revised: Alternatively, we can modify any subset of the parameters in unison, i.e. perform multi-parameter modiﬁcations .

Original: The degree of modiﬁcation is determined by the input variation/step-size hyperparameter ε ∈ [0 , 1] .
Revised: The degree of modiﬁcation is determined by the input variation/step-size hyperparameter ε ∈ [0 , 1] .

Original: In the context of pixel perturbations, the step-size corresponds to the maximum permissible change of a pixel.
Revised: In the context of pixel perturbations, the step-size corresponds to the maximum permissible change of a pixel.

Original: For
Revised: For

Original: SAEs, the value of ε is proportional to the magnitude of the geometric and textural changes induced; the effect depends on the semantic parameter under consideration.
Revised: SAEs, the value of ε is proportional to the magnitude of the geometric and textural changes induced; the effect depends on the semantic parameter under consideration.

Original: Large values of ε result in unrealistic images created by the generative model (examples of this include perturbing the mesh to the point where cars are twisted into shapes no longer resembling vehicles).
Revised: Large values of ε result in unrealistic images created by the generative model (examples of this include perturbing the mesh to the point where cars are twisted into shapes no longer resembling vehicles).

Original: To avoid such issues and to simulate realistic transformations, we use a different step-size for each semantic parameter.
Revised: To avoid such issues and to simulate realistic transformations, we use a different step-size for each semantic parameter.

Original: We test various values of ε for each semantic parameter, and report the best choice for brevity.
Revised: We test various values of ε for each semantic parameter, and report the best choice for brevity.

Original: Speciﬁcally, (a) color : ε = 0.05, ( ) weather : ε = ) foliage  ε = 0.10, (d) rotate : ε = 0.01, (e) translate : ε = 0.01, and (f) mesh : ε = 0.025.
Revised: Speciﬁcally, (a) color : ε = 0.05, (b) weather : ε = 0.25, (c) foliage : ε = 0.10, (d) rotate : ε = 0.01, (e) translate : ε = 0.01, and (f) mesh : ε = 0.025.

Original: We stress these hyperparameters were obtained after extensive visual inspection (by 3 viewers independently); norm-based approaches typically serve as a proxy for visual veriﬁcation (Sen et al., 2019).
Revised: We stress these hyperparameters were obtained after extensive visual inspection (by 3 viewers independently); norm-based approaches typically serve as a proxy for visual veriﬁcation (Sen et al., 2019).

Original: Additionally, our choice in hyperparameters enables us to use the same ground truth labels throughout our experiments; e.g. produced SAEs have bounding box coordinates that enable us to use the same ground  labels as their 
Revised: Additionally, our choice in hyperparameters enables us to use the same ground truth labels throughout our experiments; e.g., produced SAEs have bounding box coordinates that enable us to use the same ground truth labels as their benign counterparts 4 .

Revised: We also measure the Fr´echet Inception Distance (FID) scores (Heusel et al., 2017) for the generated SAEs; we observe that the score is 0.102362 (compared to 0.01651 for generating pixel perturbations), providing quantitative evidence that the SAEs are similar to the benign images they were generated from.

Original: We produce 50 SAEs for each semantic parameter combination choice.
Revised: We produce 50 SAEs for each semantic parameter combination choice.

Original: We then evaluate the efﬁcacy of generated SAEs on SqueezeDet by measuring its (a) recall percentage, and (b) mean average precision, or mAP, in percentage.
Revised: We then evaluate the efﬁcacy of generated SAEs on SqueezeDet by measuring its (a) recall percentage, and (b) mean average precision, or mAP, in percentage.

Original: These metrics have been used in earlier works (Xie et al., 2017).
Revised: These metrics have been used in earlier works (Xie et al., 2017).

Original: [Table]
Revised: [Table]

Original: From Table 1, it is clear that single parameter modiﬁcation is ineffective at generating SAEs.
Revised: From Table 1, it is clear that single parameter modiﬁcation is ineffective at generating SAEs.

Original: Thus, we generate SAEs using the multi-parameter modiﬁcation method.
Revised: Thus, we generate SAEs using the multi-parameter modiﬁcation method.

Original: To this end, we generated SAEs using the 57 remaining combinations of semantic parameters.
Revised: To this end, we generated SAEs using the 57 remaining combinations of semantic parameters.

Original: One could consider a weighted combination of different semantic parameters based on a pre-deﬁned notion of precedence.
Revised: One could consider a weighted combination of different semantic parameters based on a pre-deﬁned notion of precedence.

Original: However, we choose a non-weighted combination.
Revised: However, we choose a non-weighted combination.

Original: The results of our experiments are in Table 2.
Revised: The results of our experiments are in Table 2.

Original: For brevity, we omit most of the combinations that do not result in signiﬁcant performance degradation (and discuss the insight we gained from them in § 5.1).
Revised: For brevity, we omit most of the combinations that do not result in signiﬁcant performance degradation (and discuss the insight we gained from them in § 5.1).

Original: In the remainder of the paper, we report our evaluation using the translate + rotate + mesh parameter combination to generate SAEs.
Revised: In the remainder of the paper, we report our evaluation using the translate + rotate + mesh parameter combination to generate SAEs.

Original: [Table]
Revised: [Table]

Original:  E VALUATION
Revised:  E VALUATION

Original: We designed and carried out experiments to answer the following questions: (1) Do SAEs cause performance degradation in SqueezeDet?, (2) Can the generated SAEs be used for improving robustness?, and (3) How does thedegradation (and robustness) caused by SAEs compare to that caused by pixel perturbations?
Revised: We designed and carried out experiments to answer the following questions: (1) Do SAEs cause performance degradation in SqueezeDet?, (2) Can the generated SAEs be used for improving robustness?, and (3) Do the SAEs (generated using SqueezeDet) transfer to YOLOv3 (Redmon & Farhadi, 2018)?

Original: We use 6339 images for training our SqueezeDet model, and evaluate the model using 882 SAEs.
Revised: We use 6339 images for training our SqueezeDet model, and evaluate the model using 882 SAEs.

Original: To evaluate the robustness, we augment the training dataset with 1547 SAEs and retrain the model.
Revised: To evaluate the robustness, we augment the training dataset with 1547 SAEs and retrain the model.

Original: The various components of our framework and the datasets used are highlighted in § 4.1.
Revised: The various components of our framework and the datasets used are highlighted in § 4.1.

Original: Note that
Revised: Note that

Original: SqueezeDet’s loss function comprises three terms corresponding to (a) bounding box regresson, (b) conﬁdence score regression, and (c) classiﬁcation loss.
Revised: SqueezeDet’s loss function comprises three terms corresponding to (a) bounding box regresson, (b) conﬁdence score regression, and (c) classiﬁcation loss.

Original: In our experiments, we target the conﬁdence score regression loss term to impact the mAP and recall of the model.
Revised: In our experiments, we target the conﬁdence score regression loss term to impact the mAP and recall of the model.

Original: All code was written in python .
Revised: All code was written in python .

Original: Our experiments were performed on two servers.
Revised: Our experiments were performed on two servers.

Original: The SAE generation was carried out on a server with an NVIDIA Titan GP102 GPU, 8 CPU cores, and 15GB memory.
Revised: The SAE generation was carried out on a server with an NVIDIA Titan GP102 GPU, 8 CPU cores, and 15GB memory.

Original: All training and evaluation was carried out on a server with 264 GB memory, using NVIDIA’s GeForce RTX
Revised: All training and evaluation was carried out on a server with 264 GB memory, using NVIDIA’s GeForce RTX

Original: GPUs and 48 CPU cores.
Revised: GPUs and 48 CPU cores.

Original: Our experiments suggest that: (1) SAEs are indeed effective in degrading the performance of SqueezeDet.
Revised: Our experiments suggest that: (1) SAEs are indeed effective in degrading the performance of SqueezeDet.

Original: We also observe that the model is susceptible to changes that target the geometry of the input (cars in this case) rather than the changes in the background (refer § 5.1), (2) The generated SAEs do, in fact, help in improving model robustness.
Revised: We also observe that the model is susceptible to changes that target the geometry of the input (cars in this case) rather than the changes in the background (refer § 5.1), (2) The generated SAEs do, in fact, help in improving model robustness.

Original: Our experiments show that
Revised: Our experiments show that

Original: SAE-based data augmentation can improve mAP by up to 15 percentage points (refer § 5.2), and (3)
Revised: SAE-based data augmentation can improve mAP by up to 15 percentage points (refer § 5.2), and (3)

Original: Pixel perturbation-based augmentation is ineffective against SAEs (refer § 5.3).
Revised: SAEs generated with SqueezeDet do not transfer to YOLOv3. (see § 5.3).

Original: We do not report other metrics (classiﬁcation accuracy, background error, etc.) associated with detection as our experiments are not designed to alter them.
Revised: We do not report other metrics (classiﬁcation accuracy, background error, etc.) associated with detection as our experiments are not designed to alter them.

Original: 5. E FFECTIVENESS OF SAE S
Revised: 5. E FFECTIVENESS OF SAE S

Original: The results in Table 2 in § 4.2 demonstrate the effectiveness of SAEs, and offer two insights.
Revised: The results in Table 2 in § 4.2 demonstrate the effectiveness of SAEs, and offer two insights.

Original: First, the victim model was more susceptible to transformations that modify the geometry of the input (such as translate and mesh ) than other types of transformations.
Revised: First, the victim model was more susceptible to transformations that modify the geometry of the input (such as translate and mesh ) than other types of transformations.

Original: This has dire implications for safety-critical applications; for the cars in our inputs, modiﬁcations in the mesh parameter results in deformed cars as outputs.
Revised: This has dire implications for safety-critical applications; for the cars in our inputs, modiﬁcations in the mesh parameter results in deformed cars as outputs.

Original: These are common occurrences in sites of accidents, and need to be detected correctly.
Revised: These are common occurrences in sites of accidents, and need to be detected correctly.

Original: A combination of translations and rotations also seem to compound the degradation to the performance of the network (refer Table 2).
Revised: A combination of translations and rotations also seem to compound the degradation to the performance of the network (refer Table 2).

Original: This is most likely due to the introduction of unique angles and visual perspectives that are not frequently encountered in assembled datasets.
Revised: This is most likely due to the introduction of unique angles and visual perspectives that are not frequently encountered in assembled datasets.

Original: Unlike pixel perturbations, SAEs are easy to interpret, i.e. we are able to understand how the model fails to generalize to speciﬁc changes in input semantics.
Revised: Unlike pixel perturbations, SAEs are easy to interpret, i.e. we are able to understand how the model fails to generalize to speciﬁc changes in input semantics.

Original: Additionally, they are easier to realize i.e. the situations described above (related to translation and deformation of vehicles) occur on a daily basis.
Revised: Additionally, they are easier to realize i.e. the situations described above (related to translation and deformation of vehicles) occur on a daily basis.

Original: Intuitively, changing the geometry of the car can be viewed as targeting the perception of what a car really is – if the human can recognize that the object in question is a car but a model cannot, then the model is not exposed to the sufﬁcient variety of car shapes, positions, and orientations that it may encounter in real-world scenarios; i.e. it is unable to domain adapt (Tzeng et al., 2017).
Revised: Intuitively, changing the geometry of the car can be viewed as targeting the perception of what a car really is – if the human can recognize that the object in question is a car but a model cannot, then the model is not exposed to the sufﬁcient variety of car shapes, positions, and orientations that it may encounter in real-world scenarios; i.e. it is unable to domain adapt (Tzeng et al., 2017).

Original: The second insight we gain is that the model was more susceptible to SAEs caused by changing multiple parameters simultaneously.
Revised: The second insight we gain is that the model was more susceptible to SAEs caused by changing multiple parameters simultaneously.

Original: We evaluate the model with 882 SAEs generated using a combination of the parameters listed in § 4.2.
Revised: We evaluate the model with 882 SAEs generated using a combination of the parameters listed in § 4.2.

Original: We observe that compared to the baseline performance on non-adversarial/benign inputs (recall = 93.63, mAP = 85.95), SAEs cause a signiﬁcant performance degradation (recall = 93.17, mAP = 57.78).
Revised: We observe that compared to the baseline performance on non-adversarial/benign inputs (recall = 93.63, mAP = 85.95), SAEs cause a signiﬁcant performance degradation (recall = 93.17, mAP = 57.78).

Original: As stated before, these combinations are easily realizable, and the model’s poor performance is indicative of poor domain adaptation.
Revised: As stated before, these combinations are easily realizable, and the model’s poor performance is indicative of poor domain adaptation.

Original: 5. D ATA A UGMENTATION T O I NCREASE R OBUSTNESS
Revised: 5. D ATA A UGMENTATION T O I NCREASE R OBUSTNESS

Original: As we have established that SAEs are effective in attacking SqueezeDet, we wished to enhance the model’s robustness through data augmentation, as in Dreossi et al.
Revised: As we have established that SAEs are effective in attacking SqueezeDet, we wished to enhance the model’s robustness through data augmentation, as in Dreossi et al.

Original: (2018b).
Revised: (2018b).

Original: To this end, we carried out two sets of experiments.
Revised: To this end, we carried out two sets of experiments.

Original: In the ﬁrst, we incrementally (re)trained the benign SqueezeDet model on a combination of benign inputs and SAEs (4792+1547) for 24000 iterations.
Revised: In the ﬁrst, we incrementally (re)trained the benign SqueezeDet model on a combination of benign inputs and SAEs (4792+1547) for 24000 iterations.

Original: In the second, we tuned our benign model using just SAEs (1547) for 6000 iterations.
Revised: In the second, we tuned our benign model using just SAEs (1547) for 6000 iterations.

Original: The results of our experiments are presented in Table 3.
Revised: The results of our experiments are presented in Table 3.

Original: [Table]
Revised: [Table]

Original: It is clear that both approaches provide comparable increase in mAP while not impacting recall.
Revised: It is clear that both approaches provide comparable increase in mAP while not impacting recall.

Original: Additionally, we found that making a model robust to semantic perturbations through either procedure described earlier allowed us to achieve good performance on benign inputs.
Revised: Additionally, we found that making a model robust to semantic perturbations through either procedure described earlier allowed us to achieve good performance on benign inputs.

Original: On benign inputs, we found that for the Retrained (SAE + Benign) model, recall = 93.7 and mAP = 84.73, while for Tuned (SAE), recall = 91.9 and mAP = 79.1.
Revised: On benign inputs, we found that for the Retrained (SAE + Benign) model, recall = 93.7 and mAP = 84.73, while for Tuned (SAE), recall = 91.9 and mAP = 79.1.

Original: This is comparable to the performance of the baseline model (which was trained and validated on benign inputs), where recall = 93.6 and mAP = 86.17.
Revised: This is comparable to the performance of the baseline model (which was trained and validated on benign inputs), where recall = 93.6 and mAP = 86.17.

Original: Our results suggests that SAE-based augmentation is a promising direction for exploration; based on insight from § 3, we could formulate a framework for semantic adversarial training, similar to (Madry et al., 2017).
Revised: Our results suggests that SAE-based augmentation is a promising direction for exploration; based on insight from § 3, we could formulate a framework for semantic adversarial training, similar to (Madry et al., 2017).

Original: We leave the exact formulation to future work.
Revised: We leave the exact formulation to future work.

Revised: 5. I NVESTIGATING T RANSFERABILITY

Revised: To understand if SAEs transfer, we tuned a YOLOv3 network that was initially trained with the

Original: In the next subsection, we will compare this approach to augmentation using pixel perturbations, and a combination of both approaches.
Revised: KITTI dataset, for 50 epochs. To do so, we use the same 6339 benign re-rendered images from the VKITTI dataset (as we did before), generated by the inverse graphics framework. The mAP of this network is 91.28%.

Revised: When we tested the YOLOv3 network with the generated SAEs, we observed a drop in mAP to 87.50% (which is not as signiﬁcant as in the case with SqueezeDet).

Revised: This suggests that SAEs are effective against the network they are generated from, suggesting a tight coupling between the two.

Revised: Similar observations have been made with pixel perturbations (Tram ` er et al., 2017); further analysis is required to understand this phenomenon, which we hope to pursue in future work.

Revised: While we investigate if SAEs transfer across different models and architectures, an interesting question revolves around the transferability of SAEs across tasks.

Revised: We believe that SAEs generated to fool object detectors will not work against classiﬁcation or semantic segmentation networks as the generation process was not targeted for these tasks.

Revised: That being said, it is trivial to modify our general construction (by changing the cost/loss function) and generate SAEs for tasks such as classiﬁcation and semantic segmentation.

Revised: We leave generating such SAEs for future work.

Revised:  C ONCLUSIONS

Revised: In this paper, we describe semantic adversarial examples (SAEs), where adversaries perturb the semantics of inputs to produce outputs that are misclassiﬁed.

Revised: Such instances are easier to realize in the physical world, and are more interpretable than their traditional pixel-based counterparts.

Revised: We propose an algorithm to construct SAEs using advances in differentiable rendering, and evaluate the effectiveness of our approach.

Revised: We observe that SAEs cause performance degradation in object detector networks (SqueezeDet), and that data augmentation using SAEs increases robustness of the model.

Original: 5. P IXEL P ERTURBATIONS VS . SAE S
Revised: A A PPENDIX A. P IXEL P ERTURBATIONS VS . SAE S

Original: Our experiments suggested that pixel perturbations are more effective in degrading SqueezeDet’s performance (recall=2.9, mAP=0.05); we conjecture this is due to the larger feature space within which a solution for the optimization can be found i.e. the space of pixels is larger than the space of speciﬁc semantic parameters/transformations we consider.
Revised: Our experiments suggested that pixel perturbations are more effective in degrading SqueezeDet’s performance (recall=2.9, mAP=0.05); we conjecture this is due to the larger feature space within which a solution for the optimization can be found i.e. the space of pixels is larger than the space of speciﬁc semantic parameters/transformations we consider.

Original: To measure the robustness provided by pixel perturbations against semantic perturbations, we performed the same experiment as in § 5.2.
Revised: To measure the robustness provided by pixel perturbations against semantic perturbations, we performed the same experiment as in § 5.2.

Original: In one experiment, we retrained the benign SqueezeDet model on a combination of benign inputs and pixel perturbations (4792+1547) for 24000 iterations.
Revised: In one experiment, we retrained the benign SqueezeDet model on a combination of benign inputs and pixel perturbations (4792+1547) for 24000 iterations.

Original: In another experiment, we tuned our model using just pixel perturbations (1547) for 6000 iterations.
Revised: In another experiment, we tuned our model using just pixel perturbations (1547) for 6000 iterations.

Original: The results of our experiments are presented in Table 4.
Revised: The results of our experiments are presented in Table 4.

Original: [Table]
Revised: [Table]

Original: We observed that data augmentation using pixel perturbations does not increase the robustness to
Revised: We observed that data augmentation using pixel perturbations does not increase the robustness to

Original: SAEs.
Revised: SAEs.

Original: Pixel perturbations are more general, and do not capture the effects induced by SAEs.
Revised: Pixel perturbations are more general, and do not capture the effects induced by SAEs.

Original: Consequently, we wished to understand if we could get the best of both worlds i.e. robustness against both pixel perturbations and SAEs.
Revised: Consequently, we wished to understand if we could get the best of both worlds i.e. robustness against both pixel perturbations and SAEs.

Original: We report the results of this experiment in Appendix A.
Revised: We report the results of this experiment in Appendix A.

Original: C ONCLUSIONS

Original: In this paper, we describe semantic adversarial examples (SAEs), where adversaries perturb the semantics of inputs to produce outputs that are misclassiﬁed.

Original: Such instances are easier to realize in the physical world, and are more interpretable than their traditional pixel-based counterparts.

Original: We propose an algorithm to construct SAEs using advances in differentiable rendering, and evaluate the effectiveness of our approach.

Original: We observe that SAEs cause performance degradation in object detector networks (SqueezeDet), that data augmentation using SAEs increases robustness of the model, and that data augmentation using traditional adversarial examples ( i.e. pixel perturbations) are ineffective against SAEs.

Original: A A PPENDIX A. H YBRID T UNING
Revised: A. H YBRID T UNING

Original: To understand if we are able to obtain robustness against both pixel perturbations and SAEs, we repeated our experiments from earlier sections ( § 5.2); this time, we tuned the model robust to SAEs (obtained after 6000 iterations) using pixel perturbations for 6000 iterations - denoted Tuned (Pixel); we also tuned the model robust to pixel perturbations (obtained after 6000 iterations) using SAEs for 6000 iterations - denoted Tuned (SAE).
Revised: To understand if we are able to obtain robustness against both pixel perturbations and SAEs, we repeated our experiments from earlier sections ( § 5.2); this time, we tuned the model robust to SAEs (obtained after 6000 iterations) using pixel perturbations for 6000 iterations - denoted Tuned (Pixel); we also tuned the model robust to pixel perturbations (obtained after 6000 iterations) using SAEs for 6000 iterations - denoted Tuned (SAE).

Original: [Table]
Revised: [Table]

Original: We validated both the generated models against (a) SAEs, and (b) pixel perturbations.
Revised: We validated both the generated models against (a) SAEs, and (b) pixel perturbations.

Original: Our results are presented in Table 5.
Revised: Our results are presented in Table 5.

Original: While the order of tuning did not greatly impact robustness in the face of SAEs, it had a direct impact for pixel perturbations (as evident from Table 5b).
Revised: While the order of tuning did not greatly impact robustness in the face of SAEs, it had a direct impact for pixel perturbations (as evident from Table 5b).

Original: We explain the observation in Table 5b using manifold learning.
Revised: We explain the observation in Table 5b using manifold learning.

Original: The detector is assumed to generalize over unobserved data on the data manifold.
Revised: The detector is assumed to generalize over unobserved data on the data manifold.

Original: Due to the limited amount of sampling, the detector fails to generalize to SAEs lying on these uncovered regions; learning them aids generalization.
Revised: Due to the limited amount of sampling, the detector fails to generalize to SAEs lying on these uncovered regions; learning them aids generalization.

Original: Pixel perturbations, however, do not lie on the data manifold (due to its larger degree of freedom).
Revised: Pixel perturbations, however, do not lie on the data manifold (due to its larger degree of freedom).

Original: Tuning the model using pixel perturbation is analogous to learning a thickened manifold (with the same uncovered regions as before).
Revised: Tuning the model using pixel perturbation is analogous to learning a thickened manifold (with the same uncovered regions as before).

Original: Now, if we tune this model, the detector learns uncovered regions in a thickened (and incorrect) manifold; this impacts generalization, further making it susceptible to pixel perturbations.
Revised: Now, if we tune this model, the detector learns uncovered regions in a thickened (and incorrect) manifold; this impacts generalization, further making it susceptible to pixel perturbations.

Original: Another intuition could be based on the generality of pixel perturbations; tuning ﬁrst on such general perturbations impacts generalization (Tsipras et al., 2018).
Revised: Another intuition could be based on the generality of pixel perturbations; tuning ﬁrst on such general perturbations impacts generalization (Tsipras et al., 2018).

Original: When such a model is tuned on SAEs, we gain generalization at the expense of robustness.
Revised: When such a model is tuned on SAEs, we gain generalization at the expense of robustness.