Original: On Frank-Wolfe Adversarial Training
Revised: On Frank-Wolfe Adversarial Training

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: We develop a theoretical framework for adversarial training (AT) with FW optimization (FWAT) that reveals a geometric connection between the loss landscape and the distortion of (cid:96) ∞ FW attacks (the attack’s (cid:96) 2 norm).
Revised: We develop a theoretical framework for adversarial training (AT) with FW optimization (FWAT) that reveals a geometric connection between the loss landscape and the distortion of (cid:96) ∞ FW attacks (the attack’s (cid:96) 2 norm).

Original: Speciﬁcally, we show that high distortion of FW attacks is equivalent to low variation along the attack path.
Revised: Speciﬁcally, we show that high distortion of FW attacks is equivalent to low variation along the attack path.

Original: It is then experimentally demonstrated on various deep neural network architectures that (cid:96) ∞ attacks against robust models achieve near maximal (cid:96)distortion.
Revised: It is then experimentally demonstrated on various deep neural network architectures that (cid:96) ∞ attacks against robust models achieve near maximal (cid:96)distortion.

Original: To demonstrate the utility of our theoretical framework we develop FW-AT-Adapt, a novel adversarial training algorithm which uses simple distortion measure to adapt the number of attack steps during training.
Revised: To demonstrate the utility of our theoretical framework we develop FW-AT-Adapt, a novel adversarial training algorithm which uses simple distortion measure to adapt the number of attack steps during training.

Original: FW-AT-Adapt provides strong robustness against white- and blackbox attacks at lower training times than PGD-AT.
Revised: FW-AT-Adapt provides strong robustness against white- and blackbox attacks at lower training times than PGD-AT.

Original: 1. Introduction
Revised: 1. Introduction

Original: It is well known that small, carefully chosen input perturbations, known as adversarial perturbations, can fool deep neural networks (DNNs) into making incorrect predictions (Goodfellow et al., 2015).
Revised: It is well known that small, carefully chosen input perturbations, known as adversarial perturbations, can fool deep neural networks (DNNs) into making incorrect predictions (Goodfellow et al., 2015).

Original: Various methods have been proposed to defend against such perturbations (Kurakin et al., 2017; Ros & Doshi-Velez, 2018; Madry et al., 2018; Moosavi-Dezfooli et al., 2019).
Revised: Various methods have been proposed to defend against such perturbations (Kurakin et al., 2017; Ros & Doshi-Velez, 2018; Madry et al., 2018; Moosavi-Dezfooli et al., 2019).

Original: One of the best performing algorithms is adversarial training (AT) (Madry et al., 2018), which is formulated as a robust optimization problem (Shaham et al., 2018).
Revised: One of the best performing algorithms is adversarial training (AT) (Madry et al., 2018) which is formulated as a robust optimization problem (Shaham et al., 2018).

Original: Computation of optimal adversarial perturbations is NP-hard (Weng et al., 2018) and approximate methods are used to solve the inner maximization.
Revised: Computation of optimal adversarial perturbations is NP-hard (Weng et al., 2018) and approximate methods are used to solve the inner maximization.

Original: The most popular approximate method is projected gradient descent (PGD).
Revised: The most popular approximate method is projected gradient descent (PGD).

Original: Frank-Wolfe (FW) optimization has been recently proposed in (Chen et al., 2020) and was shown to effectively fool standard networks with less distortion.
Revised: Frank-Wolfe (FW) optimization has been recently proposed in (Chen et al., 2020) and was shown to effectively fool standard networks with less distortion.

Original: Recent work has shown that FW optimization can be efﬁciently used to generate sparse counterfactual perturbations to explain model predictions and visualize principal class features (Roberts & Tsiligkaridis, 2021).
Revised: Recent work has shown that FW optimization can be efﬁ- ciently used to generate sparse counterfactual perturbations to explain model predictions and visualize principal class features (Roberts & Tsiligkaridis, 2021).

Original: In this paper, we use the Frank-Wolfe optimization to derive a relationship between the (cid:96) 2 norm of (cid:96) ∞ adversarial perturbations (distortion) and the structure of the loss landscape; based on this relationship we propose an adaptive
Revised: In this paper, we use the Frank-Wolfe optimization to derive a relationship between the (cid:96) 2 norm of (cid:96) ∞ adversarial perturbations (distortion) and the structure of the loss landscape; based on this relationship we propose an adaptive

Original: Frank-Wolfe adversarial training (FW-AT-Adapt) method to obtain robustness.
Revised: Frank-Wolfe adversarial training (FW-AT-Adapt) method to obtain robustness.

Original: This method can achieve robustness near or above multistep PGD-AT while signiﬁcantly decreasing training time.
Revised: This method can achieve robustness near or above multistep PGD-AT while signiﬁcantly decreasing training time.

Original: 2. Background and Previous Work
Revised: 2. Background and Previous Work

Original: One of the most popular and effective defenses against adversarial attacks is adversarial training (AT) (Madry et al., 2018) which minimizes the adversarial risk
Revised: One of the most popular and effective defenses against adversarial attacks is adversarial training (AT) (Madry et al., 2018) which minimizes the adversarial risk

Original: [Equation]
Revised: [Equation]

Original: This framework was extended in the TRADES algorithm (Zhang et al., 2019) which proposes a modiﬁed loss function that captures the clean and adversarial accuracy tradeoff.
Revised: This framework was extended in the TRADES algorithm (Zhang et al., 2019) which proposes a modiﬁed loss function that captures the clean and adversarial accuracy tradeoff.

Original: Local Linearity Regularization (LLR) (Qin et al., 2019) uses an analogous approach where the adversary is chosen to maximally violate local linearity based on a ﬁrst order approximation to (1).
Revised: Local Linearity Regularization (LLR) (Qin et al., 2019) uses an analogous approach where the adversary is chosen to maximally violate local linearity based on a ﬁrst order approximation to (1).

Original: To construct adversarial attacks at a given input x , AT uses
Revised: To construct adversarial attacks at a given input x , AT uses

Original: Projected Gradient Descent (PGD) to approximate the inner maximization using a ﬁxed number of iterations:
Revised: Projected Gradient Descent (PGD) to approximate the inner maximization using a ﬁxed number of iterations:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where P B p ( (cid:15) ) is the orthogonal projection onto the constraint set.
Revised: where P B p ( (cid:15) ) is the orthogonal projection onto the constraint set.

Original: We refer to AT using K step PGD as PGD(K)-AT.
Revised: We refer to AT using K step PGD as PGD(K)-AT.

Original: The computational cost of this method is dominated by the number of steps used to approximate the inner maximization.
Revised: The computational cost of this method is dominated by the number of steps used to approximate the inner maximization.

Original: Using fewer PGD steps lowers this cost, but these amount to weaker attacks possibly leading to gradient obfuscation (Papernot et al., 2017; Uesato et al., 2018), a phenomenon where networks learn to defend against gradient-based attacks by making the loss landscape highly non-linear, and less robust models.
Revised: Using fewer PGD steps lowers this cost, but these amount to weaker attacks possibly leading to gradient obfuscation (Papernot et al., 2017; Uesato et al., 2018), a phenomenon where networks learn to defend against gradient-based attacks by making the loss landscape highly non-linear, and less robust models.

Original: Other works have modiﬁed the number of steps used to approximate (1) such as curriculum learning (Cai et al., 2018) which monitors adversarial performance during training and increases the number of attack steps as performance improves, and (Wang et al., 2019) which use a FW convergence criterion to adapt the number of attack steps at given inputs.
Revised: Other works have modiﬁed the number of steps used to approximate (1) such as curriculum learning (Cai et al., 2018) which monitors adversarial performance during training and increases the number of attack steps as performance improves, and (Wang et al., 2019) which use a FW convergence criterion to adapt the number of attack steps at given inputs.

Original: Both of these methods use PGD to generate adversarial examples and do not report improved training times.
Revised: Both of these methods use PGD to generate adversarial examples and do not report improved training times.

Original: We present Frank Wolfe Adversarial Training (FW-AT) which replaces the PGD inner optimization with a FrankWolfe optimizer.
Revised: We present Frank Wolfe Adversarial Training (FW-AT) which replaces the PGD inner optimization with a FrankWolfe optimizer.

Original: FW-AT achieves similar robustness as its
Revised: FW-AT achieves similar robustness as its

Original: PGD counterpart.
Revised: PGD counterpart.

Original: Using a closed form expression for the
Revised: Using a closed form expression for the

Original: FW attack path we derive bounds on the gradient alignment of the loss and attack path variation in terms of the distortion of the attack.
Revised: FW attack path we derive bounds on the gradient alignment of the loss and attack path variation in terms of the distortion of the attack.

Original: This key insight leads to a simple modiﬁcation of FW-AT where the step size at each epoch is adapted based on the (cid:96) 2 distortion of the attacks and is shown to provide strong robustness and faster training times.
Revised: This key insight leads to a simple modiﬁcation of FW-AT where the step size at each epoch is adapted based on the (cid:96) 2 distortion of the attacks and is shown to provide strong robustness and faster training times.

Original: 2.1. Frank-Wolfe Adversarial Attack
Revised: 2.1. Frank-Wolfe Adversarial Attack

Original: The Frank-Wolfe (FW) optimization algorithm has its origins in convex optimization though recently has been shown to perform well in more general settings (Frank & Wolfe, 1956;
Revised: The Frank-Wolfe (FW) optimization algorithm has its origins in convex optimization though recently has been shown to perform well in more general settings (Frank & Wolfe, 1956;

Original: Jaggi, 2013).
Revised: Jaggi, 2013).

Original: FW ﬁrst optimizes a linear approximation to the original problem, called a Linear Maximization
Revised: FW ﬁrst optimizes a linear approximation to the original problem, called a Linear Maximization

Original: Oracle (LMO), ¯ δ k = argmax δ ∈ B p ( (cid:15) )
Revised: Oracle (LMO), ¯ δ k = argmax δ ∈ B p ( (cid:15) )

Original: (cid:104) δ, ∇ δ L ( x + δ k , y ) (cid:105) .
Revised: (cid:104) δ, ∇ δ L ( x + δ k , y ) (cid:105) .

Original: After calling the LMO, FW takes a step using a convex combination with the current iterate, δ k +

Revised: After calling the LMO, FW takes a step using a convex combination with the current iterate, δ k +

Original: = δ k
Revised: = δ k

Original: + γ k (¯ δ k − δ k ) where γ k ∈ [0 , 1] is the step size.
Revised: + γ k (¯ δ k − δ k ) where γ k ∈ [0 , 1] is the step size.

Original: An effective choice is γ k =
Revised: An effective choice is γ k =

Original: c/ ( c + k ) for some c ≥ 1 .
Revised: c/ ( c + k ) for some c ≥ 1 .

Original: In our experiments robustness of FW-AT was not sensitive to choice of c .
Revised: In our experiments robustness of FW-AT was not sensitive to choice of c .

Original: This is in contrast to PGD-AT which can be highly sensitize to choice of step size (Wong et al., 2020).
Revised: This is in contrast to PGD-AT which can be highly sensitize to choice of step size (Wong et al., 2020).

Original: The LMO can be computed exactly for any (cid:96) p and for the (cid:96) ∞ case, LMO is given by ¯ δ k,i = (cid:15) sgn
Revised: The LMO can be computed exactly for any (cid:96) p and for the (cid:96) ∞ case, LMO is given by ¯ δ k,i = (cid:15) sgn

Original: ( ∇L k,i ) , where ∇L = ∇ δ L ( x + δ k , y ) .
Revised: ( ∇L k,i ) , where ∇L = ∇ δ L ( x + δ k , y ) .

Original: 3. Distortion of Frank-Wolfe Attacks
Revised: 3. Distortion of Frank-Wolfe Attacks

Original: We refer to the (cid:96) 2 norm of an (cid:96) ∞ attack as distortion in this section we explore the insights distortion can give us into the behavior of FW-AT.
Revised: We refer to the (cid:96) 2 norm of an (cid:96) ∞ attack as distortion in this section we explore the insights distortion can give us into the behavior of FW-AT.

Original: For the remainder of this section FWattacks will refer to (cid:96) ∞ attacks.
Revised: For the remainder of this section FWattacks will refer to (cid:96) ∞ attacks.

Original: In this setting, the maximal distortion possible is (cid:15) √ d , and we refer to (cid:107) δ (cid:107) 2 / ( (cid:15) √ d ) as the distortion ratio (or simply distortion) of the attack δ .
Revised: In this setting, the maximal distortion possible is (cid:15) √ d , and we refer to (cid:107) δ (cid:107) 2 / ( (cid:15) √ d ) as the distortion ratio (or simply distortion) of the attack δ .

Original: 3.1. Adversarial Training Rapidly Increases Distortion
Revised: 3.1. Adversarial Training Rapidly Increases Distortion

Original: Low distortion of FW-attacks was observed for standard models in (Chen et al., 2020) but robust models were not considered.
Revised: Low distortion of FW-attacks was observed for standard models in (Chen et al., 2020) but robust models were not considered.

Original: We analyze the distortion ratio of FW attacks on three architectures trained with standard cross-entropy training and PGD(10)-AT on CIFAR-10.
Revised: We analyze the distortion ratio of FW attacks on three architectures trained with standard cross-entropy training and PGD(10)-AT on CIFAR-10.

Original: We analyze the distortion ratio of (cid:15) = 8 / 255 attacks using FW(20) on the CIFAR10 test dataset.
Revised: We analyze the distortion ratio of (cid:15) = 8 / 255 attacks using FW(20) on the CIFAR10 test dataset.

Original: Figure 1a shows that, while adversarial perturbations of standard models have small distortion, robust models produce attacks that are nearly maximally distorted.
Revised: Figure 1a shows that, while adversarial perturbations of standard models have small distortion, robust models produce attacks that are nearly maximally distorted.

Original: This phenomena occurs across three different architectures and is further supported by our theory below.
Revised: This phenomena occurs across three different architectures and is further supported by our theory below.

Original: We note for PGD attacks the distortion ratio can be trivially maximized with a large step size α , and thus this connection between distortion and robustness does not exist for PGD optimization.
Revised: We note for PGD attacks the distortion ratio can be trivially maximized with a large step size α , and thus this connection between distortion and robustness does not exist for PGD optimization.

Original: In Figure 1b we run three epochs of FW-AT for varying number of steps and monitor the distortion.
Revised: In Figure 1b we run three epochs of FW-AT for varying number of steps and monitor the distortion.

Original: Initially there is high variation but after merely one epoch the distortion of the FW attacks has converged.
Revised: Initially there is high variation but after merely one epoch the distortion of the FW attacks has converged.

Original: As we will see in Theorem 1 this indicates that the loss gradients are rapidly aligning during FW-AT.
Revised: As we will see in Theorem 1 this indicates that the loss gradients are rapidly aligning during FW-AT.

Original: 3.2. Multistep, High Distortion Attacks are Inefﬁcient
Revised: 3.2. Multistep, High Distortion Attacks are Inefﬁcient

Original: Our main tool in analyzing the distortion of FW attacks, and a prime reason FW-AT is more mathematically transparent than PGD-AT, is a representation of the FW attack as a convex combination of the LMO iterates.
Revised: Our main tool in analyzing the distortion of FW attacks, and a prime reason FW-AT is more mathematically transparent than PGD-AT, is a representation of the FW attack as a convex combination of the LMO iterates.

Original: We refer to the steps taken during the optimization as the attack path.
Revised: We refer to the steps taken during the optimization as the attack path.

Original: Proposition 1.
Revised: Proposition 1.

Original: The FW Attack with step sizes γ k =
Revised: The FW Attack with step sizes γ k =

Original: c/ ( c + k ) for some c ≥ 1 yields the following adversarial perturbation after K steps
Revised: c/ ( c + k ) for some c ≥ 1 yields the following adversarial perturbation after K steps

Original: [Equation]
Revised: [Equation]

Original: where α l = γ l (cid:81) K − 1 i = l +1 (1 − γ i ) ∈ [0 , 1] are non-decreasing in l , and sum to unity.
Revised: where α l = γ l (cid:81) K − [Equation] = l +1 (1 − γ i ) ∈ [0 , 1] are non-decreasing in l , and sum to unity.

Original: Using this representation we can derive connections between the distortion of the attack and the variation along the attack path.
Revised: Using this representation we can derive connections between the distortion of the attack and the variation along the attack path.

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider a FW attack δ K .
Revised: Consider a FW attack δ K .

Original: Let cos β lj be the directional cosine between sgn ( ∇ δ L ( x + δ l , y )) and sgn ( ∇ δ L ( x + δ j , y )) .
Revised: Let cos β lj be the directional cosine between sgn ( ∇ δ L ( x + δ l , y )) and sgn ( ∇ δ L [Equation] x + δ j , y )) .

Original: The distortion ratio of the adversarial perturbation δ K is:
Revised: The distortion ratio of the adversarial perturbation δ K is:

Original: [Equation]
Revised: [Equation]

Original: We can summarize the spirit of Theorem 1 by:
Revised: We can summarize the spirit of Theorem 1 by:

Original: Higher distortion is equivalent to lower gradient variation throughout the attack path.
Revised: Higher distortion is equivalent to lower gradient variation throughout the attack path.

Original: Concretely, the accumulation of sign changes between every step of the attack decreases distortion.
Revised: Concretely, the accumulation of sign changes between every step of the attack decreases distortion.

Original: Following this logic further we are able quantify the distance between different step attacks in terms of the ﬁnal distortion.
Revised: Following this logic further we are able quantify the distance between different step attacks in terms of the ﬁnal distortion.

Original: [Figure]
Revised: [Figure]

Original: Theorem 2.
Revised: Theorem 2.

Original: Let the same conditions as Theorem 1 hold and K > 1 .
Revised: Let the same conditions as Theorem 1 hold and K > 1 .

Original: Assume the distortion ratio of the adversarial perturbation satisﬁes:
Revised: Assume the distortion ratio of the adversarial perturbation satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: with η ∈ (0 , 1).
Revised: with η ∈ (0 , 1)

Revised: [Equation]

Original: Then,the deviation of δ K and intermediate steps δ k 0 , for 1 ≤ k 0 ≤ K satisfy:
Revised: Then,the deviation of δ K and intermediate steps δ k 0 , for 1 ≤ k 0 ≤ K satisfy:

Original: [Equation]
Revised: [Equation]

Original: The consequences of 2 can be summarized as:
Revised: The consequences of 2 can be summarized as:

Original: Multistep attacks with high distortion are inefﬁcient.
Revised: Multistep attacks with high distortion are inefﬁcient.

Original: This suggests that during FW-AT using a large number of steps to approximate the adversarial risk results in diminishing returns once high distortion of the attacks is attained.
Revised: This suggests that during FW-AT using a large number of steps to approximate the adversarial risk results in diminishing returns once high distortion of the attacks is attained.

Original: Figure 1b shows that the distortion of attacks reaches a maximal value in the early stages of training suggesting that much of the computation used in the later steps of the optimization is wasted.
Revised: Figure 1b shows that the distortion of attacks reaches a maximal value in the early stages of training suggesting that much of the computation used in the later steps of the optimization is wasted.

Original: Inspired by this insight we design a
Revised: Inspired by this insight we design a

Original: FW-AT algorithm which drops the number of attack steps based the distortion of a multistep attack (FW-AT-Adapt).
Revised: FW-AT algorithm which drops the number of attack steps based the distortion of a multistep attack (FW-AT-Adapt).

Original: Additional bounds are shown in the Appendix which assert that, for batches with high distortion, the gradients and thus the weight updates obtained by large step attacks are near those of a low step attack.
Revised: Additional bounds are shown in the Appendix which assert that, for batches with high distortion, the gradients and thus the weight updates obtained by large step attacks are near those of a low step attack.

Original: Thus, FW-AT-Adapt is expected to achieve a similar level of robustness to non-adaptive methods.
Revised: Thus, FW-AT-Adapt is expected to achieve a similar level of robustness to non-adaptive methods.

Original: Proofs are included in the Appendix.
Revised: Proofs are included in the Appendix.

Original: 4. Frank-Wolfe Adversarial Training Algorithm
Revised: 4. Frank-Wolfe Adversarial Training Algorithm

Original: Details of Adaptive Frank-Wolfe adversarial training method (FW-AT-Adapt-E) are provided in Algorithm 2.
Revised: Details of Adaptive Frank-Wolfe adversarial training method (FW-AT-Adapt-E) are provided in Algorithm 2.

Original: The overall algorithm is almost identical to PGD-AT with two slight modiﬁcations: 
Revised: The overall algorithm is almost identical to PGD-AT with two slight modiﬁcations: Algorithm

Original: Input: Network f θ , input ( x, y ) , steps K .
Revised: Input: Network f θ , input ( x, y ) , steps K .

Original: δ =for 0 ≤ k < K do ¯ δ = (cid:15) sgn ( ∇ δ L ( f θ ( x + δ ) , y ))
Revised: δ =for 0 ≤ k < K do ¯ δ = (cid:15) sgn ( ∇ δ L ( f θ ( x + δ ) , y ))

Original: δ =
Revised: δ =

Original: δ + c c + k
Revised: δ + c c + k

Original: (¯ δ − δ ) end for
Revised: (¯ δ − δ ) end for

Original: Return: δ
Revised: Return: δ

Original: The adversarial attack is approximated via a FW optimization scheme (Alg. 1) 2.
Revised: The adversarial attack is approximated via a FW optimization scheme (Alg. 1) 2.

Original: For the ﬁrst N b batches of each epoch, the distortion of a K step attack is monitored.
Revised: For the ﬁrst N b batches of each epoch, the distortion of a K step attack is monitored.

Original: If its mean across these batches is above a threshold r then the number of attack steps is dropped to K 0 for the remainder of the epoch.
Revised: If its mean across these batches is above a threshold r then the number of attack steps is dropped to K 0 for the remainder of the epoch.

Original: We also experiment with periodic distortion monitoring (FW-AT-Adapt-P) where we perform a distortion check on a single batch when the epoch is 0, 30, and 60% complete.
Revised: We also experiment with periodic distortion monitoring (FW-AT-Adapt-P) where we perform a distortion check on a single batch when the epoch is 0, 30, and 60% complete.

Original: Theorem 2 guarentees that high-step high distortion attacks are near low step attacks implying that FW-AT-Adapt should train similarly robust models to AT.
Revised: Theorem 2 guarentees that high-step high distortion attacks are near low step attacks implying that FW-AT-Adapt should train similarly robust models to AT.

Original: A detailed expression of this can be found in Cor.
Revised: A detailed expression of this can be found in Cor.

Original: 1 in the appendix.
Revised: 1 in the appendix.

Original: 5. Experimental Results
Revised: 5. Experimental Results

Original: We evaluate the performance of our proposed FW-AT-Adapt against standard training, and PGD-AT (Madry et al., 2018).
Revised: We evaluate the performance of our proposed FW-AT-Adapt against standard training, and PGD-AT (Madry et al., 2018).

Original: All networks were trained by ﬁne-tuning for 30 epochs off a standard model via SGD optimization with a learning rate of 0.1 which was decreased to 0.01 after 15 epochs.
Revised: All networks were trained by ﬁne-tuning for 30 epochs off a standard model via SGD optimization with a learning rate of 0.1 which was decreased to 0.01 after 15 epochs.

Original: We record the time to train the full 30 epochs and models were selected based on their checkpoint which obtained the highest adversarial accuracy on the test set.
Revised: We record the time to train the full 30 epochs and models were selected based on their checkpoint which obtained the highest adversarial accuracy on the test set.

Original: Each conﬁguration is run 3 times and the median time is reported.
Revised: Each conﬁguration is run 3 times and the median time is reported.

Original: The performance metric used is the accuracy on the CIFAR-10 test set after
Revised: The performance metric used is the accuracy on the CIFAR-10 test set after

Original: Input: Network f θ , data D , epoch learning rate η t , high steps K 1 , low steps K 0 , max distortion r , number of monitoring batches B m .
Revised: Input: Network f θ , data D , epoch learning rate η t , high steps K 1 , low steps K 0 , max distortion r , number of monitoring batches B m .

Original: Result: Robust model weights θ .
Revised: Result: Robust model weights θ .

Original: N b , d m =
Revised: N b , d m =

Original: K =
Revised: K =

Original: Kfor each batch ( x, y ) ∼ D do δ = FW-Attack ( x, y ; K, p = ∞ ) if N b ≤ B m then d m =
Revised: Kfor each batch ( x, y ) ∼ D do δ = FW-Attack ( x, y ; K, p = ∞ ) if N b ≤ B m then d m =

Original: d m + (cid:107) δ (cid:107)if N b
Revised: d m + (cid:107) δ (cid:107)if N b

Original: =
Revised: =

Original: B m and d m /B m > r then
Revised: B m and d m /B m > r then

Original: K = Kend if end if θ = θ − η t 1 | B | (cid:80) i
Revised: K = Kend if end if θ = θ − η t 1 | B | (cid:80) i

Original: ∈
Revised: ∈

Original: B ∇ θ L ( f θ ( x i + δ i ) , y i )
Revised: B ∇ θ L ( f θ ( x i + δ i ) , y i )

Original: N b = N b +end for(a) (cid:15) = 8 /(b)
Revised: N b = N b +end for

Original: (cid:15) =
Revised: [Figure]

Original: 16 / the attack is applied (adversarial accuracy).
Revised: the attack is applied (adversarial accuracy).

Original: All PGD(K) attacks of size (cid:15) with are computed with step size 2 .
Revised: All PGD(K) attacks of size (cid:15) with are computed with step size 2 .

Original: 5 (cid:15)/K similar to (Madry et al., 2018).
Revised: 5 (cid:15)/K similar to (Madry et al., 2018).

Original: To determine whether there are step size, drop step, and max distortion ratio combinations, K - K 0 - r , for which FW-AT-Adapt-E/P achieves superior robustness/train time trade offs compared to PGD-AT, a hyperparameter sweep across K ∈ { 4 , 5 , 7 , 10 } , K 0 ∈ { 1 , 2 , 3 } , and r ∈ { 0 . 91 , 0 . 92 , . .
Revised: To determine whether there are step size, drop step, and max distortion ratio combinations, K - K 0 - r , for which FW-AT-Adapt-E/P achieves superior robustness/train time trade offs compared to PGD-AT, a hyperparameter sweep across K ∈ { 4 , 5 , 7 , 10 } , K 0 ∈ { 1 , 2 , 3 } , and r ∈ { 0 . 91 , 0 . 92 , . .

Original: 96 } are compared against PGD(1)- through PGD(7)-AT, and PGD(10)-AT.
Revised: 96 } are compared against PGD(1)- through PGD(7)-AT, and PGD(10)-AT.

Original: We set B m =for all FW-AT-Adapt-E models.
Revised: We set B m =for all FW-AT-Adapt-E models.

Original: Based on Figure1b we allow the models to do traditional FW-AT for one epoch and then begin the adaptation.
Revised: Based on Figure1b we allow the models to do traditional FW-AT for one epoch and then begin the adaptation.

Original: We note on CIFAR-10 AT with (cid:15) = 8 / 255 performance of AT seems to saturate with a low number of steps so we experimented with adding an extra epoch of FW-AT before beginning adaptation.
Revised: We note on CIFAR-10 AT with (cid:15) = 8 / 255 performance of AT seems to saturate with a low number of steps so we experimented with adding an extra epoch of FW-AT before beginning adaptation.

Original: We then evaluated all models against a PGD(50) attack.
Revised: We then evaluated all models against a PGD(50) attack.

Original: Figure 2 shows the performance front which start with the lowest training time models and adds models sequentially in time if their adversarial accuracy is strictly greater than the previous point on the front.
Revised: Figure 2 shows the performance front which start with the lowest training time models and adds models sequentially in time if their adversarial accuracy is strictly greater than the previous point on the front.

Original: Both FW-AT-Adapt-E and
Revised: Both FW-AT-Adapt-E and

Original: FW-AT-Adapt-P exhibit superior performance trade-offs to PGD-AT.
Revised: FW-AT-Adapt-P exhibit superior performance trade-offs to PGD-AT.

Original: Notably in the case of (cid:15) = 8 / 255 (Fig.
Revised: Notably in the case of (cid:15) = 8 / 255 (Fig.

Original: 2a ) FW-Adapt-E’s performance front is strictly superior to
Revised: 2a ) FW-Adapt-E’s performance front is strictly superior to

Original: PGD-AT’s achieving points with the same robustness as
Revised: PGD-AT’s achieving points with the same robustness as

Original: PGD(10)-AT at half the training time.
Revised: PGD(10)-AT at half the training time.

Original: For (cid:15) = 16 /
Revised: For (cid:15) = 16 /

Original: PGD(10)-AT (Fig. 2b ) remains optimal with respect to adversarial accuracy but FW-Adapt can still achieve competitive robustness at half the training time.
Revised: PGD(10)-AT (Fig. 2b ) remains optimal with respect to adversarial accuracy but FW-Adapt can still achieve competitive robustness at half the training time.

Original: Gradient masking is evaluated using a similar evaluation protocol as in (Moosavi-Dezfooli et al., 2019) inspired by (Uesato et al., 2018).
Revised: Gradient masking is evaluated using a similar evaluation protocol as in (Moosavi-Dezfooli et al., 2019) inspired by (Uesato et al., 2018).

Original: Table 1 shows that our method achieves high adversarial accuracy when evaluated against PGD attacks of increasing strength.
Revised: Table 1 shows that our method achieves high adversarial accuracy when evaluated against PGD attacks of increasing strength.

Original: Our models are evaluated against Square (Andriushchenko et al., 2020); a black-box gradient-free attack repeatedly perform queries to construct adversarial images.
Revised: Our models are evaluated against Square (Andriushchenko et al., 2020); a black-box gradient-free attack repeatedly perform queries to construct adversarial images.

Original: Table 1 contains the adversarial accuracy obtained under PGD(100) white-box attack and blackbox attacks against (cid:96) ∞ -robust FW-Adapt and PGD(10) models for a batch of 1000 test examples.
Revised: Table 1 contains the adversarial accuracy obtained under PGD(100) white-box attack and blackbox attacks against (cid:96) ∞ -robust FW-Adapt and PGD(10) models for a batch of 1000 test examples.

Original: For (cid:15) = 8 / 255 ,
Revised: For (cid:15) = 8 / 255 ,

Original: FW-Adapt-E was used with parameters 10-2-0.94, and for (cid:15)
Revised: FW-Adapt-E was used with parameters 10-2-0.94, and for (cid:15)

Original: = 16 / 255 , FW-Adapt-E parameters were 10-2-0.91.
Revised: = 16 / 255 , FW-Adapt-E parameters were 10-2-0.91.

Original: The proposed FW-AT-Adapt defense achieves similar robustness to PGD-AT.
Revised: The proposed FW-AT-Adapt defense achieves similar robustness to PGD-AT.

Original: Further gradient masking evaluations are included in the Appendix.
Revised: Further gradient masking evaluations are included in the Appendix.

Original: [Table]
Revised: [Table]

Original: 6. Conclusion
Revised: 6. Conclusion

Original: FW (cid:96) ∞ attacks against robust models have higher (cid:96) 2 distortions than standard ones.
Revised: FW (cid:96) ∞ attacks against robust models have higher (cid:96) 2 distortions than standard ones.

Original: We derive a theoretical connection between loss geometry and distortion of FW attacks which explains this phenomenon and bound the variation along the attack path in terms of the distortion.
Revised: We derive a theoretical connection between loss geometry and distortion of FW attacks which explains this phenomenon and bound the variation along the attack path in terms of the distortion.

Original: Inspired by this connection, we propose an adaptive AT algorithm (FW-ATAdapt) that achieves high robustness with lower training time than PGD-AT against strong white- and black-box attacks, and is resistant to gradient obfuscation.
Revised: Inspired by this connection, we propose an adaptive AT algorithm (FW-ATAdapt) that achieves high robustness with lower training time than PGD-AT against strong white- and black-box attacks, and is resistant to gradient obfuscation.

Original: We hope this work encourages future research on the connection between
Revised: We hope this work encourages future research on the connection between

Original: Frank-Wolfe optimization and adversarial robustness.
Revised: Frank-Wolfe optimization and adversarial robustness.

Original: 7. Appendix 7.1. Theoretical Results and Proofs
Revised: 7. Appendix 7.1. Theoretical Results and Proofs

Original: While loss functions L ( x + δ, y ) in deep neural networks are non-convex in general, we make the following assumption.
Revised: While loss functions L ( x + δ, y ) in deep neural networks are non-convex in general, we make the following assumption.

Original: Assumption 1.
Revised: Assumption 1.

Original: The function L has L -Lipschitz continuous gradients on B p ( (cid:15) ) , i.e., (cid:107)∇L ( x + u, y ) −∇L
Revised: The function L has L -Lipschitz continuous gradients on B p ( (cid:15) ) , i.e., (cid:107)∇L ( x + u, y ) −∇L

Original: ( x + v, y )
Revised: ( x + v, y )

Original: (cid:107) ≤
Revised: (cid:107) ≤

Original: L (cid:107) u − v (cid:107) , ∀ u, v ∈ B p ( (cid:15) ) .
Revised: L (cid:107) u − v (cid:107) , ∀ u, v ∈ B p ( (cid:15) ) .

Original: Assumption 1 is a standard assumption for the non-convex setting and has been made in several works (Lacoste-Julien, 2016; Chen et al., 2020).
Revised: Assumption 1 is a standard assumption for the non-convex setting and has been made in several works (Lacoste-Julien, 2016; Chen et al., 2020).

Original: For standard ERM training loss smoothing has been observed to occur in over parameterized (Allen-Zhu et al., 2019; Zou & Gu, 2019; Cao & Gu, 2020) and batch normalized (Santurkar et al., 2018) DNNs.
Revised: For standard ERM training loss smoothing has been observed to occur in over parameterized (Allen-Zhu et al., 2019; Zou & Gu, 2019; Cao & Gu, 2020) and batch normalized (Santurkar et al., 2018) DNNs.

Original: For robust models the process of adversarial training has been noted to signiﬁcantly smooth the loss (Moosavi-Dezfooli et al., 2019; Qin et al., 2019).
Revised: For robust models the process of adversarial training has been noted to signiﬁcantly smooth the loss (Moosavi-Dezfooli et al., 2019; Qin et al., 2019).

Original: an example comparing standard and robust loss landscapes is given in Figure 3.
Revised: an example comparing standard and robust loss landscapes is given in Figure 3.

Original: Moreover, the distortion plots of Figure 1 (b) together with Theorem 1 suggest that FW-AT quickly regularizes the loss landscape.
Revised: Moreover, the distortion plots of Figure 1 (b) together with Theorem 1 suggest that FW-AT quickly regularizes the loss landscape.

Original: Corollary 1.
Revised: Corollary 1.

Original: Consider a batch update of FW-AT Algorithm 2 where the high distortion condition of Thm.
Revised: Consider a batch update of FW-AT Algorithm 2 where the high distortion condition of Thm.

Original: 2 holds on average on examples in a batch B , i.e. for some small η ∈ (0 , 1) : 
Revised: 2 holds on average on examples in a batch B , i.e. for some small η ∈ (0 , 1) : | B | (cid:88) i ∈ B (cid:107) [Equation] [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: (cid:15) √

Original:  where δ i ( θ, K ) denotes the K -step FW adversarial perturbation for the i -th example in the batch B .
Revised: [Equation] ≥ (cid:112) [Equation] − η (6) where δ i ( θ, K ) denotes the K -step FW adversarial perturbation for the i -th example in the batch B .

Original: Let the SGD model weight gradient be given by:
Revised: Let the SGD model weight gradient be given by:

Original: [Equation]
Revised: [Equation]

Original: Given Assumption 1 holds, the model weights SGD update using adversarial perturbations δ K and δ k 0 are bounded as:
Revised: Given Assumption 1 holds, the model weights SGD update using adversarial perturbations δ K and δ k 0 are bounded as:

Original: [Equation]
Revised: [Equation]

Original: where L is the local Lipschitz constant.
Revised: where L is the local Lipschitz constant.

Original: Bound (7) asserts that in the high distoriton setting the gradients, and thus the weight updates, obtained by a high step attack are near those of a low step attack.
Revised: Bound (7) asserts that in the high distoriton setting the gradients, and thus the weight updates, obtained by a high step attack are near those of a low step attack.

Original: Therfore, it is expected to achieve a similar level of adversarial robustness using the proposed adaptive algorithm.
Revised: Therfore, it is expected to achieve a similar level of adversarial robustness using the proposed adaptive algorithm.

Original: 7.2. Loss Landscape
Revised: 7.2. Loss Landscape

Original: It has been shown experimentally that AT robust models and geometric regularization methods that increase adversarial robustness, have more regular loss landscapes than
Revised: It has been shown experimentally that AT robust models and geometric regularization methods that increase adversarial robustness, have more regular loss landscapes than

Original: [Figure]
Revised: [Figure]

Original: their non-robust counterparts (Lyu et al., 2015; Ros & DoshiVelez, 2018; Moosavi-Dezfooli et al., 2019; Qin et al., 2019).
Revised: their non-robust counterparts (Lyu et al., 2015; Ros & DoshiVelez, 2018; Moosavi-Dezfooli et al., 2019; Qin et al., 2019).

Original: Figure 3 shows sample loss surfaces for a standard model,
Revised: Figure 3 shows sample loss surfaces for a standard model,

Original: PGD(10)-AT, and FW(10)-AT.
Revised: PGD(10)-AT, and FW(10)-AT.

Original: The white circle denotes the original image, and the red circle denotes the (cid:96) ∞ adversarial perturbation at (cid:15)
Revised: The white circle denotes the original image, and the red circle denotes the (cid:96) ∞ adversarial perturbation at (cid:15)

Original: = 8 / 255 .
Revised: = 8 / 255 .

Original: One axis is aligned to the adversarial perturbation direction and the other is aligned to a random orthogonal direction.
Revised: One axis is aligned to the adversarial perturbation direction and the other is aligned to a random orthogonal direction.

Original: The loss surface of standard models is highly non-linear and the loss varies signiﬁcantly in this small neighborhood, with the largest variation occurs in the adversarial direction.
Revised: The loss surface of standard models is highly non-linear and the loss varies signiﬁcantly in this small neighborhood, with the largest variation occurs in the adversarial direction.

Original: Figure 3 (b) and (c) demonstrate that robust models offer more resistance to such local loss variation and the variation is primarily in the direction of the adversarial attack.
Revised: Figure 3 (b) and (c) demonstrate that robust models offer more resistance to such local loss variation and the variation is primarily in the direction of the adversarial attack.

Original: These results suggest high resistance to gradient obfuscation.
Revised: These results suggest high resistance to gradient obfuscation.

Original: 7.3. Adversarial Robustness: White-Box Evaluations
Revised: 7.3. Adversarial Robustness: White-Box Evaluations

Original: To further evaluate FW-AT-Adapt we choose strong performers from the previous experiment to test against a larger variety of adversarial attacks, including white-box untargeted and targeted attacks towards a random class r ∼ U ( { 1 , . .
Revised: To further evaluate FW-AT-Adapt we choose strong performers from the previous experiment to test against a larger variety of adversarial attacks, including white-box untargeted and targeted attacks towards a random class r ∼ U ( { 1 , . .

Original: . , K }\ y ) .
Revised: . , K }\ y ) .

Original: The white-box setup considers attack techniques that have full access to the model parameters and are constrained by the same maximum perturbation size (cid:15) .
Revised: The white-box setup considers attack techniques that have full access to the model parameters and are constrained by the same maximum perturbation size (cid:15) .

Original: The classiﬁcation margin is deﬁned as
Revised: The classiﬁcation margin is deﬁned as

Original: M ( x, y ) = log p
Revised: M ( x, y ) = log p

Original: y
Revised: y

Original: ( x ) − max j (cid:54) =
Revised: ( x ) − max j (cid:54) =

Original: y log p
Revised: y log p

Original: j
Revised: j

Original: ( x ) .
Revised: ( x ) .

Original: The following white-box attacks are used for evaluating adversarial robustness:
Revised: The following white-box attacks are used for evaluating adversarial robustness:

Original: (UL) Untargeted-loss: max δ ∈ B ( (cid:15) )
Revised: (UL) Untargeted-loss: max δ ∈ B ( (cid:15) )

Original: L ( x + δ, y ) (TL)
Revised: L ( x + δ, y ) (TL)

Original: Random Targeted-loss: min δ ∈ B ( (cid:15) )
Revised: Random Targeted-loss: min δ ∈ B ( (cid:15) )

Original: L ( x + δ, r ) (UM) Untargeted-margin: min δ ∈ B ( (cid:15) )
Revised: L ( x + δ, r ) (UM) Untargeted-margin: min δ ∈ B ( (cid:15) )

Original: M ( x + δ, y ) (TM)
Revised: M ( x + δ, y ) (TM)

Original: Random Targeted-margin: max δ ∈ B ( (cid:15) )
Revised: Random Targeted-margin: max δ ∈ B ( (cid:15) )

Original: M ( x + δ, r )
Revised: M ( x + δ, r )

Original: Table 2 reports robustness results on CIFAR-10 for these (cid:96) ∞ attacks at (cid:15)
Revised: Table 2 reports robustness results on CIFAR-10 for these (cid:96) ∞ attacks at (cid:15)

Original: = 8 / 255 , 16 / 255 for a ResNet18 architecture trained at (cid:15)
Revised: = 8 / 255 , 16 / 255 for a ResNet18 architecture trained at (cid:15)

Original: = 8 / 255 , 16 / 255 respectively.
Revised: = 8 / 255 , 16 / 255 respectively.

Original: FW-AT-Adapt achieves robustness competitive with PGD-AT for untargeted and targeted attacks at lower training times.
Revised: FW-AT-Adapt achieves robustness competitive with PGD-AT for untargeted and targeted attacks at lower training times.

Original: 7.4. Gradient Masking Evaluations
Revised: 7.4. Gradient Masking Evaluations

Original: Figure 4 compares the margin computed using Square attack and PGD(100) attack for a large batch of test points.
Revised: Figure 4 compares the margin computed using Square attack and PGD(100) attack for a large batch of test points.

Original: The margin captures conﬁdence, and is positive for correct predictions and negative for misclassiﬁcations.
Revised: The margin captures conﬁdence, and is positive for correct predictions and negative for misclassiﬁcations.

Original: The margin was computed with black-box Square Attack for the y-axis and white-box PGD(100) attack for x-axis on a set of 1000 test points.
Revised: The margin was computed with black-box Square Attack for the y-axis and white-box PGD(100) attack for x-axis on a set of 1000 test points.

Original: Points near the line y
Revised: Points near the line y

Original: =
Revised: =

Original: x indicate both types of attacks found similar adversarial perturbations, while points below the line shown in red imply that Square Attack identiﬁed stronger attacks than PGD.
Revised: x indicate both types of attacks found similar adversarial perturbations, while points below the line shown in red imply that Square Attack identiﬁed stronger attacks than PGD.

Original: It is observed that both methods lead to a similar margin except on a small subset of points for FW-AT-Adapt-E.
Revised: It is observed that both methods lead to a similar margin except on a small subset of points for FW-AT-Adapt-E.

Original: For (cid:15) = 8 / 255 , FW-AT-Adapt-E has 13 / 1000 = 1 .
Revised: For (cid:15) = 8 / 255 , FW-AT-Adapt-E has 13 / 1000 = 1 .

Original: 3% and PGD-AT has 23 / 1000 = 2 .
Revised: 3% and PGD-AT has 23 / 1000 = 2 .

Original: 3% points in red.
Revised: 3% points in red.

Original: For (cid:15) = 16 / 255 , FW-AT-Adapt-E has 130 / 1000 = 13% and
Revised: For (cid:15) = 16 / 255 , FW-AT-Adapt-E has 130 / 1000 = 13% and

Original: PGD-AT has 120 / 1000 = 12% points in red.
Revised: PGD-AT has 120 / 1000 = 12% points in red.

Original: The results in
Revised: The results in

Original: Table 1 and Fig.
Revised: Table 1 and Fig.

Original: 4 further verify that FW-AT improves the true robustness and does not suffer from grading masking or obfuscation.
Revised: 4 further verify that FW-AT improves the true robustness and does not suffer from grading masking or obfuscation.

Original: 7.5. Proof of Proposition
Revised: 7.5. Proof of Proposition

Original: Proof.
Revised: Proof.

Original: The LMO solution is given by ¯ δ k
Revised: The LMO solution is given by ¯ δ k

Original: =
Revised: =

Original: (cid:15) φ p ( ∇ δ L ( x + δ k , y )) and the update becomesδ k +1 =
Revised: (cid:15) φ p ( ∇ [Equation] L ( x + δ k , y )) and the update becomes

Original: δ k + γ k (¯ δ k − δ k ) = (1 − γ k ) δ k + γ k (cid:15) φ p ( ∇ δ L ( x + δ k , y ))
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: AT and

Original: FW-AT-Adapt-E (cid:96)

Original: ∞ robust networks trained on ResNet18 architecture.

Original: FW-AT-Adapt-E exhibits a high resistance to gradient masking.

Original: Method ( (cid:15) = 8 / 255 )
Revised: Method ( (cid:15) = 8 / 255 )

Original: Clean
Revised: Clean

Original: UL
Revised: UL

Original: TL
Revised: TL

Original: UM
Revised: UM

Original: TM
Revised: TM

Original: Params
Revised: Params

Original: Time (mins)
Revised: Time (mins)

Original: Standard 94.
Revised: Standard 94.

Revised: [Table]

Revised: [Table]

Original: n/a n/a
Revised: n/a n/a

Original: FW
Revised: FW

Original: [Table]
Revised: [Table]

Original: AT
Revised: P 84.

Original: [Table]
Revised: [Table]

Original: Adapt-P 84.

Original: [Table]
Revised: [Table]

Original: -E 83.

Original: 23.57 FW

Original: [Table]
Revised: [Table]

Original: -E 82.

Original: 33.06 PGD

Original: [Table]
Revised: [Table]

Original: 21.04 PGD-AT 82.

Original: 29.71 PGD-AT 82.

Original: Method ( (cid:15) = 16 / 255 )
Revised: Method ( (cid:15) = 16 / 255 )

Original: Clean
Revised: Clean

Original: UL
Revised: UL

Original: TL
Revised: TL

Original: UM
Revised: UM

Original: TM
Revised: [Table] [Table]

Original: Params
Revised: [Table] [Table] [Table] [Table]

Original: Time (mins)
Revised: [Table]

Original: FW
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Adapt-P 62.
Revised: [Table] [Table] [Table] [Table]

Original: 24.94 FW

Original: [Table]
Revised: [Table]

Original: Adapt-E 60.

Original: [Table]
Revised: [Table]

Original: Using induction on this relation yields after K steps:
Revised: Using induction on this relation yields after K steps:

Original: [Equation]
Revised: [Equation]

Original: where δ 0 is the initial point which affects both terms in (8) and γ k
Revised: where δ 0 is the initial point which affects both terms in (8) and γ k

Original: = c/ ( c + k ) for k ≥ 0 .
Revised: = c/ ( c + k ) for k ≥ 0 .

Original: Since γ 0 = 1 , the ﬁrst term vanishes and (8) simpliﬁes to
Revised: Since γ 0 = 1 , the ﬁrst term vanishes and (8) simpliﬁes to

Original: [Equation]
Revised: [Equation]

Original: where the coefﬁcients are
Revised: where the coefﬁcients are

Original: [Equation]
Revised: [Equation]

Original: Since γ l ∈ [0 , 1] , it follows that α l ∈ [0 , 1] .
Revised: Since γ l ∈ [0 , 1] , it follows that α l ∈ [0 , 1] .

Original: Induction on (10) yields that (cid:80) K − 1 l =

Revised: Induction on (10) yields that (cid:80) K − 1 l =

Original: α l = 1 .
Revised: α l = 1 .

Original: Furthermore, α l ≤ α l + follows from:
Revised: Furthermore, α l ≤ α l + follows from:

Original: [Equation]
Revised: [Equation]

Original: Thus, the sequence α l is non-decreasing in l .
Revised: Thus, the sequence α l is non-decreasing in l .

Original: Since the coefﬁcients sum to unity, (9) is in the convex hull of the generated LMO sequence { φ p ( ∇ δ L ( x + δ l )) :
Revised: Since the coefﬁcients sum to unity, (9) is in the convex hull of the generated LMO sequence { φ p ( ∇ δ L ( x + δ l )) :

Original: l = 0 , . .
Revised: l = 0 , . .

Original: , K − 1 } .
Revised: , K − 1 } .

Original: 7.6. Proof of Theorem
Revised: 7.6. Proof of Theorem

Original: Proof.
Revised: Proof.

Original: From Proposition 1, we obtain the following decomposition of the adversarial perturbation:
Revised: From Proposition 1, we obtain the following decomposition of the adversarial perturbation:

Original: [Equation]
Revised: [Equation]

Original: To bound the magnitude of the adversarial perturbation, we have
Revised: To bound the magnitude of the adversarial perturbation, we have

Original: [Equation]
Revised: [Equation]

Original: where we use the shorthand notation s l = sgn ( ∇ δ L ( x + δ l , y )) .
Revised: where we use the shorthand notation s l = sgn ( ∇ δ L ( x + δ l , y )) .

Original: The squared (cid:96) 2 norm in the above is bounded as:
Revised: The squared (cid:96) 2 norm in the above is bounded as:

Original: [Equation]
Revised: [Equation]

Original: where we used (cid:107) s l (cid:107)
Revised: where we used (cid:107) s l (cid:107)

Original: 2 = √ d and from Proposition( (cid:80) l
Revised: 2 = √ d and from Proposition( (cid:80) l

Original: α l ) 2 = 1 .
Revised: α l ) 2 = 1 .

Original: The ﬁnal step follows from symmetry.
Revised: The ﬁnal step follows from symmetry.

Original: This concludes the proof.
Revised: This concludes the proof.

Original: 7.7. Proof of Theorem
Revised: 7.7. Proof of Theorem

Original: Proof.
Revised: Proof.

Original: From Theorem 1 and the lower bound on the distortion, it follows that:
Revised: From Theorem 1 and the lower bound on the distortion, it follows that:

Original: [Equation]
Revised: [Equation]

Original: Letting s i = sgn
Revised: Letting s i = sgn

Original: ( ∇L ( x + δ i , y )) and expanding the squared difference of signed gradients:
Revised: ( ∇L ( x + δ i , y )) and expanding the squared difference of signed gradients:

Original: [Equation]
Revised: [Equation]

Original: Using (12) into (11),(cid:88)
Revised: Using (12) into (11),

Original: [Equation]
Revised: [Equation]

Original: For the FGSM deviation bound, i.e., k 0 = 1 , we have by the triangle inequality:
Revised: For the FGSM deviation bound [Equation] i [Equation]

Revised: e.

Original: [Equation]
Revised: [Equation]

Original: (cid:107) (cid:15)
Revised: k 0 = [Equation] we have by the triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: Using Cauchy-Schwarz inequality
Revised: Using Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: where we used the non-decreasing property of the sequence { α l } l and the bound (13) [Equation]
Revised: where we used the non-decreasing property of the sequence { α l } l and the bound (13).

Original: This concludes the ﬁrst part.
Revised: This concludes the ﬁrst part.

Original: Given 1 ≤ k 0 ≤ K , we have via using Proposition 1 twice:
Revised: Given 1 ≤ k 0 ≤ K , we have via using Proposition 1 twice:

Original: [Equation]
Revised: [Equation]

Original: 16) where α l = γ l [Equation]
Revised: where α l = γ l (cid:81) K − 1 i = l +1 (1 − γ i ) , 0 ≤ l ≤ K − 1 and ˜ α j = γ j (cid:81) k 0 − 1 i =

Original: K − 1 i = l + [Equation] 1 − γ i 
Revised: l +1 (1 − γ i ) , 0 ≤ j ≤ k 0 − 1 .

Revised: Taking the (cid:96) 2 norm of both sides of (16) and using the triangle inequality, we obtain:

Original: [Equation]
Revised: [Equation]

Original: , 0 ≤ l ≤ K − 1 and ˜ α j =
Revised: Using the Cauchy-Schwarz inequality yields:

Original: [Equation]
Revised: [Equation]

Original: (cid:81)

Original: [Equation]
Revised: [Equation]

Original: Taking the (cid:96) 2 norm of both sides of (16) and using the triangle inequality
Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Equation]

Original: we obtain:

Original: [Equation]
Revised: [Equation]

Original: Using the Cauchy-Schwarz inequality yields:

Original:  [Equation]
Revised: ≤ (cid:118)(cid:117)(cid:117)(cid:116) K − 1 (cid:88) [Equation]

Original: where we used (13) and the non-decreasing sequence { α l } implies min l<j { α l α j }
Revised: where we used (13) and the non-decreasing sequence { α l } implies min l<j { α l α j }

Original: = α 

Revised: = α

Original: α 1 .
Revised: α 1 .

Original: This concludes the proof of the second part.
Revised: This concludes the proof of the second part.

Original: 7.8. Proof of Corollary
Revised: 7.8. Proof of Corollary

Original: Proof.
Revised: Proof.

Original: Using the triangle inequality and the L -Lipschitz continuous loss gradient assumption:
Revised: Using the triangle inequality and the L -Lipschitz continuous loss gradient assumption:

Original: [Equation]
Revised: [Equation]

Original: The average distortion condition yields via Proposition(with the superscript ( i ) denoting the i -th example variables):
Revised: The average distortion condition yields via Proposition(with the superscript ( i ) denoting the i -th example variables):

Original: [Equation]
Revised: [Equation]

Original: Using Jensen’s inequality (and the concavity of the square root function) further yields after some algebra:
Revised: Using Jensen’s inequality (and the concavity of the square root function) further yields after some algebra:

Original: [Equation]
Revised: [Equation]

Original: Borrowing the relation (12) from the proof of Theorem 2, we further obtain:
Revised: Borrowing the relation (12) from the proof of Theorem [Equation] we further obtain:

Original: [Equation]
Revised: [Equation]

Original: Using the relation (16)
Revised: Using the relation (16)

Original: [Equation]
Revised: [Equation]

Original: it follows:
Revised: it follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation] where we used (a) triangle inequality, (b) Cauchy-Schwarz and (c) Jensen’s inequality.
Revised: where we used (a) triangle inequality, (b) Cauchy-Schwarz and (c) Jensen’s inequality [Equation]

Original: From (18), it follows that:
Revised: From (18), it follows that:

Original: [Equation]
Revised: [Equation]

Revised: l =

Revised: [Equation]

Original: Combining (20) with (19) yields:
Revised: Combining (20) with (19) yields:

Original: [Equation]
Revised: [Equation]

Original: where C
Revised: where C

Original: [Equation]
Revised: [Equation]

Original: Using this bound in (17) concludes the proof.
Revised: Using this bound in (17) concludes the proof.

Original: 7.9. Convergence Analysis
Revised: 7.9. Convergence Analysis

Original: Loss functions L ( x + δ, y ) in deep neural networks are nonconvex in general.
Revised: Loss functions L ( x + δ, y ) in deep neural networks are nonconvex in general.

Original: For a targeted attack that aims to fool the classiﬁer to predict a speciﬁc label, without loss of generality, we seek to minimize the loss f ( δ ) =
Revised: For a targeted attack that aims to fool the classiﬁer to predict a speciﬁc label, without loss of generality, we seek to minimize the loss f ( δ ) =

Original: L ( x + δ, y (cid:48) ) over a (cid:96) p constraint set.
Revised: L ( x + δ, y [Equation] ) over a (cid:96) p constraint set.

Original: The untargeted case follows similarly.
Revised: The untargeted case follows similarly.

Original: 1 For general non-convex constrained optimization, the Frank-Wolfe gap given by (Frank & Wolfe, 1956):
Revised: 1 For general non-convex constrained optimization, the Frank-Wolfe gap given by (Frank & Wolfe, 1956):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: is non-negative in general and zero at stationary points.
Revised: is non-negative in general and zero at stationary points.

Original: The convergence of FW on non-convex functions has been studied in (Lacoste-Julien, 2016) and recently improved for strongly convex constraints in (Rector-Brooks et al., 2019).
Revised: The convergence of FW on non-convex functions has been studied in (Lacoste-Julien, 2016) and recently improved for strongly convex constraints in (Rector-Brooks et al., 2019).

Original: Assumption 2.
Revised: Assumption 2.

Original: The function f has L -Lipschitz continuous gradients on B p ( (cid:15) ) , i.e., (cid:107)∇ f ( u ) − ∇ f ( v ) (cid:107) ≤ L (cid:107) u − v (cid:107) , ∀ u, v ∈ B p ( (cid:15) ) .
Revised: The function f has L -Lipschitz continuous gradients on B p ( (cid:15) ) , i.e., (cid:107)∇ f ( u ) − ∇ f ( v ) (cid:107) ≤ L (cid:107) u − v (cid:107) , ∀ u, v ∈ B p ( (cid:15) ) .

Original: Assumption 2 is a standard assumption for the non-convex setting and has been made in several works (Lacoste-Julien, 2016; Chen et al., 2020).
Revised: Assumption 2 is a standard assumption for the non-convex setting and has been made in several works (Lacoste-Julien, 2016; Chen et al., 2020).

Original: A recent study (Santurkar et al., 2018) shows that the batch normalization layer used in modern neural networks makes the loss much smoother.
Revised: A recent study (Santurkar et al., 2018) shows that the batch normalization layer used in modern neural networks makes the loss much smoother.

Original: Other recent works (Allen-Zhu et al., 2019; Zou & Gu, 2019;
Revised: Other recent works (Allen-Zhu et al., 2019; Zou & Gu, 2019;

Original: Cao & Gu, 2020) showed that the loss is semi-smooth for overparameterized DNNs.
Revised: Cao & Gu, 2020) showed that the loss is semi-smooth for overparameterized DNNs.

Original: Furthermore, the process of adversarial training smooths the loss landscape in comparison to standard models signiﬁcantly as Fig.
Revised: Furthermore, the process of adversarial training smooths the loss landscape in comparison to standard models signiﬁcantly as Fig.

Original: 3 illustrates and other works have noted this phenomenon as well (MoosaviDezfooli et al., 2019;
Revised: 3 illustrates and other works have noted this phenomenon as well (MoosaviDezfooli et al., 2019;

Original: Qin et al., 2019).
Revised: Qin et al., 2019).

Original: Given Assumption 2 and the compactness of the constraint sets, all limit points of FW are stationary points (Bertsekas, 1999).
Revised: Given Assumption 2 and the compactness of the constraint sets, all limit points of FW are stationary points (Bertsekas, 1999).

Original: The convergence rate of FW to a stationary point for optimization over arbitrary convex sets was ﬁrst shown in (Lacoste-Julien, 2016) given by
Revised: The convergence rate of FW to a stationary point for optimization over arbitrary convex sets was ﬁrst shown in (Lacoste-Julien, 2016) given by

Original: [Equation]
Revised: [Equation]

Original: where h 0 =
Revised: where h 0 =

Original: f
Revised: f

Original: ( δ 0 )
Revised: ( δ 0 )

Original: − min δ ∈ B ( (cid:15) )
Revised: − min δ ∈ B ( (cid:15) )

Original: f ( δ ) is the initial global suboptimality.
Revised: f ( δ ) is the initial global suboptimality.

Original: It follows that larger (cid:15) imply a larger diameter and more iterations may be needed to converge 2 .
Revised: It follows that larger (cid:15) imply a larger diameter and more iterations may be needed to converge 2 .

Original: This result implies that an approximate stationary point can be found with gap less than (cid:15) 0 in at most O (1 /(cid:15) 20 ) iterations.
Revised: This result implies that an approximate stationary point can be found with gap less than (cid:15) 0 in at most O (1 /(cid:15) 20 ) iterations.

Original: Theorem 4 in (Rector-Brooks et al., 2019) shows that for smooth non-convex functions over strongly convex constraint sets,
Revised: Theorem 4 in (Rector-Brooks et al., 2019) shows that for smooth non-convex functions over strongly convex constraint sets,

Original: FW yields an improved convergence rate O (cid:0) 1 t (cid:1) , which importantly does not hold for the (cid:96) ∞ constraint. 
Revised: FW yields an improved convergence rate O (cid:0) 1 t (cid:1) , which importantly does not hold for the (cid:96) ∞ constraint. Acknowledgements

Revised: Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artiﬁcial

Original: [Table]
Revised: Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.